A Microscopic Image Mosaic and Fusion Method and System with High Degrees of Freedom

Abstract

The present invention provides a microscopic image stitching and fusion method and system with high degrees of freedom, which relates to the technical field of image stitching and fusion. On the premise of determining the microscopic main image and determining the image ImgLast that is finally stitched into the microscopic main image, considering the overlapping regions at other positions between the image to be stitched and the microscopic main image except for the image ImgLast during the actual image stitching process, if the overlapping regions are multi-regions and polygon overlaps, the accurate relative displacement amount between the image ImgLast and the image to be stitched is determined, which guides the calculation of the weight distribution values when fusing each point in the overlapping region between the image to be stitched and the microscopic main image. Taking the weight distribution values as a benchmark, the gray values of the overlapping image after fusing each point of the image to be stitched and the microscopic main image at the same position are calculated, the fused image block is determined and pasted into the image to be stitched, and then stitching and fusion are performed. Compared with the traditional situation that only adapts to rectangular regions, stitching to be performed is achieved with higher degrees of freedom.

Description

Technical Field

[0001] The present invention relates to the technical field of image stitching and fusion, and more specifically, to a microscopic image stitching and fusion method and system with high degrees of freedom. Background Art

[0002] In the fields of scientific research, medical clinics, etc., in order to obtain the microscopic morphology of the target to be detected, it is necessary to rely on a camera and lens with sufficient resolution to find a suitable observation image. With the continuous development of microscope technology in microscopic applications, especially on the horizontal plane, microscopic images under high magnification can provide more abundant image information than low-magnification images. However, as the magnification of the lens increases, the field of view of the image also decreases, making it impossible to place the region of interest in the same field of view at a relatively high magnification. This brings great obstacles to the observation and analysis of large targets under high magnification, and the processing and analysis of microscope images become increasingly important.

[0003] At present, the operator can observe the separated high-magnification target images separately and then complete the image reconstruction through imagination. If further image analysis is required, only a few images of the target fields of view can be randomly selected to obtain approximate results, and accurate analysis results of the images cannot be obtained, which brings a lot of inconvenience to image analysis.

[0004] Image stitching technology refers to the technology of stitching two or more images with overlapping parts into a high-resolution and wide-angle image containing the information of each image through image preprocessing, image registration, and image fusion technologies. This technology is widely used in medical imaging, remote sensing technology, virtual reality, video editing, etc. With the development of technologies such as computer vision, its importance has become increasingly prominent. In addition, image stitching technology is also used for video compression, stitching video frames into high-resolution images, removing redundant parts, reducing storage space, and increasing transmission speed, and searching for a given target using the phase correlation method to achieve image indexing. The most core technologies of image stitching are divided into two aspects, namely image registration and image fusion. At present, there are many algorithms for image registration with high generality, robustness, and real-time performance, while there are relatively few algorithms for image fusion with good effects, high robustness, and high adaptability.

[0005] In the prior art, a microscopic image stitching method and system are proposed. The image acquisition unit is controlled to sequentially translate and acquire microscopic images at various positions of the sample. The size of the overlapping region between adjacent images is equal to a set value. Then, source feature points within the overlapping region of the source image are searched, and corresponding target feature points are determined within the microscopic image corresponding to the overlapping region of the target image. The target image is the microscopic image after current stitching. All feature points within a window centered on any target feature point are searched, the distances between the corresponding source feature points and each feature point within the window are calculated, and it is judged whether the source feature point matches the feature point corresponding to the minimum distance according to the ratio of the minimum distance to the second minimum distance; the source image and the target image are stitched according to the matching result, and the stitched image is used as the target image; and so on until the last microscopic image is stitched to the target image. However, currently, this stitching method only focuses on the fusion of a simple rectangular overlapping region between the image to be stitched and the two images acquired at the previous position. However, when actually stitching the microscopic image to be stitched with the target image, since the position where the image to be stitched is incorporated into the main image is random, this process also needs to consider the overlapping regions of the image to be stitched and the already stitched image at other positions except the image at the previous position. Therefore, the overlapping region is not a completely regular rectangular region, but may be a complex irregular multi-region or polygonal background. Therefore, this stitching and fusion method based on a rectangular overlapping region has a high limitation, restricting the position where the image to be stitched is incorporated and the stitching effect, and is prone to generating seams and having poor fusion performance. Summary of the Invention

[0006] To solve the problem that the existing image stitching and fusion method has high limitations and poor fusion performance when facing the fusion of the image to be stitched and a multi-region polygonal background, the present invention proposes a microscopic image stitching and fusion method and system with high degrees of freedom, which realizes the incorporation of the image to be stitched into any position of the image at the previous position with high degrees of freedom and ensures seamless fusion of the overlapping parts of the images.

[0007] To achieve the above technical effects, the technical solution of the present invention is as follows:

[0008] A microscopic image stitching and fusion method with high degrees of freedom, the method comprising the following steps:

[0009] S1. Taking the image randomly and preliminarily stitched by several acquired microscopic images as the microscopic main image, and determining the image ImgLast that is the last one incorporated into the microscopic main image during the preliminary stitching process;

[0010] S2. The microscopic image acquisition device moves to acquire a microscopic image in another field of view that has a certain overlapping region with the image ImgLast in the microscopic main image, and taking this microscopic image as the image to be stitched;

[0011] S3. Determine the accurate relative displacement between the image ImgLast and the image to be stitched;

[0012] S4. Preprocess the image to be stitched;

[0013] S5. Determine the weight distribution values when fusing each point in the overlapping area between the preprocessed image to be stitched and the microscopic main image;

[0014] S6. Based on the weight distribution values, determine the gray values of the overlapping image after fusing each point of the image to be stitched and the microscopic main image at the same position, and determine the fused image block;

[0015] S7. Paste the image block into the image to be stitched, and directly perform image stitching on the image to be stitched and the microscopic main image based on the relative displacement.

[0016] Compared with the traditional method, traditional stitching and fusion can only solve the fusion of the simple rectangular overlapping area between the image to be stitched and the image ImgLast that is finally stitched into the microscopic main image. This kind of stitching and fusion limits the position where the image to be stitched is stitched and the stitching effect. However, in the actual process of image stitching, since the position where the image to be stitched is stitched into the microscopic main image is random, this process also needs to consider the overlapping areas of the image to be stitched and the microscopic main image at other positions except the image ImgLast. When the image to be stitched overlaps not only with the image ImgLast but also with the microscopic main image, this part of the content involves the stitching and fusion problem of multi-region and polygon overlap, which cannot be solved by the traditional method. In this technical solution, under the requirement of microscopic images, guided by the weight distribution values, compared with the traditional situation that only adapts to rectangular regions, the image to be stitched is stitched from the image ImgLast with a higher degree of freedom. Thus, the seamless stitching from small-field images to large-field images is successfully completed.

[0017] Preferably, let the microscopic main image be represented as ImgSrc, determine the region of ImgSrc, denoted as RegSrc, let the image to be stitched be represented as ImgAdd, determine the region of ImgAdd, denoted as RegAdd, the image to be stitched is an image with a fixed width and height, let the region of the image ImgLast be RegLast, the overlapping region between RegSrc and RegAdd is denoted as RegOverlap, and the range of the overlapping region is 10% - 80%, satisfying RegOverlap = {Regs|Regs∈RegSrc And Regs∈RegAdd}, where RegOverlap contains one region or multiple regions, that is, Size(RegOverlap)≥1.

[0018] Preferably, the specific process of step S3 includes:

[0019] S31. Coarsely match the image ImgLast with the image to be stitched;

[0020] S32. On the basis of the coarse match, perform an accurate match between the image ImgLast and the image to be stitched to obtain the accurate relative displacement between the image ImgLast and the image to be stitched.

[0021] Preferably, the method for coarsely matching the image ImgLast with the image to be stitched is the phase correlation method. Through the phase correlation method, the overlapping area between the image ImgLast and the image to be stitched is initially determined. By the coarse match, the relative distance between the point at the upper left corner of the image to be stitched and the image ImgLast is first determined, that is, the position of the overlapping area is initially determined.

[0022] Preferably, image feature points are extracted within the initially determined overlapping area between the image ImgLast and the image to be stitched. The image feature points include the positions of similar content points and the similarity degrees of the similar content points in the image ImgLast and the image to be stitched. The higher the similarity degree, the smaller the Euclidean distance between the similar content points;

[0023] First, set a threshold, use the KNN algorithm to screen out the image feature points to be matched that are greater than the threshold, and then, through the RANSAC algorithm, screen out the image feature points with higher similarity from the image feature points to be matched. Finally, use the screened image feature points to obtain the accurate relative displacement between the image ImgLast and the image to be stitched. Specifically:

[0024] Suppose there are q groups of obtained similar content points, and the image coordinates of each group of similar content points are expressed as (x j , y j ), where x j represents the abscissa and y j represents the ordinate, j = 1, 2,..., q. Subtract the corresponding abscissas and ordinates between the q groups of similar content points to obtain multiple groups of position offsets. After excluding outliers from the multiple groups of position offsets, calculate the mean value to obtain the accurate relative displacement between the image ImgLast and the image to be stitched;

[0025] When the positional relationship between the image ImgLast and the microscopic main image is determined, and the accurate relative displacement between the image ImgLast and the image to be stitched is determined, and the position offset between the image to be stitched and the microscopic main image ImgSrc is determined, then the specific position of the overlapping area RegOverlap between RegSrc and RegAdd can be obtained.

[0026] Here, generally, there is more than one extracted image feature point. Two images with only a small number of similar points may have false similar points. When there are no similar points after screening, it is considered that the two images do not overlap at all. The image ImgLast is equivalent to the image ImgAdd to be stitched initially. When the initial stitching is completed, the positional relationship between the image ImgLast and the microscopic main image ImgSrc has been determined. At this time, only by determining the positional relationship between the current image ImgAdd to be stitched and the image ImgLast can the positional relationship between ImgAdd and ImgSrc be determined. From the positional offset value of the image ImgAdd to be stitched relative to the microscopic main image ImgSrc, the overlapping region RegOverlap between ImgAdd and ImgSrc can be further obtained. After each image stitching, the merged region of ImgSrc and ImgAdd is calculated, and the merged region is the region of the new ImgSrc. When the positional relationship between ImgAdd and ImgSrc has been determined, that is, the upper left corner point of the rectangle has been determined, and the width and height of the image ImgAdd to be stitched are also known, that is, the region to be stitched (the region of ImgAdd) is also known. Taking the intersection of the ImgSrc region and the ImgAdd region gives the overlapping region of the two.

[0027] Here, on the basis of the high-degree-of-freedom fusion technology, in order to complete more efficient and higher-precision image stitching and achieve a more perfect image stitching effect, in terms of positioning, the phase correlation method is used for rough matching and extraction of ORB features, and the KNN algorithm and the RANSAC algorithm are used to screen the accurate matching of the matching points to achieve a more accurate position positioning.

[0028] Preferably, in step S3, by using the method of hand-eye calibration to convert the mechanical absolute coordinates into image pixel coordinates, the accurate relative displacement amount between the image ImgLast and the image to be stitched is determined, which helps to accelerate the image stitching and fusion.

[0029] Preferably, in step S4, the image to be stitched is preprocessed by means of flat-field correction.

[0030] Here, due to uneven light when the camera captures images, inconsistent responses between the center and the edge of the lens, etc., the resulting adverse effect of uneven brightness distribution of the acquired images (usually manifested as bright in the middle and dark around the edges) can be solved by flat-field correction.

[0031] Preferably, in step S5, the process of determining the weight distribution values of each point during the fusion within the overlapping region between the preprocessed image to be stitched and the microscopic main image is as follows:

[0032] S51. Classify the contour line sets of the overlapping regions between the pre - processed images to be stitched and the microscopic main image. Classify the edges belonging to the images to be stitched into the edge line set EdgeLines, where EdgeLines = {EL1, EL2,... EL m}, EL i is the edge line of EdgeLines, and m is the number of edge lines in the edge line set EdgeLines; classify the edges belonging to the edges of the microscopic main image ImgSrc into the inner line set InnerLines, where InnerLines = {IL1, IL2,... IL n}, IL i is the edge line of InnerLines, and n is the number of inner lines in the inner line set InnerLines;

[0033] S52. Let the overlapping region be represented as RegOverlap(x, y), and calculate the minimum distance d0(x, y) between each point in the overlapping region and the inner line set InnerLines, and the minimum distance d1(x, y) between each point and the edge line set EdgeLines;

[0034] S53. According to the distance values described in S52, calculate the weight distribution values when each point in the overlapping region RegOverlap(x, y) is fused. The expression is:

[0035]

[0036] where weight(x, y) represents the weight distribution value when each point is fused, which is simply distributed from 1 to 0, and GPU parallel computing is used in the calculation process;

[0037] In step S6, the expression for the gray value of the overlapping image after fusing each point of the image to be stitched at the same position with the microscopic main image is:

[0038] ImgStitch(x, y) = (1 - weight) * ImgAdd(x, y)+weight * ImgSrc(x, y).

[0039] Here, in order to complete image stitching and fusion more efficiently, GPU concurrent computing is used to make the stitching and fusion process meet the real - time requirements.

[0040] Preferably, the process of calculating the minimum distance d0(x, y) between each point in the overlapping region and the inner line set InnerLines, and the minimum distance d1(x, y) between each point and the edge line set EdgeLines is:

[0041] Let \(p\) represent the coordinate points of the overlapping region, expressed as \(p=(x,y)\). Define the distance from \(p\) to the line segment \(EL\) as \(dist(p,EL)\), then \(dist(p,EL)=\min(\|p - p_1\|_2,\|p - p_2\|_2,\|p - p proj \|_2)\), where \(p_1\) is one endpoint of the line segment \(EL\), \(p_2\) is the other endpoint of the line segment \(EL\), and \(p proj is the projection point of point \(p\) on the line segment \(EL\);

[0042] Then the minimum distance from \(p\) to InnerLines is:

[0043] \(d_0(x,y)=\min(dist(p,IL i ))

[0044] where \(i = 1,2,\cdots,n\), and \(n\) is the number of inner lines in the inner line set InnerLines;

[0045] The minimum distance from \(p\) to EdgeLines is:

[0046] \(d_1(x,y)=\min(dist(p,EL i ))

[0047] where \(i = 1,2,\cdots,m\), and \(m\) is the number of edge lines in the edge line set EdgeLines.

[0048] This application also proposes a high - degree - of - freedom microscopic image stitching and fusion system, and the system includes:

[0049] An image preparation unit, which is used to take the image randomly and preliminarily stitched by several collected microscopic images as the microscopic main image, and determine the image ImgLast that is the last one stitched into the microscopic main image during the process of preliminary stitching;

[0050] A microscopic image acquisition control unit, which is used to control the movement of the microscopic image acquisition device, acquire a microscopic image in another field of view and having a certain overlapping region with the image ImgLast in the microscopic main image, and take this microscopic image as the image to be stitched;

[0051] A relative displacement amount calculation unit, which is used to determine the accurate relative displacement amount between the image ImgLast and the image to be stitched;

[0052] A pre - processing unit, which is used to pre - process the image to be stitched;

[0053] A weight distribution calculation unit, which is used to determine the weight distribution values when fusing each point in the overlapping region between the pre - processed image to be stitched and the microscopic main image;

[0054] The grayscale value fusion calculation unit determines the grayscale values of the overlapping image formed by fusing each point of the image to be stitched at the same position with the microscopic main image based on the weight distribution value, and determines the fused image block.

[0055] The stitching unit pastes the image block into the image to be stitched, and directly stitches the image to be stitched and the microscopic main image based on the relative displacement amount.

[0056] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0057] The present invention provides a method and system for stitching and fusing microscopic images with high degrees of freedom. On the premise of preparing the microscopic main image in advance and determining the last image ImgLast to be stitched into the microscopic main image, when considering the actual image stitching process, for the overlapping areas at other positions between the image to be stitched and the microscopic main image except for the image ImgLast, if the overlapping areas are multi-regions or polygon overlaps, the accurate relative displacement amount between the image ImgLast and the image to be stitched is determined, which guides the calculation of the weight distribution value when fusing each point in the overlapping area between the image to be stitched and the microscopic main image. Taking the weight distribution value as a reference, the grayscale values of the overlapping image formed by fusing each point of the image to be stitched at the same position with the microscopic main image are calculated, the fused image block is determined and pasted into the image to be stitched and then stitched and fused. Compared with the traditional situation that only adapts to rectangular areas, the image to be stitched can be stitched into the microscopic main image with higher degrees of freedom. Description of the Drawings

[0058] Figure 1 It represents a schematic flow chart of the method for stitching and fusing microscopic images with high degrees of freedom proposed in Embodiment 1 of the present invention;

[0059] Figure 2 It represents a schematic diagram of polygon region stitching and fusion proposed in Embodiment 1 of the present invention;

[0060] Figure 3 It represents a matching diagram of image feature points between the image ImgLast and the image to be stitched proposed in Embodiment 1 of the present invention;

[0061] Figure 4 It represents a result diagram of the splicing process of the transverse section of the dicotyledonous root and stem proposed in Embodiment 1 of the present invention;

[0062] Figure 5 It represents a result diagram of the completion of the splicing of the transverse section of the dicotyledonous root and stem proposed in Embodiment 1 of the present invention;

[0063] Figure 6 It represents a schematic structural diagram of the system for stitching and fusing microscopic images with high degrees of freedom proposed in Embodiment 3 of the invention. Detailed Embodiments

[0064] The accompanying drawings are only for illustrative purposes and should not be construed as limiting the patent;

[0065] To better illustrate this embodiment, some parts of the accompanying drawings are omitted, enlarged or reduced, and do not represent the actual size;

[0066] For those skilled in the art, it is understandable that some well-known content descriptions in the accompanying drawings may be omitted.

[0067] The technical solution of the present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0068] The description of the positional relationship in the accompanying drawings is only for illustrative purposes and should not be construed as limiting the patent;

[0069] Embodiment 1

[0070] As Figure 1 shown, this embodiment proposes a microscopic image stitching and fusion method with high degrees of freedom, and the method includes the following steps:

[0071] S1. Use the image randomly and preliminarily stitched by several collected microscopic images as the microscopic main image, and determine the image ImgLast that is finally stitched into the microscopic main image during the preliminary stitching process;

[0072] S2. Move the microscopic image acquisition device to acquire a microscopic image in another field of view that has a certain overlapping area with the image ImgLast in the microscopic main image, and use this microscopic image as the image to be stitched;

[0073] In this embodiment, the microscopic image acquisition device is a microscope. Let the microscopic main image be denoted as ImgSrc, determine the area of ImgSrc, denoted as RegSrc, let the image to be stitched be denoted as ImgAdd, determine the area of ImgAdd, denoted as RegAdd, the image to be stitched is an image with a fixed width and height, which is determined by a top-left corner point and a set of width and height. The top-left corner point determines the position of the image, and the width and height are the width and height of the image. Let the area of the image ImgLast be RegLast, and the overlapping area of RegSrc and RegAdd is denoted as RegOverlap. The range of the overlapping area is 10% - 80%, and it satisfies RegOverlap = {Regs|Regs ∈ RegSrc And Regs ∈ RegAdd}, where RegOverlap contains one area or multiple areas, that is, Size(RegOverlap) ≥ 1. Figure 2Schematic diagram of a regional polygon for a single splicing. Among them, the marked area 1 in the figure is the area RegSrc where the microscopic main image ImgSrc is located, area 2 is the area RegLast where the image ImgLast is located, area 3 is the area RegAdd where the image ImgAdd to be spliced is located, and area 4 is the overlapping area RegOverlap between RegSrc and RegLast.

[0074] Capture an image under the current field of view, translate the microscope objective to other positions, and take another image. There is a certain range of overlap in position and content between the two images, approximately between 10% and 80%. The microscopic main image and the points inside it are represented as ImgSrc(x, y), where x and y refer to the image coordinates. The image ImgLast and the points inside it are represented as ImgLast(x, y), and the image ImgAdd to be spliced and the points inside it are represented as ImgAdd(x, y).

[0075] S3. Determine the accurate relative displacement amount between the image ImgLast and the image to be spliced.

[0076] Here, the specific process of step S3 includes:

[0077] S31. Coarsely match the image ImgLast and the image to be spliced.

[0078] S32. On the basis of the coarse match, perform an accurate match between the image ImgLast and the image to be spliced to obtain the accurate relative displacement amount between the image ImgLast and the image to be spliced.

[0079] The method for coarsely matching the image ImgLast and the image to be spliced is the phase correlation method. Through the phase correlation method, the overlapping area between the image ImgLast and the image to be spliced is initially determined. By the coarse match, the relative distance between the point at the upper left corner of the image to be spliced and the image ImgLast is first determined, that is, the position of the overlapping area is initially determined.

[0080] In this embodiment, the phase correlation method utilizes the relative displacement relationship between two images and performs a Fourier transform to the frequency domain, and obtains its normalized power spectrum through the formula in its frequency domain. The normalized power spectrum is an exponential function. By inverse-transforming the exponential function to the spatial domain, an impulse function can be obtained, thereby determining the translation amounts x0 and y0. Let ImgLast(x, y) and ImgAdd(x, y) be the two images, and there is a translation relationship in the overlapping part between them. The relative horizontal and vertical translation amounts are x0 and y0, then there is

[0081] f1(x, y) = f2(x - x0, y - y0)

[0082] Among them, f1 is the previously completed stitched small image ImgLast, f2 is the small image ImgAdd to be stitched, and there is a translation relationship between f1 and f2;

[0083] Performing Fourier transform on the above formula gives

[0084]

[0085] Among them, the coordinate system in the frequency domain range after Fourier transform. The normalized power spectrum is

[0086]

[0087] Among them, F * is the complex conjugate function of F; the normalized power spectrum is an exponential function. Performing inverse Fourier transform on it to the spatial domain, the following equation is obtained:

[0088]

[0089] The right side of the above formula is an impulse function. The peak position corresponding to the impulse function reflects the correlation between the two images, that is, this process determines x0 and y0. After inverse Fourier transform of the left side formula, it is an impulse function. According to the properties of the impulse function, the offset values x0 and y0 reflect the position of the impulse function. What is observed on the image is a white point with a maximum gray value (the largest gray value) in a uniformly gray image, and this white point position is (x0, y0), and this position is the upper left corner position of the overlapping area, thus determining the translation parameters x0, y0. Therefore, we can roughly match the position offset between the two images and calculate the overlapping area between ImgLast and ImgAdd.

[0090] Extract image feature points within the overlapping area of the initially determined image ImgLast and the image to be stitched. The methods include sift, orb, surf algorithms, and the extraction method belongs to common knowledge and is universal, so it will not be elaborated here. Image feature points include the positions of similar content points between the image ImgLast and the image to be stitched, and the similarity of the similar content points. The higher the similarity, the smaller the Euclidean distance between the similar content points;

[0091] First, set a threshold, use the KNN algorithm to screen out the image feature points to be matched that are greater than the threshold, and then use the RANSAC algorithm to screen out the image feature points with higher similarity from the image feature points to be matched. Finally, use the screened image feature points to obtain the accurate relative displacement between the image ImgLast and the image to be stitched. Figure 3 It is a matching schematic diagram of the feature points of the image ImgLast and the image ImgAdd to be stitched. Specifically:

[0092] Suppose there are q groups of similar content points obtained, and the image coordinates of each group of similar content points are represented as (x j , y j ), where x j represents the abscissa and y j represents the ordinate, j = 1, 2,..., q. Subtract the corresponding abscissas and ordinates between the q groups of similar content points to obtain multiple groups of position offsets. After excluding outliers from the multiple groups of position offsets, calculate the mean value to obtain the accurate relative displacement between the image ImgLast and the image to be stitched;

[0093] Determine the positional relationship between the image ImgLast and the microscopic main image. When determining the accurate relative displacement between the image ImgLast and the image to be stitched, determine the position offset between the image to be stitched and the microscopic main image ImgSrc, and then obtain the specific position of the overlapping region RegOverlap between RegSrc and RegAdd.

[0094] Here, generally, more than one image feature point is extracted. Two images with only a small number of similar points may be false similar points. When there are no similar points after screening, it is considered that the two images do not overlap at all. The image ImgLast is equivalent to the image to be stitched ImgAdd in the initial preliminary stitching. When the preliminary stitching is completed, the positional relationship between the image ImgLast and the microscopic main image ImgSrc has been determined. At this time, only by determining the positional relationship between the current image to be stitched ImgAdd and the image ImgLast can the positional relationship between ImgAdd and ImgSrc be determined. From the position offset value of the image to be stitched ImgAdd relative to the microscopic main image ImgSrc, the overlapping region RegOverlap between ImgAdd and ImgSrc can be further obtained. After each image is stitched, calculate the merged region of ImgSrc and ImgAdd. The merged region is the new ImgSrc region. When the positional relationship between ImgAdd and ImgSrc has been determined, that is, the upper left corner point of the rectangle has been determined, and the width and height of the image to be stitched ImgAdd are also known, that is, the region to be stitched (the region of ImgAdd) is also known. Taking the intersection of the ImgSrc region and the ImgAdd region gives the overlapping region of the two.

[0095] On the basis of the high-degree-of-freedom fusion technology, in order to complete more efficient and higher-precision image stitching and achieve a more perfect image stitching effect, in terms of positioning, use the phase correlation method for rough matching and extract ORB features, and adopt the KNN algorithm and the RANSAC algorithm to screen matching points for fine matching to achieve more accurate position positioning.

[0096] S4. Preprocess the images to be stitched; in this embodiment, the images to be stitched are preprocessed by flat-field correction. Due to uneven light during camera image acquisition, inconsistent responses between the center and edges of the lens, etc., the resulting images have an uneven brightness distribution, usually manifested as bright in the middle and dark around the edges. This problem can be solved using flat-field correction.

[0097] In actual implementation, the two-point correction method is usually adopted. First, the image acquisition device performs a dark-field exposure to obtain pixel offsets. Then, under uniform illumination, an image of a uniformly gray object is taken to obtain a uniform image. Finally, the dark-field image is subtracted from the uniform light-field image, and the relative calibration method is used to correct the image gain.

[0098] S5. Determine the weight distribution values when fusing each point in the overlapping region between the preprocessed images to be stitched and the microscopic main image;

[0099] In step S5, the process of determining the weight distribution values when fusing each point in the overlapping region between the preprocessed images to be stitched and the microscopic main image is as follows:

[0100] S51. Classify the contour line sets of the overlapping region between the preprocessed images to be stitched and the microscopic main image. The edges belonging to the images to be stitched are classified into the edge line set EdgeLines, where EdgeLines = {EL1, EL2,... EL m}, EL i is the edge line of EdgeLines, and m is the number of edge lines in the edge line set EdgeLines; the edges belonging to the edges of the microscopic main image ImgSrc are classified into the inner line set InnerLines, where InnerLines = {IL1, IL2,... IL n}, IL i is the edge line of InnerLines, and n is the number of inner lines in the inner line set InnerLines;

[0101] S52. Let the overlapping region be represented as RegOverlap(x, y), and calculate the minimum distance d0(x, y) between each point in the overlapping region and the inner line set InnerLines, and the minimum distance d1(x, y) between each point and the edge line set EdgeLines; the calculation process is as follows:

[0102] Let p represent the coordinate point of the overlapping region, expressed as p = (x, y), and define the distance from p to the line segment EL as dist(p, EL), then dist(p, EL) = min(||p - p1||2, ||p - p2||2, ||p - p proj ||2), where p1 is one endpoint of the line segment EL, p2 is the other endpoint of the line segment EL, and pproj is the projection point of point p on line segment EL;

[0103] Then the minimum distance from p to InnerLines is:

[0104] d0(x, y) = min(dist(p, IL i ))

[0105] where i = 1, 2,... n, and n is the number of inner lines in the inner line set InnerLines;

[0106] The minimum distance from p to EdgeLines is:

[0107] d1(x, y) = min(dist(p, EL i ))

[0108] where i = 1, 2,... m, and m is the number of edge lines in the edge line set EdgeLines.

[0109] S53. According to the distance values described in S52, calculate the weight distribution values when each point in the overlapping region RegOverlap(x, y) is fused. The expression is:

[0110]

[0111] where weight(x, y) represents the weight distribution value when each point is fused, which is simply distributed from 1 to 0. In the calculation process, GPU parallel computing is adopted. Since the calculation cost of the weight distribution map is very high, it is difficult for a single CPU processor to meet the real-time requirements of image stitching. Therefore, this process is completed using GPU parallel computing, which can greatly improve the stitching efficiency.

[0112] In addition, under the guidance of the corresponding distribution map of the weight distribution map value, when looking in with the naked eye, it is a seamless fused image, while when there is no weight distribution guidance, the image has an obvious sense of fragmentation with dividing lines.

[0113] S6. Based on the weight distribution values, determine the gray values of the overlapping images after the points at the same position of the images to be stitched are fused with the microscopic main image, and determine the fused image blocks;

[0114] In step S6, the expression for the gray values of the overlapping images after the points at the same position of the images to be stitched are fused with the microscopic main image is:

[0115] ImgStitch(x, y) = (1 - weight) * ImgAdd(x, y) + weight * ImgSrc(x, y).

[0116] S7. Paste the image block into the image to be stitched. Based on the relative displacement amount, directly stitch the image to be stitched and the microscopic main image. For the transverse section image of the dicotyledonous rhizome under the microscope, using the above process, the result image of the transverse section stitching process of the dicotyledonous rhizome is as shown in Figure 4 shown, and the result image of the end of the transverse section stitching of the dicotyledonous rhizome is as shown in Figure 5 shown. It can be seen that the method proposed in this embodiment looks like a seamless fusion image to the naked eye.

[0117] Embodiment 2

[0118] In this embodiment, except for being consistent with the overall process idea of Embodiment 1, for the calculation of the accurate relative displacement amount between the image ImgLast and the image to be stitched, the method of converting the mechanical absolute coordinates into image pixel coordinates by using hand-eye calibration can be used to determine the accurate relative displacement amount between the image ImgLast and the image to be stitched, which helps to accelerate the image stitching and fusion.

[0119] Embodiment 3

[0120] See Figure 6 , this embodiment proposes a high-degree-of-freedom microscopic image stitching and fusion system, and the system includes:

[0121] An image preparation unit, which is used to use the image randomly and preliminarily stitched by a number of collected microscopic images as the microscopic main image, and determine the image ImgLast that is finally stitched into the microscopic main image during the preliminary stitching process;

[0122] A microscopic image acquisition control unit, which is used to control the movement of the microscopic image acquisition device to acquire a microscopic image in another field of view that has a certain overlapping area with the image ImgLast in the microscopic main image, and use this microscopic image as the image to be stitched;

[0123] A relative displacement amount calculation unit, which is used to determine the accurate relative displacement amount between the image ImgLast and the image to be stitched;

[0124] A preprocessing unit, which is used to preprocess the image to be stitched;

[0125] A weight distribution calculation unit, which is used to determine the weight distribution value when fusing each point in the overlapping area of the preprocessed image to be stitched and the microscopic main image;

[0126] A gray value fusion calculation unit, based on the weight distribution value, determines the gray value of the overlapping image after fusing each point of the image to be stitched and the microscopic main image at the same position, and determines the fused image block;

[0127] The splicing unit pastes the image blocks into the image to be spliced, and directly performs image splicing on the image to be spliced and the microscopic main image based on the relative displacement amount.

[0128] Obviously, the above-mentioned embodiments of the present invention are only examples for clearly explaining the present invention, and are not limitations on the implementation manners of the present invention. For those of ordinary skill in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all implementation manners here. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the claims of the present invention.

Claims

1. A microscopic image stitching and fusion method with high degrees of freedom, characterized in that, The method includes the following steps: S1. Use the image randomly and preliminarily stitched by several collected microscopic images as the microscopic main image, and determine the image ImgLast that is the last one stitched into the microscopic main image during the preliminary stitching process. S2. Move the microscopic image acquisition device to acquire a microscopic image in another field of view that has a certain overlapping area with the image ImgLast in the microscopic main image, and use this microscopic image as the image to be stitched. S3. Determine the accurate relative displacement amount between the image ImgLast and the image to be stitched. Extract image feature points within the overlapping area of the preliminarily determined image ImgLast and the image to be stitched. The image feature points include the positions of the similar content points and the similarity degrees of the similar content points in the image ImgLast and the image to be stitched. The higher the similarity degree, the smaller the Euclidean distance between the similar content points. First, set a threshold, use the KNN algorithm to screen out the image feature points to be matched that are greater than the threshold, and then use the RANSAC algorithm to further screen out the image feature points with higher similarity from the image feature points to be matched. Finally, use the screened image feature points to calculate the accurate relative displacement amount between the image ImgLast and the image to be stitched. Specifically: Suppose that a total of q groups of similar content points are obtained, and the image coordinates of each group of similar content points are expressed as ( x j , y j ), where x j represents the abscissa, y j represents the ordinate, j = 1, 2,..., q. Subtract the corresponding abscissas and ordinates between the q groups of similar content points to obtain multiple groups of position offsets. After excluding outliers from the multiple groups of position offsets, calculate the mean value to obtain the accurate relative displacement between the image ImgLast and the image to be stitched; The positional relationship between the image ImgLast and the microscopic main image is determined. When the accurate relative displacement amount between the image ImgLast and the image to be stitched is determined, the positional offset amount between the image to be stitched and the microscopic main image ImgSrc is determined, and then the specific position of the overlapping area RegOverlap between RegSrc and RegAdd is obtained. Among them, the microscopic main image is represented as ImgSrc, the area of ImgSrc is determined and set as RegSrc, the image to be stitched is represented as ImgAdd, and the area of ImgAdd is determined and set as RegAdd. S4. Preprocess the image to be stitched. S5. Determine the weight distribution values when fusing each point within the overlapping area of the preprocessed image to be stitched and the microscopic main image. S6. Based on the weight distribution values, determine the gray values of the overlapping image after fusing each point of the image to be stitched and the microscopic main image at the same position, and determine the fused image block. S7. Paste the image block into the image to be stitched, and directly perform image stitching on the image to be stitched and the microscopic main image based on the relative displacement amount.

2. The high-degree-of-freedom microscopic image stitching and fusion method according to claim 1, characterized in that The image to be stitched is an image with a fixed width and height. Let the area of the image ImgLast be RegLast, and the overlapping area between RegSrc and RegAdd be represented as RegOverlap. The range of the overlapping area is 10% - 80%, and it satisfies RegOverlap = {Regs|Regs∈RegSrc AndRegs∈RegAdd}, where RegOverlap contains one or more areas, that is, Size(RegOverlap)≥1, and Regs represents the set element representation form of the overlapping area RegOverlap.

3. The high-degree-of-freedom microscopic image stitching and fusion method according to claim 2, wherein The specific process of step S3 includes: S31. Coarsely match the image ImgLast and the image to be stitched. S32. On the basis of rough matching, perform precise matching between the image ImgLast and the image to be stitched to obtain the accurate relative displacement between the image ImgLast and the image to be stitched.

4. The high-degree-of-freedom microscopic image stitching and fusion method according to claim 3, characterized in that The method for rough matching between the image ImgLast and the image to be stitched is the phase correlation method. Through the phase correlation method, the overlapping area between the image ImgLast and the image to be stitched is initially determined.

5. The method for stitching and fusing microscopic images with high degrees of freedom according to claim 2, wherein In step S3, the accurate relative displacement between the image ImgLast and the image to be stitched is determined by using the hand-eye calibration method to convert the mechanical absolute coordinates into image pixel coordinates.

6. The method for stitching and fusing microscopic images with high degrees of freedom according to claim 1, wherein In step S4, the image to be stitched is preprocessed by means of flat-field correction.

7. The method for micro-image stitching and fusion with high degrees of freedom according to claim 1, wherein In step S5, the process of determining the weight distribution values when each point in the overlapping area between the preprocessed image to be stitched and the microscopic main image is fused is as follows: S51. Classify the contour line sets of the overlapping regions between the image to be stitched after preprocessing and the microscopic main image, and classify the edges belonging to the image to be stitched into the edge line set EdgeLines, where, , is the edge line of EdgeLines, and m is the number of edge lines in the edge line set EdgeLines; classify the edges belonging to the edge of the microscopic main image ImgSrc into the inner line set InnerLines, where, , is 's edge line, and n is the number of inner lines in the inner line set InnerLines; Let the overlapping region be characterized as RegOverlap(x,y), and calculate the minimum distances from each point within the overlapping region to the set of inner lines InnerLines and the minimum distances to the set of edge lines EdgeLines ; S53. According to the distance values described in S52, calculate the weight distribution values when each point in the overlapping area RegOverlap(x, y) is fused. The expression is: Among them, represents the weight distribution value when each point is fused, which is simply distributed from 1 to 0, and GPU parallel computing is adopted in the calculation process; In step S6, the expression for the gray value of the overlapping image after each point of the image to be stitched at the same position is fused with the microscopic main image is: 。 8. The method for stitching and fusing microscopic images with high degrees of freedom according to claim 7, wherein Calculate the minimum distances between each point in the overlapping area and the inner line set InnerLines and the minimum distances from the edge line set EdgeLines The process is as follows: Let p represent the coordinate points of the overlapping region, expressed as , and define the distance from p to the line segment EL as , then , where is one endpoint of the line segment EL, is the other endpoint of the line segment EL, is the projection point of point p on the line segment EL; Then the minimum distance from p to InnerLines is: , where n is the number of inner lines in the set of inner lines InnerLines; The minimum distance from p to EdgeLines is: Among them, , where m is the number of edge lines in the edge line set EdgeLines.

9. A microscopic image stitching and fusion system with high degrees of freedom, characterized in that, The system includes: An image preparation unit, which is used to take the image randomly and preliminarily stitched by several collected microscopic images as the microscopic main image, and determine the image ImgLast that is the last one stitched into the microscopic main image during the process of preliminary stitching. A microscopic image acquisition control unit, which is used to control the movement of the microscopic image acquisition device to acquire a microscopic image in another field of view that has a certain overlapping area with the image ImgLast in the microscopic main image, and take this microscopic image as the image to be stitched. A relative displacement calculation unit, which is used to determine the accurate relative displacement between the image ImgLast and the image to be stitched. Extract image feature points in the initially determined overlapping area between the image ImgLast and the image to be stitched. The image feature points include the positions of the similar content points and the similarity degrees of the similar content points between the image ImgLast and the image to be stitched. The higher the similarity degree, the smaller the Euclidean distance between the similar content points. First, set a threshold, use the KNN algorithm to screen out the image feature points to be matched that are greater than the threshold, and then use the RANSAC algorithm to screen out the image feature points with higher similarity from the image feature points to be matched. Finally, use the screened image feature points to obtain the accurate relative displacement between the image ImgLast and the image to be stitched. Specifically: Suppose there are q groups of similar content points obtained, and the image coordinates of each group of similar content points are represented as ( x j , y j ), where x j represents the abscissa, y j represents the ordinate, j = 1, 2,..., q. Subtract the corresponding abscissas and ordinates between the q groups of similar content points to obtain multiple groups of position offsets. After excluding outliers from the multiple groups of position offsets, calculate the mean value to obtain the accurate relative displacement between the image ImgLast and the image to be stitched; The positional relationship between the image ImgLast and the microscopic main image is determined. When the accurate relative displacement amount between the image ImgLast and the image to be stitched is determined, the positional offset amount between the image to be stitched and the microscopic main image ImgSrc is determined, and then the specific position of the overlapping region RegOverlap between RegSrc and RegAdd is obtained; among them, the microscopic main image is represented as ImgSrc, the region of ImgSrc is determined, denoted as RegSrc, the image to be stitched is represented as ImgAdd, and the region of ImgAdd is determined, denoted as RegAdd; The preprocessing unit is used to preprocess the image to be stitched; The weight distribution calculation unit is used to determine the weight distribution values when fusing each point in the overlapping region between the preprocessed image to be stitched and the microscopic main image; The gray value fusion calculation unit, based on the weight distribution values, determines the gray values of the overlapping image after fusing each point of the image to be stitched and the microscopic main image at the same position, and determines the fused image block; The stitching unit pastes the image block into the image to be stitched, and directly performs image stitching on the image to be stitched and the microscopic main image based on the relative displacement amount.

Images