Microscope image intelligent stitching method, device, medium and equipment

Abstract

The present invention discloses a method, device, medium, and equipment for intelligently stitching microscope images acquired using a rolling shutter. The method discloses an intelligent stitching method for microscope images based on an electronic rolling shutter, including: an automatic stitching process based on the camera characteristics of the electronic rolling shutter during the movement of the microscope stage; a real-time detection method for image motion blur; calculation and dynamic display of various states of the stitching process, and coordination of the stage movement speed for effective image stitching; and a dual-projection automatic stitching algorithm. The present invention is simple to use and operate, effectively expanding the application range of electronic rolling shutter cameras by intelligently identifying and automatically stitching images acquired during the stage movement of the electronic rolling shutter camera. This allows a conventional microscope coupled with a popular digital camera to perform the same function as a digital slide scanner in scanning digital slides.

Description

Technical Field

[0001] The present invention relates to the technical field of optical microscope image processing, and in particular to a method, device, medium and equipment for intelligently stitching microscope images acquired based on a rolling shutter. Background Art

[0002] Conventional digital cameras use shutter exposure to create images, controlling the effective exposure time of the photosensitive chip through the shutter. There are two main exposure methods for digital camera chips: global shutter exposure and rolling shutter exposure. Global shutter exposure is achieved by exposing the entire scene at the same time, turning exposure on and off for each digital pixel in the camera simultaneously. The value of each pixel in the image sensor is then read out to form a complete image. The advantage of global shutter exposure is that all pixels are exposed simultaneously, resulting in uniform and stable exposure for each pixel, without smearing. Compared to a global shutter, a rolling shutter is achieved by exposing the photosensitive chip row by row. At the beginning of imaging, the photosensitive chip is sequentially scanned and exposed row by row, and the value of each pixel of the image sensor is read out row by row until all pixels are scanned, exposed, and read out, forming a complete image. The advantage of rolling shutter exposure is a shorter exposure time and simpler hardware design for the pixel readout process. However, its disadvantage is that because the images are exposed and captured row by row at different times, if the image is moving, there will be a phenomenon of smearing. The impact of the smearing depends on the speed of the image motion and the exposure time of the entire image. Currently, most digital cameras use economical electronic rolling shutter cameras.

[0003] Patent document CN111279673A discloses "Image Stitching with Electronic Rolling Shutter Correction." The capture system includes multiple electronic rolling shutter camera image sensors to acquire images. A parallax correction map is determined by compensating for epipolar lines. A distortion map is determined using the parallax correction map and the electronic rolling shutter correction map. The electronic rolling shutter correction map is applied to the output of the parallax correction map using the distortion map. This series of correction image compensation methods corrects for image distortion caused by the electronic rolling shutter camera acquisition. Images from two or more camera sensors are then stitched together to generate a composite stitched image. This method requires the configuration of multiple cameras, increasing system complexity and cost.

[0004] Patent document CN201380059462 discloses a system and method for asynchronously sequencing microscope devices while simultaneously capturing images using a rolling shutter camera. The system employs a rolling shutter camera to capture images, specifically by assembling a rolling shutter camera within a microscope. The system also utilizes a stepper motor to move the stage. The system is characterized by synchronously controlling the stage's movement with the stepper motor based on the rolling shutter's exposure end time pulse, while only receiving image data from the shared exposure period associated with the rolling shutter exposure signal. This system is complex and costly, and is used only for synchronous image acquisition using a rolling shutter camera, without involving image stitching.

[0005] Patent document CN202011053664 discloses “A method for rapid stitching and fusion of digital microscope images”, which includes four steps: acquiring microscopic images from a multimodal imaging system, preprocessing the images (through an algorithm), registering the images, and fusing the images to obtain the final stitched image. The system also includes a stepper motor to control the moving stage, obtaining the overlapping area of ​​the two images according to the displacement of the stepper motor, and then using the Fourier transform method to accurately match the images and evaluate the similarity of the overlapping areas. This method must use an automatic stage to obtain image displacement, which increases the system cost and has poor adaptability. At the same time, its image matching based on Fourier transform is computationally complex and increases the amount of calculation.

[0006] In summary, it is necessary to provide a method, device, medium and equipment for intelligent stitching of microscope images acquired based on rolling shutter, which is universally applicable to manual and automatic optical microscopes, has low computational complexity and high stitching accuracy. Summary of the Invention

[0007] The object of the present invention is to provide an intelligent stitching method for microscope images acquired based on a rolling shutter.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] Microscope image intelligent stitching method, including:

[0010] S1, real-time exposure of the camera to obtain real-time dynamic images of the microscope, the camera is an electronic rolling shutter camera;

[0011] S2, performing real-time detection of image motion blur on the acquired real-time dynamic image to determine whether the image quality meets the preset evaluation standard. If so, proceed to S3; otherwise, return to S1;

[0012] S3, extracting a first image and a second image, wherein the first image is a currently acquired dynamic image, and the second image is an image extracted from the same position of the spliced ​​large image as the current dynamic image;

[0013] S4. Project the initial stitching results of the current image and the previous stitched image in the horizontal direction and the vertical direction of the image respectively to obtain projection histograms in two directions, perform horizontal and vertical one-dimensional correlation analysis based on the projection histograms to obtain the horizontal and vertical offsets of the image, and thus stitch the images.

[0014] Furthermore, in S1, the stage has a working feed rate and a fast feed rate;

[0015] When the current real-time dynamic image has been stitched, the stage switches from the working speed to the fast speed and moves quickly;

[0016] When the stage moves to the boundary of the currently stitched large image, the stage switches from the fast-forward rate to the working rate until the stitching of the next image to be stitched is completed.

[0017] Furthermore, S2 includes:

[0018] S21, converting the current image from an RGB image to a grayscale image;

[0019] S22, sampling the converted grayscale image, performing edge detection, extracting image clarity parameters and normalizing;

[0020] It uses the Laplace edge detection algorithm to detect edges, and introduces the improved Laplace operator. The sum of the absolute values ​​of the second-order partial derivatives is taken. For the two-dimensional image function f(x,y), the Laplace operator of the image is:

[0021]

[0022] Using difference instead of differential for calculation, considering the texture change of microscopic image, using variable step size between pixels to calculate the second-order partial derivative, we have.

[0023] ML(x,y)=|2f(x,y)-f(x-step,y)-f(x+step,y)|+|2f(x,y)-f(x,y-step)-f(x,y+step|,

[0024] Pick

[0025] Where T is a threshold. Only Laplace values ​​greater than T are accumulated. Finally, the detection parameters are obtained and compared with the detection threshold. If the detection threshold is reached, the image quality is judged to meet the preset evaluation criteria and the detection parameters are saved to the detection parameter library. Otherwise, the image quality is judged to not meet the preset evaluation criteria.

[0026] Furthermore, the variable step size step and the threshold T are positively correlated according to the noise level of the sequence image.

[0027] Furthermore, S3 includes:

[0028] S31, detecting image features, texture features, morphological features, and spatial relationship features containing rich local information in the current image, composing data structures and descriptors based on these detected feature points, and constructing basic data elements for image matching and splicing;

[0029] S32, after obtaining the feature points and feature description of the current image, perform feature matching with the previous image. If the match is successful, proceed to step S34; otherwise, proceed to step S33; the feature library is a collection of feature points of all previous images;

[0030] S33, feature matching is performed with the feature points in the feature library, and in the matching process, the matching is performed gradually from the current image position to the feature points of the surrounding adjacent images. If the match is successful, the process proceeds to step S34; otherwise, it means that the current image has moved out of the range of the stitched image, and the status display needs to be refreshed, indicating that the stage needs to be moved back to the original stitched image position;

[0031] S34, extracting excellent matching points from the feature points of the current image and storing them in a feature library for use in subsequent image matching;

[0032] S35. Calculate the coverage of successfully matched feature points, determine the matching area and current image position information, calculate the movement distance between the image and the previous stitched image, and the position in the stitched large image, determine the absolute offset distance of the image movement, thereby extracting the first image and the second image, and enter the flowchart of the dual-projection automatic stitching algorithm.

[0033] S4 uses a dual-projection automatic stitching algorithm, which includes further projecting the current image and the previous stitched image in the horizontal and vertical directions according to the initial stitching results to obtain projection histograms in two directions. Based on the histograms, horizontal and vertical one-dimensional correlation analysis is performed to obtain the horizontal and vertical offsets of the image, thereby stitching the images to achieve sub-pixel stitching accuracy, improving image stitching accuracy, and reducing the two-dimensional correlation analysis to two one-dimensional correlation analyses, greatly improving stitching efficiency.

[0034] Furthermore, let the horizontal projection difference histogram of the first image be the reference histogram T X , whose size is K; the horizontal projection difference histogram of the second image is the reference histogram S X , whose size is M; the benchmark histogram T X Overlaid on the reference histogram S X Upward translation, reference histogram T XThe search image under the cover is sub-image S x i , i is the lower left corner pixel of this sub-image in S X The coordinates in the graph are used as reference points, and the similarity measure is,

[0035]

[0036] Where, the numerator is the subgraph S x i With the benchmark histogram T X The cross-correlation of the denominator is the subgraph S x i The energy of , normalized to , is

[0037]

[0038] When R x When (i) takes the maximum value, (i) is the correct matching position on the x-axis;

[0039] Similarly, let the horizontal projection difference histogram of the first image be the reference histogram T y , whose size is L; the horizontal projection difference histogram of the second image is the reference histogram S y , whose size is N; the benchmark histogram T y Overlaid on the reference histogram S y Upward translation, reference histogram T y The search image under the cover is sub-image S y j , j is the lower left corner pixel of this sub-image in S y The coordinates in the graph are used as reference points, and the similarity measure is,

[0040]

[0041] Where, the numerator is the subgraph S y j With the benchmark histogram T y The cross-correlation of the denominator is the subgraph S y j The energy of , normalized to , is

[0042]

[0043] When R y When (j) takes the maximum value, (j) is the correct matching position on the y-axis.

[0044] Furthermore, the dynamic display of the current image and the corresponding operation prompts are displayed on the human-computer interaction interface for the user to operate and complete the image stitching; the dynamic display of the current image includes one or more of the following: the current image is a blurred image, the current image is a focused image, the offset distance and positioning of the current image and the previous stitched image, the matching area of ​​the current image and the previous image, the current image and the previous image are not in the matching area, and the current image is in the position of the stitched surrounding images.

[0045] Another object of the present invention is to provide a microscope image intelligent stitching device, the device comprising:

[0046] A microscope having a stage movable along an X / Y axis and a focusing unit;

[0047] A camera, wherein the camera is an electronic rolling shutter camera;

[0048] A computer device having a processor and a human-computer interaction unit; the processor is connected to the camera to execute the aforementioned intelligent microscope image stitching method; the interactive interface of the human-computer interaction unit displays dynamic images and operation prompts during the execution process.

[0049] Another object of the present invention is to provide a computer-readable storage medium, wherein the storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the aforementioned intelligent stitching method for microscope images.

[0050] Another object of the present invention is to provide a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded by the processor and executed by the aforementioned intelligent stitching method for microscope images.

[0051] After adopting the above technical solution, the present invention has the following advantages compared with the background technology:

[0052] 1. The present invention only requires one camera to acquire real-time images. Furthermore, the image acquisition process is independent of the stage's degree of automation, making it universally applicable to most optical microscopes. For example, a conventional microscope equipped with a popular digital camera can perform the same function as a digital slide scanner scanning digital slides. In the dual-projection automatic stitching algorithm, the present invention reduces high-precision image stitching from two-dimensional spatial correlation analysis calculations to two one-dimensional spatial correlation analysis calculations, thereby increasing image stitching speed, accuracy, and precision.

[0053] 2. After the current image is stitched together, the stage can move quickly across most of the stitched field of view and quickly move to the vicinity of the next field of view to be stitched together. At this time, the movement speed needs to be reduced to reduce the impact of rolling shutter exposure on image movement, obtain an image that meets the requirements, and complete the stitching of the next field of view. This cycle is repeated to form a rhythmic stitching process, which not only achieves efficient image acquisition but also ensures the high quality of the acquired image.

[0054] 3. The present invention avoids image focus deviation by introducing an improved Laplace operator. At the same time, a variable step size between pixels and a threshold are introduced in the calculation process, which significantly improves the detection effect. The detection results have good unimodality, unbiasedness, a very obvious change trend near the focal plane, and high sensitivity.

[0055] 4. The present invention takes into account the optimization processing of feature library matching in the image matching method, and gradually matches the feature points of the surrounding adjacent images from the current image position, which greatly reduces the time spent on traversing the feature library. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 The figure is a flow chart of the microscope image intelligent stitching method of the present invention.

[0057] Figure 2 Comparison diagram of global shutter exposure and rolling shutter exposure principles.

[0058] Figure 3 Flowchart for real-time detection of image motion blur.

[0059] Figure 4 Flowchart for image feature extraction and image matching.

[0060] Figure 5 Flowchart of the dual-projection automatic stitching algorithm.

[0061] Figure 6 Screen display showing various states during the dynamic splicing process.

[0062] Figure 7 Schematic diagram of computer equipment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0064] Example 1

[0065] Please refer to Figure 1As shown, the present invention discloses a method for intelligent stitching of microscope images, which includes four core steps as shown in the flowchart:

[0066] S1. Image acquisition: During the movement of the stage, the camera's rolling shutter maintains continuous exposure, and the processor acquires the real-time dynamic image of the exposure;

[0067] S2, detection of image motion blur: transform the image, perform edge detection on the image using the Laplace edge detection algorithm, and determine whether the image quality meets the preset evaluation criteria. If so, proceed to S3, otherwise return to S1;

[0068] S3, image processing and matching: extracting a first image and a second image, where the first image is a currently acquired dynamic image, and the second image is an image extracted from the same position of the spliced ​​large image at the same position as the current dynamic image;

[0069] S4. Automatic stitching of dual-projection images: The initial stitching results of the current image and the previous stitched image are projected in the horizontal and vertical directions of the image respectively to obtain projection histograms in two directions. Based on the projection histograms, horizontal and vertical one-dimensional correlation analysis is performed to obtain the horizontal and vertical offsets of the image, thereby stitching the images.

[0070] In S1, the stage has a working feed rate and a fast forward rate. Thus, when the current real-time dynamic image is stitched, the stage switches from the working feed rate to the fast forward rate and moves quickly. When the stage moves to the boundary of the currently stitched large image, the stage switches from the fast forward rate to the working feed rate until the next image to be stitched is completed.

[0071] To understand the advantages of setting different feed rates for the stage, please refer to Figure 2 As shown, Figure 2 This diagram compares the principles of global shutter and rolling shutter exposure. A global shutter camera starts exposing all lines simultaneously at the start of exposure, and ends exposure simultaneously at the end of exposure, ensuring that all pixels in the entire image are exposed for the same duration. In contrast, a rolling shutter camera uses different exposure times for each line of the camera's image sensor. For example, the second line's exposure starts and ends with a fixed delay from the first, the second line's exposure ends with a fixed delay from the first, the third line's exposure starts with a fixed delay from the second, the third line's exposure ends with a fixed delay from the second, and so on, until exposure of the Nth line begins and ends. This exposure mechanism of a rolling shutter camera has no effect on static images, but does affect moving images. The faster the moving image, the greater the effect, resulting in image smearing.

[0072] Therefore, the rolling shutter camera's exposure during microscope stage movement, speed, and exposure significantly impact image quality. To ensure that the acquired image meets both image quality and stitching requirements, the stage's movement speed must be adjusted and reduced during image stitching. After stitching the current image, the stage can move rapidly across the majority of the stitched field of view and quickly move to the vicinity of the next field of view to be stitched. At this point, the stage's speed must be reduced to minimize the impact of the rolling shutter exposure on image movement, obtain an image that meets the requirements, and complete the stitching of the next field of view. This cycle repeats, forming a rhythmic stitching process. This process ensures both image quality and acquisition speed by enabling the electronic rolling shutter camera to detect image motion blur in real time during stage movement, calculate and dynamically display various states of the current image, and coordinate the adjustment of the stage's movement speed.

[0073] S2 is a real-time detection method for image motion blur. It segments the target area of ​​the image acquired during the movement of the microscope stage. By introducing an improved Laplace operator, it calculates the image focus quality and determines the degree of image blur. According to the evaluation criteria, it determines whether the current image meets the quality threshold and is accepted, or does not meet the image quality threshold and is rejected.

[0074] Specifically, it includes:

[0075] S21, converting the current image from an RGB image to a grayscale image;

[0076] S22. Sampling the converted grayscale image, performing edge detection, extracting image clarity parameters and normalizing the image.

[0077] In the prior art, there are many image edge detection methods, such as Roberts, Prewitt, Sobel, Laplace, Canny operators, etc. This implementation adopts an optimized Laplace edge detection algorithm. The Laplace edge detection algorithm is a second-order partial derivative that can obtain high-frequency components to detect sharp edges. The linear differential Laplace operator can be used as an estimator of high-frequency components. Since the second-order partial derivatives in the x-direction and the y-direction may have opposite signs and cancel each other out, the focus of the image will deviate. We introduce the improved Laplace operator and take the sum of the absolute values ​​of the second-order partial derivatives, then:

[0078]

[0079] Differences are generally used instead of differentials for calculation. Previous edge detection functions generally approximated the Laplace operator as a 3*3 operator. Here, considering the texture variations of microscopic images, a variable step size between pixels is used to calculate the second-order partial derivative.

[0080] Right now:

[0081] Pick

[0082] Where T is a threshold, and only Laplace values ​​greater than T are included in the accumulation.

[0083] Conventional edge detection using the Laplace operator ignores the role of step size and threshold. However, the introduction of step size and threshold significantly improves detection performance, resulting in good unimodal properties, unbiasedness, a very clear trend of change near the focal plane, and high sensitivity.

[0084] It's easy to understand that larger step sizes and thresholds don't necessarily mean better focusing results. Generally, for image sequences with minimal noise interference, the step size and threshold can be smaller; whereas for image sequences with significant noise and brightness variations, the step size and threshold can be slightly larger. For example, with a high-magnification objective, the step size and threshold can be smaller, while with a low-magnification objective, the step size and threshold can be slightly larger.

[0085] Adjust the Laplace operator coefficients based on the type of microscopy section to achieve optimal detection results. The software interface is refreshed with the detection result parameters, dynamically displaying the sharpness and focus clarity of the current image. The detection parameters are compared with the detection threshold. If the detection threshold is reached, the image quality is determined to meet the preset evaluation criteria. The detection parameters are saved to the detection parameter library, and the next step of the image feature extraction and image matching process is entered. Otherwise, the image quality is determined to not meet the preset evaluation criteria, and the interface status is refreshed to return to microscope operation.

[0086] S3 is used for image feature extraction and matching. Its methods include extracting image geometric features, density features, texture features, and data structures and descriptors that describe image features, which contain rich local information. This process constructs the basic data elements for image recognition, decision-making, and intelligent splicing, and forms a feature library. The matching process performs feature matching between the current image and the previous spliced ​​image, or performs feature matching within the feature library. The image's movement distance and its position within the spliced ​​image are calculated, and the absolute offset distance of the image movement is determined.

[0087] S3 includes:

[0088] S31, detecting image features, texture features, morphological features, and spatial relationship features containing rich local information in the current image, composing data structures and descriptors based on these detected feature points, and constructing basic data elements for image matching and splicing;

[0089] S32: After obtaining the feature points and feature description of the current image, perform feature matching with the previous image. If the match is successful, proceed to step S34; otherwise, it means that there is no overlapping area between the current image and the previous image, and proceed to step S33 to perform feature matching with the feature points in the feature library; the feature library is a collection of feature points of all previous images, and as the stitching distance increases, the feature library also expands.

[0090] S33, feature matching is performed with the feature points in the feature library, and in the matching process, the feature points of the current image are gradually matched towards the feature points of the surrounding adjacent images. If the match is successful, the process proceeds to step S34; otherwise, the state is refreshed, and the stage is prompted or controlled to return to the original stitched image position;

[0091] S34. Extract excellent matching points from the feature points of the current image and store them in a feature library for use in subsequent image matching. Excellent matching points can be extracted using the Harris corner detection method, and feature parameters (such as the number of adjacent corner points, the quality level of the corner points, and the minimum distance feature parameter between two corner points) are set.

[0092] S35. Calculate the coverage of the successfully matched feature points, determine the matching area and the current image position information, calculate the movement distance between the image and the previous stitched image, and the position in the stitched large image, thereby extracting the first image and the second image and entering the dual-projection automatic stitching process.

[0093] The dual-projection automatic stitching algorithm described in S4 includes further projecting the current image and the previous stitched image in the horizontal direction and the vertical direction of the image according to the results of the initial stitching to obtain projection histograms in two directions, performing two-dimensional horizontal and vertical correlation analysis based on the histograms to obtain the horizontal and vertical offsets of the image, thereby stitching the images to achieve sub-pixel stitching accuracy, improve image stitching accuracy, and reduce the two-dimensional correlation analysis to two one-dimensional correlation analyses, thereby greatly improving stitching efficiency.

[0094] At the same time, during the stitching process, image memory capacity is dynamically expanded as the stitched image gradually expands, adaptively stretching the image size and display ratio. In other words, dynamic image memory expansion means initially allocating a small amount of memory. As the stitched image grows, additional memory is allocated as the memory becomes insufficient. This ensures that the allocated memory is used appropriately, improving system performance while preserving valuable system memory resources, allowing them to be used for other system software processes.

[0095] To facilitate understanding of the superiority of the dual-projection automatic stitching algorithm of this embodiment, a feasible method is first introduced.

[0096] One feasible method for high-precision sub-pixel image stitching of two images is to use two-dimensional correlation analysis of images. The method is as follows: suppose the base image T (the size of T is K×L) is superimposed on the reference image S (the size of S is M×N) and translated. The search image covered by the base image T is called the sub-image S. i,j , i, j are the coordinates of the lower left corner pixel of this sub-image in the S image as reference points. It is not difficult to understand that the value range of i and j is 1≤i≤M-K+1, 1≤j≤N-L+1.

[0097] Now we can compare T and S i,j If the two are consistent, then T and S i,j The difference is zero. So we can use the following measure to measure T and S i,j The similarity of , then,

[0098]

[0099] Expand the formula, and we have

[0100]

[0101] The third term on the right side of the formula represents the total energy of the reference image T, which is a constant and has nothing to do with (i, j). The first term is the energy of the sub-image covered by T, which changes slowly with the position of (i, j). The second term is the cross-correlation between the sub-image and the reference image, which changes with (i, j). T and S i,j This value is the largest when matching, so the following related function can be used to measure similarity, then,

[0102]

[0103] Normalized to:

[0104]

[0105] It is the base graph T and subgraph S i,j According to the Schwarz inequality, R(i,j) takes values ​​between 0 and 1, and only when the ratio S i,j When (m,n) / T(m,n) is a constant, R(i,j) reaches its maximum value (equal to 1). According to the similarity principle, the larger the R(i,j) value, the closer the two are. When R(i,j) reaches its maximum value, (i,j) is the correct matching position.

[0106] This type of two-dimensional image correlation analysis can produce high-precision sub-pixel image stitching results, but it requires a lot of computing time. This implementation proposes an optimization scheme that decomposes the two-dimensional image correlation analysis into two one-dimensional image correlation analyses, which can greatly optimize the stitching performance and increase the stitching speed. The method is as follows: the first image and the second image are projected horizontally and vertically respectively to obtain four projected one-dimensional sequence data. After differentiating these four projected one-dimensional sequence data, they are organized into the form of a one-dimensional grayscale projection difference histogram. Therefore, a one-dimensional correlation analysis can be performed. The horizontal projection difference histogram of the first image is correlated with the horizontal projection difference histogram of the second image to obtain a high-precision optimal coupling point, that is, the sub-pixel x offset value. The vertical projection difference histogram of the first image is correlated with the vertical projection difference histogram of the second image to obtain a high-precision optimal coupling point, that is, the sub-pixel y offset value.

[0107] The specific method is:

[0108] Let the horizontal projection difference histogram of the first image be the reference histogram T X , whose size is K; the horizontal projection difference histogram of the second image is the reference histogram S X , whose size is M; the benchmark histogram T X Overlaid on the reference histogram S X Upward translation, reference histogram T X The search image under the cover is sub-image S x i , i is the lower left corner pixel of this sub-image in S X Taking the coordinates in the figure as reference points, it is not difficult to understand that the value range of i is 1≤i≤M.

[0109] In this way, we can compare T X and S X i If the two are consistent, then T X and S X i The difference is zero. So we can use the following measure to measure T X and S x i degree of similarity.

[0110] Then the similarity measure is,

[0111]

[0112] Then its expansion is

[0113]

[0114] In the formula, the third term on the right represents the reference graph TX The total energy is a constant, independent of (i), and the first term is T X The energy of the sub-image covered by it changes slowly with the position of (i). The second term is the cross-correlation between the sub-image and the reference image, which changes with (i). X and S X i This value is maximized when matching, so the following related functions can be used as similarity measures:

[0115]

[0116] Normalized to:

[0117]

[0118] The above formula is the reference graph T X With subgraph S X i According to the Schwarz inequality, Rx(i) takes values ​​between 0 and 1, and only when the ratio S X i (m) / T X When (m) is a constant, Rx(i) reaches its maximum value (equal to 1). According to the similarity principle, the larger the value of Rx(i), the closer the two are. When Rx(i) reaches its maximum value, (i) is the correct matching position on the x-axis.

[0119] Similarly, let the first vertical projection difference histogram be the reference histogram T y (T y The size of L) is superimposed on the second vertical projection difference histogram as the reference histogram S y (S y The size of N) is translated on the reference graph T y The search image under the cover is called sub-image S y j , j is the lower left corner pixel of this sub-image in S y Taking the coordinates in the figure as reference points, it is not difficult to understand that the value range of j is 1≤j≤L.

[0120] Now we can compare T y and S y j If the two are consistent, then T y and S y j The difference is zero. So we can use the following measure to measure T y and S y j degree of similarity.

[0121]

[0122] Its expanded form is,

[0123]

[0124] The third term on the right side represents the total energy of the reference graph T, which is a constant and has nothing to do with (j). The first term is T y The energy of the sub-image covered by it changes slowly with the position (j). The second term is the cross-correlation between the sub-image and the reference image, which changes with (j). T and S i This value is maximized when matching, so the following related functions can be used as similarity measures:

[0125]

[0126] Normalized to:

[0127]

[0128] The above formula is the correlation coefficient between the base graph Ty and the subgraph Syj. According to Schwarz’s inequality, R y (j) takes values ​​between 0 and 1 and is only valid for the ratio S y j When (n) / T(n) is a constant, R y (j) takes the maximum value (equal to 1). According to the similarity principle, R y (j) The larger the value, the closer the two are. When R y When (j) takes the maximum value, (j) is the correct matching position on the y-axis.

[0129] According to the obtained x and y matching offsets, the current image is fused into the stitched large image. The fusion of this part can adopt a gradual in-and-out image gradient fusion method. Its implementation can refer to patent document CN202011053664, that is, the fusion adopts an optimized nonlinear interpolation method and a configurable fusion area overlap parameter alpha value to obtain the best fusion effect.

[0130] Save the fused stitched image and related parameters, refresh the dynamic display status, and start the next round of operations.

[0131] Example 2

[0132] Another object of the present invention is to provide a device for intelligently stitching microscope images, which includes a microscope, a camera, and a computer device.

[0133] The microscope has a stage movable along the X / Y axis and a focusing unit. The stage can be an automatic stage or a manual stage, and the focusing unit can be a manually rotated focusing knob or an automatic focusing unit, which is not specifically limited in this application.

[0134] The camera is an electronic rolling shutter camera, and a popular digital camera can be used.

[0135] A computer device comprising a processor and a human-computer interaction unit, which may be a desktop computer, a laptop computer, or a smart mobile device. The processor is connected to the camera in a manner configured to execute the microscope image intelligent stitching method described in Example 1; the human-computer interaction unit's interactive interface displays dynamic image information and operational prompts during execution.

[0136] like Figure 6 As shown in the figure, the human-computer interaction interface is a software screen display diagram for displaying various states in the dynamic stitching process. Frame A represents the previous frame image; Frame B represents the image of the current frame; Frame C indicates that the clarity of the current dynamic image is not enough, and it is necessary to manually fine-tune the focus of the microscope to obtain a clear dynamic image; Frame D indicates that the stitching point can no longer be found in the current image, and the X / Y axis can be moved to find the stitching point again.

[0137] Example 3

[0138] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one instruction or at least one program. The at least one instruction or the at least one program is loaded and executed by a processor to implement the intelligent stitching method of microscope images as described in the previous embodiment 1.

[0139] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media provided in this application and used in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAM bus dynamic RAM (RDRAM), etc.

[0140] Example 4

[0141] Please refer to Figure 7 As shown, another object of the present invention is to provide a computer device, which includes a processor and a memory. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The computer program is loaded by the processor and executes the microscope image intelligent stitching method as described in Example 1. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data such as interface access methods. The network interface of the computer device is used to communicate with an external terminal through a network connection.

[0142] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

[0143] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A microscope image intelligent stitching method, characterized in that: include: S1, real-time exposure of the camera to obtain real-time dynamic images of the microscope, the camera is an electronic rolling shutter camera; S2, performing real-time detection of image motion blur on the acquired real-time dynamic image to determine whether the image quality meets the preset evaluation standard. If so, proceed to S3; otherwise, return to S1; S3, extracting a first image and a second image, wherein the first image is a currently acquired dynamic image, and the second image is an image extracted from the same position of the spliced ​​large image as the current dynamic image; S4, projecting the current dynamic image and an image extracted from the same position of the stitched large image at the same position as the current dynamic image in the horizontal direction and the vertical direction of the image respectively to obtain projection histograms in the two directions, performing horizontal and vertical one-dimensional correlation analysis based on the projection histograms to obtain horizontal and vertical offsets of the images, thereby stitching the images; In S1, the stage has a working feed rate and a fast feed rate; When the current real-time dynamic image has been stitched, the stage switches from the working speed to the fast speed and moves quickly; When the stage moves to the boundary of the currently stitched large image, the stage switches from the fast-forward rate to the working-forward rate until the stitching of the next image to be stitched is completed.

2. The microscope image intelligent stitching method according to claim 1, wherein S2 include: S21, converting the current image from an RGB image to a grayscale image; S22, sampling the converted grayscale image, performing edge detection, extracting image clarity parameters and normalizing; It uses the Laplace edge detection algorithm to detect edges, and introduces an improved Laplace operator to take the sum of the absolute values ​​of the second-order partial derivatives for the two-dimensional image function. , the Laplace operator of the image is, Using difference instead of differential for calculation, considering the texture change of microscopic image, using variable step size between pixels to calculate the second-order partial derivative, we have: ， Pick , Where T is a threshold. Only Laplace values ​​greater than T are accumulated. Finally, the detection parameters are obtained and compared with the detection threshold. If the detection threshold is reached, it is determined that the image quality meets the preset evaluation criteria and the detection parameters are saved in the detection parameter library. Otherwise, it is determined that the image quality does not meet the preset evaluation criteria.

3. The method for intelligent microscopic image stitching according to claim 2, wherein: The variable step size step and the threshold T are positively correlated according to the noise level of the sequence image.

4. The microscope image intelligent stitching method according to claim 1, wherein: S4 is specifically: Let the horizontal projection difference histogram of the first image be the reference histogram T X , whose size is K; the horizontal projection difference histogram of the second image is the reference histogram S X , whose size is M; the benchmark histogram T X Overlaid on the reference histogram S X Upward translation, reference histogram T X The search image under the cover is sub-image S x i , i is the lower left corner pixel of this sub-image in S X The coordinates in the graph are used as reference points, and the similarity measure is, ， Where, the numerator is the subgraph S x i With the benchmark histogram T X The cross-correlation of the denominator is the subgraph S x i The energy of , normalized to , is ， When R x (i) When the value is the largest, i is the correct matching position on the x-axis; Similarly, let the vertical projection difference histogram of the first image be the reference histogram T y , whose size is L; the vertical projection difference histogram of the second image is the reference histogram S y , whose size is N; the benchmark histogram T y Overlaid on the reference histogram S y Upward translation, reference histogram T y The search image under the cover is sub-image S y j , j is the lower left corner pixel of this sub-image in S y The coordinates in the graph are used as reference points, and the similarity measure is, ， Where, the numerator is the subgraph S y j With the benchmark histogram T y The cross-correlation of the denominator is the subgraph S y j The energy of , normalized to , is ， When R y When (j) takes the maximum value, j is the correct matching position on the y-axis.

5. The method for intelligent microscopic image stitching according to claim 1, wherein: The dynamic display of the current image and the corresponding operation prompts are displayed on the human-computer interaction interface for the user to operate and complete the image stitching; the dynamic display of the current image includes one or more of the following: the current image is a blurred image, the current image is a focused image, the offset distance and positioning of the current image and the previous stitched image, the matching area of ​​the current image and the previous image, the current image and the previous image are not in the matching area, and the current image is in the position of the stitched surrounding images.

6. Microscope image intelligent stitching device, characterized in that: The device comprises: A microscope having a stage movable along an X / Y axis and a focusing unit; A camera, wherein the camera is an electronic rolling shutter camera; A computer device comprising a processor and a human-computer interaction unit; the processor is connected to the camera and is used to execute the microscope image intelligent stitching method according to any one of claims 1 to 5; the interactive interface of the human-computer interaction unit displays dynamic images and operation prompts during execution.

7. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the microscope image intelligent stitching method according to any one of claims 1 to 5.

8. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded by the processor and executes the microscope image intelligent stitching method according to any one of claims 1 to 5.

Images