Microscopic image processing method, device and equipment
By extracting and matching multiple frames of images to be processed, combined with the image fusion algorithm, the problem of great uncertainty in image stitching is solved, and the stitching quality and efficiency are improved.
Patent Information
- Application Number
- CN202411876682.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-19
- Publication Date
- 2025-05-02
AI Technical Summary
There is great uncertainty in the current image stitching method, resulting in low quality of stitching images.
By extracting features of multiple frames of images to be processed, filtering intermediate feature points, and using feature descriptors to perform feature matching, aligning the images to the target coordinate system, and then using image fusion algorithm for stitching and fusion.
Improve the splicing quality and splicing efficiency, reduce image distortion after splicing, and eliminate obvious suture lines.
Smart Images
Figure CN119919934A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image splicing, and in particular to a microscopic image processing method, device and equipment. Background Art
[0002] Image stitching technology is to stitch a series of partially overlapping pictures of the same scene into a large, wide-angle, high-resolution image. The stitched image is required to be as close to the original image as possible, with the smallest possible image distortion and no obvious stitching lines. Image stitching requires that there must be an overlapping area between the adjacent boundaries of the two images, and the overlapping area represents the same scene content. In cell observation research, slide images of cultured cells can be obtained through an automatic optical microscope. Because the field of view of the microscope is much smaller than the size of the slide, only a series of partially overlapping cell images can be obtained. Therefore, it is necessary to accurately stitch these cell images in order to obtain more accurate information on cell growth, cell density, etc. However, the current image stitching method often directly translates the image, resulting in great uncertainty, which affects the final stitching quality. Summary of the invention
[0003] The purpose of the present invention is to solve the problem that the current image stitching method has great uncertainty, resulting in low quality of the stitched image, and provide a microscopic image processing method, device and equipment.
[0004] The technical solution of the embodiment of the present application is implemented as follows: A first aspect of an embodiment of the present application provides a microscopic image processing method, comprising: Perform feature extraction on multiple frames of images to be processed to obtain intermediate feature points; the images to be processed include common feature points and non-common feature points; The intermediate feature points are screened, and feature matching is performed on the screened intermediate feature points in different images to be processed using feature descriptors, so as to align multiple frames of the images to be processed to a target coordinate system; In the target coordinate system, multiple frames of the to-be-processed images are spliced and fused using an image fusion algorithm to obtain a target image.
[0005] Optionally, before extracting features from multiple frames of images to be processed to obtain intermediate feature points, the method further includes: Preprocessing is performed on the multiple frames of images to be processed; the preprocessing includes denoising, contrast enhancement and smoothing.
[0006] Optionally, the filtering of the intermediate feature points, and using feature descriptors to perform feature matching on the filtered intermediate feature points in different images to be processed, and aligning multiple frames of the images to be processed to a target coordinate system, includes: Using a random sampling consensus algorithm to eliminate erroneous matching points in the intermediate feature points, and determine common feature points between different images to be processed; The common feature points are matched by using feature descriptors, and multiple frames of the images to be processed are aligned to a target coordinate system.
[0007] Optionally, the performing feature matching on the common feature points by using feature descriptors to align multiple frames of the to-be-processed images to a target coordinate system includes: Determine the initial translation amount between the common feature points by using a feature descriptor combined with a Fourier transform method; Optimizing the initial translation amount based on a hill climbing algorithm to obtain a final translation amount; The final translation amount is used to align multiple frames of the to-be-processed images to a target coordinate system.
[0008] A second aspect of the present application provides a microscopic image processing device, comprising: a feature extraction module, a feature matching module and an image fusion module; wherein: The feature extraction module is configured to extract features from multiple frames of images to be processed to obtain intermediate feature points; the images to be processed include common feature points and non-common feature points; The feature matching module is configured to screen the intermediate feature points, and use feature descriptors to perform feature matching on the screened intermediate feature points in different images to be processed, so as to align multiple frames of the images to be processed to a target coordinate system; The image fusion module is configured to use an image fusion algorithm to splice and fuse multiple frames of the to-be-processed images in the target coordinate system to obtain a target image.
[0009] Optionally, a preprocessing module is further included, and the preprocessing module is configured as follows: Preprocessing is performed on the multiple frames of images to be processed; the preprocessing includes denoising, contrast enhancement and smoothing.
[0010] Optionally, the feature matching module is specifically configured as follows: Using a random sampling consensus algorithm to eliminate erroneous matching points in the intermediate feature points, and determine common feature points between different images to be processed; The common feature points are matched by using feature descriptors, and multiple frames of the images to be processed are aligned to a target coordinate system.
[0011] Optionally, the feature matching module is specifically configured as follows: Determine the initial translation amount between the common feature points by using a feature descriptor combined with a Fourier transform method; Optimizing the initial translation amount based on a hill climbing algorithm to obtain a final translation amount; The final translation amount is used to align multiple frames of the to-be-processed images to a target coordinate system.
[0012] A third aspect of an embodiment of the present application provides an electronic device, comprising a processor and a memory; the memory stores a computer program, wherein the computer program implements the microscopic image processing method described in the first aspect when executed by the processor.
[0013] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.
[0014] Compared with the prior art, the technical solution provided by this application has the following beneficial effects: The present invention provides a microscopic image processing method, device and equipment, which extract features from multiple frames of images to be processed to obtain intermediate feature points, screen the intermediate feature points, and use feature descriptors to perform feature matching on the screened intermediate feature points in different images to be processed, align the multiple frames of images to be processed to a target coordinate system, and use an image fusion algorithm to splice and fuse the multiple frames of images to be processed in the target coordinate system to obtain a target image. By matching the common feature points between different images to be processed, the coordinates of the features are unified, thereby achieving fusion transition of the overlapping areas of different images to be processed, completing image splicing, and improving the splicing quality and splicing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1 A schematic diagram of a process of a microscopic image processing method provided in an embodiment of the present application; Figure 2 A schematic diagram of the software architecture of PyImageJ provided in the embodiment of the present application; Figure 3 A schematic diagram of the structure of a microscopic image processing device provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0016] Below, embodiments of the present application will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present application. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present application. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present application.
[0017] The terms used herein are only for describing specific embodiments and are not intended to limit the present application. The terms "include", "comprising", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.
[0018] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.
[0019] Some block diagrams and / or flow charts are shown in the accompanying drawings. It should be understood that some blocks or combinations thereof in the block diagrams and / or flow charts may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that these instructions, when executed by the processor, may create a device for implementing the functions / operations described in these block diagrams and / or flow charts.
[0020] Cell morphology can reflect the morphology, structure, function and other characteristics of cells, and is of great significance for evaluating the physiological and pathological states of cells and studying cell biological processes. In terms of cell image recognition algorithms, deep learning is one of the most popular technologies at present. Neural network models such as convolutional neural networks (CNN) and recurrent neural networks (RNN) have been widely used in cell image classification and recognition tasks, and have achieved good results. The basis for cell image recognition is to obtain accurate images.
[0021] In some embodiments, see Figure 1 , Figure 1 A schematic diagram of a process flow of a microscopic image processing method provided in an embodiment of the present application; A microscopic image processing method provided in an embodiment of the present application comprises: S110, extracting features from multiple frames of images to be processed to obtain intermediate feature points; the images to be processed include common feature points and non-common feature points.
[0022] In this embodiment, the image to be processed may be a glass slide image of cultured cells. Multiple frames of images to be processed are acquired through an automatic optical microscope. Considering that the field of view of the microscope is much smaller than the size of the glass slide, only a series of partially overlapping cell images can be acquired. The intermediate feature points may be one or more types of feature points on the image to be processed. Commonly used feature extraction algorithms include SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), etc. Here, the specific feature extraction algorithm is not limited and can be selected according to actual needs. It can be foreseen that there are common feature points between a frame of the image to be processed and one or more adjacent frames of the image, that is, feature points in the overlapping area, and there are non-common feature points, that is, feature points in the non-overlapping area.
[0023] In some embodiments, at S110, before extracting features from multiple frames of images to be processed to obtain intermediate feature points, the method further includes: Preprocessing is performed on multiple frames of images to be processed; the preprocessing includes denoising, contrast enhancement and smoothing.
[0024] Here, before extracting features from multiple frames of images to be processed, the images to be processed need to be preprocessed first, so as to better improve image quality and reduce subsequent feature errors. Preprocessing can include operations such as denoising, contrast enhancement, and smoothing.
[0025] S120, screening the intermediate feature points, and using feature descriptors to perform feature matching on the screened intermediate feature points in different images to be processed, so as to align multiple frames of images to be processed to a target coordinate system.
[0026] Feature descriptors are vectors used to describe pixel information around feature points in an image. They usually contain key information around feature points, such as gradients and textures, which can be used to measure similarities in images. After screening the intermediate feature points and removing some noise points, feature descriptors are used to complete feature point matching. Multiple frames of images to be processed can be aligned to the same coordinate system to complete preliminary stitching.
[0027] In some embodiments, S120, screening the intermediate feature points, and using feature descriptors to perform feature matching on the screened intermediate feature points in different images to be processed, and aligning multiple frames of images to be processed to a target coordinate system, includes: The random sampling consistency algorithm is used to eliminate the wrong matching points in the intermediate feature points and determine the common feature points between different images to be processed; Feature descriptors are used to match common feature points and align multiple frames of images to be processed to the target coordinate system.
[0028] In this embodiment, the random sampling consensus algorithm RANSAC is used to remove erroneous matching points in the intermediate feature points. After RANSAC filtering, common feature points can be obtained, and these common feature points have corresponding matching points in different images to be processed. Through these matching points, the transformation matrix between different images to be processed can be determined, thereby aligning multiple frames of images to be processed to the target coordinate system based on the transformation matrix. For multiple frames of images to be processed, repeating the above steps can ensure that each frame of the image to be processed is aligned relative to a common target coordinate system. It should be noted that the threshold and number of iterations of RANSAC can be adjusted according to the specific situation to achieve the best effect.
[0029] In some embodiments, feature matching is performed on common feature points using feature descriptors to align multiple frames of images to be processed to a target coordinate system, including: The initial translation between common feature points is determined by using feature descriptors combined with Fourier transform method; The initial translation amount is optimized based on the hill climbing algorithm to obtain the final translation amount; The final translation amount is used to align multiple frames of images to be processed to the target coordinate system.
[0030] Here, the initial translation obtained by Fourier transform is used as the initial estimate of the hill climbing algorithm. Starting from the initial translation, small changes in translation are tried in the neighborhood, and the translation that maximizes the objective function value is selected as the new estimate. This process is repeated until the change in the objective function value is less than a threshold or the maximum number of iterations is reached. The objective function can be a similarity measure between images. The final translation calculated for each frame of the image to be processed can be transformed into the target coordinate system.
[0031] S130, in the target coordinate system, using an image fusion algorithm to stitch and fuse multiple frames of images to be processed to obtain a target image.
[0032] In this embodiment, image fusion algorithms such as weighted average and Laplace pyramid fusion can be used to stitch and fuse the images to be processed. Since there may be problems such as uneven illumination and color differences in the image stitching process, it is necessary to perform color correction and remove stitching traces on the stitched panoramic image to improve the stitching quality. After the overall target image is obtained by stitching, the cells can be counted to determine their density, thereby analyzing the cell growth.
[0033] In an alternative embodiment, see Figure 2 , Figure 2Schematic diagram of the software architecture of PyImageJ provided in the embodiment of the present application; the above-mentioned microscopic image processing method can be applied to image processing software. The image processing software is designed based on the pyimagej library of the Python environment. One of the main advantages of the pyimagej library is the ability to use ImageJ and ImageJ2 related functions on the Python software ecosystem. The blue part on the left shows the Python environment and the sample Python application. The red part on the right shows the ImageJ2 software stack, which contains sample plug-ins running in a special Python integrated Java virtual machine (JVM). In the Python environment, PyImageJ uses JPype (from the scyjava layer) to create a Python integrated JVM that will run the ImageJ2 software stack. In the Java environment, this encapsulated JVM contains all user-requested Java libraries, including ImageJ, ImageJ2, and other plug-ins, such as plug-ins from Fiji and / or other ImageJ update sites. The Python top-level PyImageJ provides access to the ImageJ2 gateway and Python convenience functions. The imglyb on the Python side interfaces with the ImgLib2 on the Java side. Finally, the Pythonscyjava layer provides basic components such as JVM configuration and type conversion. Based on the PyImageJ plug-in, you can use imagej related functions in python to complete the software design. For the stitching part, its basic module includes input parameter settings and output parameter settings. The input parameters include grid width, grid height, file name mode, row start number, column start number, row start number, image path, etc.
[0034] The embodiment of the present invention extracts features from multiple frames of images to be processed to obtain intermediate feature points, screens the intermediate feature points, and uses feature descriptors to perform feature matching on the screened intermediate feature points in different images to be processed, aligns the multiple frames of images to be processed to a target coordinate system, and uses an image fusion algorithm to splice and fuse the multiple frames of images to be processed in the target coordinate system to obtain a target image. By matching the common feature points between different images to be processed, the coordinates of the features are unified, thereby achieving fusion transition of the overlapping areas of different images to be processed, completing image splicing, and improving the splicing quality and splicing efficiency.
[0035] In some embodiments, see Figure 3 , Figure 3 The present invention provides a schematic diagram of a microscopic image processing device according to an embodiment of the present invention; the present invention provides a microscopic image processing device 300, including: a feature extraction module 310, a feature matching module 320 and an image fusion module 330; wherein, The feature extraction module 310 is configured to extract features from multiple frames of images to be processed to obtain intermediate feature points; the images to be processed include common feature points and non-common feature points; The feature matching module 320 is configured to screen the intermediate feature points, and use the feature descriptors to perform feature matching on the screened intermediate feature points in different images to be processed, so as to align multiple frames of images to be processed to the target coordinate system; The image fusion module 330 is configured to use an image fusion algorithm to stitch and fuse multiple frames of images to be processed in a target coordinate system to obtain a target image.
[0036] In some embodiments, the microscopic image processing device 300 further includes a pre-processing module, and the pre-processing module is configured as follows: Preprocessing is performed on multiple frames of images to be processed; the preprocessing includes denoising, contrast enhancement and smoothing.
[0037] In some embodiments, the feature matching module 320 is specifically configured as follows: The random sampling consistency algorithm is used to eliminate the wrong matching points in the intermediate feature points and determine the common feature points between different images to be processed; Feature descriptors are used to match common feature points and align multiple frames of images to be processed to the target coordinate system.
[0038] In some embodiments, the feature matching module 320 is specifically configured as follows: The initial translation between common feature points is determined by using feature descriptors combined with Fourier transform method; The initial translation amount is optimized based on the hill climbing algorithm to obtain the final translation amount; The final translation amount is used to align multiple frames of images to be processed to the target coordinate system.
[0039] The microscopic image processing device provided in the embodiment of the present application can implement each process in the embodiment corresponding to the above-mentioned microscopic image processing method, and will not be described again here to avoid repetition.
[0040] It should be noted that the microscopic image processing device provided in the embodiment of the present application and the microscopic image processing method provided in the embodiment of the present application are based on the same application concept, so the specific implementation of this embodiment can refer to the implementation of the aforementioned microscopic image processing method, and the repeated parts will not be repeated.
[0041] In some embodiments, see Figure 4 , Figure 4The electronic device 400 provided in the embodiment of the present application includes a processor 410 and a memory 420; the memory 420 stores a computer program, wherein the computer program implements the above-mentioned microscopic image processing method when executed by the processor.
[0042] Specifically, the processor 410 may include, for example, a general-purpose microprocessor, an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 410 may also include an onboard memory for cache purposes. The processor 410 may be a single processing unit or multiple processing units for executing different actions of the method flow according to the embodiment of the present application.
[0043] The memory 420 may be any medium capable of containing, storing, conveying, propagating or transmitting instructions. For example, the memory 420 may include, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device, component or propagation medium. Specific examples of the memory 420 include: a magnetic storage device, such as a magnetic tape or a hard disk (HDD); an optical storage device, such as a compact disk (CD-ROM); a random access memory (RAM) or flash memory; and / or a wired / wireless communication link.
[0044] The present application also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned microscopic image processing method is implemented. The computer-readable medium may be included in the device / apparatus / system described in the above-mentioned embodiment; or it may exist independently without being assembled into the device / apparatus / system. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed, the method according to the embodiment of the present application is implemented.
[0045] According to an embodiment of the present application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, optical cable, radio frequency signal, etc., or any suitable combination of the above.
[0046] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in the various embodiments and / or claims of the present application may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present application. Therefore, the scope of the present application should not be limited to the above-described embodiments, but should be determined not only by the appended claims, but also by the equivalents of the appended claims.
Claims
1. A microscopic image processing method, characterized in that: include: Extract features from multiple frames of images to be processed to obtain intermediate feature points; the images to be processed include common feature points and non-common feature points; The intermediate feature points are screened, and feature matching is performed on the screened intermediate feature points in different images to be processed using feature descriptors, so as to align multiple frames of the images to be processed to a target coordinate system; In the target coordinate system, multiple frames of the to-be-processed images are spliced and fused using an image fusion algorithm to obtain a target image.
2. The microscopic image processing method according to claim 1, characterized in that: Before extracting features from multiple frames of images to be processed to obtain intermediate feature points, the method further includes: The multiple frames of images to be processed are preprocessed; the preprocessing includes denoising, contrast enhancement and smoothing.
3. The microscopic image processing method according to claim 1, characterized in that: The screening of the intermediate feature points, and using feature descriptors to perform feature matching on the screened intermediate feature points in different images to be processed, and aligning multiple frames of the images to be processed to a target coordinate system, include: Using a random sampling consensus algorithm to eliminate erroneous matching points in the intermediate feature points, and determine common feature points between different images to be processed; The common feature points are matched by using feature descriptors, and multiple frames of the images to be processed are aligned to a target coordinate system.
4. The microscopic image processing method according to claim 3, characterized in that: The using feature descriptors to perform feature matching on the common feature points and aligning multiple frames of the to-be-processed images to a target coordinate system includes: Determine the initial translation amount between the common feature points by using a feature descriptor combined with a Fourier transform method; Optimizing the initial translation amount based on a hill climbing algorithm to obtain a final translation amount; The final translation amount is used to align multiple frames of the to-be-processed images to a target coordinate system.
5. A microscopic image processing device, characterized in that: include: Feature extraction module, feature matching module and image fusion module; among them, The feature extraction module is configured to extract features from multiple frames of images to be processed to obtain intermediate feature points; the images to be processed include common feature points and non-common feature points; The feature matching module is configured to screen the intermediate feature points, and use feature descriptors to perform feature matching on the screened intermediate feature points in different images to be processed, so as to align multiple frames of the images to be processed to a target coordinate system; The image fusion module is configured to use an image fusion algorithm to splice and fuse multiple frames of the to-be-processed images in the target coordinate system to obtain a target image.
6. The microscopic image processing device according to claim 5, characterized in that: It also includes a preprocessing module, wherein the preprocessing module is configured as follows: The multiple frames of images to be processed are preprocessed; the preprocessing includes denoising, contrast enhancement and smoothing.
7. The microscopic image processing device according to claim 5, characterized in that: The feature matching module is specifically configured as follows: Using a random sampling consensus algorithm to eliminate erroneous matching points in the intermediate feature points, and determine common feature points between different images to be processed; The common feature points are matched by using feature descriptors, and multiple frames of the images to be processed are aligned to a target coordinate system.
8. The microscopic image processing device according to claim 7, characterized in that: The feature matching module is specifically configured as follows: Determine the initial translation amount between the common feature points by using a feature descriptor combined with a Fourier transform method; Optimizing the initial translation amount based on a hill climbing algorithm to obtain a final translation amount; The final translation amount is used to align multiple frames of the to-be-processed images to a target coordinate system.
9. An electronic device comprising a processor and a memory; the memory stores a computer program, wherein: The computer program implements the microscopic image processing method according to any one of claims 1 to 4 when executed by the processor.
10. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.