Image enhancement method based on photoacoustic microscopy imaging

Abstract

Disclosed in the present invention is an image enhancement method based on photoacoustic microscopy imaging. The method comprises: executing multiple photoacoustic microscopy scannings on a target object to obtain three-dimensional data of the target, and obtaining scanning results of the target on a time axis by means of continuous scanning imaging; before performing a projection operation on a single imaging result, for the three-dimensional data, within a set search range, combining signals of adjacent areas in space dimension; and for the scanning results on the time axis, within a set search range, combining signals of adjacent areas in time dimension, so as to obtain an enhanced image. The present invention effectively improves the image signal-to-noise ratio of a photoacoustic microscopy system, and improves the calculation efficiency.

Description

An image enhancement method based on photoacoustic microscopy Technical Field

[0001] The present invention relates to the technical field of photoacoustic imaging, and more particularly to an image enhancement method based on photoacoustic microscopy. Background Art

[0002] With the rapid development of medical imaging technology, photoacoustic imaging, as a non-invasive imaging method, has emerged as a leader in medical imaging due to its combination of the high contrast of optical imaging and the high resolution and penetration depth of ultrasound imaging. Photoacoustic imaging surpasses the limitations of traditional optical imaging (~1 mm), extending penetration depth to the centimeter level, making it one of the fastest-growing technologies in medical imaging. Current imaging systems typically perform maximum projection on scanned three-dimensional images to generate projection images, which are then processed by algorithms. This approach often results in a limited signal-to-noise ratio (SNR) for the images obtained.

[0003] In existing technologies, there are two common image enhancement methods. One is based on the 3D Block Matching algorithm (BM3D) to denoise raw data, using the self-similarity of the image to perform frequency domain transformation to denoise and improve the image's signal-to-noise ratio. The other is deep learning, which uses convolutional neural networks to enhance the image's signal-to-noise ratio using low-noise images within the field of view as real datasets.

[0004] After analysis, the existing image enhancement solutions have the following main defects:

[0005] (1) The collaborative filtering and Wiener filtering operations that constitute the algorithm take a long time. When the noise intensity is large, the denoised image often loses details such as edges and textures, resulting in image blur and other problems.

[0006] (2) Deep learning requires a large amount of image data to prepare training sets, validation sets, and test sets, which results in a lot of time spent on data collection. In addition, the training process is computationally intensive and has high computational time costs.

[0007] (3) Existing solutions all process the single-frame maximum projection image obtained after scanning, which loses spatial information.

[0008] (4) For photoacoustic (PA) imaging technology, especially photoacoustic microscopy (PAM), existing image enhancement methods still face challenges. For example, PAM can perform submicron imaging for vascular level analysis, but due to the influence of background noise, it is still difficult to observe microvascular networks such as capillaries.

[0009] Summary of the Invention

[0010] The purpose of the present invention is to overcome the above-mentioned shortcomings of the prior art and provide an image enhancement method based on photoacoustic microscopy. The method comprises the following steps:

[0011] Perform multiple photoacoustic microscopy scans on the target object to obtain three-dimensional data of the target, and obtain the scanning results of the target on the time axis through continuous scanning imaging;

[0012] Before performing a projection operation on a single imaging result, synthesizing signals of adjacent areas within a set search range on the three-dimensional data in a spatial dimension;

[0013] With respect to the scanning result on the time axis, within a set search range, signals of adjacent areas are merged in the time dimension to obtain an enhanced image.

[0014] Compared with existing technologies, the present invention offers advantages in that its photoacoustic microscopy-based image enhancement method fully utilizes the characteristics of three-dimensional photoacoustic microscopy data, exploits the high spatial and temporal correlations of individual voxels, and uses an improved non-local means (NLM) algorithm to process images, significantly improving image quality and the signal-to-noise ratio of the imaging system. Furthermore, compared with deep learning methods, the present invention does not require extensive data training and can fully utilize the spatial and temporal dimensions of the acquired data.

[0015] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0017] FIG1 is a flow chart of an image enhancement method based on photoacoustic microscopy according to one embodiment of the present invention;

[0018] FIG2 is a schematic diagram of a process of an image enhancement method based on photoacoustic microscopy according to an embodiment of the present invention;

[0019] FIG3 is a flowchart of processing original three-dimensional data in the depth dimension according to one embodiment of the present invention;

[0020] FIG4 is a flow chart of processing a maximum projection graph in a time axis dimension according to one embodiment of the present invention;

[0021] FIG5 is a schematic diagram showing experimental results of an image enhancement method based on photoacoustic microscopy according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0023] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0024] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0025] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0026] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0027] The present invention provides a continuous frame four-dimensional image processing algorithm (hereinafter referred to as seq-NLM) based on NLM technology, which is used for image denoising and enhancement to improve the image signal-to-noise ratio. In this invention, the non-local means (NLM) algorithm is applied in both the spatial depth direction and the temporal axis, combining the characteristics of the data obtained by PAM. This algorithm expands the range of neighborhood selection and obtains better similar block selection results. This invention fully utilizes the three-dimensional structure of photoacoustic microscopy data and combines the continuity of the data along the time axis to improve the performance of traditional NLM algorithms.

[0028] Specifically, referring to FIG1 and FIG2 , the provided image enhancement method based on photoacoustic microscopy includes the following steps:

[0029] In step S110 , three-dimensional data information of the target is obtained by photoacoustic microscopy scanning, and a scanning result of the target on a time axis is obtained by continuous scanning imaging.

[0030] For example, photoacoustic microscopy can obtain three-dimensional information about an object or target, and through continuous scanning, it can produce temporally correlated scan results. Because blood vessels are spatially distributed and structurally continuous, images obtained using this scanning system exhibit a high degree of spatiotemporal continuity. The combination of three-dimensional spatial information and temporal continuity can be considered four-dimensional data processing for consecutive frames.

[0031] Step S120 : for the three-dimensional data information, merge the signals of adjacent areas in the spatial dimension.

[0032] Taking advantage of the continuity of photoacoustic microscopy 3D data, when performing denoising on the 3D data layer by layer, the output result is optimized by combining the tomographic results of adjacent depths. In one embodiment, a pixel-by-pixel processing algorithm is used. For example, to restore the imaging quality, before performing the maximum projection operation on the results of a single imaging, the NLM algorithm is first applied to the original 3D data in the spatial dimension, synthesizing the relevant signals of adjacent regions within a certain search range.

[0033] in, represents the pixel result after denoising, v(j,h) represents the original pixel with noise, ω(i,j) represents the weight relationship between two pixels, I represents the plane range that the algorithm can search in the original data, H represents the depth range that the algorithm can search in the original data, i represents the pixel index in the target depth plane, j represents the pixel index in a certain depth plane, and h represents the depth index.

[0034] In order to reasonably utilize adjacent signals, the weight ω is introduced before signal synthesis to suppress the interference of irrelevant items. For example, the weight ω(i,j) is set to:

[0035] Among them, Z(i) represents the normalization coefficient, g represents the smoothing coefficient, and V(i) represents the neighborhood similarity block of pixel i. Only when the neighborhood similarity is high can it be said that the similarity of the two pixels is high, which can be expressed as ‖V(i)-V(j,h)‖ 2 Measuring the similarity between signals can effectively evaluate the correlation between signals in adjacent regions, which can be expressed as:

[0036] Where d represents the radius of the similarity block.

[0037] Different from the traditional maximum projection map acquisition, the present invention first utilizes the continuity of vascular data to search and weighted average the data in a three-dimensional range, making full use of the three-dimensional data obtained by scanning, effectively expanding the selection range of similar blocks in the algorithm, and effectively reducing the noise intensity in the original data.

[0038] Step S130 : Based on the scanning result on the time axis, signals of adjacent regions are merged in the time dimension.

[0039] For example, a non-local means algorithm can be used to merge signals in the temporal dimension. Specifically, after performing a maximum projection operation on the three-dimensional data obtained from each round of scanning, a single frame image can be obtained, and continuous dynamic video can be obtained through scanning. Due to the high-speed characteristics of the imaging system, it can be assumed that there is still continuity between different frames. To improve the imaging quality of the maximum projection image, the maximum projection images that are continuously scanned and processed in step S120 are stacked in chronological order. The NLM algorithm is used in the temporal dimension to synthesize the relevant signals of adjacent regions within a certain range, expressed as:

[0040] in, represents the pixel result after denoising, v(j,t) represents the original pixel with noise, ω(i,j) represents the weight relationship between two pixels, I represents the plane range that the algorithm can search in the original data, T represents the time range that the algorithm can search in the original data, i represents the pixel index at the target moment, j represents the pixel index at a certain moment, and t represents the time index.

[0041] In order to reasonably utilize adjacent signals, the weight ω is introduced before signal synthesis to suppress the interference of irrelevant items.

[0042] Where Z(i) represents the normalization coefficient, g represents the smoothing coefficient, and V(i) represents the range of similar blocks. Use ‖V(i)-V(j,t)‖ 2 Measuring the similarity between signals can effectively evaluate the correlation between signals in adjacent areas, which can be expressed as:

[0043] Where d represents the radius of the similarity block.

[0044] It should be noted that the weight ω involved in steps S120 and S130 may also adopt other forms, for example, set to a fixed value based on experience or experiments, and the similarity between signals may also adopt other measurement standards, such as structural similarity.

[0045] Existing NLM algorithms are usually used for denoising and enhancement of single images. When processing photoacoustic microscopy images, extending the range of similar blocks to the time axis range can effectively expand the selection range of similar blocks in the algorithm, making it easier to find similar pixels for weighted synthesis.

[0046] Figure 3 is a flowchart for processing raw 3D data in the depth dimension. I(m,n,h) represents the raw 3D data obtained by photoacoustic microscopy scanning, m and n represent the image size, and the detection depth is h. Ds represents the search block range, ds represents the similarity block size, and Hs represents the depth range. The 3D data I[i-Ds:i+Ds,j-Ds:j+Ds,h-Hs:h+Hs] represents the search range Sz of the NLM algorithm. The 2D data I[i-ds:i+ds,j-ds:j+ds,h] represents the size of the reference block R, and the 2D data I[p-ds:p+ds,q-ds:q+ds,z] represents the size of the similarity block S. Similar blocks are sequentially searched within the search range. Weights A are calculated based on the degree of similarity between the similar blocks and the reference block, and then a weighted sum is performed to update the target pixel I(i,j).

[0047] Figure 4 is a flowchart for processing the maximum projection image along the time axis, where I(m,n,t) is the maximum projection image to be processed at time t, and m and n are the image sizes. Ds is the search block range, ds is the similarity block size, and Ts is the time axis range. The three-dimensional data I[i-Ds:i+Ds,j-Ds:j+Ds,t-Ts:t+Ts] represents the search range Sz of the NLM algorithm. The two-dimensional data I[i-ds:i+ds,j-ds:j+ds,t] represents the size of the reference block R, and the two-dimensional data I[p-ds:p+ds,q-ds:q+ds,τ] represents the size of the similarity block S. Similar blocks are sequentially searched within the search range. The weight A is calculated based on the degree of similarity between the similar blocks and the reference block, and then a weighted sum is performed to update the target pixel I(i,j).

[0048] To further validate the effectiveness of the present invention, experimental simulations were conducted. For example, photoacoustic microscopy imaging experiments were performed on blood vessels in the ears and cerebral cortex of nude mice. As shown in Figure 5, the experimental results demonstrate that the resulting images exhibit a denoising and enhancement effect, helping to improve the signal-to-noise ratio of the PAM system.

[0049] In summary, compared with the prior art, the present invention has the following advantages:

[0050] (1) The present invention proposes a Seq-NLM algorithm that fully utilizes the data information of three-dimensional voxels for image processing of photoacoustic microscopy, effectively improving the image signal-to-noise ratio of the photoacoustic microscopy system. The present invention utilizes this data characteristic and uses the Seq-NLM algorithm to obtain images with excellent signal-to-noise ratio.

[0051] (2) The algorithm of the present invention does not use complex filtering algorithms and can still obtain excellent results, thus reducing time costs.

[0052] (3) The algorithm in the present invention does not rely on a large amount of data sets, but only requires processing of the original photoacoustic data, which reduces the difficulty of data acquisition and the computational burden.

[0053] (4) The seq-NLM algorithm proposed in the present invention is easy to implement and has high computational efficiency, and can be easily extended to other PAM systems based on devices such as scanning galvanometers.

[0054] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0055] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0056] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0057] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0058] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0059] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0060] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0061] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0062] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. An image enhancement method based on photoacoustic microscopy, The following steps are involved: Perform multiple photoacoustic microscopy scans on the target object to obtain the three-dimensional data of the target, and obtain the scanning results of the target on the time axis through continuous scanning imaging; Before performing a projection operation on a single imaging result, synthesizing signals of adjacent areas within a set search range for the three-dimensional data in a spatial dimension; With respect to the scanning result on the time axis, within a set search range, signals of adjacent regions are merged on the time dimension to obtain an enhanced image.

2. The method according to claim 1, It is characterized in that In the spatial dimension, the following formula is used to synthesize the signals of adjacent areas within the set search range of the three-dimensional data: in, represents the pixel result after synthesis, v(j,h) represents the original pixel before synthesis, ω(i,j) represents the weight between two pixels, I represents the set search plane range, H represents the search depth range, i represents the pixel index in the target depth plane, j represents the pixel index in a certain depth plane, and h represents the depth index.

3. The method according to claim 2, It is characterized in that The weight ω(i,j) between the two pixels is set to: Among them, Z(i) represents the normalization coefficient, g represents the smoothing coefficient, V(i) represents the neighborhood similarity block of pixel i, ‖V(i)-V(j,h)‖ 2 is the similarity between signals.

4. The method according to claim 3, It is characterized in that The similarity between the signals ‖V(i)-V(j,h)‖ 2 Set to: Among them, d represents the similarity block radius and k is the pixel index.

5. The method according to claim 1, It is characterized in that For the scanning results on the time axis, within the set search range, the following formula is used to merge the signals of adjacent areas in the time dimension: in, represents the pixel result after merging, v(j,t) represents the original pixel before merging, ω(i,j) represents the weight between two pixels, I represents the plane range of the search, T represents the time range of the search, i represents the pixel index at the target moment, j represents the pixel index at a certain moment, and t represents the time index.

6. The method according to claim 5, It is characterized in that The weight between the two pixels is set as: Where Z(i) represents the normalization coefficient, g represents the smoothing coefficient, and V(i) represents the similarity block range. ‖V(i)-V(j,t)‖ 2 is the similarity between signals.

7. The method according to claim 6, It is characterized in that The similarity between the signals ‖V(i)-V(j,t)‖ 2 Set to: Among them, d represents the similarity block radius and k is the pixel index.

8. The method according to claim 1, It is characterized in that The target object is a blood vessel.

9. A computer-readable storage medium having a computer program stored thereon, in, When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of being executed on the processor is stored in the memory, It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images