Photoacoustic-imaging-based method and apparatus for real-time naked-eye visualization of blood vessels on body surface

Abstract

Disclosed in the present invention are a photoacoustic-imaging-based method and apparatus for real-time naked-eye visualization of blood vessels on a body surface. The method comprises: generating a pulsed laser, and focusing same on a tissue surface to form a linear light spot; collecting a generated photoacoustic signal, and performing computer processing and reconstruction on same to obtain a photoacoustic blood vessel image; projecting the reconstructed photoacoustic blood vessel image to the tissue surface by means of a projector combined with an approximate ellipse fitting curved surface projection algorithm; and collecting a projection image of the tissue surface in real time by means of an RGBD camera, transmitting the projection image to a computer, using a learning-based target tracking algorithm to calculate the pose of the tissue surface in real time, and when the tissue surface undergoes an involuntary movement, performing blood vessel image tracking and projection. In the present invention, real-time naked-eye visualization of blood vessels on a body surface is implemented by means of photoacoustic imaging, such that the imaging depth and resolution are high, and the scanning range is large and flexible; and by using a target tracking algorithm, a blood vessel image can still be projected in-situ onto a tissue surface in real time when tissue moves, such that real-time visualization is more accurate.

Description

Real-time naked-eye visualization method and device for blood vessel surface based on photoacoustic imaging Technical Field

[0001] The present invention belongs to the technical field of photoacoustic imaging technology and computer vision, and specifically relates to a method and device for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging. Background Art

[0002] Accurate positioning of blood vessels is crucial in vascular-related surgeries. Currently, there are many ways to image blood vessels, such as Doppler ultrasound, computed tomography angiography, magnetic resonance angiography, and transmission near-infrared imaging. Among these vascular imaging methods, Doppler ultrasound has low sensitivity to microvessels and cannot image high-resolution microvascular images. Computed tomography angiography requires intravenous injection of contrast agents, which may cause complications if patients are allergic to the contrast agents. In addition, this imaging method emits ionizing radiation, which can cause harm to the human body. Magnetic resonance angiography also requires intravenous injection of contrast agents, which can cause complications. Although transmission near-infrared imaging can image blood vessels non-destructively, its imaging depth is shallow and it is mostly used for imaging superficial veins in the skin. Photoacoustic computed tomography (PCT) using multiple array elements can not only image blood vessels non-destructively, but also has the characteristic of specific absorption of light by blood vessels, which gives PAT an inherent advantage in vascular imaging. It can image blood vessels with high resolution using PAT. At the same time, PAT uses a linear spot for photoacoustic signal excitation, has a deep penetration depth, and uses a multi-element signal reception method, which has a large imaging range. Therefore, PAT has the advantages of large depth and high resolution imaging.

[0003] In traditional image navigation, images are presented on a 2D screen, separated from the actual tissue surface. The integration of vascular images and the actual tissue surface relies entirely on the experience of medical staff. With the advancement of technology, many devices have emerged that can directly integrate vascular images with the tissue surface, allowing blood vessels to be visualized on the body surface. However, these current vascular surface visualization devices suffer from shallow imaging depth and low resolution. Furthermore, even though many devices use cameras for positioning, they do not process curved projections, resulting in inaccurate projections of blood vessels on curved tissues.

[0004] Patent CN 104665766 A discloses a portable venous imaging device. This patent utilizes an infrared imaging device to image venous vessels and a projector to project the vessels onto the skin surface for visualization. While this venous imaging device is convenient to use, the patent utilizes infrared imaging, which, due to its projected illumination and diffuse reflection reception, results in a shallow imaging depth and low resolution. Therefore, this patent can only be used for imaging superficial veins in the skin. Furthermore, this patent lacks a target positioning device, potentially resulting in low accuracy in vascular projection.

[0005] Patent CN 112773333 A discloses a portable vascular imaging device. This patent utilizes near-infrared imaging to image superficial veins on the skin, projecting the vascular images onto the skin surface using a projector, while also utilizing a camera module for positioning. The near-infrared imaging employed in this patent also suffers from shallow imaging depth and poor resolution. Furthermore, even with the use of a camera for positioning, this patent fails to employ target tracking or curved surface projection strategies to improve vascular projection accuracy.

[0006] Patent CN 104116496 A discloses a medical 3D venous augmented reality device and method. The patent describes a pair of glasses that fuse venous images with the tissue surface. Wearing the glasses allows visualization of venous images on the tissue surface. A drawback of this approach is that the augmented reality image cannot be viewed with the naked eye; the surface display of blood vessels is only visible to the wearer and cannot be shared with other members of the surgical team. Furthermore, the patent focuses solely on venous visualization and does not address deeper microvascular networks.

[0007] Patent CN 210842997 U discloses a clinical blood collection pad, and patent CN 214632123 U discloses a near-infrared blood vessel projector. Both patents use near-infrared imaging, which has the disadvantages of shallow imaging depth and low resolution.

[0008] Summary of the Invention

[0009] The main purpose of the present invention is to provide a real-time naked-eye visualization method and device for vascular surface based on photoacoustic imaging to address the problems of shallow imaging depth, poor imaging resolution, and inaccurate naked-eye visualization of blood vessels in current vascular surface visualization technology.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] In one aspect of the present invention, a method for real-time naked-eye visualization of blood vessels based on photoacoustic imaging is provided, comprising the following steps:

[0012] Generate pulsed laser and focus it on the tissue surface through the photoacoustic imaging probe to form a light spot;

[0013] The photoacoustic signal is collected by the photoacoustic imaging probe and processed and reconstructed by a computer to obtain the photoacoustic vascular image;

[0014] The reconstructed photoacoustic vascular image is projected onto the tissue surface by a projector combined with an approximate ellipse fitting surface projection algorithm;

[0015] The projection image of the tissue surface is collected in real time by an RGBD camera and transmitted to a computer. The position of the tissue surface is solved in real time using a learning-based target tracking algorithm. When the tissue surface moves involuntarily, vascular image tracking and projection are performed.

[0016] As a preferred technical solution, the pulsed laser is generated and focused on the tissue surface to form a light spot via a photoacoustic imaging probe, specifically:

[0017] The pulse controller is controlled by a computer to send out a pulse signal, which in turn drives the pulse laser to generate pulse laser. The generated pulse laser is coupled into the optical fiber bundle through the optical fiber coupler. The optical fiber bundle is connected to the photoacoustic imaging probe. After passing through the photoacoustic imaging probe, the pulse laser forms a focal line spot on the tissue surface.

[0018] As a preferred technical solution, the approximate ellipse fitting surface projection algorithm is specifically as follows:

[0019] Performing three-dimensional surface reconstruction of the tissue surface using the RGBD camera;

[0020] Mathematical modeling of three-dimensional surface models;

[0021] The coordinates of the edge vertices and the highest point of the imaging area are solved for the three-dimensional surface model. The distances x, y, and z between two points in three-dimensional space are solved based on these coordinates. An ellipse model is established with x as the major semi-axis of the ellipse and z as the minor semi-axis of the ellipse. The perimeter of the ellipse is fitted using an approximate ellipse fitting method. This fitted curved edge is the actual curved edge distance from the edge vertex of the imaging area to the highest point. The projected vascular image is transformed in proportion to the ratio of the ellipse curved edge length to the ellipse major axis to achieve projection of the two-dimensional vascular image on the curved surface tissue.

[0022] As a preferred technical solution, the learning-based target tracking algorithm specifically includes:

[0023] Input layer for inputting images;

[0024] An object recognition network that extracts features using the MobileNet network. Specifically, three additional convolutional layers of 1x1, 3x3, and 5x5 are added before the original 1x1 convolutional layer of the MobileNet network, and a pooling layer is added after the original 1x1 convolutional layer to adjust the size and number of channels of the feature map.

[0025] The feature extraction network for feature point extraction through the CNN network is as follows: on the basis of the original 3x3 convolution layer of the CNN network, a 3x3 convolution layer, a 5x5 convolution layer and two pooling layers are added in parallel. That is, one path of the output of the original 3x3 convolution layer is connected to the fully connected layer through the 3x3 convolution layer, and the other path is connected to the fully connected layer through the newly added 3x3 convolution layer and pooling layer; the output of the newly added 3x3 convolution layer is also connected to the fully connected layer through the newly added 5x5 convolution layer and pooling layer;

[0026] The attention mechanism inputs the output feature maps of the target recognition network and the feature extraction network into two attention modules respectively; the attention mechanism uses a gating mechanism to adjust the importance of the feature maps.

[0027] As a preferred technical solution, the device calibration method of the projector and RGBD camera is specifically as follows:

[0028] A customized standard photoacoustic imaging sample and an image reconstructed after sample photoacoustic imaging are used to extract feature points from the sample image captured by the RGBD camera and the photoacoustic image reconstructed after imaging. Calibration is completed by solving the transformation relationship between the corresponding feature points on the sample image and the photoacoustic image. The customized standard photoacoustic imaging sample is made by arranging carbon rods in agar in a checkerboard shape.

[0029] As a preferred technical solution, no markers are used for registration after photoacoustic imaging. The tissue surface is captured by the RGBD camera in an arbitrary posture and transmitted to a computer. The computer extracts the feature points of the tissue surface in an arbitrary posture, estimates the registration relationship between the vascular image and the tissue surface, and performs an affine transformation on the projected vascular image, which is finally projected by a projector.

[0030] Another aspect of the present invention provides a device for real-time naked-eye visualization of blood vessels based on photoacoustic imaging, comprising a pulsed laser generator, a photoacoustic imaging probe, an acquisition and amplification circuit, a computer, a projector, and an RGBD camera.

[0031] The pulse laser generating device is connected to the photoacoustic imaging probe and is used to generate pulse laser and focus the pulse laser on the tissue surface through the photoacoustic imaging probe to form a light spot;

[0032] The photoacoustic imaging probe is also used to collect the generated photoacoustic signals and transmit them to the computer via the collection and amplification circuit;

[0033] The computer is used to process and reconstruct photoacoustic vascular images and is loaded with an approximate ellipse fitting surface projection algorithm and a learning-based target tracking algorithm;

[0034] The projector is used to project the reconstructed photoacoustic blood vessel image onto the tissue surface by combining an approximate ellipse fitting surface projection algorithm;

[0035] The RGBD camera is used to align the projected photoacoustic vascular image with the tissue surface in combination with a learning-based target tracking algorithm, and to perform reprojection when the tissue surface moves involuntarily.

[0036] As a preferred technical solution, the pulse laser generating device includes a pulse controller, a pulse laser, a fiber coupler, and a fiber bundle connected in sequence; the pulse controller is controlled by a computer to emit a pulse signal, which in turn drives the pulse laser to generate pulse laser. The generated pulse laser is coupled into the fiber bundle through the fiber coupler, and the fiber bundle is connected to the photoacoustic imaging probe. After passing through the photoacoustic imaging probe, the pulse laser is focused on the tissue surface to form a linear spot.

[0037] As a preferred technical solution, the photoacoustic imaging probe includes a cylindrical lens, a reflector, and a multi-element ultrasonic transducer array. The optical fiber bundle is focused into a linear light spot by the cylindrical lens, which is reflected 45 degrees by two reflectors and then vertically enters the tissue surface. The multi-element ultrasonic transducer array is located next to the cylindrical lens and directly above the focused linear light spot to receive the generated photoacoustic signal; the cylindrical lens is placed in a precision slide bracket, and the size of the focused linear light spot is adjusted by adjusting the distance between the precision slide and the optical fiber bundle, thereby adjusting the imaging depth and imaging resolution; the multi-element ultrasonic transducer array consists of 128 ultrasonic transducers with a main frequency of 10MHz.

[0038] As a preferred technical solution, the approximate ellipse fitting surface projection algorithm is specifically as follows:

[0039] Performing three-dimensional surface reconstruction of the tissue surface using the RGBD camera;

[0040] Mathematical modeling of three-dimensional surface models;

[0041] The coordinates of the edge vertices and the highest point of the imaging area are calculated for the three-dimensional surface model. The distances x, y, and z between two points in three-dimensional space are calculated using these coordinates. An ellipse model is established with x as the major semi-axis of the ellipse and z as the minor semi-axis of the ellipse. The perimeter of the ellipse is fitted using an approximate ellipse fitting method. This fitted curved edge is the actual curved edge distance from the edge vertex of the imaging area to the highest point. The projected vascular image is proportionally transformed using the ratio of the ellipse curved edge length to the ellipse major axis, thereby achieving projection of the two-dimensional vascular image onto the curved surface tissue.

[0042] The learning-based target tracking algorithm includes:

[0043] Input layer for inputting images;

[0044] An object recognition network that extracts features using the MobileNet network. Specifically, three additional convolutional layers of 1x1, 3x3, and 5x5 are added before the original 1x1 convolutional layer of the MobileNet network, and a pooling layer is added after the original 1x1 convolutional layer to adjust the size and number of channels of the feature map.

[0045] The feature extraction network for feature point extraction through the CNN network is as follows: on the basis of the original 3x3 convolution layer of the CNN network, a 3x3 convolution layer, a 5x5 convolution layer and two pooling layers are added in parallel. That is, one path of the output of the original 3x3 convolution layer is connected to the fully connected layer through the 3x3 convolution layer, and the other path is connected to the fully connected layer through the newly added 3x3 convolution layer and pooling layer; the output of the newly added 3x3 convolution layer is also connected to the fully connected layer through the newly added 5x5 convolution layer and pooling layer;

[0046] The attention mechanism inputs the output feature maps of the target recognition network and the feature extraction network into two attention modules respectively; the attention mechanism uses a gating mechanism to adjust the importance of the feature maps.

[0047] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0048] (1) The present invention has a deeper vascular imaging depth and higher imaging resolution: Photoacoustic computed tomography uses a large spot to irradiate, has a deep penetration depth, and utilizes the specific light absorption characteristics of blood vessels to have high sensitivity for blood vessel and microvascular detection;

[0049] (2) The scanning range of the present invention is larger and more flexible: the present invention can use a mechanical scanning structure to drive the probe for scanning, and can also scan and image the tissue by hand. The imaging method is flexible and the imaging range is not limited;

[0050] (3) The present invention provides more accurate real-time visualization of blood vessels on the body surface: a target tracking algorithm based on deep learning is used to identify the tissue surface, and a customized feature extraction network is used to extract features from the tissue surface, so that the posture solution is more accurate when the tissue surface moves involuntarily, and the blood vessel image can still be projected in real time on the tissue surface when the tissue moves. In addition, combined with an approximate elliptical surface algorithm, the blood vessel image can be projected more accurately on the curved tissue surface.

[0051] (4) The blood vessel positioning of the present invention has three-dimensional depth information: the fusion of the three-dimensional model of the surgical surface and the three-dimensional photoacoustic blood vessel can provide the position and structural relationship between the surgical surface and the arm blood vessels, and can also provide the depth information of the subcutaneous blood vessels.

[0052] (5) The installation and deployment of the present invention is simpler: real-time blood vessel visualization on the body surface can be achieved using only a commercial projector and a commercial RGBD camera.

[0053] (6) The present invention has a more flexible way of visualizing blood vessels: photoacoustic imaging only needs to be completed once, and the blood vessel image can be used continuously without the need for additional fixing devices for real-time imaging. In addition, the target tracking algorithm can recognize any patient posture, and the blood vessel image can be accurately visualized on the body surface in real time without the need for additional fixing devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] FIG1 is a schematic structural diagram of a device for real-time naked-eye visualization of blood vessels based on photoacoustic imaging according to an embodiment of the present invention;

[0055] FIG2 is a schematic structural diagram of a photoacoustic imaging probe according to an embodiment of the present invention;

[0056] FIG3 is a schematic diagram of a flow chart of a target tracking algorithm according to an embodiment of the present invention;

[0057] FIG4 is a schematic diagram of a flow chart of an approximate elliptical surface projection algorithm according to an embodiment of the present invention;

[0058] FIG5 is a schematic structural diagram of an ellipse model according to an embodiment of the present invention;

[0059] FIG6 is a schematic diagram of a neural network structure of a target tracking algorithm according to an embodiment of the present invention.

[0060] Explanation of the accompanying figures: 1. Pulse controller; 2. Pulsed laser; 3. Pulsed laser beam; 4. Fiber coupler; 5. Fiber bundle; 6. Photoacoustic imaging probe; 6-1. Cylindrical lens; 6-2. Reflector; 6-3 Laser beam, 6-4 Multi-element ultrasonic transducer; 6-5. Precision slide; 7. Focus line spot; 8. Tissue surface; 9. Amplifying circuit; 10. Data acquisition system; 11. Computer; 12. Projector; 13. RGBD camera. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0062] Example 1

[0063] As shown in Figure 1, this embodiment provides a real-time naked-eye visualization device for vascular surface based on photoacoustic imaging, including a pulse controller 1, a pulse laser 2, a pulse laser beam 3, a fiber coupler 4, a fiber bundle 5, a photoacoustic imaging probe 6, a focal line spot 7, a tissue surface 8, an amplifier circuit 9, a data acquisition system 10, a computer 11, a projector 12, and an RGBD camera 13; the computer 11 communicates with the pulse controller 1, controls the pulse controller 1 to generate a pulse signal to drive the pulse laser 2 to emit a pulse laser beam 3, and the laser beam is coupled into the fiber bundle 5 through the fiber coupler 4. The fiber bundle is connected to a photoacoustic imaging probe 6 to form a focal line spot 7 on the tissue surface 8 for exciting a photoacoustic signal. The excited photoacoustic signal is received by the photoacoustic imaging probe, amplified by the amplifier circuit 9, and then collected and stored by the data acquisition system 10, and finally a photoacoustic vascular image is reconstructed on the computer 11. After a complete imaging, the projector 12 can be used in combination with an approximate elliptical surface fitting algorithm to project the photoacoustic vascular image onto the tissue surface 8 of any posture. The RGBD camera 13 can be used in combination with a target tracking algorithm based on deep learning to perform vascular image registration and target tracking. This allows the vascular image to be accurately projected onto the tissue surface 8 when the tissue surface 8 moves involuntarily, achieving real-time naked-eye visualization.

[0064] Furthermore, as shown in FIG2 , the photoacoustic imaging probe 6 includes a cylindrical lens 6-1, two reflectors 6-2, and a 128-element multi-element ultrasonic transducer 6-4. The laser beam 6-3 emitted from the optical fiber bundle is focused by the cylindrical lens 6-1 and then reflected by the reflector 6-2, so that the focal line spot 7 is located just below the multi-element ultrasonic transducer 6-4. The cylindrical lens 6-1 is connected to a precision slide 6-5. By adjusting the slide 6-5, the distance between the cylindrical lens 6-1 and the optical fiber bundle can be adjusted to adjust the size of the focal line spot 7, thereby adjusting different imaging depths and imaging resolutions.

[0065] Furthermore, the multi-element ultrasonic transducer array is composed of 128 ultrasonic transducers with a main frequency of 10 MHz.

[0066] Furthermore, when the photoacoustic imaging probe 6 performs photoacoustic imaging on tissue, different imaging modes can be selected according to different scenarios and different needs. The tissue can be scanned and imaged in a handheld manner, or the photoacoustic imaging probe 6 can be placed on a two-dimensional motor structure platform for mechanical scanning and imaging.

[0067] Furthermore, the deep learning-based target tracking algorithm includes two networks: a target recognition network and a feature extraction network. The target tracking algorithm process is shown in Figure 3. First, the RGBD camera 13 captures an image of the tissue surface 8 at a frame rate of 30fps. The captured image is then input into the target recognition network. This network is the product of training the tissue surface 8 and non-tissue surfaces using an improved lightweight MobileNet network. Through this target recognition network, the tissue surface 8 can be detected and identified, and attention is then focused on the tissue surface area while eliminating interference from the non-tissue surface. The camera image is then input into the feature extraction network to extract features from the tissue surface 8 area. This network is the product of customized training of feature points on the tissue surface 8 using an improved convolutional neural network (CNN) to address the problem of weak texture on the skin tissue. After feature extraction is completed, the pose is solved and moment judgment is performed. If the pose changes, the projected image needs to be transformed so that the vascular image can still be accurately aligned when reprojected onto the tissue surface 8.

[0068] The improved lightweight MobileNet network is shown in the dotted box on the left side of Figure 6. Specifically, three additional convolutional layers of 1x1, 3x3, and 5x5 are added before the original 1x1 convolutional layer of the MobileNet network (as shown in the dark gray box on the left side of Figure 6), and a pooling layer is added after the original 1x1 convolutional layer (as shown in the dark gray box on the left side of Figure 6) to adjust the size and number of channels of the feature map; the three newly added convolutional layers are used to enhance the performance of feature extraction and recognition of the target, and the one pooling layer is used to prevent overfitting;

[0069] The improved convolutional neural network CNN is shown in the dotted box on the right side of Figure 6. Specifically, a 3x3 convolutional layer, a 5x5 convolutional layer and two pooling layers are added in parallel on the basis of the original 3x3 convolutional layer of the CNN network (as shown in the dark gray box on the right side of Figure 6). That is, one path of the output of the original 3x3 convolutional layer is connected to the fully connected layer through the 3x3 convolutional layer, and the other path is connected to the fully connected layer through the newly added 3x3 convolutional layer and pooling layer; the output of the newly added 3x3 convolutional layer is also connected to the fully connected layer through the newly added 5x5 convolutional layer and pooling layer; the newly added two convolutional layers can better extract features, and the newly added two pooling layers can improve the scale invariance and rotation invariance when extracting features from the target.

[0070] Furthermore, the approximate elliptical surface projection algorithm includes three-dimensional surface imaging of the tissue surface 8, precise mathematical modeling, and approximate elliptical fitting. The algorithm flow chart of the approximate elliptical surface projection algorithm is shown in Figure 4. First, the RGBD camera 13 captures RGB images and depth, and then uses the captured images to perform dense three-dimensional surface reconstruction of the tissue surface 8. The reconstructed model can easily obtain the three-dimensional coordinates of the vertex A at the edge of the imaging area and the highest point B of the model. The coordinates can then be used to calculate the x, y, and z distances between A and B in three-dimensional space. An elliptical model is constructed using x as the major semi-axis of the ellipse and z as the minor semi-axis of the ellipse. The specific geometric model is shown in Figure 5. The circumference of the ellipse is calculated using an approximate elliptical fitting method. The calculated 1 / 4 ellipse circumference is the curved distance between A and B in a two-dimensional vertical plane. By magnifying the projected image using the ratio of the 1 / 4 ellipse circumference to the major semi-axis, accurate projection onto the curved tissue surface can be achieved.

[0071] Furthermore, the real-time naked-eye visualization device of the vascular surface based on photoacoustic imaging can also perform three-dimensional surface reconstruction of the surgical surface through the RGBD camera 13. The reconstructed three-dimensional surface model can be fused with the three-dimensional photoacoustic vascular image to view the vascular image, the position of the subcutaneous blood vessels and the position of the arm surface, the structural relationship and the vascular depth information in the three-dimensional point cloud space, to provide assistance for preoperative planning.

[0072] Furthermore, the projector 12 and RGBD camera 13 form a visual projection tracking device that can be freely set at any height directly above the tissue surface 8 and connected to the computer 11. By adjusting the height from the tissue surface 8, the range of vascular surface visualization can be adjusted. After each height adjustment, a simple device calibration is required on the computer 11 to complete the configuration of the visual projection tracking device. The device calibration method is specifically as follows:

[0073] A customized standard photoacoustic imaging sample and an image reconstructed after sample photoacoustic imaging are used to extract feature points from the sample image captured by the RGBD camera 13 and the photoacoustic image reconstructed after imaging, and calibration is completed by solving the transformation relationship between the corresponding feature points on the sample image and the photoacoustic image; the customized standard photoacoustic imaging sample is made by arranging 20 carbon rods with a length of 30 mm and a diameter of 0.5 mm in a checkerboard shape in agar.

[0074] Furthermore, after photoacoustic imaging, no markers are used for registration. The tissue surface 8 can be captured by the RGBD camera 13 in any posture and transmitted to the computer 11. In the computer 11, the feature points of the tissue surface 8 in any posture can be extracted, the registration relationship between the vascular image and the tissue surface 8 can be accurately estimated, and the projected vascular image can be affine transformed. Finally, after projection by the projector, it can be accurately visualized in situ on the skin surface with the naked eye.

[0075] It should be noted here that the device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure is divided into different functional modules to complete all or part of the functions described above. The device is a real-time naked-eye visualization method of the vascular surface based on photoacoustic large-depth and high-resolution imaging that can be applied to the following embodiments.

[0076] Example 2

[0077] In this embodiment, a method for real-time naked-eye visualization of blood vessels based on photoacoustic imaging is provided. This method can apply the real-time naked-eye visualization device for blood vessels based on photoacoustic imaging of the above embodiment, and includes the following steps:

[0078] S1, generate a pulsed laser beam 3 and focus it on the tissue surface 8 through the photoacoustic imaging probe 6 to form a linear spot, specifically:

[0079] A computer 11 controls a pulse controller 1 to emit a pulse signal, thereby driving a pulse laser 2 to generate a pulse laser beam 3. The generated pulse laser beam 3 is coupled into an optical fiber bundle 5 through an optical fiber coupler 4. The optical fiber bundle 5 is connected to a photoacoustic imaging probe 6. After passing through the photoacoustic imaging probe 6, the pulse laser beam 3 forms a focal line spot 7 on a tissue surface 8.

[0080] S2, collecting the photoacoustic signals generated by the photoacoustic imaging probe 6 and processing and reconstructing them by the computer 11 to obtain a photoacoustic vascular image;

[0081] S3, projecting the reconstructed photoacoustic blood vessel image onto the tissue surface 8 by the projector 12 in combination with an approximate ellipse fitting surface projection algorithm;

[0082] Furthermore, the approximate ellipse fitting surface projection algorithm is specifically as follows:

[0083] Performing three-dimensional surface reconstruction of the tissue surface 8 using the RGBD camera 13;

[0084] Mathematical modeling of three-dimensional surface models;

[0085] The coordinates of the edge vertices and the highest point of the imaging area are calculated for the three-dimensional surface model. The distances x, y, and z between two points in three-dimensional space are calculated using these coordinates. An ellipse model is established with x as the major semi-axis of the ellipse and z as the minor semi-axis of the ellipse. The perimeter of the ellipse is fitted using an approximate ellipse fitting method. This fitted curved edge is the actual curved edge distance from the edge vertex of the imaging area to the highest point. The projected vascular image is proportionally transformed using the ratio of the ellipse curved edge length to the ellipse major axis, thereby achieving projection of the two-dimensional vascular image onto the curved surface tissue.

[0086] S4. The projection image of the tissue surface 8 is collected in real time by the RGBD camera 13 and transmitted to the computer 11. The posture of the tissue surface 8 is solved in real time using the learning-based target tracking algorithm. When the tissue surface 8 moves involuntarily, vascular image tracking and projection are performed.

[0087] Furthermore, the learning-based target tracking algorithm specifically includes:

[0088] (1) Input layer for inputting images;

[0089] (2) The target recognition network that extracts features through the MobileNet network is shown in the dotted box on the left side of Figure 6. Specifically, three additional convolutional layers of 1x1, 3x3, and 5x5 are added before the original 1x1 convolutional layer of the MobileNet network (as shown in the dark gray box on the left side of Figure 6), and a pooling layer is added after the original 1x1 convolutional layer (as shown in the dark gray box on the left side of Figure 6) to adjust the size and number of channels of the feature map; the three newly added convolutional layers are used to enhance the performance of feature extraction and recognition of the target, and the one pooling layer is used to prevent overfitting;

[0090] (3) The feature extraction network for feature point extraction through the CNN network is shown in the dotted box on the right side of Figure 6. Specifically, a 3x3 convolution layer, a 5x5 convolution layer and two pooling layers are added in parallel on the basis of the original 3x3 convolution layer of the CNN network (as shown in the dark gray box on the right side of Figure 6). That is, one path of the output of the original 3x3 convolution layer is connected to the fully connected layer through the 3x3 convolution layer, and the other path is connected to the fully connected layer through the newly added 3x3 convolution layer and pooling layer; the output of the newly added 3x3 convolution layer is also connected to the fully connected layer through the newly added 5x5 convolution layer and pooling layer; the newly added two convolution layers can better extract features, and the newly added two pooling layers can improve the scale invariance and rotation invariance when extracting features from the target.

[0091] (4) Attention mechanism, which inputs the output feature maps of the target recognition network and the feature extraction network into two attention modules respectively; the attention mechanism uses a gating mechanism to adjust the importance of the feature maps

[0092] Furthermore, the projector 12 and RGBD camera 13 form a visual projection tracking device that can be freely set at any height directly above the tissue surface 8 and connected to the computer 11. By adjusting the height from the tissue surface 8, the range of vascular surface visualization can be adjusted. After each height adjustment, a simple device calibration is required on the computer 11 to complete the configuration of the visual projection tracking device. The device calibration method is specifically as follows:

[0093] A customized standard photoacoustic imaging sample and an image reconstructed after sample photoacoustic imaging are used to extract feature points from the sample image captured by the RGBD camera 13 and the photoacoustic image reconstructed after imaging, and calibration is completed by solving the transformation relationship between the corresponding feature points on the sample image and the photoacoustic image; the customized standard photoacoustic imaging sample is made by arranging 20 carbon rods with a length of 30 mm and a diameter of 0.5 mm in a checkerboard shape in agar.

[0094] Furthermore, after photoacoustic imaging, no markers are used for registration. The tissue surface 8 is captured by the RGBD camera 13 in an arbitrary posture and transmitted to the computer 11. In the computer 11, the feature points of the tissue surface 8 in the arbitrary posture are extracted, the registration relationship between the vascular image and the tissue surface 8 is estimated, and the projected vascular image is affine transformed, and finally projected by the projector.

[0095] Furthermore, only one complete photoacoustic imaging is required to locate and track the tissue surface 8 through the RGBD camera 13 in combination with the target tracking algorithm, so that the imaging results can be used for naked-eye visualization of surface blood vessels.

[0096] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0097] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging, characterized in that: The following steps are involved: Generate pulsed laser and focus it on the tissue surface through the photoacoustic imaging probe to form a light spot; The photoacoustic signal is collected by the photoacoustic imaging probe and processed and reconstructed by a computer to obtain a photoacoustic vascular image; The reconstructed photoacoustic vascular image is projected onto the tissue surface by a projector combined with an approximate ellipse fitting surface projection algorithm; The projection image of the tissue surface is collected in real time by an RGBD camera and transmitted to a computer. The posture of the tissue surface is solved in real time using a learning-based target tracking algorithm. When the tissue surface moves involuntarily, vascular image tracking and projection are performed.

2. The method for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging according to claim 1, characterized in that: The pulsed laser is generated and focused on the tissue surface to form a light spot through the photoacoustic imaging probe, specifically: The pulse controller is controlled by a computer to send out a pulse signal, which in turn drives the pulse laser to generate a pulse laser. The generated pulse laser is coupled into the optical fiber bundle through the optical fiber coupler. The optical fiber bundle is connected to the photoacoustic imaging probe. After the pulse laser passes through the photoacoustic imaging probe, a focal line spot is formed on the tissue surface.

3. The method for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging according to claim 1, characterized in that: The approximate ellipse fitting surface projection algorithm is specifically as follows: Using the RGBD camera to perform three-dimensional surface reconstruction on the tissue surface; Mathematical modeling of three-dimensional surface models; The coordinates of the edge vertices and the highest point of the imaging area are solved for the three-dimensional surface model. The distances x, y, and z between two points in the three-dimensional space are solved with these coordinates. The ellipse model is established with x as the major semi-axis of the ellipse and z as the minor semi-axis of the ellipse. The perimeter of the ellipse is fitted by the approximate ellipse fitting method. The curved edge of this fitting That is, the actual curved edge distance from the edge vertex of the imaging area to the highest point. The projected vascular image is transformed in proportion to the ratio of the length of the ellipse curved edge to the major axis of the ellipse to realize the projection of the two-dimensional vascular image on the curved surface tissue.

4. The method for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging according to claim 1, characterized in that: The learning-based target tracking algorithm specifically includes: Input layer for inputting images; The object recognition network uses the MobileNet network for feature extraction, specifically: adding three additional convolutional layers of 1x1, 3x3, and 5x5 before the original 1x1 convolutional layer of the MobileNet network, and adding a pooling layer after the original 1x1 convolutional layer to adjust the size and number of channels of the feature map; A feature extraction network for extracting feature points through a CNN network, specifically: adding a 3x3 convolution layer, a 5x5 convolution layer and two pooling layers in parallel on the basis of the original 3x3 convolution layer of the CNN network, that is, one path of the output of the original 3x3 convolution layer is connected to the fully connected layer through the 3x3 convolution layer, and the other path is connected to the fully connected layer through the newly added 3x3 convolution layer and the pooling layer; the output of the newly added 3x3 convolution layer is also connected to the fully connected layer through the newly added 5x5 convolution layer and the pooling layer; The attention mechanism inputs the output feature maps of the target recognition network and the feature extraction network into two attention modules respectively; the attention mechanism uses a gating mechanism to adjust the importance of the feature maps.

5. The method for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging according to claim 1, characterized in that: The device calibration method of the projector and the RGBD camera is specifically as follows: A customized standard photoacoustic imaging sample and an image reconstructed after sample photoacoustic imaging are used to extract feature points in the sample image acquired by the RGBD camera and the photoacoustic image reconstructed after imaging, and calibration is completed by solving the transformation relationship between the corresponding feature points on the sample image and the photoacoustic image; the customized standard photoacoustic imaging sample is made by arranging carbon rods in agar in a checkerboard shape.

6. The method for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging according to claim 1, characterized in that: After photoacoustic imaging, no markers are used for registration. The tissue surface is captured by the RGBD camera in an arbitrary posture and transmitted to a computer. Feature points of the tissue surface in an arbitrary posture are extracted in the computer, the registration relationship between the vascular image and the tissue surface is estimated, and the projected vascular image is affine transformed, and finally projected by a projector.

7. A real-time naked-eye visualization device for blood vessel surface based on photoacoustic imaging, characterized in that: It includes a pulse laser generator, a photoacoustic imaging probe, an acquisition and amplification circuit, a computer, a projector and an RGBD camera; The pulse laser generating device is connected to the photoacoustic imaging probe and is used to generate pulse laser and focus the pulse laser on the tissue surface through the photoacoustic imaging probe to form a light spot; The photoacoustic imaging probe is also used to collect the generated photoacoustic signals and transmit them to the computer via the collection and amplification circuit; The computer is used for processing and reconstructing photoacoustic blood vessel images, and is loaded with an approximate ellipse fitting surface projection algorithm and a learning-based target tracking algorithm; The projector is used to project the reconstructed photoacoustic blood vessel image onto the tissue surface in combination with an approximate ellipse fitting surface projection algorithm; The RGBD camera is used to align the projected photoacoustic vascular image with the tissue surface in combination with a learning-based target tracking algorithm, and to reproject when involuntary movement occurs on the tissue surface.

8. The device for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging according to claim 7, characterized in that: The pulse laser generating device comprises a pulse controller, a pulse laser, a fiber coupler and a fiber bundle which are connected in sequence. The pulse controller is controlled by a computer to send out a pulse signal, thereby driving the pulse laser to generate a pulse laser. The generated pulse laser is coupled into the fiber bundle through the fiber coupler. The fiber bundle is connected to a photoacoustic imaging probe. After passing through the photoacoustic imaging probe, the pulse laser is focused on the tissue surface to form a line spot.

9. The device for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging according to claim 7, characterized in that: The photoacoustic imaging probe comprises a cylindrical lens, a reflector and a multi-element ultrasonic transducer array. The optical fiber bundle is focused into a line spot by the cylindrical lens, and then vertically enters the tissue surface after being reflected by two reflectors at 45 degrees. The multi-element ultrasonic transducer array is located next to the cylindrical lens and directly above the focused line spot, and is used to receive the generated photoacoustic signal; the cylindrical lens is placed in a precision slide bracket, and the spot size of the focused line spot is adjusted by adjusting the distance between the precision slide and the optical fiber bundle, thereby adjusting the imaging depth and imaging resolution; the multi-element ultrasonic transducer array consists of 128 ultrasonic transducers with a main frequency of 10 MHz.

10. The device for real-time naked-eye visualization of blood vessel surface based on photoacoustic imaging according to claim 7, characterized in that: The approximate ellipse fitting surface projection algorithm is specifically as follows: Using the RGBD camera to perform three-dimensional surface reconstruction on the tissue surface; Mathematical modeling of three-dimensional surface models; The coordinates of the edge vertices and the highest point of the imaging area are solved for the three-dimensional surface model, and the distances x, y, and z between two points in the three-dimensional space are solved with these coordinates. An ellipse model is established with x as the major semi-axis of the ellipse and z as the minor semi-axis of the ellipse. The perimeter of the ellipse is fitted by an approximate ellipse fitting method, and this fitted curved edge is the actual curved edge distance from the edge vertex of the imaging area to the highest point. The projected vascular image is transformed in a corresponding proportion with the ratio of the length of the ellipse curved edge to the major axis of the ellipse, so as to realize the projection of the two-dimensional vascular image on the curved surface tissue. The learning-based target tracking algorithm includes: Input layer for inputting images; The object recognition network uses the MobileNet network for feature extraction, specifically: adding three additional convolutional layers of 1x1, 3x3, and 5x5 before the original 1x1 convolutional layer of the MobileNet network, and adding a pooling layer after the original 1x1 convolutional layer to adjust the size and number of channels of the feature map; A feature extraction network for extracting feature points through a CNN network, specifically: adding a 3x3 convolution layer, a 5x5 convolution layer and two pooling layers in parallel on the basis of the original 3x3 convolution layer of the CNN network, that is, one path of the output of the original 3x3 convolution layer is connected to the fully connected layer through the 3x3 convolution layer, and the other path is connected to the fully connected layer through the newly added 3x3 convolution layer and the pooling layer; the output of the newly added 3x3 convolution layer is also connected to the fully connected layer through the newly added 5x5 convolution layer and the pooling layer; The attention mechanism inputs the output feature maps of the target recognition network and the feature extraction network into two attention modules respectively; the attention mechanism uses a gating mechanism to adjust the importance of the feature maps.

Images