Image imaging method and device based on medical image, electronic equipment and medium

Abstract

The embodiment of the invention discloses an image imaging method and device based on a medical image, electronic equipment and a medium. A specific embodiment of the method comprises the following steps: connecting clamp equipment for signal synchronization; in response to determining that the signal receiving end of the clamp equipment receives the host acquisition signal, sending a trigger signal to the ultrasonic probe and the optical fiber probe, and determining current timestamp information of the ultrasonic probe and the optical fiber probe to obtain current ultrasonic timestamp information and current optical timestamp information; in response to determining that the ultrasonic probe and the optical fiber probe receive the trigger signal, determining an imaged optical image and an imaged ultrasonic image according to the current ultrasonic timestamp information and the current optical timestamp information to obtain an optical image and an ultrasonic image; and displaying the fused image to a monitor for an operator to operate and process. According to the embodiment, the imaging period is shortened, large errors of the image are avoided, the imaging accuracy is improved, and consumption of computer resources is reduced.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to an imaging method, device, electronic device, and medium based on medical imaging. Background Art

[0002] With the rapid development of imaging technology, the accuracy requirements of medical imaging are getting higher and higher. Medical imaging is a technology for imaging images. For example, in the medical field, after a health check of a patient, multiple images need to be generated, and the imaged images are obtained by printing the images of the examined patient parts. At present, medical imaging is usually achieved by reflecting high-frequency sound waves and light waves.

[0003] However, when the above method is adopted, the following technical problems often occur:

[0004] When using high-frequency sound waves and light waves to achieve imaging, it is impossible to ensure that the imaging time of the two is consistent, so that when fusing the images, more computer resources are needed to align them, resulting in a longer imaging cycle, which may also extend the patient's examination time, thereby affecting the patient's experience. It may also cause large errors in the imaging, thereby affecting the accuracy of the imaging.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose imaging methods, devices, electronic devices and media based on medical images to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide an imaging method based on medical imaging, the method comprising: connecting a fixture device for signal synchronization, wherein the fixture device comprises: an ultrasonic probe interface, an optical fiber probe interface and a signal receiving end, the ultrasonic probe interface is an interface for connecting the fixture device with the ultrasonic probe, the optical fiber probe interface is an interface for connecting the fixture device with the optical fiber probe, and the ultrasonic probe and the optical fiber probe are fixed in spatial position; in response to determining that the signal receiving end of the fixture device receives a host acquisition signal, sending a trigger signal to the ultrasonic probe and the optical fiber probe, and determining that ... The method comprises the following steps: obtaining the current ultrasonic timestamp information of the ultrasonic probe and the optical fiber probe, obtaining the current ultrasonic timestamp information and the current optical timestamp information; in response to determining that the ultrasonic probe and the optical fiber probe have received the trigger signal, determining the imaging optical image and the imaging ultrasonic image according to the current ultrasonic timestamp information and the current optical timestamp information, and obtaining the optical image and the ultrasonic image; performing data alignment processing on the optical image and the ultrasonic image, and obtaining the aligned optical image and the aligned ultrasonic image; performing image fusion on the aligned optical image and the aligned ultrasonic image, and obtaining the fused image; and displaying the fused image on a monitor for the operator to operate and process.

[0009] In a second aspect, some embodiments of the present disclosure provide an imaging device based on medical imaging, the device comprising: a connecting unit, configured to connect a fixture device for signal synchronization, wherein the fixture device comprises: an ultrasonic probe interface, an optical fiber probe interface and a signal receiving end, the ultrasonic probe interface is an interface for connecting the fixture device to the ultrasonic probe, the optical fiber probe interface is an interface for connecting the fixture device to the optical fiber probe, and the ultrasonic probe and the optical fiber probe are fixed in spatial position; a sending unit, configured to send a trigger signal to the ultrasonic probe and the optical fiber probe in response to determining that the signal receiving end of the fixture device receives a host acquisition signal, and to determine the ultrasonic probe and the optical fiber probe. The ultrasonic probe and the optical fiber probe receive the trigger signal, and determine the post-imaging optical image and the post-imaging ultrasonic image according to the current ultrasonic timestamp information and the current optical timestamp information, so as to obtain the optical image and the ultrasonic image; the processing unit is configured to perform data alignment processing on the optical image and the ultrasonic image, so as to obtain the aligned optical image and the aligned ultrasonic image; the fusion unit is configured to perform image fusion on the aligned optical image and the aligned ultrasonic image, so as to obtain the fused image; the display unit is configured to display the fused image on the monitor for the operator to operate and process.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0012] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the imaging method based on medical images of some embodiments of the present disclosure, the imaging cycle is shortened, large errors in the image are avoided, the accuracy of imaging is improved, and the consumption of computer resources is reduced. Specifically, the reason for the large errors in the image, the reduced accuracy of imaging, and the waste of time resources is that when high-frequency sound waves and light waves are used to realize image imaging, it is impossible to ensure that the imaging time of the two is consistent, so that when the image is fused, more computer resources need to be consumed to align, resulting in a long imaging cycle, which may also extend the patient's examination time, thereby affecting the patient's experience. It may also cause large errors in imaging, thereby affecting the accuracy of imaging. Based on this, the imaging method based on medical images of some embodiments of the present disclosure, first, connects a fixture device for signal synchronization, wherein the fixture device includes: an ultrasonic probe interface, an optical fiber probe interface and a signal receiving end, the ultrasonic probe interface is an interface for connecting the fixture device with the ultrasonic probe, the optical fiber probe interface is an interface for connecting the fixture device with the optical fiber probe, and the ultrasonic probe and the optical fiber probe are fixed in space. Thus, the fixture device can be used to achieve signal synchronization of imaging, which is conducive to shortening the imaging cycle. Afterwards, in response to determining that the signal receiving end of the fixture device receives the host acquisition signal, a trigger signal is sent to the ultrasonic probe and the optical fiber probe, and the current timestamp information of the ultrasonic probe and the optical fiber probe is determined to obtain the current ultrasonic timestamp information and the current optical timestamp information. Thus, the imaging time of the two can be determined by the current ultrasonic timestamp information and the current optical timestamp information. Next, in response to determining that the ultrasonic probe and the optical fiber probe receive the trigger signal, the optical image after imaging and the ultrasonic image after imaging are determined according to the current ultrasonic timestamp information and the current optical timestamp information to obtain the optical image and the ultrasonic image. Thus, it can be ensured that the imaging time of the two is consistent, avoiding the need to consume more computer resources to align when the images are subsequently fused, and shortening the imaging cycle. Secondly, the optical image and the ultrasonic image are data aligned to obtain the aligned optical image and the aligned ultrasonic image. Thus, it is more time-saving to align, avoiding the consumption of more computer resources when the images are subsequently fused. Then, the aligned optical image and the aligned ultrasonic image are image fused to obtain the fused image. In this way, it is possible to avoid consuming too much computer resources, shorten the imaging cycle, and shorten the patient's examination time, thereby improving the patient's experience. Finally, the fused image is displayed on a monitor for the operator to operate and process. Therefore, the imaging cycle is shortened, large errors in the image are avoided, the accuracy of the imaging is improved, and the consumption of computer resources is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flow chart of some embodiments of the medical imaging method based on medical images according to the present disclosure;

[0015] Figure 2 is a schematic structural diagram of some embodiments of an imaging device based on medical imaging according to the present disclosure;

[0016] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure.

[0017] Figure 4 It is a comparison diagram of system imaging effects according to some embodiments of the imaging method based on medical images disclosed in the present invention.

[0018] Figure 5 It is an imaging usage scenario diagram according to some embodiments of the medical image imaging method disclosed in the present invention.

[0019] Figure 6 It is an imaging test scene diagram according to some embodiments of the medical image imaging method disclosed in the present invention. DETAILED DESCRIPTION

[0020] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0021] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0022] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0023] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0024] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0025] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0026] Figure 1 This is a process 100 of some embodiments of the medical imaging method disclosed in the present invention. The medical imaging method comprises the following steps:

[0027] Step 101, connecting a fixture device for signal synchronization.

[0028] In some embodiments, the execution entity (for example, a computing device) of the medical image-based imaging method can be connected to a fixture device for signal synchronization via a wireless connection, wherein the fixture device includes: an ultrasonic probe interface, an optical fiber probe interface and a signal receiving end, the ultrasonic probe interface is an interface for connecting the fixture device to the ultrasonic probe, the optical fiber probe interface is an interface for connecting the fixture device to the optical fiber probe, and the ultrasonic probe and the optical fiber probe are fixed in spatial position.

[0029] Here, the signal receiving end is used to receive acquisition signals from the ultrasound host and the optical coherence tomography scanner host. The ultrasound host is connected to the fixture device via an ultrasonic probe. The optical coherence tomography scanner host is connected to the fixture device via an optical fiber probe. The acquisition signal may refer to a signal for starting image acquisition. The ultrasound host may refer to a host of a medical ultrasound instrument. The optical coherence tomography scanner host may refer to a host of the optical coherence tomography scanner. When the signal receiving end of the fixture device receives acquisition signals from the ultrasound host and the optical coherence tomography scanner host, the fixture device synchronizes the acquisition signals to the ultrasound probe interface and the optical fiber probe interface, respectively. The connection may refer to a wireless connection.

[0030] For example, Figure 6 The following is an imaging test scene diagram according to some embodiments of the medical imaging method disclosed in the present invention. The fixture device aggregates the fiber optic probe and the ultrasound probe (ie, ultrasonic probe) together to facilitate imaging and measuring the same tissue.

[0031] It should be noted that the above-mentioned wireless connection methods may include but are not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0032] It should be noted that the above-mentioned computing device can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or it can be implemented as a single server or a single terminal device. For example, the computing device can be the above-mentioned target terminal. When the computing device is embodied as software, it can be installed in the hardware devices listed above. It can be implemented as multiple software or software modules for providing distributed services, for example, or it can be implemented as a single software or software module. No specific limitation is made here.

[0033] Step 102, in response to determining that the signal receiving end of the above-mentioned clamp device receives the host acquisition signal, sends a trigger signal to the above-mentioned ultrasonic probe and the above-mentioned optical fiber probe, and determines the current timestamp information of the above-mentioned ultrasonic probe and the above-mentioned optical fiber probe, and obtains the current ultrasonic timestamp information and the current optical timestamp information.

[0034] In some embodiments, the execution entity may send a trigger signal to the ultrasonic probe and the optical fiber probe in response to determining that the signal receiving end of the fixture device has received the host acquisition signal, and determine the current timestamp information of the ultrasonic probe and the optical fiber probe to obtain the current ultrasonic timestamp information and the current optical timestamp information.

[0035] Here, the trigger signal may refer to a signal that triggers the ultrasonic probe and the optical fiber probe to operate. For example, the trigger signal may refer to a "start" signal. The host acquisition signal may refer to an acquisition signal sent by the host. The current timestamp information of the ultrasonic probe is the same as the current timestamp information of the optical fiber probe. For example, the current timestamp information may refer to 1701003000000000000 (nanoseconds).

[0036] As an example, the execution subject may determine the current timestamp information of the ultrasonic probe and the optical fiber probe through a high-precision event timer to obtain the current ultrasonic timestamp information and the current optical timestamp information.

[0037] As another example, the execution subject may determine the current timestamp information of the ultrasonic probe and the optical fiber probe through the time.time_ns() function in Python to obtain the current ultrasonic timestamp information and the current optical timestamp information.

[0038] Step 103, in response to determining that the ultrasonic probe and the optical fiber probe receive the trigger signal, determine the post-imaging optical image and the post-imaging ultrasonic image according to the current ultrasonic timestamp information and the current optical timestamp information to obtain the optical image and the ultrasonic image.

[0039] In some embodiments, the execution entity may determine the post-imaging optical image and the post-imaging ultrasonic image in response to determining that the ultrasonic probe and the optical fiber probe have received the trigger signal, according to the current ultrasonic timestamp information and the current optical timestamp information, to obtain the optical image and the ultrasonic image.

[0040] Here, the ultrasonic image may be an image of a human body part collected by an ultrasonic probe of a medical ultrasonic instrument, and the optical image may be an image of a human body part collected by an optical fiber probe of an optical coherence tomography scanner.

[0041] Optionally, the execution subject may determine the post-imaging optical image and the post-imaging ultrasonic image according to the current ultrasonic timestamp information and the current optical timestamp information in response to determining that the ultrasonic probe and the optical fiber probe have received the trigger signal, and obtain the optical image and the ultrasonic image by the following steps:

[0042] The first step is to receive an ultrasonic pulse signal emitted by the ultrasonic probe in response to determining that the ultrasonic probe receives the trigger signal and the current ultrasonic timestamp information and the current optical timestamp information correspond to the same time.

[0043] Here, the same moment may refer to the same time. The ultrasonic pulse signal may refer to a transient signal in which ultrasonic waves generate a very high energy peak in a very short time (eg, nanoseconds or microseconds).

[0044] The second step is to convert the ultrasonic pulse signal into an ultrasonic digital signal, and send the ultrasonic digital signal to the ultrasonic probe.

[0045] Here, the ultrasonic digital signal may refer to a digital signal of ultrasonic waves.

[0046] As an example, the execution subject may convert the ultrasonic pulse signal into an ultrasonic digital signal through an analog-to-digital converter (ADC).

[0047] In a third step, in response to determining that the ultrasonic probe has received the ultrasonic digital signal, a post-imaging ultrasonic image is determined according to the ultrasonic digital signal to obtain an ultrasonic image.

[0048] As an example, the execution subject may use an ultrasound imaging algorithm to perform imaging determination on the ultrasound digital signal (eg, a grayscale mapping algorithm) to determine an ultrasound image after imaging, thereby obtaining an ultrasound image.

[0049] The fourth step is to receive the optical pulse signal emitted by the optical fiber probe in response to determining that the optical fiber probe receives the trigger signal and the current ultrasonic timestamp information and the current optical timestamp information correspond to the same time.

[0050] The fifth step is to convert the optical pulse signal into an optical digital signal, and send the optical digital signal to the optical fiber probe.

[0051] Here, the optical pulse signal may refer to a transient signal of an optical coherence tomography scanner that generates a very high energy peak in a very short time (eg, nanoseconds or microseconds). The optical digital signal may refer to a digital signal of an optical pulse.

[0052] In a sixth step, in response to determining that the optical fiber probe receives the optical digital signal, a post-imaging optical image is determined according to the optical digital signal to obtain an optical image.

[0053] Step 104: perform data alignment processing on the optical image and the ultrasonic image to obtain an aligned optical image and an aligned ultrasonic image.

[0054] In some embodiments, the execution entity may perform data alignment processing on the optical image and the ultrasonic image to obtain an aligned optical image and an aligned ultrasonic image.

[0055] Here, the aligned optical image may refer to an optical image aligned with an ultrasound image. The aligned ultrasound image may refer to an optical image aligned with an optical image.

[0056] For example, Figure 4 It is a system imaging test diagram according to some embodiments of the medical imaging method based on the present disclosure. It is displayed as a grid, the two pictures above the grid are related to ultrasound images, and the two pictures below the grid are related to optical images. The two pictures on the left side of the grid are the original ultrasound image and the original optical image, and the two pictures on the right side of the grid are elasticity measurements corresponding to two imaging modes, namely the aligned optical image and the aligned ultrasound image.

[0057] Optionally, the execution subject may perform data alignment processing on the optical image and the ultrasonic image through the following steps to obtain an aligned optical image and an aligned ultrasonic image:

[0058] The first step is to determine the timestamp difference between the current ultrasound timestamp information and the current optical timestamp information corresponding to the optical image and the ultrasound image.

[0059] As an example, the execution subject may subtract the current optical timestamp information from the current ultrasonic timestamp information to obtain a timestamp difference.

[0060] In a second step, in response to determining that the timestamp difference is greater than a preset value, pixel shift adjustment is performed on the optical image based on the ultrasound image to obtain an adjusted image.

[0061] Here, the preset value may refer to a preset value. For example, the preset value may refer to 0. The timestamp difference may represent the distance between the timestamps of the optical image and the ultrasound image. For example, the timestamp difference may refer to 1. The timestamp difference may also refer to -1.

[0062] As an example, in response to determining that the timestamp difference is greater than a preset value, the execution entity may perform a left pixel shift adjustment on the optical image based on the ultrasound image to obtain an adjusted image.

[0063] In a third step, in response to determining that the timestamp difference is less than a preset value, pixel shift adjustment is performed on the ultrasound image based on the optical image to obtain an adjusted image.

[0064] As an example, in response to determining that the timestamp difference is less than a preset value, the execution entity may perform a pixel right shift adjustment on the optical image based on the ultrasound image to obtain an adjusted image.

[0065] The fourth step is to extract feature points from the optical image and the ultrasonic image corresponding to the adjusted image to generate feature points of the optical image and the ultrasonic image.

[0066] As an example, the execution subject may extract feature points of the optical image corresponding to the adjusted image using Scale Invariant Feature Transform (SIFT) to generate feature points of the optical image, and extract feature points of the ultrasound image using Harris corner point detection to generate feature points of the ultrasound image.

[0067] The fifth step is to perform feature point matching on the optical image feature points and the ultrasonic image feature points to obtain feature matching results.

[0068] As an example, the execution subject may use a feature matching algorithm (e.g., K-dimensional tree matching) to match the feature points of the optical image and the feature points of the ultrasound image to obtain a feature matching result. The feature matching result represents the correspondence between the feature points of the optical image and the feature points of the ultrasound image. The correspondence means that each feature point is collected at the same time and at the same part. For example, the feature point of the optical image is (X i , Y i ). The subscript i above represents the number of feature points of the optical image. The range of the subscript i above is 1, 2, 3, .... The above X and the above Y represent the horizontal coordinate and vertical coordinate of the feature point in the image respectively. The feature points of the ultrasound image are The superscript 1 represents the identifier of the ultrasonic image feature point.

[0069] Step 6: Determine the alignment transformation matrix based on the above feature matching results.

[0070] As an example, the execution subject may use the least square method to determine the alignment transformation matrix of the feature matching result to obtain the alignment transformation matrix.

[0071] In the seventh step, the alignment transformation matrix is ​​applied to the optical image and the ultrasonic image to obtain an aligned optical image and an aligned ultrasonic image.

[0072] Step 105 , fusing the aligned optical image and the aligned ultrasonic image to obtain a fused image.

[0073] In some embodiments, the execution entity may fuse the aligned optical image and the aligned ultrasound image to obtain a fused image.

[0074] Here, the fused image may refer to an image obtained by fusing feature points of the aligned optical image and the aligned ultrasonic image.

[0075] Optionally, the execution subject may fuse the aligned optical image and the aligned ultrasound image through the following steps to obtain a fused image:

[0076] In the first step, the aligned optical image and the aligned ultrasonic image are denoised to obtain a denoised optical image and a denoised ultrasonic image.

[0077] As an example, the execution subject may use a median filter algorithm to remove salt and pepper noise from the aligned optical image to obtain a denoised optical image. A wavelet denoising algorithm may be used to remove speckle noise from the aligned ultrasound image to obtain a denoised ultrasound image.

[0078] In the second step, according to the weight addition function and the preset fusion weight, the denoised optical image and the denoised ultrasonic image are fused to obtain a fused image.

[0079] Here, the preset fusion weight is a preset weight of the fused image. For example, the preset fusion weight is a=0.5. Assuming that P is the fused image, N is the denoised optical image, and M is the denoised ultrasound image, the denoised optical image and the denoised ultrasound image are fused by the weight addition function P=a×N+(1-a)×M to obtain a fused image.

[0080] In the process of adopting technical solutions to solve the problems mentioned in the background technology, the following problems are often accompanied:

[0081] There are often differences between images at different scales or different spatial dimensions, which makes it difficult to adapt when fusing optical images and ultrasound images, resulting in poor image fusion quality. This may cause loss or redundancy of information in the fused image. Also, because the weights of optical images and ultrasound images cannot be reasonably allocated, image features cannot be fully utilized, resulting in low utilization of image features.

[0082] In the face of the above technical problems, the inventors decided to adopt the following solutions:

[0083] Optionally, the execution subject may fuse the aligned optical image and the aligned ultrasound image through the following steps to obtain a fused image:

[0084] In the first step, feature points are extracted from the aligned optical image and the aligned ultrasonic image to generate an optical image feature point set and an ultrasonic image feature point set.

[0085] As an example, the execution subject may extract feature points from the aligned optical image and the aligned ultrasound image by using a scale-invariant feature transform (SIFT) algorithm to generate an optical image feature point set and an ultrasound image feature point set.

[0086] In the second step, feature descriptors are constructed for the optical image feature point set and the ultrasonic image feature point set to generate optical image feature descriptors and ultrasonic image feature descriptors.

[0087] As an example, the execution subject can construct feature descriptors based on the gradient amplitude and direction information of the surrounding pixels of the feature point set of the optical image and the feature point neighborhood of the ultrasonic image feature point set, respectively, to generate the optical image feature descriptor and the ultrasonic image feature descriptor. The feature descriptor is a vector, and the feature descriptor is used to characterize the characteristics of the local image around the feature point. The gradient amplitude characterizes the degree of change of the grayscale value of the image pixel in space, and the direction information characterizes the direction in which the grayscale changes fastest.

[0088] The third step is to perform feature point matching on the optical image feature point set and the ultrasonic image feature point set according to the optical image feature descriptor and the ultrasonic image feature descriptor to obtain a feature point pair group.

[0089] As an example, the execution subject may first merge the optical image feature descriptor with the optical image feature point set to obtain a merged optical image feature point set. Merge the ultrasound image feature descriptor with the ultrasound image feature point set to obtain a merged ultrasound image feature point set. Then, feature matching is performed on the merged optical image feature point set and the merged ultrasound image feature point set using a feature matching algorithm (e.g., K-dimensional tree matching) to obtain a feature point pair group. Here, there is a corresponding relationship between the feature points in the feature point pair group, and the corresponding relationship means that the feature points are collected at the same time and the same part.

[0090] The fourth step is to screen the effective feature points of the above feature point pairs to obtain effective feature point pairs.

[0091] As an example, the execution subject may use a random sample consensus (RANSAC) algorithm to screen effective feature points from the feature point pair group to obtain an effective feature point pair group. The effective feature point pair group is a feature point pair group after excluding erroneous feature points from the feature point pair group.

[0092] The fifth step is to determine the spatial transformation matrix from the aligned optical image to the aligned ultrasonic image based on the effective feature point pair group.

[0093] Each element in the above spatial transformation matrix corresponds to a pixel in the image, and the value of the element represents the grayscale value of the pixel.

[0094] As an example, the execution subject may mark the effective feature point pairs in the pixel points of the aligned optical image and the aligned ultrasound image to obtain the marked optical image and the marked ultrasound image. Then, the space transformation matrix is ​​determined for the marked optical image and the marked ultrasound image according to the gray value of each pixel point to obtain the space transformation matrix.

[0095] As an example, the execution subject may perform fitting processing on the coordinates of the effective feature point pair group by the least square method, determine the values ​​of each parameter in the matrix, and obtain the space transformation matrix. The fitting processing may refer to minimizing the sum of squared errors.

[0096] In the sixth step, coordinate transformation is performed on the aligned optical image and the aligned ultrasonic image according to the spatial transformation matrix to obtain a transformed ultrasonic image and a transformed optical image.

[0097] As an example, the above-mentioned execution entity can transform the coordinates of the above-mentioned aligned optical image and the above-mentioned aligned ultrasonic image into the form of homogeneous coordinates, and then perform matrix multiplication operation on the above-mentioned aligned optical image and the above-mentioned aligned ultrasonic image transformed into the form of homogeneous coordinates by the above-mentioned spatial transformation matrix to obtain a transformed ultrasonic image and a transformed optical image.

[0098] In the seventh step, the information entropy of the transformed ultrasonic image and the transformed optical image is determined to obtain the information entropy of the ultrasonic image and the information entropy of the optical image.

[0099] As an example, the execution subject can traverse all pixels of the transformed ultrasound image and count the number of occurrences of each gray value, which is recorded as n(i). Assuming that the total number of pixels in the ultrasound image is N, the probability of the occurrence of a pixel with a gray value of i is Then the information entropy of ultrasound image is Optical image information entropy H 光学 Similarly, it will not be described here.

[0100] In the eighth step, in response to determining that the ultrasonic image information entropy is greater than the optical image information entropy, weight adjustment is performed on the transformed ultrasonic image to obtain an adjusted ultrasonic image.

[0101] As an example, the above execution entity can be expressed as follows: β 调整 =1-α 调整 The weight of the transformed ultrasonic image is adjusted to obtain an adjusted ultrasonic image.

[0102] In the ninth step, the adjusted ultrasound image is superimposed with the transformed optical image to obtain a superimposed image as a fused image.

[0103] Here, the above-mentioned superposition may refer to overlapping.

[0104] The relevant contents in the first step to the eighth step above are an inventive point of the present disclosure, which solves the technical problem mentioned in the background technology that "the image fusion quality is poor, the information of the fused image is lost or redundant, and the utilization rate of the image features is low". The factors that lead to the poor image fusion quality, which may cause the information of the fused image to be lost or redundant, and the low utilization rate of the image features are often as follows: images at different scales or different spatial dimensions often have differences, which are difficult to adapt when fusing optical images and ultrasonic images, resulting in poor image fusion quality, which may cause the information of the fused image to be lost or redundant. Because the weights of the optical image and the ultrasonic image cannot be reasonably allocated, the image features cannot be fully utilized, resulting in a low utilization rate of the image features. If the above factors are solved, the effect of improving the image fusion quality, avoiding the loss or redundancy of the information of the fused image, and improving the utilization rate of the image features can be achieved. In order to achieve this effect, first, feature point extraction is performed on the above-mentioned aligned optical image and the above-mentioned aligned ultrasonic image to generate an optical image feature point set and an ultrasonic image feature point set. The feature descriptor is constructed for the above-mentioned optical image feature point set and the above-mentioned ultrasonic image feature point set to generate an optical image feature descriptor and an ultrasonic image feature descriptor. According to the above-mentioned optical image feature descriptor and the above-mentioned ultrasonic image feature descriptor, the feature point matching is performed on the above-mentioned optical image feature point set and the ultrasonic image feature point set to obtain a feature point pair group. The above-mentioned feature point pair group is screened for effective feature points to obtain an effective feature point pair group. Therefore, the difference of images at different scales or different spatial dimensions can be avoided by feature point extraction, feature descriptor construction and effective feature point screening, which is convenient for subsequent adaptation when the optical image and the ultrasonic image are fused. According to the above-mentioned effective feature point pair group, the spatial transformation matrix from the above-mentioned aligned optical image to the above-mentioned aligned ultrasonic image is determined. Therefore, the fusion quality of the image can be improved by the spatial transformation matrix to avoid the loss or redundancy of information in the fused image. According to the above-mentioned spatial transformation matrix, the coordinate transformation of the above-mentioned aligned optical image and the above-mentioned aligned ultrasonic image is obtained to obtain a transformed ultrasonic image and a transformed optical image. The information entropy of the above-mentioned transformed ultrasonic image and the above-mentioned transformed optical image is determined to obtain the information entropy of the ultrasonic image and the information entropy of the optical image. Thus, the information loss or redundancy of the fused image can be avoided. In response to determining that the information entropy of the ultrasonic image is greater than the information entropy of the optical image, the weight of the transformed ultrasonic image is adjusted to obtain the adjusted ultrasonic image. Thus, the weights of the optical image and the ultrasonic image can be reasonably allocated to improve the utilization rate of the image features. The adjusted ultrasonic image is superimposed with the transformed optical image to obtain a superimposed image as the fused image. Therefore, the fusion quality of the image can be improved, the information loss or redundancy of the fused image can be avoided, and the utilization rate of the image features can be improved.

[0105] Step 106, displaying the fused image on a monitor for an operator to operate.

[0106] In some embodiments, the execution entity may display the fused image on a monitor for an operator to perform operations.

[0107] Here, the monitor may refer to a medical device monitor. For example, in an ultrasound examination device, the image of the internal structure of the human body acquired by the ultrasound probe will be displayed on the monitor for doctors to observe and diagnose. The operator may refer to a medical staff.

[0108] Optionally, the execution subject may display the fused image on a monitor through the following steps for an operator to perform operation processing:

[0109] In the first step, in response to determining that the contrast of the fused image is greater than a preset contrast threshold, the fused image is displayed on a monitor for an operator to perform the following operation processing steps:

[0110] The first sub-step is, in response to determining that the fused image is clicked, marking the click coordinates to the fused image.

[0111] Here, the preset contrast threshold may refer to a preset maximum contrast value. For example, the preset contrast threshold may refer to 0.4. The click coordinate may refer to a two-dimensional coordinate where the operator clicks the fused image. For example, the click coordinate may refer to (20, 50).

[0112] The second sub-step is, in response to determining that the fused image is double-clicked, enlarging the fused image.

[0113] As an example, the execution subject may perform difference amplification on the fused image by bilinear interpolation amplification.

[0114] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: through the imaging method based on medical images of some embodiments of the present disclosure, the imaging cycle is shortened, large errors in the image are avoided, the accuracy of imaging is improved, and the consumption of computer resources is reduced. Specifically, the reason for the large errors in the image, the reduced accuracy of imaging, and the waste of time resources is that when high-frequency sound waves and light waves are used to realize image imaging, it is impossible to ensure that the imaging time of the two is consistent, so that when the image is fused, more computer resources need to be consumed to align, resulting in a long imaging cycle, which may also extend the patient's examination time, thereby affecting the patient's experience. It may also cause large errors in imaging, thereby affecting the accuracy of imaging. Based on this, the imaging method based on medical images of some embodiments of the present disclosure, first, connects a fixture device for signal synchronization, wherein the fixture device includes: an ultrasonic probe interface, an optical fiber probe interface and a signal receiving end, the ultrasonic probe interface is an interface for connecting the fixture device with the ultrasonic probe, the optical fiber probe interface is an interface for connecting the fixture device with the optical fiber probe, and the ultrasonic probe and the optical fiber probe are fixed in space. Thus, the fixture device can be used to achieve signal synchronization of imaging, which is conducive to shortening the imaging cycle. Afterwards, in response to determining that the signal receiving end of the fixture device receives the host acquisition signal, a trigger signal is sent to the ultrasonic probe and the optical fiber probe, and the current timestamp information of the ultrasonic probe and the optical fiber probe is determined to obtain the current ultrasonic timestamp information and the current optical timestamp information. Thus, the imaging time of the two can be determined by the current ultrasonic timestamp information and the current optical timestamp information. Next, in response to determining that the ultrasonic probe and the optical fiber probe receive the trigger signal, the optical image after imaging and the ultrasonic image after imaging are determined according to the current ultrasonic timestamp information and the current optical timestamp information to obtain the optical image and the ultrasonic image. Thus, it can be ensured that the imaging time of the two is consistent, avoiding the need to consume more computer resources to align when the images are subsequently fused, and shortening the imaging cycle. Secondly, the optical image and the ultrasonic image are data aligned to obtain the aligned optical image and the aligned ultrasonic image. Thus, it is more time-saving to align, avoiding the consumption of more computer resources when the images are subsequently fused. Then, the aligned optical image and the aligned ultrasonic image are image fused to obtain the fused image. In this way, it is possible to avoid consuming too much computer resources, shorten the imaging cycle, and shorten the patient's examination time, thereby improving the patient's experience. Finally, the fused image is displayed on a monitor for the operator to operate and process. Therefore, the imaging cycle is shortened, large errors in the image are avoided, the accuracy of the imaging is improved, and the consumption of computer resources is reduced.

[0115] Further references Figure 2As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an imaging method based on medical imaging. These device embodiments are Figure 1 Corresponding to the method embodiments shown, the device can be specifically applied to various electronic devices.

[0116] like Figure 2 As shown, some embodiments of the imaging device 200 based on medical imaging include: a connection unit 201, a sending unit 202, a determination unit 203, a processing unit 204, a fusion unit 205 and a display unit 206. The connection unit 201 is configured to connect a fixture device for signal synchronization, wherein the fixture device includes: an ultrasonic probe interface, an optical fiber probe interface and a signal receiving end, the ultrasonic probe interface is an interface for connecting the fixture device with the ultrasonic probe, the optical fiber probe interface is an interface for connecting the fixture device with the optical fiber probe, and the ultrasonic probe and the optical fiber probe are fixed in spatial position; the sending unit 202 is configured to send a trigger signal to the ultrasonic probe and the optical fiber probe in response to determining that the signal receiving end of the fixture device receives a host acquisition signal, and to determine the current timestamp information of the ultrasonic probe and the optical fiber probe, and obtain the current ultrasonic timestamp information and current optical timestamp information; a determination unit 203, configured to determine the imaging optical image and the imaging ultrasonic image according to the current ultrasonic timestamp information and the current optical timestamp information in response to determining that the ultrasonic probe and the optical fiber probe have received the trigger signal, so as to obtain an optical image and an ultrasonic image; a processing unit 204, configured to perform data alignment processing on the optical image and the ultrasonic image to obtain an aligned optical image and an aligned ultrasonic image; a fusion unit 205, configured to perform image fusion on the aligned optical image and the aligned ultrasonic image to obtain a fused image; a display unit 206, configured to display the fused image on a monitor for an operator to perform operation processing.

[0117] It is understood that the units described in the device 200 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 200 and the units included therein, and will not be described in detail here.

[0118] Reference below Figure 3 , which shows a schematic structural diagram of an electronic device (such as a computing device) 300 suitable for implementing some embodiments of the present disclosure. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0119] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 304. In RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, ROM 302, and RAM 304 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0120] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0121] Figure 5 The following is an imaging scene diagram of some embodiments of the imaging method based on medical images disclosed in the present invention. The diagram shows a scene diagram in which imaging and measurement are being performed.

[0122] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the functions defined in the method of some embodiments of the present disclosure are executed.

[0123] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0124] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0125] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: connects to a fixture device for signal synchronization, wherein the fixture device includes: an ultrasonic probe interface, an optical fiber probe interface and a signal receiving end, the ultrasonic probe interface is an interface for connecting the fixture device to the ultrasonic probe, the optical fiber probe interface is an interface for connecting the fixture device to the optical fiber probe, and the ultrasonic probe and the optical fiber probe are fixed in spatial position; in response to determining that the signal receiving end of the fixture device receives a host acquisition signal, sends a trigger signal to the ultrasonic probe and the optical fiber probe, and determines The current timestamp information of the ultrasonic probe and the optical fiber probe is used to obtain the current ultrasonic timestamp information and the current optical timestamp information; in response to determining that the ultrasonic probe and the optical fiber probe have received the trigger signal, the imaging optical image and the imaging ultrasonic image are determined according to the current ultrasonic timestamp information and the current optical timestamp information to obtain the optical image and the ultrasonic image; data alignment is performed on the optical image and the ultrasonic image to obtain the aligned optical image and the aligned ultrasonic image; the aligned optical image and the aligned ultrasonic image are image-fused to obtain the fused image; the fused image is displayed on a monitor for the operator to operate and process.

[0126] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0127] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising: a connection unit, a sending unit, a determination unit, a processing unit, a fusion unit and a display unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the display unit may also be described as "a unit that displays the above-mentioned fused image on a monitor for an operator to perform operation processing".

[0129] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0130] The above descriptions are only some preferred embodiments of the present disclosure and the explanation of the technical principles used above. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. An imaging method based on medical imaging, comprising: Connecting a fixture device for signal synchronization, wherein the fixture device includes: an ultrasonic probe interface, an optical fiber probe interface and a signal receiving end, the ultrasonic probe interface is an interface for connecting the fixture device with the ultrasonic probe, the optical fiber probe interface is an interface for connecting the fixture device with the optical fiber probe, and the ultrasonic probe and the optical fiber probe are fixed in space; In response to determining that the signal receiving end of the fixture device receives the host acquisition signal, sending a trigger signal to the ultrasonic probe and the optical fiber probe, and determining current timestamp information of the ultrasonic probe and the optical fiber probe to obtain current ultrasonic timestamp information and current optical timestamp information; In response to determining that the ultrasonic probe and the optical fiber probe receive the trigger signal, determining a post-imaging optical image and a post-imaging ultrasonic image according to the current ultrasonic timestamp information and the current optical timestamp information, and obtaining an optical image and an ultrasonic image; Performing data alignment processing on the optical image and the ultrasonic image to obtain an aligned optical image and an aligned ultrasonic image; fusing the aligned optical image and the aligned ultrasonic image to obtain a fused image; The fused image is displayed on a monitor for an operator to perform operations.

2. The method according to claim 1, wherein: The step of fusing the aligned optical image and the aligned ultrasonic image to obtain a fused image comprises: De-noising the aligned optical image and the aligned ultrasonic image to obtain a de-noised optical image and a de-noised ultrasonic image; According to the weight addition function and the preset fusion weight, the denoised optical image and the denoised ultrasonic image are fused to obtain a fused image.

3. The method according to claim 1, wherein: Displaying the fused image on a monitor for an operator to perform operation processing includes: In response to determining that the contrast of the fused image is greater than a preset contrast threshold, the fused image is displayed on a monitor for an operator to perform the following operation processing steps: In response to determining that the fused image is clicked, marking the click coordinates to the fused image; In response to determining that the fused image is double-clicked, the fused image is enlarged.

4. The method according to claim 1, wherein: The step of performing data alignment processing on the optical image and the ultrasonic image to obtain an aligned optical image and an aligned ultrasonic image includes: Determine a timestamp difference between current ultrasound timestamp information and current optical timestamp information corresponding to the optical image and the ultrasound image; In response to determining that the timestamp difference is greater than a preset value, performing pixel translation adjustment on the optical image based on the ultrasound image to obtain an adjusted image; In response to determining that the timestamp difference is less than a preset value, performing pixel translation adjustment on the ultrasound image based on the optical image to obtain an adjusted image; Extracting feature points of the optical image and the ultrasonic image corresponding to the adjusted image to generate optical image feature points and ultrasonic image feature points; Performing feature point matching on the optical image feature points and the ultrasonic image feature points to obtain a feature matching result; Determining an alignment transformation matrix according to the feature matching result; The alignment transformation matrix is ​​applied to the optical image and the ultrasound image to obtain an aligned optical image and an aligned ultrasound image.

5. The method according to claim 1, wherein: In response to determining that the ultrasonic probe and the optical fiber probe receive the trigger signal, determining the post-imaging optical image and the post-imaging ultrasonic image according to the current ultrasonic timestamp information and the current optical timestamp information, and obtaining the optical image and the ultrasonic image, comprises: In response to determining that the ultrasonic probe receives the trigger signal and the current ultrasonic timestamp information and the current optical timestamp information correspond to the same time, receiving an ultrasonic pulse signal transmitted by the ultrasonic probe; Converting the ultrasonic pulse signal into an ultrasonic digital signal, and sending the ultrasonic digital signal to the ultrasonic probe; In response to determining that the ultrasonic probe receives the ultrasonic digital signal, determining a post-imaging ultrasonic image according to the ultrasonic digital signal to obtain an ultrasonic image; In response to determining that the optical fiber probe receives the trigger signal and the current ultrasonic timestamp information and the current optical timestamp information correspond to the same time, receiving the optical pulse signal emitted by the optical fiber probe; Converting the optical pulse signal into an optical digital signal, and sending the optical digital signal to the optical fiber probe; In response to determining that the optical fiber probe receives the optical digital signal, an optical image after imaging is determined according to the optical digital signal to obtain an optical image.

6. An imaging device based on medical imaging, comprising: A connection unit is configured to connect a fixture device for signal synchronization, wherein the fixture device includes: an ultrasonic probe interface, an optical fiber probe interface and a signal receiving end, the ultrasonic probe interface is an interface for connecting the fixture device with the ultrasonic probe, the optical fiber probe interface is an interface for connecting the fixture device with the optical fiber probe, and the ultrasonic probe and the optical fiber probe are fixed in space; a sending unit, configured to send a trigger signal to the ultrasonic probe and the optical fiber probe in response to determining that the signal receiving end of the clamp device receives the host acquisition signal, and to determine current timestamp information of the ultrasonic probe and the optical fiber probe to obtain current ultrasonic timestamp information and current optical timestamp information; a determining unit configured to determine, in response to determining that the ultrasonic probe and the optical fiber probe receive the trigger signal, a post-imaging optical image and a post-imaging ultrasonic image according to the current ultrasonic timestamp information and the current optical timestamp information, to obtain an optical image and an ultrasonic image; a processing unit configured to perform data alignment processing on the optical image and the ultrasonic image to obtain an aligned optical image and an aligned ultrasonic image; a fusion unit configured to fuse the aligned optical image and the aligned ultrasound image to obtain a fused image; The display unit is configured to display the fused image on a monitor for an operator to perform operation processing.

7. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

8. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images