Pose determination and image fusion method, device and equipment of ultrasonic probe in ray system

Abstract

The invention relates to the technical field of treatment assistance based on medical images, and provides a pose determination and image fusion method, device and equipment for an ultrasonic probe in a ray system, a marker does not need to be placed on the ultrasonic probe, and the situation that the accuracy of a real pose is affected due to improper selection when multiple selectable similarity measurement modes are selected can be avoided. The method comprises the following steps: determining a real two-dimensional contour of an ultrasonic probe according to a real two-dimensional image obtained by imaging the ultrasonic probe by a ray system; acquiring a virtual three-dimensional model of the ultrasonic probe; adjusting the speculated pose of the ultrasonic probe in the ray system until the similarity between a virtual two-dimensional contour obtained by performing two-dimensional projection on the virtual three-dimensional model according to the speculated pose and a real two-dimensional contour is smaller than or equal to a similarity threshold value; and determining the real pose of the ultrasonic probe in the ray system according to the speculated pose obtained after the adjustment is finished.

Description

Technical Field

[0001] The present application relates to the field of medical imaging-based treatment assistance technology, and in particular to a method for determining the posture of an ultrasound probe in a radiation system and a device thereof, a method for fusing ultrasound images and radiation images and a device thereof, computer equipment, storage media, and computer program products. Background Art

[0002] Medical imaging-based treatment-assisted technologies include image-guided therapy (IGT). IGT utilizes medical imaging to plan and execute treatment procedures, offering advantages such as reduced invasiveness, fewer complications, and improved treatment efficiency.

[0003] Different modalities of medical imaging equipment can capture different information, necessitating the fusion of multiple modalities to better assist in treatment. For example, during cardiac interventional surgery using transesophageal ultrasound, surgeons can use radiographic systems (such as C-arms) to obtain real-time radiographic images. These images carry spatial information, reflecting the spatial position of objects such as blood vessels, the heart, and surgical instruments. The high temporal and spatial resolution of radiographic images enables surgeons to precisely manipulate tiny catheters and other interventional instruments. However, radiographic images have limited ability to visualize soft tissue and generally only provide two-dimensional projections of the interventional scene, lacking a direct view of volumetric geometry. Ultrasound imaging can be used to capture this critical information, providing not only a three-dimensional perspective but also clear soft tissue imaging. By fusing radiographic and ultrasound images, surgeons can simultaneously determine the spatial position of soft tissue and interventional instruments in real time.

[0004] When fusing radiographic and ultrasound images, it's necessary to determine the true position of the ultrasound probe within the radiographic system. Some methods require placing specific markers on the ultrasound probe, which is labor-intensive. Other methods, while marker-free, offer a variety of similarity metrics to choose from. However, improperly selecting a similarity metric can affect the accuracy of the true position. Traditional techniques struggle to balance ease of use with the impact of the chosen similarity metric on the accuracy of the results. Summary of the Invention

[0005] Based on this, it is necessary to provide a method and device for determining the position of an ultrasound probe in a radiation system, a method and device for fusing ultrasound images and radiation images, a computer device, a storage medium and a computer program product to address the above technical problems.

[0006] The present application provides a method for determining the position and posture of an ultrasound probe in a ray system, the method comprising:

[0007] determining a true two-dimensional contour of the ultrasound probe based on a true two-dimensional image obtained by imaging the ultrasound probe through a ray system;

[0008] Acquiring a virtual three-dimensional model of the ultrasound probe;

[0009] Adjusting the inferred posture of the ultrasound probe in the ray system until a similarity between a virtual two-dimensional contour obtained by two-dimensionally projecting the virtual three-dimensional model according to the inferred posture and the real two-dimensional contour is less than or equal to a similarity threshold;

[0010] The actual position of the ultrasound probe in the ray system is determined according to the inferred position obtained at the end of the adjustment.

[0011] In one embodiment, determining the true two-dimensional contour of the ultrasound probe based on a true two-dimensional image obtained by imaging the ultrasound probe with a ray system includes:

[0012] Acquire a real two-dimensional image obtained by imaging the ultrasound probe through a ray system;

[0013] Perform contour segmentation on the real two-dimensional image to obtain the real two-dimensional contour of the ultrasound probe.

[0014] In one embodiment, performing a two-dimensional projection on the virtual three-dimensional model according to the inferred posture to obtain a virtual two-dimensional contour includes:

[0015] According to the inferred posture, the outline of the virtual three-dimensional model is converted from the ultrasound probe coordinate system to the ray system coordinate system;

[0016] Performing two-dimensional projection on the outline of the virtual three-dimensional model in the ray system coordinate system to obtain a virtual two-dimensional outline.

[0017] In one embodiment, performing a two-dimensional projection on the outline of the virtual three-dimensional model in the ray system coordinate system to obtain a virtual two-dimensional outline includes:

[0018] A projection algorithm based on computer graphics principles is used to perform two-dimensional projection on the outline of the virtual three-dimensional model in the ray system coordinate system to obtain a virtual two-dimensional outline.

[0019] In one embodiment, the method further comprises:

[0020] Obtaining a distance statistic value according to the distance between each virtual two-dimensional contour point of the virtual two-dimensional contour and the real two-dimensional contour;

[0021] When the distance statistic is less than or equal to a distance threshold, it is determined that the similarity between the virtual two-dimensional contour and the real two-dimensional contour is less than or equal to a similarity threshold.

[0022] In one embodiment, the virtual two-dimensional contour includes at least one of a virtual two-dimensional contour of a housing of the ultrasound probe and a virtual two-dimensional contour of internal components of the ultrasound probe.

[0023] The present application provides a device for determining the position and posture of an ultrasound probe in a ray system, the device comprising:

[0024] A true contour extraction module, configured to determine a true two-dimensional contour of the ultrasound probe based on a true two-dimensional image obtained by imaging the ultrasound probe with a ray system;

[0025] A virtual model acquisition module, configured to acquire a virtual three-dimensional model of the ultrasound probe;

[0026] an inferred posture adjustment module, configured to adjust the inferred posture of the ultrasound probe in the ray system until a similarity between a virtual two-dimensional contour obtained by two-dimensionally projecting the virtual three-dimensional model according to the inferred posture and the real two-dimensional contour is less than or equal to a similarity threshold;

[0027] The real posture determination module is used to determine the real posture of the ultrasound probe in the ray system according to the inferred posture obtained at the end of the adjustment.

[0028] The present application provides a method for fusing ultrasound images and radiographic images, the method comprising:

[0029] Acquiring an ultrasound image obtained by an ultrasound probe and a radiographic image obtained by a radiographic system;

[0030] According to the method for determining the position and posture of the ultrasound probe in the ray system described in the above embodiment, the true position and posture of the ultrasound probe in the ray system is obtained;

[0031] According to the true position, the ultrasound image is converted from an ultrasound probe coordinate system to a ray system coordinate system;

[0032] The ultrasound image and the radiographic image in the ray system coordinate system are fused to obtain a fused image.

[0033] The present application provides a device for fusing ultrasound images and radiographic images, the device comprising:

[0034] An image acquisition module, configured to acquire an ultrasonic image obtained by an ultrasonic probe and a radiographic image obtained by a radiographic system;

[0035] A real posture acquisition module, configured to obtain the real posture of the ultrasound probe in the ray system according to the method for determining the posture of the ultrasound probe in the ray system described in the above embodiment;

[0036] An ultrasound image conversion module, configured to convert the ultrasound image from an ultrasound probe coordinate system to a ray system coordinate system according to the true posture;

[0037] An image fusion module is used to fuse the ultrasound image and the radiographic image in the ray system coordinate system to obtain a fused image.

[0038] The present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the above method.

[0039] The present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program is used by a processor to execute the above method.

[0040] The present application provides a computer program product having a computer program stored thereon, wherein the computer program is used by a processor to execute the above method.

[0041] The solution provided in this application determines the true two-dimensional contour of the ultrasound probe based on the true two-dimensional image obtained by imaging the ultrasound probe with a ray system; obtains a virtual three-dimensional model of the ultrasound probe; adjusts the inferred position of the ultrasound probe in the ray system until the similarity between the virtual two-dimensional contour obtained by two-dimensionally projecting the virtual three-dimensional model according to the inferred position and the true two-dimensional contour is less than or equal to a similarity threshold; and determines the true position of the ultrasound probe in the ray system based on the inferred position obtained at the end of the adjustment. This solution determines the true position of the ultrasound probe in the ray system based on the contour of the ultrasound probe, without the need to place specific markers on the ultrasound probe, making it easy to operate. Moreover, by directly comparing similarity measures based on the dimension of contour geometric features, it avoids the impact of improper selection of multiple optional similarity measures on the accuracy of the true position. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 1 is a flow chart of a method for determining the position and posture of an ultrasound probe in a ray system according to an embodiment;

[0044] Figure 21 is a flow chart of a method for fusing ultrasound images and radiographic images in one embodiment;

[0045] FIG3( a ) is a schematic diagram of a process for iteratively calculating a pose in one embodiment;

[0046] FIG3( b ) is a schematic diagram of changes in contour similarity during an iterative calculation process in one embodiment;

[0047] Figure 4 1 is a structural block diagram of a device for determining the position and posture of an ultrasound probe in a ray system according to an embodiment;

[0048] Figure 5 is a structural block diagram of a device for fusing ultrasound images and radiographic images in one embodiment;

[0049] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0051] The present application provides a method for determining the position and posture of an ultrasonic probe in a ray system. The method comprises: Figure 1 The steps included in the method can be executed by a computer device.

[0052] Step S101 : determining a real two-dimensional contour of the ultrasound probe according to a real two-dimensional image obtained by imaging the ultrasound probe with a ray system.

[0053] A radiographic system can use radiation to image an object, which can be referred to as the imaged object. A radiographic system can perform two-dimensional radiographic imaging of the imaged object. In two-dimensional radiographic imaging, the radiation emitted by the radiographic system penetrates the imaged object, and the radiation attenuates in the imaged object before reaching the detector. The radiation signal sensed by the detector is converted into an electrical signal or digital signal, which is then processed and reconstructed to obtain a two-dimensional radiographic image. This two-dimensional radiographic image shows the projection of the imaged object in a certain direction and is formed by a real projection of the imaged object. To a certain extent, it reflects the real projection information of the imaged object. Therefore, this two-dimensional radiographic image can be referred to as a real two-dimensional image. A radiographic system that uses X-rays for imaging can be referred to as an X-ray system. An example of an X-ray system is a C-arm device.

[0054] The ray system can image the ultrasound probe using two-dimensional ray imaging, thereby obtaining a true two-dimensional image of the ultrasound probe. After obtaining the true two-dimensional image, the contour of the ultrasound probe can be extracted from the true two-dimensional image. This contour can be called the true two-dimensional contour of the ultrasound probe.

[0055] Step S102: Acquire a virtual three-dimensional model of the ultrasound probe.

[0056] The virtual 3D model of an ultrasound probe primarily reflects information such as the expected structure and dimensions of the ultrasound probe and is considered an expected model of the ultrasound probe. In contrast, the measured 3D model of an ultrasound probe primarily reflects information such as the actual structure and dimensions of the ultrasound probe and is considered an actual measurement model of the ultrasound probe.

[0057] In some scenarios, the 3D model of the ultrasound probe in design software can be used as a virtual 3D model of the ultrasound probe. Design software primarily uses computer technology to assist in design and drawing. Methods for obtaining the 3D model of the ultrasound probe in design software include, but are not limited to, obtaining it from the ultrasound probe manufacturer.

[0058] In some scenarios, an ultrasound probe can be imaged using 3D radiography, and the resulting 3D model can be used as the actual 3D model of the ultrasound probe. 3D radiography, based on 2D radiography, acquires 2D projection data from multiple angles and uses reconstruction algorithms to construct a 3D model of the imaged object, as is done with CT (Computed Tomography) technology.

[0059] This embodiment performs a 2D projection on the virtual 3D model of the ultrasound probe to generate a virtual 2D contour. Compared to digitally reconstructing radiographic images from a measured 3D model (such as that obtained using CT), this significantly reduces computational complexity and alleviates the computational burden. Digitally reconstructed radiographs are known as Digitally Reconstructed Radiographs (DRRs).

[0060] Step S103 , adjusting the inferred posture of the ultrasound probe in the ray system until the similarity between the virtual two-dimensional contour obtained by two-dimensionally projecting the virtual three-dimensional model according to the inferred posture and the real two-dimensional contour is less than or equal to a similarity threshold.

[0061] To determine the true position of the ultrasound probe within the ray system, an iterative calculation can be performed. During the iterative calculation, a pose is first assumed. A 2D projection of the virtual 3D model is performed based on this pose to obtain a virtual 2D contour. The similarity between the virtual 2D contour and the true 2D contour is then determined to be less than or equal to a similarity threshold. If not, a new pose is assumed and the process is repeated. The pose assumed during the iterative calculation is called the inferred pose.

[0062] Specifically, during the iterative calculation, the inferred position of the ultrasound probe in the ray system can be adjusted. Each time the inferred position is adjusted, a two-dimensional projection of the virtual three-dimensional model can be performed according to the inferred position to obtain a virtual two-dimensional contour. A determination is then made as to whether the similarity between the virtual two-dimensional contour and the real two-dimensional contour is less than or equal to a similarity threshold. If the similarity between the virtual two-dimensional contour and the real two-dimensional contour is greater than the similarity threshold, the inferred position can be adjusted again. If the similarity between the virtual two-dimensional contour and the real two-dimensional contour is less than or equal to the similarity threshold, the adjustment of the inferred position can be stopped.

[0063] Step S104: determining the actual position of the ultrasound probe in the ray system according to the estimated position obtained at the end of the adjustment.

[0064] As described above, when adjusting the inferred posture, the virtual three-dimensional model is projected two-dimensionally according to the inferred posture to obtain a virtual two-dimensional contour. If the similarity between the virtual two-dimensional contour and the real two-dimensional contour is less than or equal to the similarity threshold, the adjustment of the inferred posture can be stopped, and the inferred posture obtained by this adjustment (the inferred posture obtained at the end of the adjustment) is determined as the real posture of the ultrasound probe in the ray system.

[0065] The estimated position and actual position of the ultrasound probe in the ray system both belong to the position of the ultrasound probe in the ray system. The position of the ultrasound probe in the ray system includes: the position of the ultrasound probe in the ray system and the attitude of the ultrasound probe in the ray system. The position of the ultrasound probe in the ray system matches the position of the ultrasound probe coordinate system in the ray system coordinate system; the position may include the coordinates of three coordinate axes. The attitude of the ultrasound probe in the ray system matches the attitude of the ultrasound probe coordinate system in the ray system coordinate system; the attitude may include the yaw angle, pitch angle, and roll angle.

[0066] In the case where the position includes the coordinates of three coordinate axes and the attitude includes the yaw angle, the pitch angle and the roll angle, the posture of the ultrasound probe in the ray system belongs to a six-degree-of-freedom posture.

[0067] In the above-mentioned method for determining the position and posture of an ultrasound probe in a radiographic system, the true two-dimensional contour of the ultrasound probe is determined based on the true two-dimensional image obtained by imaging the ultrasound probe through the radiographic system; a virtual three-dimensional model of the ultrasound probe is obtained; the inferred position and posture of the ultrasound probe in the radiographic system is adjusted until the similarity between the virtual two-dimensional contour obtained by two-dimensionally projecting the virtual three-dimensional model according to the inferred position and the true two-dimensional contour is less than or equal to a similarity threshold; and the true position and posture of the ultrasound probe in the radiographic system is determined based on the inferred position and posture obtained at the end of the adjustment. This solution determines the true position and posture of the ultrasound probe in the radiographic system based on the contour of the ultrasound probe, without requiring the placement of specific markers on the ultrasound probe and is simple to operate. Furthermore, by directly comparing similarity measures based on the dimension of contour geometric features, it avoids the possibility of inappropriate selection of multiple available similarity measures that could affect the accuracy of the true position and posture.

[0068] In one embodiment, determining the true two-dimensional contour of the ultrasound probe based on a true two-dimensional image obtained by imaging the ultrasound probe with a ray system includes:

[0069] A real two-dimensional image is obtained by imaging the ultrasound probe with a ray system; and contour segmentation is performed on the real two-dimensional image to obtain a real two-dimensional contour of the ultrasound probe.

[0070] The X-ray system can image the ultrasound probe using two-dimensional radiographic imaging, thereby obtaining a true two-dimensional image of the ultrasound probe. After obtaining the true two-dimensional image, a segmentation model developed using deep learning can be used to segment the true two-dimensional image to obtain the contour of the ultrasound probe, which can be referred to as the true two-dimensional contour of the ultrasound probe. The segmentation model can use a Unet network architecture. Unet network can be abbreviated as U-shaped network.

[0071] In this embodiment, after acquiring a real two-dimensional image obtained by imaging the ultrasound probe with a ray system, a real two-dimensional contour reflecting the real situation of the ultrasound probe can be obtained by performing contour segmentation on the real two-dimensional image for subsequent contour similarity comparison. This eliminates the need to analyze other content in the image, simplifies calculations, and avoids the influence of other content in the image on the comparison results.

[0072] In one embodiment, performing a two-dimensional projection on the virtual three-dimensional model according to the inferred posture to obtain a virtual two-dimensional contour includes:

[0073] According to the inferred posture, the outline of the virtual three-dimensional model is converted from the ultrasound probe coordinate system to the ray system coordinate system; the outline of the virtual three-dimensional model in the ray system coordinate system is projected two-dimensionally to obtain a virtual two-dimensional outline.

[0074] The virtual 3D model of the ultrasound probe obtained above is located in the ultrasound probe coordinate system. Applying the inferred pose to the outline of the virtual 3D model in the ultrasound probe coordinate system can transform the outline of the virtual 3D model from the ultrasound probe coordinate system to the ray system coordinate system, thereby obtaining the outline of the virtual 3D model in the ray system coordinate system. Next, a 2D projection is performed on the outline of the virtual 3D model in the ray system coordinate system to obtain a virtual 2D outline.

[0075] In this embodiment, the contour of the virtual three-dimensional model is converted to the ray system coordinate system according to the inferred posture, and the contour of the virtual three-dimensional model is projected in the ray system coordinate system to obtain the virtual two-dimensional contour. There is no need to convert the non-contour part of the virtual three-dimensional model, which further reduces the computational burden.

[0076] In one embodiment, performing a two-dimensional projection on the outline of the virtual three-dimensional model in the ray system coordinate system to obtain a virtual two-dimensional outline includes:

[0077] By using a projection algorithm based on computer graphics principles, the outline of the virtual three-dimensional model in the ray system coordinate system is projected two-dimensionally to obtain a virtual two-dimensional outline.

[0078] Projection algorithms based on computer graphics principles include, but are not limited to, depth buffering algorithms (also known as Z-buffering algorithms). Depth buffering algorithms are projection algorithms based on computer graphics. Leveraging GPU hardware and software optimizations, they enable rapid two-dimensional projection of the outlines of virtual three-dimensional models.

[0079] The contour of the virtual three-dimensional model in the ray system coordinate system is projected two-dimensionally using a projection algorithm based on computer graphics principles. The obtained projection result can be called a virtual two-dimensional contour.

[0080] In this embodiment, a projection algorithm based on computer graphics principles performs a two-dimensional projection on the outline of the virtual three-dimensional model in the ray system coordinate system to obtain a virtual two-dimensional outline. This projection is based on a precise mathematical description of the model. Compared with the projection of the measured three-dimensional model obtained by CT technology, there is no need to simulate the integral calculation process of light passing through the volume data, and no operations such as interpolation and sampling of the volume data are involved. The calculation is simple, which reduces the computational burden.

[0081] In one embodiment, the method for determining the position and posture of the ultrasound probe in the ray system provided by the present application further includes:

[0082] A distance statistic is obtained based on the distance between each virtual two-dimensional contour point of the virtual two-dimensional contour and the real two-dimensional contour; when the distance statistic is less than or equal to a distance threshold, it is determined that the similarity between the virtual two-dimensional contour and the real two-dimensional contour is less than or equal to a similarity threshold.

[0083] The virtual two-dimensional contour includes a plurality of points, which can be referred to as virtual two-dimensional contour points. The distance between each virtual two-dimensional contour point and the real two-dimensional contour is calculated and counted to obtain a distance statistic value.

[0084] Among them, the real two-dimensional contour includes several points, which can be called real two-dimensional contour points. For any virtual two-dimensional contour point, the distance between the virtual two-dimensional contour point and each real two-dimensional contour point can be calculated, and the real two-dimensional contour point with the smallest distance from the virtual two-dimensional contour point can be determined. The real two-dimensional contour point is called the nearest real two-dimensional contour point corresponding to the virtual two-dimensional contour point; the distance between each virtual two-dimensional contour point and the corresponding nearest real two-dimensional contour point is counted, and the statistical method includes but is not limited to the summation method, thereby obtaining the distance statistical value.

[0085] Among them, the real two-dimensional contour is a line. For any virtual two-dimensional contour point, the shortest distance between the virtual two-dimensional contour point and the real two-dimensional contour can be calculated by the shortest distance calculation method from point to line; the shortest distance between each virtual two-dimensional contour point and the real two-dimensional contour is counted, and the statistical method includes but is not limited to the summation method, thereby obtaining the distance statistical value.

[0086] The process of adjusting the estimated position of the ultrasound probe in the ray system is introduced by obtaining a statistical distance value by counting the distance between each virtual two-dimensional contour point and the corresponding nearest real two-dimensional contour point.

[0087] The outline of the virtual 3D model includes several points, which can be called virtual 3D outline points. The i-th virtual 3D outline point can be recorded as , the virtual 3D contour point set can be recorded as , the i-th virtual 3D contour point and virtual 3D contour point set The relationship is: .

[0088] Method 1:

[0089] You can preset the default pose value T0 and record the pose change as △T. Adjust the pose change △T at least once. Each time you adjust the pose change △T, you can perform the following steps:

[0090] According to the product of the posture change △T of this adjustment and the posture default value T0, the estimated posture of this adjustment is . The estimated pose Acting on each virtual 3D contour point of the ultrasound probe coordinate system, thereby transforming the contour of the virtual 3D model from the ultrasound probe coordinate system to the ray system coordinate system; for example, the inferred pose Acting on the i-th virtual 3D contour point , which can be expressed as Then, we can use the projection algorithm based on computer graphics to perform two-dimensional projection on each virtual three-dimensional contour point in the ray system coordinate system to obtain several virtual two-dimensional contour points in the ray system coordinate system, which can be expressed as , It is a projection function that converts a 3D point into a 2D point. Calculate and count the distance between each virtual 2D contour point and the corresponding nearest real 2D contour point to obtain the distance statistics; if the distance statistics is greater than the distance threshold, the pose change △T can be adjusted again; if the distance statistics is less than or equal to the distance threshold, it is determined that the similarity between the virtual 2D contour and the real 2D contour is less than or equal to the similarity threshold, and the pose change △T adjustment can be stopped, and the product of the adjusted pose change △T and the pose default value T0 is calculated. , which is determined as the adjustment result, can be used as the true position of the ultrasound probe in the ray system.

[0091] The above process can be expressed as follows:

[0092] .

[0093] The above formula can be written in the standard least squares form and solved using the second-order gradient iteration of Gauss-Newton. j is the i-th virtual 3D contour point The nearest real two-dimensional contour point. Let the real two-dimensional contour point set be , the jth virtual 3D contour point q j and the real 2D contour point set The relationship is: .argmin represents the minimization function.

[0094] Method 2:

[0095] The estimated pose can be denoted as T. The estimated pose T is adjusted at least once. Each time the estimated pose T is adjusted, the following steps can be performed:

[0096] The adjusted estimated pose T is applied to each virtual 3D contour point in the ultrasound probe coordinate system, thereby converting the contour of the virtual 3D model from the ultrasound probe coordinate system to the ray system coordinate system; for example, the estimated pose T is applied to the i-th virtual 3D contour point , which can be expressed as Then, we can use the projection algorithm based on computer graphics to perform two-dimensional projection on each virtual three-dimensional contour point in the ray system coordinate system to obtain several virtual two-dimensional contour points in the ray system coordinate system, which can be expressed as , The projection function converts 3D points into 2D points. The distance between each virtual 2D contour point and the corresponding nearest real 2D contour point is calculated and counted to obtain a distance statistic. If the distance statistic is greater than the distance threshold, the next adjustment to the inferred pose T can be made. If the distance statistic is less than or equal to the distance threshold, it is determined that the similarity between the virtual 2D contour and the real 2D contour is less than or equal to the similarity threshold, and the adjustment of the inferred pose T can be stopped. The adjusted inferred pose T is determined as the adjustment result, which can be used as the real pose of the ultrasound probe in the ray system.

[0097] The above process can be expressed as follows:

[0098] .

[0099] The above formula can be written in the standard least squares form and solved using the second-order gradient iteration of Gauss-Newton. j is the i-th virtual 3D contour point The nearest real two-dimensional contour point. Let the real two-dimensional contour point set be , the jth virtual 3D contour point q j and the real 2D contour point set The relationship is: .argmin represents the minimization function.

[0100] In this embodiment, the distance between each virtual two-dimensional contour point and the real two-dimensional contour is counted to obtain a distance statistic value. If the distance statistic value is less than or equal to the distance threshold, it means that the virtual two-dimensional contour is close to the real two-dimensional contour, and the similarity between the virtual two-dimensional contour and the real two-dimensional contour is less than or equal to the similarity threshold, thereby avoiding the influence of individual virtual two-dimensional contour points on the results and improving robustness.

[0101] In one embodiment, the virtual two-dimensional contour includes at least one of a virtual two-dimensional contour of a housing of the ultrasound probe and a virtual two-dimensional contour of internal components of the ultrasound probe.

[0102] If the virtual two-dimensional outline includes the virtual two-dimensional outline of the ultrasound probe housing, the real two-dimensional outline includes the real two-dimensional outline of the ultrasound probe housing. If the virtual two-dimensional outline includes the virtual two-dimensional outline of the ultrasound probe internal components, the real two-dimensional outline includes the real two-dimensional outline of the ultrasound probe internal components. If the virtual two-dimensional outline includes the virtual two-dimensional outline of the ultrasound probe housing and the virtual two-dimensional outline of the ultrasound probe internal components, the real two-dimensional outline includes the real two-dimensional outline of the ultrasound probe housing and the real two-dimensional outline of the ultrasound probe internal components.

[0103] When the virtual two-dimensional contour includes the virtual two-dimensional contour of the ultrasound probe housing and the virtual two-dimensional contour of the internal components of the ultrasound probe, the points of the virtual two-dimensional contour of the ultrasound probe housing can be referred to as virtual two-dimensional contour points of the housing, and the points of the virtual two-dimensional contour of the internal components of the ultrasound probe can be referred to as virtual two-dimensional contour points of the internal components. Correspondingly, the points of the real two-dimensional contour of the ultrasound probe housing can be referred to as real two-dimensional contour points of the housing, and the points of the real two-dimensional contour of the internal components of the ultrasound probe housing can be referred to as real two-dimensional contour points of the internal components.

[0104] The distance between each virtual 2D contour point of the shell and the real 2D contour of the ultrasound probe shell can be calculated and counted to obtain a shell distance statistic; the distance between each virtual 2D contour point of the internal component and the real 2D contour of the ultrasound probe shell can be calculated and counted to obtain an internal component distance statistic; the relative size of the contour weights between the ultrasound probe shell and the real 2D contour of the ultrasound probe internal component can be determined based on the relative size of the clarity and / or relative size of the completeness between the real 2D contour of the ultrasound probe shell and the real 2D contour of the ultrasound probe internal component to obtain a contour weight corresponding to the ultrasound probe shell and a contour weight corresponding to the ultrasound probe internal component; the contour weight corresponding to the ultrasound probe shell is multiplied by the shell distance statistic to obtain a first product result, and the contour weight corresponding to the ultrasound probe internal component is multiplied by the internal component distance statistic to obtain a second product result; the first product result and the second product result are summed to obtain a distance statistic. If the distance statistic is less than or equal to a distance threshold, it is determined that the similarity between the virtual 2D contour and the real 2D contour is less than or equal to a similarity threshold.

[0105] In the real two-dimensional contour of the ultrasound probe shell and the real two-dimensional contour of the internal components of the ultrasound probe, the relative size of the clarity is consistent with the relative size of the contour weight; for example, if the clarity of the real two-dimensional contour of the ultrasound probe shell is greater than the clarity of the real two-dimensional contour of the internal components of the ultrasound probe, then the contour weight corresponding to the ultrasound probe shell is greater than the contour weight corresponding to the internal components of the ultrasound probe; for another example, if the clarity of the real two-dimensional contour of the ultrasound probe shell is less than the clarity of the real two-dimensional contour of the internal components of the ultrasound probe, then the contour weight corresponding to the ultrasound probe shell is less than the contour weight corresponding to the internal components of the ultrasound probe.

[0106] In the real two-dimensional contour of the ultrasound probe shell and the real two-dimensional contour of the internal components of the ultrasound probe, the relative size of the completeness is consistent with the relative size of the contour weight; for example, if the completeness of the real two-dimensional contour of the ultrasound probe shell is greater than the completeness of the real two-dimensional contour of the internal components of the ultrasound probe, then the contour weight corresponding to the ultrasound probe shell is greater than the contour weight corresponding to the internal components of the ultrasound probe; for another example, if the completeness of the real two-dimensional contour of the ultrasound probe shell is less than the completeness of the real two-dimensional contour of the internal components of the ultrasound probe, then the contour weight corresponding to the ultrasound probe shell is less than the contour weight corresponding to the internal components of the ultrasound probe.

[0107] The relative size of the contour weight can be determined based on at least one of the relative size of the clarity and the relative size of the completeness; according to the relative size of the contour weight, the contour weight corresponding to the ultrasound probe shell and the contour weight corresponding to the internal components of the ultrasound probe can be determined.

[0108] In this embodiment, when similarity measurement is performed based on contour geometric features, the used contour includes at least one of the contour of the ultrasound probe housing and the contour of the internal components of the ultrasound probe, which can improve the flexibility of contour similarity measurement.

[0109] The present application provides a method for fusing ultrasound images and radiographic images, the method comprising: Figure 2 The steps included in the method can be executed by a computer device.

[0110] Step S201 : Acquire an ultrasound image obtained by an ultrasound probe and a radiographic image obtained by a radiographic system.

[0111] For example, in cardiac interventional surgery using transesophageal ultrasound, the radiation system can use a C-arm device; the ultrasound probe is placed in the esophagus, and the ultrasound probe can closely explore the deep structure of the heart from the back to the front; the C-arm device can help the surgeon clearly observe the contours of the heart, the direction of blood vessels, the location of lesions, and the position and movement trajectory of the interventional device in the heart.

[0112] The image obtained by the ultrasound probe can be called an ultrasound image, and the ultrasound image is located in the ultrasound probe coordinate system; the image obtained by the ray system can be called a ray image, and the ray image is located in the ray system coordinate system.

[0113] Step S202 : According to the method for determining the position and posture of the ultrasound probe in the ray system described in the above embodiment, the actual position and posture of the ultrasound probe in the ray system is obtained.

[0114] The above embodiment introduces a method for determining the position and posture of an ultrasound probe in a ray system. According to this method, the actual position and posture of the ultrasound probe in the ray system can be determined.

[0115] Step S203 : transforming the ultrasound image from the ultrasound probe coordinate system to the ray system coordinate system according to the actual posture.

[0116] After obtaining the true position of the ultrasound probe in the ray system, the true position can be applied to the ultrasound image to transform the ultrasound image from the ultrasound probe coordinate system to the ray system coordinate system.

[0117] Step S204 : Fusing the ultrasound image and the radiographic image in the ray system coordinate system to obtain a fused image.

[0118] After converting the ultrasound image from the ultrasound probe coordinate system to the ray system coordinate system, both the ultrasound image and the ray image are located in the ray system coordinate system. At this time, the ultrasound image and the ray image can be fused to obtain a fused image. The fused image can not only provide the operator with spatial information involving blood vessels, heart and interventional devices, but also clearly display soft tissue and provide a direct observation perspective of volumetric geometric structures.

[0119] In the above-mentioned method for fusing ultrasound images and radiographic images, a method for determining the position of an ultrasound probe in a radiographic system is introduced according to the above-mentioned embodiment to determine the true position of the ultrasound probe in the radiographic system. There is no need to place specific markers on the ultrasound probe, and the operation is simple. Moreover, the comparison is made directly from the similarity measurement method in the dimension of contour geometric features, avoiding improper selection of multiple optional similarity measurement methods to affect the accuracy of the true position; and, based on the true position, the ultrasound image and the radiographic image can be fused to obtain a fused image, which not only provides the operator with spatial information involving blood vessels, heart and interventional devices, but also can clearly display soft tissue and provide a direct observation perspective of volumetric geometric structures.

[0120] To better understand the above method, the following describes in detail an application example of the method for determining the position of an ultrasound probe in a radiographic system and the method for fusing ultrasound and radiographic images provided by this application. This application example uses a C-arm device used in transesophageal ultrasound cardiac interventional surgery as an example. The C-arm device uses X-rays for imaging.

[0121] This application embodiment can locate the position of an ultrasound probe based on X-ray data without a marker. In this application embodiment, object tracking and matching based on a 3D template are used to determine the true position of the ultrasound probe within the C-arm device.

[0122] Specifically, this application embodiment utilizes a virtual 3D model of the ultrasound probe (which may include a virtual 3D model of the ultrasound probe housing and a virtual 3D model of its internal components) combined with the actual 2D contour of the ultrasound probe in the X-ray image to perform 3D and 2D contour matching. Once the matching is complete, the true position of the ultrasound probe in the C-arm device is obtained. This application embodiment uses a similarity metric based on the geometric features of the contours, independent of the chosen similarity metric method, enabling real-time and robust matching and tracking. Furthermore, the virtual 3D model projection process significantly reduces the computational complexity compared to the integral operation of DRR.

[0123] FIG3 (a) is a flow chart of object tracking and matching based on a three-dimensional template used in this application embodiment. Specifically, the flow chart shown in FIG3 (a) may include the following steps:

[0124] Acquire a true 2D image of the ultrasound probe imaged by the C-arm device; this true 2D image is an X-ray image. Using a segmentation model derived through deep learning, perform contour segmentation on the true 2D image to obtain the true 2D contour of the ultrasound probe.

[0125] Obtain a virtual 3D model of the ultrasound probe. The optimization unit shown in Figure 3(a) adjusts the estimated position of the ultrasound probe in the C-arm device. Based on the estimated position obtained at the end of the adjustment, the actual position of the ultrasound probe in the C-arm device is determined.

[0126] Among them, each time the inferred posture is adjusted, the following steps can be performed: according to the inferred posture, the outline of the virtual three-dimensional model is converted from the ultrasound probe coordinate system to the C-arm device coordinate system; using a projection algorithm based on computer graphics principles, the outline of the virtual three-dimensional model in the C-arm device coordinate system is two-dimensionally projected to obtain a virtual two-dimensional outline; according to the distance between each virtual two-dimensional contour point of the virtual two-dimensional contour and the real two-dimensional contour, a distance statistic is obtained; when the distance statistic is greater than the distance threshold, it is determined that the similarity between the virtual two-dimensional contour and the real two-dimensional contour is greater than the similarity threshold, and the next adjustment of the inferred posture is performed; when the distance statistic is less than or equal to the distance threshold, it is determined that the similarity between the virtual two-dimensional contour and the real two-dimensional contour is less than or equal to the similarity threshold, and the adjustment of the inferred posture is stopped.

[0127] The outline of the virtual 3D model includes several points, which can be called virtual 3D outline points. The i-th virtual 3D outline point can be recorded as , the virtual 3D contour point set can be recorded as , the i-th virtual 3D contour point and virtual 3D contour point set The relationship is: .

[0128] The default value of the posture can be preset as T0, and the posture change is recorded as △T. The posture change △T is adjusted at least once. The optimization process of the above optimization unit can be expressed as follows:

[0129] .

[0130] When expressed in this formula, each time the posture change △T is adjusted, the steps performed can be described as: According to the product of the posture change △T adjusted this time and the posture default value T0, the estimated posture of this adjustment is obtained as . The estimated pose Acting on each virtual 3D contour point of the ultrasound probe coordinate system, thereby transforming the contour of the virtual 3D model from the ultrasound probe coordinate system to the C-arm device coordinate system; for example, the inferred posture Acting on the i-th virtual 3D contour point , which can be expressed as Then, a projection algorithm based on computer graphics can be used to perform a two-dimensional projection on each virtual three-dimensional contour point in the C-arm device coordinate system to obtain several virtual two-dimensional contour points in the C-arm device coordinate system, which can be expressed as , It is a projection function that converts a 3D point into a 2D point. Calculate and count the distance between each virtual 2D contour point and the corresponding nearest real 2D contour point to obtain the distance statistics; if the distance statistics is greater than the distance threshold, the pose change △T can be adjusted again; if the distance statistics is less than or equal to the distance threshold, it is determined that the similarity between the virtual 2D contour and the real 2D contour is less than or equal to the similarity threshold, and the pose change △T adjustment is stopped, and the product of the adjusted pose change △T and the pose default value T0 is calculated. , which is determined as the adjustment result, can be used as the actual posture of the ultrasound probe in the C-arm device.

[0131] The estimated pose can be denoted as T. The estimated pose T is adjusted at least once. The optimization process of the above optimization unit can be expressed as follows:

[0132] .

[0133] When expressed in this formula, each time the estimated posture is adjusted, the steps performed can be described as: applying the adjusted estimated posture T to each virtual three-dimensional contour point in the ultrasound probe coordinate system, thereby converting the contour of the virtual three-dimensional model from the ultrasound probe coordinate system to the C-arm device coordinate system; for example, applying the estimated posture T to the i-th virtual three-dimensional contour point , which can be expressed as Then, a projection algorithm based on computer graphics can be used to perform a two-dimensional projection on each virtual three-dimensional contour point in the C-arm device coordinate system to obtain several virtual two-dimensional contour points in the C-arm device coordinate system, which can be expressed as , This is a projection function that converts 3D points into 2D points. The distance between each virtual 2D contour point and the corresponding nearest real 2D contour point is calculated and counted to obtain a distance statistic. If the distance statistic is greater than the distance threshold, the next adjustment to the inferred pose T can be made. If the distance statistic is less than or equal to the distance threshold, it is determined that the similarity between the virtual 2D contour and the real 2D contour is less than or equal to the similarity threshold, and the adjustment of the inferred pose T is stopped. The adjusted inferred pose T is determined as the adjustment result, which can be used as the actual pose of the ultrasound probe in the C-arm device.

[0134] Refer to Figure 3(b), which shows the changes in contour similarity when iteratively calculating poses using the method provided in this application embodiment. The solid line in Figure 3(b) represents the true 2D contour of the ultrasound probe, and the dashed line in Figure 3(b) represents the virtual 2D contour of the ultrasound probe. From left to right, the leftmost estimated pose has the lowest accuracy, and the leftmost virtual 2D contour has the lowest similarity to the true 2D contour. The rightmost estimated pose has the highest accuracy, and the rightmost virtual 2D contour has the highest similarity to the true 2D contour.

[0135] After determining the true position of the ultrasound probe within the C-arm, the ultrasound probe can be used to obtain an ultrasound image of the surgical object. This image can then be converted from the probe coordinate system to the C-arm coordinate system. Furthermore, an X-ray image can be obtained from the C-arm using two-dimensional X-ray imaging of the surgical object. The ultrasound and X-ray images can then be fused within the C-arm coordinate system to create a fused image. This fused image not only provides the surgeon with spatial information regarding blood vessels, the heart, and interventional devices, but also clearly displays soft tissue and provides a direct perspective for observing volumetric geometric structures.

[0136] In this application embodiment, when determining the true position of the ultrasound probe in the X-ray system, there is no need to place specific markers on the ultrasound probe, which is easy to operate. Moreover, the comparison is made directly from the similarity measurement method in the dimension of contour geometric features, avoiding the influence of improper selection of multiple optional similarity measurement methods on the accuracy of the true position. Moreover, based on the true position, the ultrasound image and the X-ray image can be fused to obtain a fused image, which not only provides the operator with spatial information involving blood vessels, heart and interventional devices, but also can clearly display soft tissue and provide a direct observation perspective of volumetric geometric structures.

[0137] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0138] Based on the same inventive concept, an embodiment of the present application further provides an apparatus for determining the position of an ultrasonic probe in a ray system, which is used to implement the aforementioned method for determining the position of an ultrasonic probe in a ray system. The solution provided by this apparatus is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the embodiments of the apparatus for determining the position of one or more ultrasonic probes in a ray system provided below can be found in the aforementioned limitations of the method for determining the position of an ultrasonic probe in a ray system, and will not be further elaborated here.

[0139] In one embodiment, Figure 4 As shown, a device for determining the position of an ultrasound probe in a ray system is provided, comprising:

[0140] A true contour extraction module 401 is configured to determine a true two-dimensional contour of the ultrasound probe based on a true two-dimensional image obtained by imaging the ultrasound probe with a ray system;

[0141] A virtual model acquisition module 402 is configured to acquire a virtual three-dimensional model of the ultrasound probe;

[0142] an inferred posture adjustment module 403, configured to adjust the inferred posture of the ultrasound probe in the ray system until a similarity between a virtual two-dimensional contour obtained by two-dimensionally projecting the virtual three-dimensional model according to the inferred posture and the real two-dimensional contour is less than or equal to a similarity threshold;

[0143] The real posture determination module 404 is used to determine the real posture of the ultrasound probe in the ray system according to the inferred posture obtained at the end of the adjustment.

[0144] In one embodiment, the true contour extraction module 401 is further configured to:

[0145] A real two-dimensional image is obtained by imaging the ultrasound probe with a ray system; and contour segmentation is performed on the real two-dimensional image to obtain a real two-dimensional contour of the ultrasound probe.

[0146] In one embodiment, the device further comprises a virtual contour acquisition module, configured to:

[0147] According to the inferred posture, the outline of the virtual three-dimensional model is converted from the ultrasound probe coordinate system to the ray system coordinate system; and the outline of the virtual three-dimensional model in the ray system coordinate system is two-dimensionally projected to obtain a virtual two-dimensional outline.

[0148] In one embodiment, the virtual contour acquisition module is further configured to:

[0149] A projection algorithm based on computer graphics principles is used to perform two-dimensional projection on the outline of the virtual three-dimensional model in the ray system coordinate system to obtain a virtual two-dimensional outline.

[0150] In one embodiment, the device further includes a contour similarity determination module, configured to:

[0151] A distance statistic is obtained based on the distance between each virtual two-dimensional contour point of the virtual two-dimensional contour and the real two-dimensional contour; when the distance statistic is less than or equal to a distance threshold, it is determined that the similarity between the virtual two-dimensional contour and the real two-dimensional contour is less than or equal to a similarity threshold.

[0152] In one embodiment, the virtual two-dimensional contour includes at least one of a virtual two-dimensional contour of a housing of the ultrasound probe and a virtual two-dimensional contour of internal components of the ultrasound probe.

[0153] Each module in the apparatus for determining the position and posture of an ultrasound probe in a radiographic system may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0154] Based on the same inventive concept, embodiments of the present application also provide an ultrasound image and radiographic image fusion device for implementing the ultrasound image and radiographic image fusion method described above. The solution provided by this device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more ultrasound image and radiographic image fusion device embodiments provided below can be found in the limitations of the ultrasound image and radiographic image fusion method described above and will not be further elaborated here.

[0155] In one embodiment, Figure 5 As shown, a device for fusing ultrasound images and radiographic images is provided, comprising:

[0156] An image acquisition module 501 is configured to acquire an ultrasound image obtained by an ultrasound probe and a radiographic image obtained by a radiographic system;

[0157] A real posture acquisition module 502 is configured to obtain the real posture of the ultrasound probe in the ray system according to the method for determining the posture of the ultrasound probe in the ray system described in the above embodiment;

[0158] An ultrasound image conversion module 503 is configured to convert the ultrasound image from an ultrasound probe coordinate system to a ray system coordinate system according to the real posture;

[0159] The image fusion module 504 is configured to fuse the ultrasound image and the radiographic image in the ray system coordinate system to obtain a fused image.

[0160] Each module in the aforementioned ultrasound image and radiographic image fusion apparatus may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0161] In an exemplary embodiment, a computer device is provided, the internal structure of which can be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the above method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a method introduced in the above embodiment.

[0162] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0163] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0164] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0165] In one embodiment, a computer program product is provided, on which a computer program is stored. The computer program is used by a processor to execute the steps in the above-mentioned various method embodiments.

[0166] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0167] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0168] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0169] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for determining the position and posture of an ultrasonic probe in a ray system, characterized in that: The method comprises: determining a true two-dimensional contour of the ultrasound probe based on a true two-dimensional image obtained by imaging the ultrasound probe through a ray system; Acquiring a virtual three-dimensional model of the ultrasound probe; Adjusting the inferred posture of the ultrasound probe in the ray system until a similarity between a virtual two-dimensional contour obtained by two-dimensionally projecting the virtual three-dimensional model according to the inferred posture and the real two-dimensional contour is less than or equal to a similarity threshold; The actual position of the ultrasound probe in the ray system is determined according to the inferred position obtained at the end of the adjustment.

2. The method according to claim 1, characterized in that Determining a true two-dimensional contour of the ultrasound probe based on a true two-dimensional image obtained by imaging the ultrasound probe through a ray system includes: Acquire a real two-dimensional image obtained by imaging the ultrasound probe through a ray system; Perform contour segmentation on the real two-dimensional image to obtain the real two-dimensional contour of the ultrasound probe.

3. The method according to claim 1, characterized in that Performing a two-dimensional projection on the virtual three-dimensional model according to the inferred posture to obtain a virtual two-dimensional contour includes: According to the inferred posture, the outline of the virtual three-dimensional model is converted from the ultrasound probe coordinate system to the ray system coordinate system; Performing two-dimensional projection on the outline of the virtual three-dimensional model in the ray system coordinate system to obtain a virtual two-dimensional outline.

4. The method according to claim 3, characterized in that Performing a two-dimensional projection on the outline of the virtual three-dimensional model in the ray system coordinate system to obtain a virtual two-dimensional outline includes: A projection algorithm based on computer graphics principles is used to perform two-dimensional projection on the outline of the virtual three-dimensional model in the ray system coordinate system to obtain a virtual two-dimensional outline.

5. The method according to claim 1, wherein The method further comprises: Obtaining a distance statistic value according to the distance between each virtual two-dimensional contour point of the virtual two-dimensional contour and the real two-dimensional contour; When the distance statistic is less than or equal to a distance threshold, it is determined that the similarity between the virtual two-dimensional contour and the real two-dimensional contour is less than or equal to a similarity threshold.

6. The method according to any one of claims 1 to 5, characterized in that The virtual two-dimensional contour includes at least one of a virtual two-dimensional contour of a housing of the ultrasound probe and a virtual two-dimensional contour of internal components of the ultrasound probe.

7. A method for fusing ultrasound images and radiographic images, characterized in that: The method comprises: Acquiring an ultrasound image obtained by an ultrasound probe and a radiographic image obtained by a radiographic system; The method for determining the position and posture of an ultrasonic probe in a ray system according to any one of claims 1 to 6, obtaining the true position and posture of the ultrasonic probe in the ray system; According to the true position, the ultrasound image is converted from an ultrasound probe coordinate system to a ray system coordinate system; The ultrasound image and the radiographic image in the ray system coordinate system are fused to obtain a fused image.

8. A device for determining the position of an ultrasonic probe in a ray system, characterized in that: The device comprises: A true contour extraction module, configured to determine a true two-dimensional contour of the ultrasound probe based on a true two-dimensional image obtained by imaging the ultrasound probe with a ray system; A virtual model acquisition module, configured to acquire a virtual three-dimensional model of the ultrasound probe; an inferred posture adjustment module, configured to adjust the inferred posture of the ultrasound probe in the ray system until a similarity between a virtual two-dimensional contour obtained by two-dimensionally projecting the virtual three-dimensional model according to the inferred posture and the real two-dimensional contour is less than or equal to a similarity threshold; The real posture determination module is used to determine the real posture of the ultrasound probe in the ray system according to the inferred posture obtained at the end of the adjustment.

9. A device for fusing ultrasound images and radiographic images, characterized in that: The device comprises: An image acquisition module, configured to acquire an ultrasonic image obtained by an ultrasonic probe and a radiographic image obtained by a radiographic system; A real posture acquisition module, configured to obtain the real posture of the ultrasound probe in the ray system according to the method for determining the posture of the ultrasound probe in the ray system according to any one of claims 1 to 6; An ultrasound image conversion module, configured to convert the ultrasound image from an ultrasound probe coordinate system to a ray system coordinate system according to the true posture; An image fusion module is used to fuse the ultrasound image and the radiographic image in the ray system coordinate system to obtain a fused image.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images