Ultrasonic three-dimensional model registration method, device, equipment and computer program product

Abstract

The invention is suitable for the technical field of medical treatment, and provides an ultrasonic three-dimensional model registration method, device and equipment and a computer program product. The ultrasonic three-dimensional model registration method comprises the following steps: acquiring a real-time ultrasonic image obtained by scanning an inspection object; obtaining a to-be-registered three-dimensional model of the inspection object; and performing rigid registration and non-rigid registration on the real-time ultrasonic image and the to-be-registered three-dimensional model, and determining a deformed three-dimensional model. According to the embodiment of the invention, the three-dimensional model can be associated with the ultrasonic image, the three-dimensional model can conform to the organ or tissue form presented by the ultrasonic image through registration, more accurate anatomical structure information is provided, and the real-time performance is improved.

Description

Technical Field

[0001] The present application belongs to the field of medical technology, and in particular, relates to a method, device, equipment and computer program product for ultrasound three-dimensional model registration. Background Art

[0002] Ultrasound (US) images are widely used in large-scale screening and routine examinations, interventional treatments, and intraoperative navigation due to their real-time nature. However, the image quality of two-dimensional ultrasound images is limited, and there are only slice data, which places a great operational burden on the operator. Some related technologies use three-dimensional ultrasound to provide overall structural information of the object under examination. Three-dimensional ultrasound is essentially a method of capturing two-dimensional images at different angles through an ultrasound probe and generating a volumetric image, providing a three-dimensional view of an organ or tissue. However, three-dimensional ultrasound equipment has the problems of high cost and slow scanning speed, making it difficult to meet real-time requirements. Summary of the invention

[0003] The embodiments of the present application provide a method, apparatus, device and computer program product for ultrasound three-dimensional model registration, which can associate a three-dimensional model with an ultrasound image. Through registration, the three-dimensional model can conform to the organ or tissue morphology presented in the ultrasound image, provide more accurate anatomical structure information, and improve real-time performance.

[0004] A first aspect of an embodiment of the present application provides a method for ultrasound three-dimensional model registration, comprising: acquiring a real-time ultrasound image obtained by scanning an inspection object; acquiring a three-dimensional model to be registered of the inspection object; performing rigid registration and non-rigid registration on the real-time ultrasound image and the three-dimensional model to be registered to determine a deformed three-dimensional model.

[0005] In some embodiments of the first aspect, acquiring the three-dimensional model to be registered of the examination object includes: determining phase information of the real-time ultrasound image; and determining the three-dimensional model to be registered in a three-dimensional model sequence according to the phase information.

[0006] In some embodiments of the first aspect, determining the phase information of the real-time ultrasound image includes: acquiring an ultrasound image sequence, the ultrasound image sequence including the real-time ultrasound image and historical ultrasound images; determining, based on the historical ultrasound images, a correspondence between morphological indicators presented in the historical ultrasound images and time; and determining the phase information of the real-time ultrasound image based on the correspondence.

[0007] In some embodiments of the first aspect, rigidly registering the real-time ultrasound image and the three-dimensional model to be registered includes: rigidly registering the real-time ultrasound image and the three-dimensional model to be registered based on anatomical feature point information of the examination object to obtain first registration parameters; and rigidly registering the real-time ultrasound image and the three-dimensional model to be registered based on the first registration parameters and contour information of the examination object to obtain second registration parameters.

[0008] In some embodiments of the first aspect, before rigidly registering the real-time ultrasound image and the three-dimensional model to be registered based on the anatomical feature point information of the inspection object to obtain the first registration parameter, it also includes: performing anatomical segmentation on the real-time ultrasound image to obtain multiple anatomical regions of the inspection object; and determining the positions of anatomical feature points in each anatomical region to obtain the anatomical feature point information.

[0009] In some embodiments of the first aspect, the real-time ultrasound image and the three-dimensional model to be registered are rigidly registered according to the first registration parameters and the contour information of the inspection object to obtain second registration parameters, including: using the first registration parameters as the initial pose of an iterative closest point algorithm, and performing point cloud registration on contour points according to the contour information of the inspection object to obtain the second registration parameters.

[0010] In some embodiments of the first aspect, after determining the deformed three-dimensional model, the method further includes: displaying the deformed three-dimensional model; and / or performing surgical navigation based on the deformed three-dimensional model.

[0011] A second aspect of an embodiment of the present application provides a registration device, including: an image acquisition unit, used to acquire a real-time ultrasound image obtained by scanning an inspection object; a three-dimensional model acquisition unit, used to acquire a three-dimensional model to be registered of the inspection object; and a registration unit, used to perform rigid registration and non-rigid registration on the real-time ultrasound image and the three-dimensional model to be registered, to determine a deformed three-dimensional model.

[0012] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method for ultrasound three-dimensional model registration as described in any one of the first aspects are implemented.

[0013] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned ultrasound three-dimensional model registration method are implemented.

[0014] A fifth aspect of an embodiment of the present application provides a computer program product, including a computer program. When the computer program is run, the method for ultrasound three-dimensional model registration described in the first aspect is executed.

[0015] In an embodiment of the present application, a real-time ultrasound image obtained by scanning the inspection object and a three-dimensional model of the inspection object to be registered are obtained, and rigid registration and non-rigid registration are performed on the real-time ultrasound image and the three-dimensional model to be registered, and a deformed three-dimensional model is determined, so that the deformed three-dimensional model can present the morphology of the inspection object in the real-time ultrasound image. In this way, the three-dimensional model is associated with the ultrasound image, and the three-dimensional model is made to conform to the organ or tissue morphology presented by the ultrasound image through registration, thereby providing more accurate anatomical structure information, helping to assist surgeons in screening or surgical navigation, and there is no need to fuse real-time two-dimensional images from different angles into a three-dimensional view, which can improve real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 It is a schematic diagram of the implementation process of the method for ultrasound three-dimensional model registration provided in an embodiment of the present application;

[0018] Figure 2 This is a schematic diagram of a specific implementation process of obtaining a to-be-registered three-dimensional model of an inspection object provided in an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of a specific implementation process of determining the phase information of the ultrasound image to be registered provided in an embodiment of the present application;

[0020] Figure 4 is a schematic diagram of a specific implementation process of rigid registration provided in an embodiment of the present application;

[0021] Figure 5 is a schematic diagram of the architecture of a registration system provided in an embodiment of the present application;

[0022] Figure 6 It is a structural schematic diagram of a device for ultrasound three-dimensional model registration provided in an embodiment of the present application;

[0023] Figure 7 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are protected by the present application.

[0025] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0026] In the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0028] Some related technologies use 3D ultrasound to provide overall structural information of the object under examination. 3D ultrasound essentially uses an ultrasound probe to capture 2D images at different angles and generate a volume image, providing a 3D view of an organ or tissue. However, 3D ultrasound equipment has the problems of high cost and slow scanning speed, making it difficult to meet real-time requirements.

[0029] In view of this, the present application proposes a method for ultrasound three-dimensional model registration, which can associate the three-dimensional model with the ultrasound image. Through registration, the three-dimensional model can conform to the organ or tissue morphology presented in the ultrasound image, providing more accurate anatomical structure information, which helps to assist surgeons in screening or surgical navigation. In addition, there is no need to fuse real-time two-dimensional images from different angles into a three-dimensional view, which can improve real-time performance.

[0030] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0031] Please refer to Figure 1 , Figure 1 A schematic diagram of the implementation process of a method for ultrasound three-dimensional model alignment provided in an embodiment of the present application is shown. The method can be applied to an electronic device, which may be a computer, a tablet computer, a smart phone, a medical device specifically used for image processing, or other smart devices, and the present application does not impose any restrictions on this.

[0032] Specifically, the above ultrasound three-dimensional model registration method may include the following steps S101 to S103.

[0033] Step S101, obtaining a real-time ultrasonic image obtained by scanning the inspection object.

[0034] The inspection object may refer to a tissue, an organ of a human or animal or other object that needs to be imaged.

[0035] In the implementation manner of the present application, the real-time ultrasound image is obtained by performing a real-time ultrasound scan on the inspection object, and can reflect the real-time morphology of the inspection object. In some specific scenarios, the real-time ultrasound image can be obtained by performing an ultrasound scan on the patient during surgery.

[0036] Step S102: obtaining a three-dimensional model of the inspection object to be registered.

[0037] The three-dimensional model to be registered refers to a three-dimensional model that needs to be registered to the real-time ultrasound image. The three-dimensional model can be a preoperative three-dimensional model or a standardized three-dimensional model of the examination object, and this application does not limit this. For example, when the examination object is the heart, the three-dimensional model to be registered can be a three-dimensional model of the heart obtained by scanning the patient before the operation, or it can be a standard three-dimensional model of the heart used uniformly by adults. It should be noted that the three-dimensional model to be registered can be a three-dimensional model obtained from non-ultrasound images.

[0038] The present application does not limit the method for obtaining the above-mentioned three-dimensional model to be registered. For example, it can be obtained through magnetic resonance imaging (MRI), computed tomography (CT), etc.

[0039] Step S103 , rigid registration and non-rigid registration are performed on the real-time ultrasound image and the three-dimensional model to be registered to determine the deformed three-dimensional model.

[0040] In the implementation manner of the present application, there is often a certain difference between the morphology of the inspection object presented by the three-dimensional model to be registered and the real-time morphology of the inspection object. In order to reduce this difference, the real-time ultrasound image and the three-dimensional model to be registered can be registered to determine the deformed three-dimensional model. The deformed three-dimensional model is used to present the morphology of the inspection object in the real-time ultrasound image. In other words, the three-dimensional model to be registered can be registered with the real-time ultrasound image, so that the morphology of the inspection object presented by the deformed three-dimensional model is consistent with the morphology presented by the inspection object in the real-time ultrasound image, so that the deformed three-dimensional model can reflect the real-time state of the inspection object.

[0041] In the implementation of the present application, the registration of the real-time ultrasound image and the three-dimensional model to be registered may include rigid registration and non-rigid registration. Rigid registration is intended to regard the inspection object as an object that will not be deformed, obtain rigid registration parameters between the real-time ultrasound image and the three-dimensional model to be registered, so that the matching point clouds between the real-time ultrasound image and the three-dimensional model to be registered overlap as much as possible in position and posture. Rigid registration parameters may include rotation matrices and translation matrices. In practical applications, since the inspection object of the real-time ultrasound image is usually an object such as an organ or tissue that deforms in real time, the present application adds non-rigid registration on the basis of rigid registration, so that the point cloud can be deformed during the registration process. By performing rigid and non-rigid registration on the real-time ultrasound image and the three-dimensional model to be registered, a set of registration parameters can be obtained, which can register the three-dimensional model to be registered to the real-time ultrasound image, and obtain the deformed three-dimensional model, so that the point cloud on the deformed three-dimensional model can be consistent with the point cloud matching on the real-time ultrasound image in position and posture, so that the deformed three-dimensional model can reflect the real-time state of the inspection object.

[0042] In an embodiment of the present application, a real-time ultrasound image obtained by scanning the inspection object and a three-dimensional model to be registered of the inspection object are acquired, and rigid registration and non-rigid registration are performed on the real-time ultrasound image and the three-dimensional model to be registered, and a deformed three-dimensional model is determined so that the deformed three-dimensional model can present the morphology of the inspection object in the real-time ultrasound image. In this way, the three-dimensional model is associated with the ultrasound image, and the three-dimensional model is made to conform to the organ or tissue morphology presented by the ultrasound image through registration, thereby providing more accurate anatomical structure information, which helps to assist surgeons in screening or surgical navigation.

[0043] Moreover, this method does not need to fuse real-time two-dimensional images from different angles into a three-dimensional view. A real-time ultrasound image can be used to align the preoperatively constructed three-dimensional model to be aligned, which can reduce the time required to scan ultrasound images and improve real-time performance.

[0044] In some embodiments of the present application, Figure 2As shown, obtaining the to-be-registered three-dimensional model of the inspection object may include: step S201 to step S202.

[0045] Step S201, determining phase information of a real-time ultrasound image.

[0046] The phase information is used to describe the phase to which the inspection object belongs in the real-time ultrasound image. The phase is to divide the inspection process into different time periods according to different physiological or pathological processes. For example, when the inspection object is the heart, according to the cardiac cycle, the phase may include but is not limited to the systolic phase and the diastolic phase; when the inspection object is the liver, the phase may include but is not limited to the portal venous phase and the arterial phase.

[0047] Specifically, in some embodiments of the present application, Figure 3 As shown, determining the phase information of the ultrasound image to be registered may include: step S301 to step S303.

[0048] Step S301, acquiring an ultrasound image sequence.

[0049] The ultrasound image sequence includes real-time ultrasound images and historical ultrasound images. The historical ultrasound images may be the first few frames of ultrasound images of the real-time ultrasound image, or ultrasound images obtained by a previous scan. Specifically, the ultrasound image sequence may be obtained by continuously scanning the inspection object, and the last frame of ultrasound image may be used as the real-time ultrasound image, and the other frames of ultrasound image may be used as the historical ultrasound images.

[0050] Step S302: determining the corresponding relationship between the morphological indexes presented in the historical ultrasound images and time according to the historical ultrasound images.

[0051] The morphological index may reflect the morphology of the inspection object in the historical ultrasound image, and may be the area of ​​the inspection object, the distance between the center of gravity and the outline, or other specific morphological indexes.

[0052] For each historical ultrasound image, its morphological index can be obtained through anatomical segmentation, and the corresponding relationship between the morphological index and time can be obtained by combining the sampling time or frame number of each historical ultrasound image. Anatomical segmentation can be implemented using a deep learning network or other existing image segmentation algorithms, which is not limited in this application.

[0053] Step S303: determining the phase information of the real-time ultrasound image according to the corresponding relationship.

[0054] Specifically, the above correspondence describes the correlation between morphological indicators and time. For real-time ultrasound images, their real-time morphological indicators can be obtained through anatomical segmentation, and the time to which the same morphological indicators belong can be determined using the correspondence, thereby determining the phase information of the real-time ultrasound image.

[0055] In other implementations, the user may also input the phase information of the real-time ultrasound image, or obtain the phase information by querying the tag of the ultrasound image, which is not limited in this application.

[0056] Step S202: determining the three-dimensional model to be registered in the three-dimensional model sequence according to the phase information.

[0057] In an embodiment of the present application, the 3D model sequence may include multiple 3D models, each of which may correspond to a phase. According to the phase information of the real-time ultrasound image, a 3D model with a consistent or closest phase may be selected from the 3D model sequence as the 3D model to be registered.

[0058] In this way, the three-dimensional model to be registered and the real-time ultrasound image are consistent in phase, and the phase is correlated with the morphology of the examination object. Selecting a three-dimensional model with a consistent or closest phase can reduce the error caused by phase differences in the subsequent registration and determination of the deformed three-dimensional model.

[0059] In some embodiments of the present application, for real-time ultrasound images, contour information of the object under examination can be obtained through anatomical segmentation. The contour information may include the position of the point cloud on the contour of the object under examination, which can be used for subsequent registration. Similarly, anatomical segmentation of real-time ultrasound images can be implemented using a deep learning network or other existing image segmentation algorithms, which is not limited by the present application.

[0060] Regarding the registration process of the real-time ultrasound image and the three-dimensional model to be registered, in some embodiments of the present application, the real-time ultrasound image and the three-dimensional model to be registered can be first rigidly registered, and then non-rigid registration can be further performed using the rigid registration parameters to determine the deformed three-dimensional model.

[0061] Specifically, in some embodiments of the present application, Figure 4 As shown, rigid registration of the real-time ultrasound image and the three-dimensional model to be registered may include: step S401 to step S402.

[0062] Step S401 : rigidly registering the real-time ultrasound image and the three-dimensional model to be registered according to the anatomical feature point information of the object to be examined, and obtaining a first registration parameter.

[0063] Since the spatial relationship between the real-time ultrasound image and the three-dimensional model to be registered is completely unknown, it is necessary to use the anatomical feature point information for initial registration. The first registration parameter is the rigid registration parameter obtained by the initial registration.

[0064] The anatomical feature point information is information about the anatomical feature points on the inspection object, and may include but is not limited to the positions of the anatomical feature points.

[0065] Specifically, before rigidly registering the real-time ultrasound image and the three-dimensional model to be registered based on the anatomical feature point information of the examination object to obtain the first registration parameter, it can also include: performing anatomical segmentation on the real-time ultrasound image to obtain multiple anatomical regions of the examination object, determining the positions of anatomical feature points in each anatomical region, and obtaining anatomical feature point information.

[0066] As an example, when the inspection object is the heart, each anatomical region can be a chamber, namely, the left ventricle (LV), the left atrium (LA), the right ventricle (RV) and the right atrium (RA). The center of gravity of each chamber can be selected as an anatomical feature point. According to the coordinates of the center of gravity of each chamber, the anatomical feature point information is obtained.

[0067] As another example, when the inspection object is the liver, each anatomical region can be a lobe, namely, the caudate lobe, the left lateral lobe, the left internal lobe, the right anterior lobe, and the right posterior lobe. The center point of each lobe can be selected as an anatomical feature point. According to the coordinates of the center point of each cavity, the anatomical feature point information is obtained.

[0068] For the three-dimensional model to be registered, the anatomical feature point information can be obtained based on the point cloud information.

[0069] According to the anatomical feature point information of the object under examination, the positions of the anatomical feature points that match the real-time ultrasound image and the three-dimensional model to be registered can be obtained. Using the positions of multiple matching anatomical feature points, an analytical solution can be obtained through eigenvalue decomposition. The formula for the analytical solution can be expressed as:

[0070]

[0071] in, and are the anatomical feature points of the real-time ultrasound image and the 3D model to be registered, R k is the rotation matrix, t k It should be noted that three or more groups of matching anatomical feature points can be used in the eigenvalue decomposition process.

[0072] The above formula (1) can be understood as: the sum of the differences corresponding to multiple matching anatomical feature points, where the difference corresponding to each matching anatomical feature point is obtained by converting the position of the anatomical feature point of the real-time ultrasound image into the position of the three-dimensional model using the rotation matrix and the translation matrix, and subtracting it from the actual position of the anatomical feature point of the three-dimensional model to be registered. The rotation matrix and translation matrix that minimizes the sum of the differences corresponding to multiple matching anatomical feature points is the first registration parameter.

[0073] Step S402: rigidly register the real-time ultrasound image and the three-dimensional model to be registered according to the first registration parameters and the contour information of the inspection object to obtain second registration parameters.

[0074] The contour information of the inspection object includes the position of the point cloud on the contour of the inspection object. Since the anatomical feature points are limited, the accuracy of the first registration parameter is poor. In this application, the first registration parameter obtained by the initial registration can be used as input to perform secondary rigid registration in combination with the contour information of the inspection object. The second registration parameter is the rigid registration parameter obtained by the secondary rigid registration.

[0075] Specifically, the first registration parameter may be used as the initial pose of an iterative closest point (ICP) algorithm, and point cloud registration may be performed on the contour points according to contour information of the inspection object to obtain the second registration parameter.

[0076] The process of iterative closest point algorithm is as follows:

[0077] S1, taking the first registration parameter as the current pose;

[0078] S2. For each contour point of the real-time ultrasound image, find the point closest to it in the 3D model to be registered as a pair of matching points. The distance here can be calculated using the Euclidean distance.

[0079] S3. Using all matching point pairs, estimate a registration parameter (rotation matrix R and translation vector t) by minimizing the distance between the two sets of point clouds (such as the least squares method). Specifically, it can be expressed as:

[0080]

[0081] Among them, q j is the contour point of the real-time ultrasound image. j is the surface data of the 3D model to be registered, indicating the surface data of the 3D model to be registered with q jThe point closest to the nearest point. Similar to formula (1), formula (2) is used to calculate the sum of the differences corresponding to multiple matching contour points. The difference corresponding to each matching anatomical feature point is obtained by converting the position of the contour point of the real-time ultrasound image into the position of the three-dimensional model using the rotation matrix and the translation matrix, and subtracting it from the actual position of the contour point of the three-dimensional model to be registered. The rotation matrix and translation matrix that minimizes the sum of the differences corresponding to multiple matching contour points is the obtained registration parameter.

[0082] S4. Use the obtained registration parameters as the current posture to execute steps S2 to S4 again until the preset number of iterations is reached, or the increment of the sum of the above-mentioned differences is less than the increment threshold. It can be considered that the rigid registration is completed, and the obtained registration parameters are used as the second registration parameters.

[0083] Through two rigid registrations, the real-time ultrasound image and the three-dimensional model to be registered can be fully aligned, thus improving the accuracy of the registration.

[0084] In some embodiments of the present application, non-rigid registration of a real-time ultrasound image and a three-dimensional model to be registered may be achieved by dynamic modeling or simple modeling.

[0085] Compared with dynamic modeling, simple modeling requires fewer parameters to be solved, which helps to improve real-time performance. Specifically, simple modeling may include: solving affine transformation parameters of the real-time ultrasound image and the three-dimensional model to be registered based on the contour information of the inspection object.

[0086] The formula for solving the affine transformation parameters is expressed as:

[0087]

[0088] Among them, the affine transformation parameters include the affine matrix A k ∈R 3×3 and the translation vector t k ′∈R 3 ;q j ∈R 3 , is the contour point of the real-time ultrasound image. j is the surface data of the 3D model to be registered, indicating the surface data of the 3D model to be registered with q j The closest point. Affine transformation parameters A k Allows adding soft constraints to the rotation matrix.

[0089] The above formula (3) can be understood as: the sum of the differences corresponding to multiple matching contour points, where the difference corresponding to each matching contour point is obtained by converting the position of the contour point of the real-time ultrasound image into the position of the three-dimensional model using the second registration parameter, and subtracting it from the actual position of the contour point of the three-dimensional model to be registered. The affine transformation parameter that minimizes the sum of the differences corresponding to multiple matching contour points is the affine transformation parameter (non-rigid registration parameter). The three-dimensional model to be registered is transformed using the affine transformation parameter to obtain a deformed three-dimensional model, so that the shape of the inspection object presented by the deformed three-dimensional model is consistent with the shape of the inspection object presented in the real-time ultrasound image.

[0090] In some embodiments of the present application, after determining the deformed three-dimensional model, the method may further include: displaying the deformed three-dimensional model to facilitate screening by doctors.

[0091] In some other embodiments of the present application, after determining the deformed three-dimensional model, the method may further include: performing surgical navigation according to the deformed three-dimensional model. The surgical navigation may be used to display surgical instruments in the deformed three-dimensional model, so that the doctor can know the relative position between the current surgical instruments and the inspection object by viewing the deformed three-dimensional model.

[0092] For easier understanding, please refer to Figure 5 , Figure 5 The schematic diagram of the framework of the registration system in the electronic device provided by the present application is shown. The registration system may include: a phase recognition module, an anatomical segmentation module, an anatomical registration module, a global registration module and a local registration module.

[0093] For the ultrasound image sequence obtained by intraoperative ultrasound scanning, the real-time ultrasound image can be segmented by the anatomical segmentation module to obtain contour information. In the phase recognition module, the historical ultrasound image can be segmented by the anatomical segmentation module to obtain contour information to analyze the correspondence between morphological indicators and time, and the phase information of the real-time ultrasound image can be identified using the correspondence. The phase information can be used to select the three-dimensional model to be registered in the three-dimensional model sequence. For the three-dimensional model to be registered, the anatomical registration module can use the anatomical feature point information to perform rigid registration to obtain the first registration parameter. The global registration module can use the first registration parameter and contour information to perform rigid registration to obtain the second registration information. The local registration module can use the second registration information and contour information to perform non-rigid registration to determine the deformed three-dimensional model. The deformed three-dimensional model can be output to the front-end display for display or surgical navigation.

[0094] It should be noted that, for the sake of simplicity of description, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders.

[0095] like Figure 6 Shown is a schematic structural diagram of an ultrasound three-dimensional model registration device 600 provided in an embodiment of the present application, wherein the registration device 600 is configured on an electronic device.

[0096] Specifically, the ultrasound three-dimensional model registration device 600 may include:

[0097] An image acquisition unit 601 is used to acquire a real-time ultrasonic image obtained by scanning the inspection object;

[0098] A three-dimensional model acquisition unit 602 is used to acquire a three-dimensional model of the inspection object to be registered;

[0099] The registration unit 603 is used to perform rigid registration and non-rigid registration on the real-time ultrasound image and the three-dimensional model to be registered, and determine the deformed three-dimensional model.

[0100] In some embodiments of the present application, the three-dimensional model acquisition unit 602 is specifically used to: determine the phase information of the real-time ultrasound image; and determine the three-dimensional model to be registered in the three-dimensional model sequence according to the phase information.

[0101] In some embodiments of the present application, the three-dimensional model acquisition unit 602 is specifically used to: acquire an ultrasound image sequence, the ultrasound image sequence including the real-time ultrasound image and the historical ultrasound image; determine the correspondence between the morphological indicators presented in the historical ultrasound image and time based on the historical ultrasound image; and determine the phase information of the real-time ultrasound image based on the correspondence.

[0102] In some embodiments of the present application, the registration unit 603 is specifically used to: rigidly register the real-time ultrasound image and the three-dimensional model to be registered based on the anatomical feature point information of the examination object to obtain a first registration parameter; and rigidly register the real-time ultrasound image and the three-dimensional model to be registered based on the first registration parameter and the contour information of the examination object to obtain the second registration parameter.

[0103] In some embodiments of the present application, the registration unit 603 is specifically used to: perform anatomical segmentation on the real-time ultrasound image to obtain multiple anatomical regions of the inspection object; determine the position of anatomical feature points in each anatomical region to obtain the anatomical feature point information.

[0104] In some embodiments of the present application, the registration unit 603 is specifically used to: use the first registration parameter as the initial pose of the iterative closest point algorithm, perform point cloud registration on the contour points according to the contour information of the inspection object, and obtain the second registration parameter.

[0105] In some embodiments of the present application, the registration device 600 for ultrasound three-dimensional model registration may further include a display unit for displaying the deformed three-dimensional model.

[0106] In some embodiments of the present application, the registration device 600 for ultrasound three-dimensional model registration may further include a surgical navigation unit for performing surgical navigation according to the deformed three-dimensional model.

[0107] It should be noted that, for the convenience and simplicity of description, the specific working process of the registration device 600 for the above-mentioned ultrasound three-dimensional model registration can be referred to Figures 1 to 5 The corresponding process of the method will not be repeated here.

[0108] like Figure 7 FIG. 1 is a schematic diagram of an electronic device 7 provided in an embodiment of the present application. Specifically, the electronic device 7 may include: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70, such as a registration program. When the processor 70 executes the computer program 72, the steps in the above-mentioned method embodiments for registration of the three-dimensional ultrasound model are implemented, such as Figure 1 Alternatively, when the processor 70 executes the computer program 72, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 6 The functions of the image acquisition unit 601, the three-dimensional model acquisition unit 602 and the registration unit 603 are shown.

[0109] The computer program may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 7.

[0110] For example, the computer program can be divided into: an image acquisition unit, a three-dimensional model acquisition unit, and a registration unit. The specific functions of each unit are as follows: the image acquisition unit is used to acquire a real-time ultrasound image obtained by scanning the inspection object; the three-dimensional model acquisition unit is used to acquire the three-dimensional model to be registered of the inspection object; the registration unit is used to perform rigid registration and non-rigid registration on the real-time ultrasound image and the three-dimensional model to be registered to determine the deformed three-dimensional model.

[0111] The electronic device 7 may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Figure 7 It is only an example of the electronic device 7 and does not constitute a limitation of the electronic device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 7 may also include input and output devices, network access devices, buses, etc.

[0112] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0113] The memory 71 may be an internal storage unit of the electronic device 7, such as a hard disk or memory of the electronic device 7. The memory 71 may also be an external storage device of the electronic device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7. Further, the memory 71 may also include both an internal storage unit and an external storage device of the electronic device 7. The memory 71 is used to store the computer program and other programs and data required by the electronic device 7. The memory 71 may also be used to temporarily store data that has been output or is to be output.

[0114] It should be noted that, for the convenience and brevity of description, the structure of the electronic device 7 can also refer to the specific description of the structure in the method embodiment, which will not be repeated here.

[0115] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0116] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0117] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0118] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0119] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0120] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0121] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0122] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for ultrasound three-dimensional model registration, characterized in that: include: Acquire a real-time ultrasound image obtained by scanning the inspection object; Acquire a three-dimensional model of the inspection object to be registered; The real-time ultrasound image and the three-dimensional model to be registered are rigidly registered and non-rigidly registered to determine a deformed three-dimensional model.

2. The method for ultrasound three-dimensional model registration according to claim 1, characterized in that: The step of obtaining the to-be-registered three-dimensional model of the inspection object comprises: Determining phase information of the real-time ultrasound image; According to the phase information, the three-dimensional model to be registered is determined in the three-dimensional model sequence.

3. The method for ultrasound three-dimensional model registration as claimed in claim 2, characterized in that: The determining of the phase information of the real-time ultrasound image comprises: Acquire an ultrasound image sequence, wherein the ultrasound image sequence includes the real-time ultrasound image and a historical ultrasound image; Determining, according to the historical ultrasound image, a corresponding relationship between the morphological index presented in the historical ultrasound image and time; According to the corresponding relationship, the phase information of the real-time ultrasound image is determined.

4. The method for ultrasound three-dimensional model registration according to claim 1, characterized in that: Performing rigid registration on the real-time ultrasound image and the three-dimensional model to be registered includes: According to the anatomical feature point information of the inspection object, rigidly register the real-time ultrasound image and the three-dimensional model to be registered to obtain a first registration parameter; According to the first registration parameters and the contour information of the inspection object, the real-time ultrasound image and the three-dimensional model to be registered are rigidly registered to obtain second registration parameters.

5. The method for ultrasound three-dimensional model registration as claimed in claim 4, characterized in that: Before rigidly registering the real-time ultrasound image and the three-dimensional model to be registered according to the anatomical feature point information of the inspection object to obtain the first registration parameter, the method further includes: Performing anatomical segmentation on the real-time ultrasound image to obtain multiple anatomical regions of the inspection object; The positions of the anatomical feature points are respectively determined in each anatomical region to obtain the anatomical feature point information.

6. The method for ultrasound three-dimensional model registration as claimed in claim 4, characterized in that: The step of performing rigid registration on the real-time ultrasound image and the three-dimensional model to be registered according to the first registration parameter and the contour information of the inspection object to obtain a second registration parameter includes: The first registration parameter is used as the initial pose of the iterative closest point algorithm, and point cloud registration is performed on the contour points according to the contour information of the inspection object to obtain the second registration parameter.

7. The method for ultrasound three-dimensional model registration according to claim 1, characterized in that: After determining the deformed three-dimensional model, the method further includes: displaying the deformed three-dimensional model; And / or, performing surgical navigation based on the deformed three-dimensional model.

8. An ultrasound three-dimensional model registration device, characterized in that: include: An image acquisition unit, used to acquire a real-time ultrasonic image obtained by scanning the inspection object; A three-dimensional model acquisition unit, used to acquire a three-dimensional model of the inspection object to be registered; The registration unit is used to perform rigid registration and non-rigid registration on the real-time ultrasound image and the three-dimensional model to be registered, so as to determine the deformed three-dimensional model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for ultrasound three-dimensional model registration according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables the method for ultrasound three-dimensional model registration as claimed in any one of claims 1 to 7 to be executed.

Images