Ultrasonic three-dimensional model registration method, device, equipment and computer program product

Abstract

The invention is suitable for the technical field of medical treatment, and provides an ultrasonic three-dimensional model registration method, device and equipment and a computer program product. The ultrasonic three-dimensional model registration method comprises the following steps: acquiring a to-be-registered ultrasonic image and a to-be-registered three-dimensional model; determining registration parameters between the to-be-registered ultrasonic image and the to-be-registered three-dimensional model, wherein the registration parameters are used for limiting telescopic transformation in a first direction perpendicular to the to-be-registered ultrasonic image; and according to the registration parameters, registering the to-be-registered three-dimensional model to the to-be-registered ultrasonic image, and determining a deformed three-dimensional model. According to the embodiment of the invention, the three-dimensional model can be registered to the two-dimensional ultrasonic image, and the used registration parameters limit the telescopic transformation in the first direction perpendicular to the to-be-registered ultrasonic image, so that the registration result is more accurate.

Description

Technical Field

[0001] The present application belongs to the field of medical technology, and in particular, relates to a method, device, equipment and computer program product for ultrasound three-dimensional model registration. Background Art

[0002] Ultrasound (US) images are widely used for large-scale screening and routine examinations, interventional treatments, and intraoperative navigation due to their real-time nature. However, the image quality of two-dimensional ultrasound images is limited, and only slice data is available, which places a great burden on the operator. Currently, one of the hot topics of research is to accurately align three-dimensional models with two-dimensional ultrasound images and establish the conversion relationship from three-dimensional models to two-dimensional ultrasound images. However, existing free deformation methods (such as thin plate splines, free deformation, and finite elements) are difficult to directly apply to the registration of three-dimensional models to two-dimensional ultrasound images. Summary of the invention

[0003] The embodiments of the present application provide a method, apparatus, device and computer program product for ultrasound three-dimensional model registration, which can align the three-dimensional model to a two-dimensional ultrasound image, and the registration parameters used limit the scaling transformation in the first direction perpendicular to the ultrasound image to be registered, which can make the registration result more accurate.

[0004] A first aspect of an embodiment of the present application provides a method for ultrasound three-dimensional model registration, including: acquiring an ultrasound image to be registered and a three-dimensional model to be registered; determining registration parameters between the ultrasound image to be registered and the three-dimensional model to be registered, wherein the registration parameters are used to limit the scaling transformation in a first direction perpendicular to the ultrasound image to be registered; and according to the registration parameters, registering the three-dimensional model to be registered to the ultrasound image to be registered to determine the deformed three-dimensional model.

[0005] In some embodiments of the first aspect, determining the registration parameters between the ultrasound image to be registered and the three-dimensional model to be registered includes: determining matching point pairs, each of the matching point pairs including a contour point in the ultrasound image to be registered and a grid point on the three-dimensional model to be registered that matches the contour point; using a first transformation parameter that minimizes the sum of each first distance as the registration parameter, each of the first distances being the distance between the grid point and the contour point transformed by the first transformation parameter in one of the matching point pairs, the affine matrix in the first transformation parameter being the product of a first rotation matrix, an axially symmetric matrix, and a second rotation matrix, and the element on the symmetry axis of the axially symmetric matrix corresponding to the first direction is 1.

[0006] In some embodiments of the first aspect, the first transformation parameter that will minimize the sum of each first distance is used as the alignment parameter, including: correcting each first distance using a robustness loss function; and using the first transformation parameter that will minimize the sum of each corrected first distance as the alignment parameter.

[0007] In some embodiments of the first aspect, determining the registration parameters between the ultrasound image to be registered and the three-dimensional model to be registered includes: determining matching point pairs, each of the matching point pairs including a contour point in the ultrasound image to be registered and a grid point on the three-dimensional model to be registered that matches the contour point; using a second transformation parameter that minimizes the sum of each second distance and soft constraint term as the registration parameter, each of the second distances being the distance between the grid point and the contour point transformed by the second transformation parameter in one of the matching point pairs, the soft constraint term being the product of a hyperparameter and a soft constraint on an affine matrix, and the soft constraint being used to make the affine matrix approach a rotation matrix.

[0008] In some embodiments of the first aspect, the method for ultrasound three-dimensional model registration as described in claim 1 is characterized in that the soft constraint is used to make the affine matrix orthogonal and the determinant is 1.

[0009] In some embodiments of the first aspect, the second transformation parameter that minimizes the sum of each second distance and the soft constraint item is used as the alignment parameter, including: correcting each second distance using a robustness loss function; and using the second transformation parameter that minimizes the sum of each corrected second distance and the soft constraint item as the alignment parameter.

[0010] In some embodiments of the first aspect, after determining the deformed three-dimensional model, the method further includes: displaying the deformed three-dimensional model; and / or performing surgical navigation based on the deformed three-dimensional model.

[0011] A second aspect of an embodiment of the present application provides a device for ultrasonic three-dimensional model registration, including: an acquisition unit, used to acquire an ultrasonic image to be registered and a three-dimensional model to be registered; a parameter determination unit, used to determine the registration parameters between the ultrasonic image to be registered and the three-dimensional model to be registered, the registration parameters being used to limit the scaling transformation in a first direction perpendicular to the ultrasonic image to be registered; a registration unit, used to align the three-dimensional model to be registered to the ultrasonic image to be registered according to the registration parameters, and determine the deformed three-dimensional model.

[0012] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method for ultrasound three-dimensional model registration as described in any one of the first aspects are implemented.

[0013] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned ultrasound three-dimensional model registration method are implemented.

[0014] A fifth aspect of an embodiment of the present application provides a computer program product, including a computer program. When the computer program is run, the method for ultrasound three-dimensional model registration described in the first aspect is executed.

[0015] In an embodiment of the present application, an ultrasound image to be registered and a three-dimensional model to be registered are obtained, and according to the registration parameters between the ultrasound image to be registered and the three-dimensional model to be registered, the three-dimensional model to be registered is registered with the ultrasound image to be registered, and the deformed three-dimensional model is determined, so that the deformed three-dimensional model is used to present the shape of the scanned object in the medical image to be registered. While the three-dimensional model is registered with the two-dimensional ultrasound image, since the used registration parameters can limit the scaling transformation in the first direction perpendicular to the ultrasound image to be registered during the deformation of the three-dimensional model to the deformed three-dimensional model, it can reduce the inappropriate compression of the three-dimensional model to be registered in the first direction due to the lack of information of the ultrasound image in the first direction, so that the registration result is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 It is a schematic diagram of the implementation process of the method for ultrasound three-dimensional model registration provided in an embodiment of the present application;

[0018] Figure 2 This is a schematic diagram of a first specific implementation process of determining registration parameters provided in an embodiment of the present application;

[0019] Figure 3 It is a schematic diagram of a second specific implementation process of determining the registration parameters provided in an embodiment of the present application;

[0020] Figure 4is a comparison diagram of the registration solution provided in the embodiment of the present application;

[0021] Figure 5 It is a structural schematic diagram of a device for ultrasound three-dimensional model registration provided in an embodiment of the present application;

[0022] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are protected by the present application.

[0024] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0025] In the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0027] Existing free deformation methods (such as thin plate splines, free deformation and finite elements) are difficult to directly apply to the registration of 3D models to 2D ultrasound images. This is because 2D ultrasound images only provide a single slice perspective of the scanned object and lack information perpendicular to the slice, resulting in insufficient geometric constraints on 2D ultrasound images. This limitation makes 3D models prone to overfitting and deformation field calculations appear singular.

[0028] In view of this, the present application proposes a method for ultrasound three-dimensional model registration, which can align the three-dimensional model to the two-dimensional ultrasound image, and the registration parameters used limit the scaling transformation in the first direction perpendicular to the ultrasound image to be registered, which can make the registration result more accurate.

[0029] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0030] Please refer to Figure 1 , Figure 1 A schematic diagram of the implementation process of a method for ultrasound three-dimensional model alignment provided in an embodiment of the present application is shown. The method can be applied to an electronic device, which may be a computer, a smart phone, a medical device specifically used for image processing, or other smart devices, and the present application does not impose any restrictions on this.

[0031] Specifically, the above ultrasound three-dimensional model registration method may include the following steps S101 to S103.

[0032] Step S101, obtaining an ultrasound image to be registered and a three-dimensional model to be registered.

[0033] The ultrasound image to be registered and the three-dimensional model to be registered are a two-dimensional ultrasound image and a three-dimensional model that need to be registered with each other.

[0034] In some embodiments of the present application, the ultrasound image to be registered may be obtained by real-time scanning of a scanned object, and may reflect the real-time morphology of the scanned object. The scanned object may refer to a tissue, organ, or other object that needs to be imaged of a human or animal.

[0035] In some specific scenarios, real-time ultrasound images can be obtained by performing ultrasound scanning on the patient during surgery.

[0036] The three-dimensional model to be registered can be a preoperative three-dimensional model of the same patient as the ultrasound image to be registered, or a standardized three-dimensional model of the scanned object, and this application does not impose any restrictions on this. For example, when the scanned object is the heart, the three-dimensional model to be registered can be a three-dimensional model of the heart obtained by scanning the patient before surgery, or a standard three-dimensional model of the heart used uniformly by adults. It should be noted that the three-dimensional model to be registered can be a three-dimensional model obtained from non-ultrasound images.

[0037] The present application does not limit the method for obtaining the above-mentioned three-dimensional model to be registered. For example, it can be obtained through magnetic resonance imaging (MRI), computed tomography (CT), etc.

[0038] Step S102: determining the registration parameters between the ultrasound image to be registered and the three-dimensional model to be registered.

[0039] The registration parameters may be used to limit the scaling transformation in a first direction perpendicular to the ultrasound image to be registered.

[0040] In some embodiments of the present application, the registration parameters may include an affine matrix and a translation vector for performing affine transformation and translation on the three-dimensional model to be registered.

[0041] Specifically, the registration parameters can be expressed as:

[0042] Among them, the affine matrix A k ∈R 3×3 , t k ∈R 3 is the translation vector. k is a 4×4 homogeneous form, Represents a 3*3 zero matrix.

[0043] Step S103: register the three-dimensional model to be registered with the ultrasound image to be registered according to the registration parameters, and determine the deformed three-dimensional model.

[0044] In the implementation manner of the present application, there is often a certain difference between the morphology of the scanned object presented by the three-dimensional model to be registered and the morphology of the scanned object in the ultrasound image to be registered. In order to reduce or even avoid such a difference, the three-dimensional model to be registered can be registered with the ultrasound image to be registered according to the registration parameters to determine the deformed three-dimensional model. The deformed three-dimensional model is used to present the morphology of the scanned object in the medical image to be registered. In other words, the three-dimensional model to be registered can be registered and constrained by the medical image to be registered, so that the morphology of the scanned object presented by the deformed three-dimensional model is consistent with the morphology presented by the scanned object in the medical image to be registered.

[0045] Specifically, obtain the registration parameter B k Then, treat the registered 3D model q j Transform, that is, B k q j , and obtain the deformed three-dimensional model.

[0046] In an embodiment of the present application, an ultrasound image to be registered and a three-dimensional model to be registered are obtained, and according to the registration parameters between the ultrasound image to be registered and the three-dimensional model to be registered, the three-dimensional model to be registered is registered with the ultrasound image to be registered, and the deformed three-dimensional model is determined, so that the deformed three-dimensional model is used to present the shape of the scanned object in the medical image to be registered. While the three-dimensional model is registered with the two-dimensional ultrasound image, since the used registration parameters can limit the scaling transformation in the first direction perpendicular to the ultrasound image to be registered during the deformation of the three-dimensional model to the deformed three-dimensional model, it can reduce the inappropriate compression of the three-dimensional model to be registered in the first direction due to the lack of information of the ultrasound image in the first direction, so that the registration result is more accurate.

[0047] Specifically, the present application provides two different solutions for limiting the telescopic transformation in the first direction, namely, a non-rigid registration solution with soft constraints and a non-rigid registration solution with a fixed telescopic axis. The two solutions are described in detail below.

[0048] Non-rigid registration scheme with soft constraints:

[0049] In some embodiments of the present application, Figure 2 As shown, determining the registration parameters between the ultrasound image to be registered and the three-dimensional model to be registered may include: step S201 to step S202.

[0050] Step S201, determining matching point pairs.

[0051] Each matching point pair includes a contour point in the ultrasound image to be registered and a grid point on the three-dimensional model to be registered that matches the contour point.

[0052] Specifically, contour points can be obtained by performing image segmentation on the ultrasound image. Image segmentation can be implemented using a deep learning network or other existing image segmentation algorithms, which is not limited in this application. For each contour point on the ultrasound image to be registered, the nearest three-dimensional grid point can be selected on the three-dimensional model to be registered as a matching grid point to form a matching point pair. The distance here can be calculated using the Euclidean distance.

[0053] Step S202: taking the second transformation parameter that minimizes the sum of the second distances and the soft constraint term as the registration parameter.

[0054] Each second distance is the distance between a grid point and a contour point transformed by the second transformation parameter in a matching point pair. Specifically, the contour point is transformed by using the second transformation parameter to obtain the transformed coordinates of the contour point, and the transformed coordinates and the coordinates of the grid point that forms a matching point pair with the contour point are calculated to obtain the second distance of the matching point pair. The second distances of different matching point pairs are accumulated to obtain the sum of the second distances.

[0055] The soft constraint term is the product of the hyperparameter and the soft constraint on the affine matrix. The soft constraint is used to make the affine matrix close to the rotation matrix. If the affine matrix A k is a standard rotation matrix, lacking soft constraints, then the affine matrix A k The loss of non-rigid transformation characteristics will cause compression in the first direction, resulting in inappropriate compression of the three-dimensional model to be registered. Soft constraints are added to make the affine matrix close to the rotation matrix, and the scaling transformation in the first direction is constrained.

[0056] In some embodiments of the present application, soft constraints may be used to make the affine matrix orthogonal and have a determinant of 1.

[0057] Specifically, the soft constraint can be expressed as: Rot(A k )=(c1·c2) 2 +(c1·c3) 2 +(c2·c3) 2 +(c1·c1-1) 2 +(c2·c2-1) 2 +(c3·c3-1) 2 , c1, c2 and c3 are affine matrices A k 3 columns of elements.

[0058] The second transformation parameter that minimizes the sum of each second distance and soft constraint item indicates that the transformation parameter can be used to align each pair of matching points as much as possible, and the soft constraint is used to limit the scaling transformation in the first direction. Therefore, the second transformation parameter that minimizes the sum of each second distance and soft constraint item is used as the registration parameter, which can make the registration parameter more accurate. In addition, in the two directions perpendicular to each other on the plane where the ultrasound image is located, the affine matrix A can be used to calculate the registration parameters. k The ultrasound image is transformed adaptively.

[0059] In some embodiments of the present application, a second transformation parameter that minimizes the sum of each second distance and soft constraint item is used as a registration parameter, which may include: correcting each second distance using a robustness loss function; and using the second transformation parameter that minimizes the sum of each corrected second distance and soft constraint item as the registration parameter.

[0060] Among them, the robust loss function is used to reduce abnormal noise in the ultrasound image. The absolute value loss function, Huber loss function or other existing robust loss functions can be used, and this application does not impose any restrictions on this.

[0061] Specifically, the objective function can be constructed as follows:

[0062]

[0063] Among them, q j is the jth contour point segmented from the ultrasound image, p i is with q j The distance to the nearest grid point, Ω is the distance to the grid point p i The collection of B k is the transformation parameter. ρ is the robustness loss function, λ is the hyperparameter, Rot(A k ) is a soft constraint. In this case, the transformation parameter B k There are 12 degrees of freedom.

[0064] By optimizing the objective function (1), the second transformation parameter B that minimizes the sum of the second distances and the soft constraint terms can be k , as the registration parameter B k * .

[0065] Non-rigid registration scheme with a fixed telescopic axis:

[0066] In some embodiments of the present application, Figure 3 As shown, determining the registration parameters between the ultrasound image to be registered and the three-dimensional model to be registered may include: step S301 to step S302.

[0067] Step S301, determining matching point pairs.

[0068] Each matching point pair includes a contour point in the ultrasound image to be registered and a grid point on the three-dimensional model to be registered that matches the contour point. Step S301 can refer to the description of step S201, which will not be described in detail in this application.

[0069] Step S302: taking the first transformation parameter that minimizes the sum of the first distances as the registration parameter.

[0070] Each first distance is the distance between a grid point and a contour point transformed by the first transformation parameter in one of the matching point pairs. Specifically, the contour point is transformed by using the first transformation parameter to obtain the transformed coordinates of the contour point, and the transformed coordinates and the coordinates of the grid points that form the matching point pair with the contour point are calculated to obtain the first distance of the matching point pair. The first distances of different matching point pairs are accumulated to obtain the sum of the first distances.

[0071] The affine matrix in the first transformation parameter is the product of the first rotation matrix, the axisymmetric matrix, and the second rotation matrix, and the element on the axis of symmetry of the axisymmetric matrix corresponding to the first direction is 1.

[0072] Specifically, for the affine transformation group B k , construct B k Subgroup L k , so that L k There is no scaling transformation in the direction perpendicular to the ultrasound plane, and any scaling transformation can be performed in the remaining two directions of the ultrasound plane. k , perform singular value decomposition (SVD), we can get:

[0073]

[0074] Among them, U k and V k are all orthogonal rotation matrices with determinant 1. k is an axisymmetric matrix, and the three values ​​on the axis of symmetry represent the expansion and contraction amounts in three directions. The three directions here include the aforementioned first direction and two directions perpendicular to each other on the plane where the ultrasound image is located. Affine matrix A k , which can be understood as a rotation matrix First rotate any point p, and then after the rotation, scale it along three directions according to the elements on the axis of symmetry of the axisymmetric matrix. Finally, use the rotation matrix U k Rotation. The element corresponding to the first direction on the axis of symmetry of the axisymmetric matrix is ​​1, which can limit the expansion and contraction transformation in the first direction to remain unchanged.

[0075] Based on the above description, the aforementioned registration parameters can be expressed as:

[0076]

[0077] M k =R l Σ′R r R0;

[0078] Here R0 is the fixed second rotation matrix, which transforms the grid point p of the three-dimensional model i Convert to a custom coordinate system, where the Z axis is perpendicular to the ultrasound data plane, that is, parallel to or coincident with the first direction; and the X axis and Y axis are on the plane where the ultrasound image is located. r For point R0p i Rotation on the plane where the ultrasound image is located has only one degree of freedom. The third element of the symmetry axis of Σ′ is fixed to 1, and the other two elements of the symmetry axis constitute two degrees of freedom. The other two elements can make the two directions perpendicular to each other on the plane where the ultrasound image is located to be transformed according to the ultrasound image. l ∈SO(3) is the first rotation matrix with three degrees of freedom. SO(3) refers to a special orthogonal group containing the rotation matrix R, which can be called a three-dimensional rotation group. k For B k The subgroup has 9 degrees of freedom, L k The scaling transformation in the first direction is fixed to 1, and the scaling transformation in the two directions on the plane where the ultrasound image is located is not restricted. k There are 3 degrees of freedom.

[0079] The first transformation parameter that minimizes the sum of the first distances means that using the first transformation parameter can align each pair of matching points as much as possible while limiting the scaling transformation in the first direction. Therefore, using the first transformation parameter that minimizes the sum of the first distances as the alignment parameter can make the alignment parameter more accurate.

[0080] In some embodiments of the present application, the first transformation parameter that minimizes the sum of each first distance is used as the alignment parameter, which may include: correcting each first distance using a robustness loss function; and using the first transformation parameter that minimizes the sum of each corrected first distance as the alignment parameter.

[0081] Similarly, the robustness loss function is used to reduce abnormal noise in ultrasound images, and an absolute value loss function, a Huber loss function, or other existing robustness loss functions may be used, which is not limited in this application.

[0082] Specifically, the objective function can be constructed as follows:

[0083]

[0084] Among them, q j is the jth contour point segmented from the ultrasound image, p i is with q j The distance to the nearest grid point, Ω is the distance to the grid point p i The collection of Bk is the transformation parameter. ρ is the robustness loss function. In this case, the transformation parameter B k By optimizing the objective function (2), the first transformation parameter L that minimizes the sum of the first distances can be k , as the registration parameter

[0085] The above objective function (1) and objective function (2) are equivalent in the solution process and can be transformed into a nonlinear least squares problem. The iterative solution method is used to gradually approach the minimum value and finally determine the registration parameters. By obtaining the registration parameters, the three-dimensional model to be registered can be registered with the two-dimensional ultrasound image to be registered to obtain the deformed three-dimensional model.

[0086] Compared with the non-rigid registration scheme with soft constraints, the non-rigid registration scheme with a fixed telescopic axis does not require setting hyperparameters and can be easily applied to the registration scenario.

[0087] Please refer to Figure 4 , Figure 4 The red part in the middle represents the heart chamber obtained by segmenting the medical image. 4-1 to 4-5 represent five groups of comparison images. In each group of comparison images, the first row shows the original result without adopting the registration scheme provided in this application, and the green part shows the cross-sectional view of the three-dimensional model to be registered on the plane of the ultrasound image. The second row shows the effect after processing using a non-rigid registration scheme with soft constraints, and the green part shows the cross-sectional view of the deformed three-dimensional model on the plane of the ultrasound image. The third row shows the effect after processing using a non-rigid registration scheme with a fixed telescopic axis, and the green part shows the cross-sectional view of the deformed three-dimensional model on the plane of the ultrasound image. From Figure 4 It can be clearly seen in each set of comparison images that the cross-sectional view of the deformed 3D model on the plane of the ultrasound image (the green parts in the second and third rows) fits more closely with the heart chamber represented by the red part.

[0088] In some embodiments of the present application, after determining the deformed three-dimensional model, the method may further include: displaying the deformed three-dimensional model to facilitate screening by doctors.

[0089] In some other embodiments of the present application, after determining the deformed three-dimensional model, the method may further include: performing surgical navigation according to the deformed three-dimensional model. The surgical navigation is used to display surgical instruments in the deformed three-dimensional model, so that the doctor can know the relative position between the current surgical instrument and the scanned object by viewing the deformed three-dimensional model.

[0090] It should be noted that, for the sake of simplicity of description, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited to the described order of actions, because according to the present application, certain steps can be performed in other orders.

[0091] like Figure 5 The figure is a schematic diagram of the structure of an apparatus 500 for ultrasound three-dimensional model registration provided in an embodiment of the present application. The apparatus 500 for ultrasound three-dimensional model registration is configured on an electronic device.

[0092] Specifically, the ultrasound three-dimensional model registration device 500 may include:

[0093] An acquisition unit 501 is used to acquire an ultrasound image to be registered and a three-dimensional model to be registered;

[0094] A parameter determination unit 502, configured to determine a registration parameter between the ultrasound image to be registered and the three-dimensional model to be registered, wherein the registration parameter is used to limit a scaling transformation in a first direction perpendicular to the ultrasound image to be registered;

[0095] The registration unit 503 is used to register the three-dimensional model to be registered with the ultrasound image to be registered according to the registration parameters, and determine the deformed three-dimensional model.

[0096] In some embodiments of the present application, the parameter determination unit 502 is specifically used to: determine matching point pairs, each of the matching point pairs includes a contour point in the ultrasound image to be registered and a grid point on the three-dimensional model to be registered that matches the contour point; use a first transformation parameter that minimizes the sum of each first distance as the registration parameter, each of the first distances being the distance between the grid point and the contour point after transformation by the first transformation parameter in one of the matching point pairs, the affine matrix in the first transformation parameter being the product of a first rotation matrix, an axially symmetric matrix, and a second rotation matrix, and the element on the symmetry axis of the axially symmetric matrix corresponding to the first direction is 1.

[0097] In some embodiments of the present application, the parameter determination unit 502 is specifically used to: modify each of the first distances using a robustness loss function; and use the first transformation parameter that minimizes the sum of the modified first distances as the registration parameter.

[0098] In some embodiments of the present application, the parameter determination unit 502 is specifically used to: determine matching point pairs, each of the matching point pairs includes a contour point in the ultrasound image to be registered and a grid point on the three-dimensional model to be registered that matches the contour point; a second transformation parameter that minimizes the sum of each second distance and soft constraint term is used as the registration parameter, each of the second distances being the distance between the grid point and the contour point transformed by the second transformation parameter in one of the matching point pairs, the soft constraint term being the product of a hyperparameter and a soft constraint on an affine matrix, and the soft constraint being used to make the affine matrix approach a rotation matrix.

[0099] In some embodiments of the present application, soft constraints are used to make the affine matrix orthogonal and the determinant 1.

[0100] In some embodiments of the present application, the parameter determination unit 502 is specifically used to: use a robustness loss function to correct each of the second distances; and use the second transformation parameter that minimizes the sum of each corrected second distance and the soft constraint term as the alignment parameter.

[0101] In some embodiments of the present application, the apparatus 500 for ultrasound three-dimensional model registration may further include a display unit for displaying the deformed three-dimensional model.

[0102] In some embodiments of the present application, the ultrasound three-dimensional model registration device 500 may further include a surgical navigation unit for performing surgical navigation according to the deformed three-dimensional model.

[0103] It should be noted that, for the convenience and simplicity of description, the specific working process of the above-mentioned ultrasound three-dimensional model registration device 500 can be referred to Figures 1 to 4 The corresponding process of the method will not be repeated here.

[0104] like Figure 6 FIG. 6 is a schematic diagram of an electronic device 6 provided in an embodiment of the present application. Specifically, the electronic device 6 may include: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60, such as a registration program. When the processor 60 executes the computer program 62, the steps in the above-mentioned method embodiments for registration of the three-dimensional ultrasound model are implemented, such as Figure 1 Alternatively, when the processor 60 executes the computer program 62, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Figure 5 The functions of the acquisition unit 501, the parameter determination unit 502 and the registration unit 503 are shown.

[0105] The computer program may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 6.

[0106] For example, the computer program can be divided into: an acquisition unit, a parameter determination unit and a registration unit. The specific functions of each unit are as follows: an acquisition unit, used to acquire the ultrasound image to be registered and the three-dimensional model to be registered; a parameter determination unit, used to determine the registration parameters between the ultrasound image to be registered and the three-dimensional model to be registered, and the registration parameters are used to limit the scaling transformation in the first direction perpendicular to the ultrasound image to be registered; a registration unit, used to register the three-dimensional model to be registered to the ultrasound image to be registered according to the registration parameters, and determine the deformed three-dimensional model.

[0107] The electronic device 6 may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will appreciate that Figure 6 It is only an example of the electronic device 6 and does not constitute a limitation of the electronic device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 6 may also include input and output devices, network access devices, buses, etc.

[0108] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0109] The memory 61 may be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. The memory 61 may also be an external storage device of the electronic device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. Further, the memory 61 may also include both an internal storage unit of the electronic device 6 and an external storage device. The memory 61 is used to store the computer program and other programs and data required by the electronic device 6. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0110] It should be noted that, for the convenience and brevity of description, the structure of the electronic device 6 can also refer to the specific description of the structure in the method embodiment, which will not be repeated here.

[0111] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0112] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0113] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0114] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are merely schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0115] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0117] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0118] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for ultrasound three-dimensional model registration, characterized in that: include: Acquire an ultrasound image to be registered and a three-dimensional model to be registered; Determine a registration parameter between the ultrasound image to be registered and the three-dimensional model to be registered, wherein the registration parameter is used to limit the scaling transformation in a first direction perpendicular to the ultrasound image to be registered; According to the registration parameters, the three-dimensional model to be registered is registered with the ultrasound image to be registered to determine a deformed three-dimensional model.

2. The method for ultrasound three-dimensional model registration according to claim 1, characterized in that: The determining of the registration parameters between the ultrasound image to be registered and the three-dimensional model to be registered comprises: Determine matching point pairs, each matching point pair comprising a contour point in the ultrasound image to be registered and a grid point on the three-dimensional model to be registered that matches the contour point; The first transformation parameter that minimizes the sum of the first distances is used as the registration parameter, each of the first distances is the distance between the grid point and the contour point transformed by the first transformation parameter in a matching point pair, the affine matrix in the first transformation parameter is the product of the first rotation matrix, the axially symmetric matrix and the second rotation matrix, and the element on the symmetry axis of the axially symmetric matrix corresponding to the first direction is 1.

3. The method for ultrasound three-dimensional model registration as claimed in claim 2, characterized in that: The first transformation parameter that will minimize the sum of the first distances, as the registration parameter, includes: Using a robustness loss function to correct each of the first distances; The first transformation parameter that minimizes the sum of the corrected first distances is used as the registration parameter.

4. The method for ultrasound three-dimensional model registration according to claim 1, characterized in that: The determining of the registration parameters between the ultrasound image to be registered and the three-dimensional model to be registered comprises: Determine matching point pairs, each of the matching point pairs comprising a contour point in the ultrasound image to be registered and a grid point on the three-dimensional model to be registered that matches the contour point; A second transformation parameter that minimizes the sum of each second distance and soft constraint term is used as the alignment parameter, each of the second distances is the distance between the grid point and the contour point transformed by the second transformation parameter in a matching point pair, the soft constraint term is the product of a hyperparameter and a soft constraint on an affine matrix, and the soft constraint is used to make the affine matrix close to a rotation matrix.

5. The method for ultrasound three-dimensional model registration as claimed in claim 4, characterized in that: The soft constraint is used to make the affine matrix orthogonal and have a determinant of 1.

6. The method for ultrasound three-dimensional model registration as claimed in claim 4, characterized in that: The second transformation parameter that will minimize the sum of the second distances and the soft constraint terms, as the registration parameter, includes: Using a robustness loss function to correct each of the second distances; The second transformation parameter that minimizes the sum of each corrected second distance and the soft constraint term is used as the registration parameter.

7. The method for ultrasound three-dimensional model registration according to any one of claims 1 to 6, characterized in that: After determining the deformed three-dimensional model, the method further includes: displaying the deformed three-dimensional model; And / or, performing surgical navigation based on the deformed three-dimensional model.

8. A device for ultrasound three-dimensional model registration, characterized in that: include: An acquisition unit, used for acquiring an ultrasound image to be registered and a three-dimensional model to be registered; A parameter determination unit, configured to determine registration parameters between the ultrasound image to be registered and the three-dimensional model to be registered, wherein the registration parameters include parameters for limiting a scaling transformation in a first direction perpendicular to the ultrasound image to be registered; A registration unit is used to register the three-dimensional model to be registered with the ultrasound image to be registered according to the registration parameters, and determine a deformed three-dimensional model, wherein the deformed three-dimensional model is used to present the shape of the scanned object in the medical image to be registered.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for ultrasound three-dimensional model registration according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables the method for ultrasound three-dimensional model registration as claimed in any one of claims 1 to 7 to be executed.

Images