Image registration method and device and orthopedic surgery navigation system based on image registration

Abstract

The invention provides an image registration method and device and an orthopedic surgery navigation system based on image registration. The image registration method comprises the following steps: according to a three-dimensional image, a normal position two-dimensional image and a lateral position two-dimensional image of a to-be-registered object and an internal reference of an X-ray machine, generating an initial object pose of the to-be-registered object and an initial normal position pose and an initial lateral position pose of the X-ray machine; performing registration optimization on the initial object pose, and generating a target object pose of the to-be-registered object; performing registration optimization on the initial side position pose of the X-ray machine to generate a registered target side position pose; and integrating the target object pose of the to-be-registered object and the target side pose and the initial normal pose of the X-ray machine into a registration optimization result. According to the scheme provided by the invention, the pose of the object can be optimized, the side pose of the X-ray machine can be accurately registered under the condition that the position of the X-ray machine is inaccurate, and the accuracy and usability of the registration method are improved.

Description

Technical Field

[0001] The present application relates to the field of medical imaging, and in particular to an image registration method and device, and an orthopedic surgery navigation system based on image registration. Background Art

[0002] At present, robot-assisted spinal surgery has been promoted and applied clinically. Among them, the navigation system is the key part of robot-assisted spinal surgery. In the prior art, there are mainly the following methods for robot-assisted spinal navigation: (1) navigation imaging method based on two-dimensional C-arm X-ray machine; (2) intraoperative navigation imaging method based on three-dimensional C-arm image; (3) navigation imaging method based on preoperative three-dimensional CT reconstruction and planning. Among them, the navigation imaging method based on two-dimensional C-arm X-ray machine is the most widely used, but it is not intuitive enough and the surgical efficiency is low; the intraoperative navigation imaging method based on three-dimensional C-arm image has unsatisfactory imaging quality, is expensive, and has a low penetration rate of equipment; the navigation imaging method based on preoperative three-dimensional CT reconstruction and planning cannot accurately perform the surgery without an intraoperative spatial positioning algorithm.

[0003] The navigation imaging method of preoperative 3D CT reconstruction and planning based on 2D / 3D registration can avoid the above problems, but this method requires higher 2D / 3D registration accuracy to meet clinical needs. Summary of the invention

[0004] In view of the problems existing in the prior art, the present application proposes an image registration method and device, which obtains the initial object posture through a three-dimensional image and an anteroposterior and lateral two-dimensional image and the internal reference of an X-ray machine, optimizes the initial object posture and the lateral posture of the X-ray machine according to the anteroposterior and lateral two-dimensional images, and obtains the target object posture and the optimized lateral posture of the object to be registered. The scheme proposed in the present application can not only optimize the object posture, but also optimize the lateral posture of the X-ray machine when the X-ray machine is not in the correct position, thereby improving the accuracy and ease of use of the registration method.

[0005] According to the first aspect of the present application, an image registration method is proposed, which may include the following steps: generating an initial object pose of the object to be registered and an initial anteroposterior pose and an initial lateral pose of the X-ray machine based on a three-dimensional image, an anteroposterior two-dimensional image, a lateral two-dimensional image of the object to be registered and an internal parameter of the X-ray machine; performing registration optimization on the initial object pose based on the anteroposterior two-dimensional image and the lateral two-dimensional image to generate a target object pose of the object to be registered; performing registration optimization on the initial lateral pose of the X-ray machine based on the target object pose and the lateral two-dimensional image to generate a target lateral pose of the X-ray machine after registration; integrating the target object pose of the object to be registered, the target lateral pose of the X-ray machine and the initial anteroposterior pose of the X-ray machine into a registration optimization result.

[0006] According to some embodiments, the initial lateral posture of the X-ray machine is aligned and optimized according to the target object posture and the lateral two-dimensional image to generate the target lateral posture of the X-ray machine after alignment, which may include the following steps: generating a first virtual object posture according to the target object posture and the lateral two-dimensional image; generating the target lateral posture of the X-ray machine after alignment according to the relative relationship between the first virtual object posture and the lateral two-dimensional image.

[0007] According to some embodiments, generating a first virtual object posture according to a target object posture and a lateral two-dimensional image may include the following steps: S201, taking the target object posture as the current object posture; S202, generating a first virtual lateral two-dimensional image corresponding to the current object posture; S203, calculating a first similarity between the lateral two-dimensional image and the first virtual lateral two-dimensional image; S204, taking the current object posture as the first virtual object posture when the first similarity satisfies a first object posture condition; S205, adjusting the current object posture when the first similarity does not satisfy the first object posture condition, and returning to step S202.

[0008] According to some embodiments, performing registration optimization on the initial object pose based on the anteroposterior two-dimensional image and the lateral two-dimensional image, and generating the target object pose of the object to be registered may include: generating a first temporary object pose based on the initial object pose, the anteroposterior two-dimensional image and the lateral two-dimensional image; generating the target object pose based on the first temporary object pose and the anteroposterior two-dimensional image.

[0009] According to some embodiments, generating a first temporary object pose based on an initial object pose, an anteroposterior two-dimensional image, and a lateral two-dimensional image may include: S301, taking the initial object pose as the current object pose; S302, generating a second virtual anteroposterior two-dimensional image and a second virtual lateral two-dimensional image corresponding to the current object pose; S303, calculating a second similarity between the anteroposterior two-dimensional image and the second virtual anteroposterior two-dimensional image, and a third similarity between the lateral two-dimensional image and the second virtual lateral two-dimensional image; S304, taking the current object pose as the first temporary object pose when the second similarity and the third similarity meet the second object pose condition; S305, adjusting the current object pose when the second similarity and the third similarity do not meet the second object pose condition, and returning to step S302.

[0010] According to some embodiments, generating a target object pose based on a first temporary object pose and an orthostatic two-dimensional image may include: S401, taking the first temporary object pose as the current object pose; S402, generating a third virtual orthostatic two-dimensional image corresponding to the current object pose; S403, calculating a fourth similarity between the orthostatic two-dimensional image and the third virtual orthostatic two-dimensional image; S404, taking the current object pose as the target object pose if the fourth similarity satisfies a condition of the third object pose; S405, adjusting the current object pose if the fourth similarity does not satisfy a condition of the third object pose, and returning to step S402.

[0011] According to some embodiments, adjusting the posture of the object may include: performing a rotation or translation operation on the posture of the object to be registered.

[0012] According to the second aspect of the present application, an image registration device is proposed, which may include an initial pose generation module, a target object pose generation module, a target lateral pose generation module and a registration result integration module, wherein: the initial pose generation module can be used to generate the initial object pose of the object to be registered and the initial anteroposterior pose and initial lateral pose of the X-ray machine according to the three-dimensional image, the frontal two-dimensional image, the lateral two-dimensional image of the object to be registered and the internal parameters of the X-ray machine; the target object pose generation module can be used to perform registration optimization on the initial object pose according to the frontal two-dimensional image and the lateral two-dimensional image, and generate the target object pose of the object to be registered; the target lateral pose generation module can be used to perform registration optimization on the initial lateral pose of the X-ray machine according to the target object pose and the lateral two-dimensional image, and generate the target lateral pose of the registered X-ray machine; the registration result integration module is used to integrate the target object pose of the object to be registered, the target lateral pose of the X-ray machine and the initial frontal pose of the X-ray machine into a registration optimization result.

[0013] According to the third aspect of the present application, an orthopedic surgical navigation system based on image registration is proposed, which system may include: a three-dimensional CT device, which can be used to capture a three-dimensional image of a part to be operated; an X-ray machine, which can be used to capture an anteroposterior two-dimensional image and a lateral two-dimensional image of the part to be operated, and provide the internal parameters of the X-ray machine to a processing module; the processing module can be used to determine a surgical operation plan based on the three-dimensional image; the processing module can also be used to generate a registration optimization result based on the three-dimensional image, the anteroposterior two-dimensional image, the lateral two-dimensional image and the internal parameters of the X-ray machine using the method described in the first aspect of the present application, and the registration optimization result is generated. The quantification results include the target object posture of the object to be registered, the target lateral posture of the X-ray machine and the initial anteroposterior posture of the X-ray machine; according to the target object posture, the coordinates of the surgical operation plan are converted from the CT image coordinate system to the patient coordinate system; according to the target lateral posture and the initial anteroposterior posture, the coordinates of the surgical operation plan are converted from the patient coordinate system to the three-dimensional coordinate system of the X-ray machine; according to the internal reference of the X-ray machine, the coordinates of the surgical operation plan are converted from the three-dimensional coordinate system of the X-ray machine to the two-dimensional anteroposterior X-ray coordinate system and the two-dimensional lateral X-ray coordinate system; the operating robot can be used to operate the part to be operated according to the surgical operation plan.

[0014] According to the fourth aspect of the present application, an electronic device is proposed, which may include: a processor; a memory storing a computer program, and when the computer program is executed by the processor, the processor executes the method described in the first aspect of the present application.

[0015] According to the fifth aspect of the present application, a non-transitory computer-readable storage medium is proposed, on which computer-readable instructions are stored. When the instructions are executed by a processor, the processor executes the method described in the first aspect of the present application.

[0016] The solution proposed in this application not only considers optimizing the posture of the object, but also considers optimizing the lateral posture of the X-ray machine. According to the solution proposed in this application, when the actual position of the X-ray machine is different from the calculated theoretical position, accurate image registration can still be achieved by optimizing the posture of the X-ray machine, thereby improving the availability and accuracy of image registration. At the same time, the orthopedic surgery navigation system proposed in this application uses the image registration method proposed in this application for registration, which can improve the registration accuracy, more accurately achieve intraoperative spatial positioning, and then accurately perform surgery to meet clinical needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without exceeding the scope of protection required by the present application.

[0018] Figure 1 A flowchart of the image registration method of the present application;

[0019] Figure 2 A flowchart of generating a first virtual object posture according to a target object posture and a lateral two-dimensional image in the image registration method of the present application;

[0020] Figure 3 A flowchart of generating a first temporary object posture according to an initial object posture, an anteroposterior two-dimensional image and a lateral two-dimensional image in the image registration method of the present application;

[0021] Figure 4 A flow chart of generating a target object posture according to a first temporary object posture and an orthogonal two-dimensional image in the image registration method of the present application;

[0022] Figure 5 A schematic diagram of an image registration device of the present application;

[0023] Figure 6 A schematic diagram of an orthopedic surgery navigation system based on image registration of the present application;

[0024] Figure 7 This is a structural diagram of an electronic device of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0026] Figure 1 Flow chart of the image registration method of this application. Figure 1 As shown, the method includes the following steps.

[0027] In some specific embodiments, the X-ray machine is a C-arm X-ray machine. The method proposed in the present application is described in detail below using the C-arm X-ray machine as an example.

[0028] Step S101, generating an initial object pose of the object to be registered and an initial anteroposterior pose and an initial lateral pose of the C-arm X-ray machine according to the three-dimensional image, the two-dimensional anteroposterior image, the two-dimensional lateral image and the internal parameters of the C-arm X-ray machine of the object to be registered.

[0029] In some specific embodiments, the three-dimensional image is a three-dimensional CT image taken before surgery. The three-dimensional image includes a naming field of the object to be registered and the corresponding three-dimensional image guidance information, wherein the coordinates of the points in the three-dimensional image adopt the CT image coordinate system. In some specific embodiments, the two-dimensional image is an X-ray image taken by a C-arm X-ray machine during surgery, and the two-dimensional image includes an anteroposterior two-dimensional image and a lateral two-dimensional image. The anteroposterior two-dimensional image and the lateral two-dimensional image respectively include the corresponding naming fields and two-dimensional image guidance information of the anteroposterior and lateral objects to be registered, wherein the coordinates of the points in the anteroposterior two-dimensional image and the lateral two-dimensional image adopt the X-ray two-dimensional coordinate system. In some specific embodiments, the object posture of the object to be registered is a conversion matrix of the coordinates of the object to be registered from the CT image coordinate system to the patient coordinate system.

[0030] In some specific embodiments, the C-arm X-ray machine is used to take an anteroposterior two-dimensional image and a lateral two-dimensional image during surgery. The internal parameters of the C-arm X-ray machine include the focal length of the C-arm X-ray machine, the horizontal coordinates and the vertical coordinates of the viewpoint center of the C-arm X-ray machine in the taken X-ray image, wherein the horizontal coordinates and the vertical coordinates refer to the coordinates in the X-ray two-dimensional coordinate system. In some specific embodiments, the external parameters of the C-arm X-ray machine are calculated using the PNP (Perspective-n-Point) algorithm based on the three-dimensional image, the anteroposterior two-dimensional image, the lateral two-dimensional image and the internal parameters of the C-arm X-ray machine, and the external parameters of the C-arm X-ray machine include the initial anteroposterior posture and the initial lateral posture of the C-arm X-ray machine. In some specific embodiments, the conversion matrix from the CT image coordinate system to the three-dimensional coordinate system of the X-ray machine is determined based on the three-dimensional image guidance information and the two-dimensional image guidance information of the object to be registered. In some specific embodiments, the initial object pose of the object to be registered is obtained based on the transformation matrix from the CT image coordinate system to the three-dimensional coordinate system of the X-ray machine, the external parameters of the C-arm X-ray machine, and the internal parameters of the C-arm X-ray machine.

[0031] Step S102 , performing registration optimization on the initial object posture according to the frontal two-dimensional image and the lateral two-dimensional image to generate a target object posture of the object to be registered.

[0032] In some specific embodiments, a first temporary object pose is generated according to the initial object pose, the anteroposterior two-dimensional image and the lateral two-dimensional image, and a virtual anteroposterior two-dimensional image and a virtual lateral two-dimensional image corresponding to the initial object pose are generated using a DRR (Digitally Reconstructed Radiographs) method. Optionally, the DRR method is specifically implemented by using a line integral method. In some specific embodiments, a second similarity between the virtual anteroposterior two-dimensional image and the anteroposterior two-dimensional image generated by the DRR, and a third similarity between the virtual lateral two-dimensional image and the lateral two-dimensional image are calculated. In some specific embodiments, when the second similarity and the third similarity do not meet the second object pose condition, the object pose is adjusted, and a virtual anteroposterior two-dimensional image and a virtual lateral two-dimensional image corresponding to the adjusted object pose are regenerated, and the similarity is recalculated; when the second similarity and the third similarity meet the second object pose condition, the object pose of the current object to be registered is used as the first temporary object pose. Optionally, the second object posture condition is that a difference between two adjacent iterations of the second similarity is greater than a preset second similarity threshold, and a difference between two adjacent iterations of the third similarity is greater than a preset third similarity threshold.

[0033] In some specific embodiments, the target object posture is generated according to the first temporary object posture and the orthostatic two-dimensional image. A third virtual orthostatic two-dimensional image corresponding to the first temporary object posture is generated using the DRR method, and the fourth similarity between the third virtual orthostatic two-dimensional image and the orthostatic two-dimensional image is calculated. In some specific embodiments, when the fourth similarity does not meet the third object posture condition, the first temporary object posture is adjusted, and the third virtual orthostatic two-dimensional image corresponding to the adjusted first temporary object posture is regenerated, and the fourth similarity between the adjusted third virtual orthostatic two-dimensional image and the orthostatic two-dimensional image is calculated; when the fourth similarity meets the third object posture condition, the current first temporary object posture is used as the target object posture. Optionally, the third object posture condition is that the difference between two adjacent iterations of the fourth similarity is greater than a preset fourth similarity threshold.

[0034] Step S103 , performing registration optimization on the initial lateral posture of the C-arm X-ray machine according to the posture of the target object and the lateral two-dimensional image, and generating a target lateral posture of the C-arm X-ray machine after registration.

[0035] In some specific embodiments, the PNP (Perspective-n-Point) algorithm is used to calculate the initial lateral posture of the C-arm X-ray machine according to the lateral two-dimensional image of the C-arm X-ray machine. A first virtual object posture is generated according to the target object posture and the initial lateral posture of the C-arm X-ray machine, a first virtual lateral two-dimensional image corresponding to the first virtual object posture is generated using the DRR method, and a first similarity between the first virtual lateral two-dimensional image and the lateral two-dimensional image is calculated. In some specific embodiments, when the first similarity does not meet the first object posture condition, the target object posture is adjusted, the first virtual lateral two-dimensional image corresponding to the adjusted object posture is regenerated, and the first similarity between the adjusted first virtual lateral two-dimensional image and the lateral two-dimensional image is calculated; when the first similarity meets the first object posture condition, the current object posture is used as the first virtual object posture. Optionally, the first object posture condition is that the difference between two adjacent iterations of the first similarity is greater than a preset first similarity threshold.

[0036] In some specific embodiments, a registered target lateral posture is generated based on the relative relationship between the first virtual object posture and the lateral two-dimensional image, a first transition posture is generated based on the first virtual object posture, and the first virtual object posture is inverted to obtain the inverse as the first transition posture.

[0037] In some specific embodiments, the initial lateral posture of the C-arm X-ray machine is calculated using a PNP algorithm, and the first transition posture is multiplied by the initial lateral posture of the current C-arm X-ray machine to obtain the target lateral posture of the C-arm X-ray machine after registration. In some specific embodiments, there is a relative relationship between the first virtual object posture and the initial lateral posture. The lateral posture of the C-arm X-ray machine is calculated based on the relative relationship when the object posture remains as the target object posture, and the lateral posture at this time is determined as the target lateral posture of the C-arm X-ray machine.

[0038] Step S104 , integrating the target object posture of the object to be registered, the target lateral posture of the C-arm X-ray machine, and the initial frontal posture of the C-arm X-ray machine into a registration optimization result.

[0039] In some specific embodiments, in step S104, the target object posture of the object to be registered is the conversion matrix of the coordinates from the CT image coordinate system to the patient coordinate system after registration optimization. The target lateral posture and initial anteroposterior posture of the C-arm X-ray machine are the conversion matrices of the coordinates from the patient coordinate system to the three-dimensional coordinate system of the X-ray machine after registration optimization. In some specific embodiments, in step S104, the target object posture of the object to be registered, the target lateral posture of the C-arm X-ray machine, and the initial anteroposterior posture of the C-arm X-ray machine are integrated to generate a registration optimization result, which is provided to the subsequent operator for operation.

[0040] According to Figure 1 In the illustrated implementation scheme, the method proposed in the present application determines the initial object posture of the object to be registered in step S101; in step S102, the initial object posture of the object to be registered is optimized to obtain the target object posture, that is, the conversion matrix of the coordinates from the CT image coordinate system to the patient coordinate system, thereby improving the accuracy of converting the coordinates of the points on the object to be registered from the CT image coordinate system to the patient coordinate system; in step S103, the lateral posture of the C-arm X-ray machine is optimized to improve the accuracy of converting the coordinates of the points on the object to be registered from the patient coordinate system to the C-arm three-dimensional coordinate system; in step S104, the registration optimization results are integrated to facilitate subsequent operations using the registration optimization results.

[0041] The solution proposed in this application not only considers optimizing the posture of the object, but also considers optimizing the lateral posture of the C-arm X-ray machine. According to the solution proposed in this application, when the actual position of the C-arm X-ray machine is different from the calculated theoretical position, accurate image registration can still be achieved by optimizing the posture of the C-arm X-ray machine, thereby improving the availability and accuracy of image registration.

[0042] Figure 2 This is a flow chart of generating a first virtual object posture according to the target object posture and the lateral two-dimensional image in the image registration method of the present application. Figure 2 As shown, generating a first virtual object posture according to the target object posture and the lateral two-dimensional image includes the following steps.

[0043] S201, taking the target object posture as the current object posture.

[0044] S202: Generate a first virtual lateral two-dimensional image corresponding to the current object posture.

[0045] S203: Calculate a first similarity between the lateral two-dimensional image and the first virtual lateral two-dimensional image.

[0046] S204: When the first similarity satisfies the first object posture condition, use the current object posture as the first virtual object posture.

[0047] In some specific embodiments, the PNP (Perspective-n-Point) algorithm is used to calculate the initial lateral posture of the C-arm X-ray machine according to the lateral two-dimensional image of the C-arm X-ray machine. In some specific embodiments, the current object posture is the target object posture, and a first virtual object posture is generated according to the target object posture and the initial lateral posture of the C-arm X-ray machine, and a first virtual lateral two-dimensional image corresponding to the first virtual object posture is generated using the DRR method. The first similarity between the first virtual lateral two-dimensional image and the lateral two-dimensional image is calculated, and optionally, the similarity is calculated using the normalized cross-correlation coefficient method.

[0048] The Normalized Cross-Correlation Coefficient (NCC) method is a commonly used cost function that is robust to scaling and translation of the distribution. This method first requires calculating the mean and standard deviation of the image.

[0049] The mean and variance of image K are defined as follows:

[0050]

[0051]

[0052] Among them, μ is the mean, σ is the variance, Ω is the total number of pixels, K(p) is the size of the image pixel, and p represents the position of the pixel in the image, which can be represented by two-dimensional coordinates (u, v).

[0053] Therefore, the NCC is calculated as follows:

[0054]

[0055] Among them, I, J represent two images I and J, and J' is image J after transformation T -1 The transformed image is in the same time domain space as image I.

[0056] In order to make the registration process more stable and exclude the situation where the range of X-ray and DRR are different, in some specific embodiments, for the above two images I and J, the two gradients in the X direction and Y direction are solved by Sobel operator respectively, and then the NCC of the X gradient and Y gradient images are calculated, and finally the two NCCs are summed as the image similarity. See the following formula, S_GNCC is the NCC of the gradient image.

[0057]

[0058] Among them, S_GNCC is the sum of the NCC of the two gradient images, X gradient and Y gradient, C r is the coordinate of the center position in the X direction, C c is the coordinate of the center position in the Y direction, r is the radius, S_GNCC(I1,I2,C r ,C c , r) is C r and C c The similarity of a square area with centered at 2r and side length 2r, I1 and I2 represent two images in the same coordinate system. Optionally, I1 is image I, and I2 is image J'.

[0059] At the same time, in order to avoid causing the result to deviate too far from the initial value, which may lead to some unexpected extreme situations, in some embodiments, the present application also applies a regularization method to prevent the above situation. The regularization equation adopts the following formula:

[0060]

[0061] Wherein σ is the variance of translation or rotation, Folded-Norm refers to the normal distribution of translation, and the parameters in the embodiment of the present application are selected as 50 mm for translation and 60 degrees for rotation.

[0062]

[0063] This formula ensures that the value of the regularization function is large when the independent variable is larger than the variance.

[0064] The final similarity formula is as follows:

[0065]

[0066] Where SE(3) is the three-dimensional special Euclidean group of rigid body transformation motion (rotation and translation), and θ is the position change in Euclidean space. Where S is the similarity calculation formula, R is the function of the position change of θ in Euclidean space, λ is an adjustable superderivative, usually with a weight of 0.1, and P(I;θ) is the two-dimensional projection of the three-dimensional image I after θ transformation.

[0067] In some specific embodiments, the first similarity satisfies a first object pose condition, and the current object pose is used as the first virtual object pose. Optionally, the first object pose condition is that the difference between two adjacent iterations of the first similarity is greater than a preset first similarity threshold.

[0068] S205, when the first similarity does not satisfy the first object posture condition, adjust the current object posture and return to step S202.

[0069] In some specific embodiments, the first similarity does not meet the first object pose condition, and the object pose of the object to be registered is rotated or translated. In some specific embodiments, a two-stage method is used to adjust the pose of the object to be registered, that is, first adjust the low-resolution image, then use the high-resolution image, use the covariance matrix adaptive evolution strategy optimizer to adjust the low-resolution image, and use a faster gradient-independent optimization algorithm to adjust the high-resolution image. The specific steps of the two-stage method are as follows:

[0070] In the first stage, in some specific embodiments, a fixed resolution image of 256x256 is selected, and a covariance matrix adaptive evolutionary strategy optimizer (CMA-ES) is selected. The advantage of this optimizer is that it does not need to calculate the gradient equation of the loss function, and the optimization effect is very good. It is the algorithm with the least sensitivity to initial values ​​among modern optimizers. However, this optimizer needs to calculate the loss function multiple times in each iteration, so the speed is very slow. The CMA-ES algorithm requires the input of the initial value of the equation to be optimized, the optimization parameter sequence, and the variance of each parameter.

[0071] In some specific embodiments, the initial value of the equation to be optimized used by the CMA-ES algorithm is the initial pose matrix from the patient coordinate system to the CT coordinate system solved by the PNP algorithm mentioned above; the optimization parameter list is the translation and rotation 6DOF degrees of freedom (6 parameters) of the object to be registered; the translation variance and XYZ input variance are both 5 mm, the 3 rotation variances are 15 degrees, and CMA-ES samples 100 times each time.

[0072] In the second stage, in some specific embodiments, the image size after 1 / 2 downsampling is used as the registration optimization image. If the resolution is less than 256, the original X-ray size is used directly. At this time, the slower evolutionary algorithm cannot be used as the optimizer, but for the convenience of solving, a gradient-independent optimization algorithm is still required. For example, BOBYQA, NEWUOA, etc. are all optimizers that do not require a large amount of loss function calculations in each iteration and can be used.

[0073] In some specific embodiments, when the first similarity does not satisfy the first object posture condition, the current object posture is adjusted, and the process returns to step S202 to regenerate the corresponding virtual two-dimensional image and calculate the first similarity.

[0074] Figure 3 This is a flow chart of generating a first temporary object pose according to an initial object pose, an anteroposterior two-dimensional image and a lateral two-dimensional image in the image registration method of the present application. Figure 3 As shown, generating a first temporary object posture according to the initial object posture, the frontal two-dimensional image and the lateral two-dimensional image includes the following steps.

[0075] S301, taking the initial object posture as the current object posture.

[0076] S302, generating a second virtual frontal two-dimensional image and a second virtual lateral two-dimensional image corresponding to the current object posture.

[0077] S303: Calculate a second similarity between the anteroposterior two-dimensional image and the second virtual anteroposterior two-dimensional image, and a third similarity between the lateral two-dimensional image and the second virtual lateral two-dimensional image.

[0078] S304: When the second similarity and the third similarity satisfy a second object posture condition, use the current object posture as a first temporary object posture.

[0079] In some specific embodiments, the current object posture is the initial object posture, and the DRR method is used to generate a second virtual frontal two-dimensional image and a second virtual lateral two-dimensional image corresponding to the initial object posture. In some specific embodiments, the second similarity of the second virtual frontal two-dimensional image and the frontal two-dimensional image is calculated, and the third similarity of the second virtual lateral two-dimensional image and the lateral two-dimensional image is calculated. Optionally, the similarity is calculated using the normalized mutual correlation coefficient method. In some specific embodiments, the second similarity and the third similarity meet the second object posture condition, and the current object posture is used as the first temporary object posture. Optionally, the second object posture condition is that the difference between two adjacent iterations of the second similarity is greater than a preset second similarity threshold, and the difference between two adjacent iterations of the third similarity is greater than a preset third similarity threshold.

[0080] S305 , when the second similarity and the third similarity do not satisfy the second object posture condition, adjust the current object posture and return to step S302 .

[0081] In some specific embodiments, the second similarity and the third similarity do not meet the second object posture condition, and the posture of the object to be registered is rotated or translated. Optionally, the posture of the object to be registered is adjusted using a two-stage method, that is, the low-resolution image is adjusted first, and then the high-resolution image is used, the low-resolution image is adjusted using a covariance matrix adaptive evolutionary strategy optimizer, and the high-resolution image is adjusted using a faster gradient-independent optimization algorithm. In some specific embodiments, when the second similarity and the third similarity do not meet the second object posture condition, the current object posture is adjusted, and step S302 is returned to regenerate the corresponding virtual two-dimensional image and calculate the second similarity and the third similarity.

[0082] Figure 4 This is a flow chart of generating a target object pose according to a first temporary object pose and an orthogonal two-dimensional image in the image registration method of the present application. Figure 4 As shown, generating a target object posture according to a first temporary object posture and an orthogonal two-dimensional image includes the following steps.

[0083] S401: Taking the first temporary object pose as the current object pose.

[0084] S402, generating a third virtual two-dimensional image corresponding to the current object posture.

[0085] S403: Calculate a fourth similarity between the orthostatic two-dimensional image and the third virtual orthostatic two-dimensional image.

[0086] S404: When the fourth similarity satisfies the third object posture condition, the current object posture is used as the target object posture.

[0087] In some specific embodiments, the current object posture is the first temporary object posture, and a third virtual orthostatic two-dimensional image corresponding to the first temporary object posture is generated using the DRR method, and a fourth similarity between the third virtual orthostatic two-dimensional image and the orthostatic two-dimensional image is calculated. Optionally, the similarity is calculated using a normalized cross-correlation coefficient method. In some specific embodiments, the fourth similarity satisfies the third object posture condition, and the current object posture is used as the target object posture. Optionally, the third object posture condition is that the difference between two adjacent iterations of the fourth similarity is greater than a preset fourth similarity threshold.

[0088] S405: When the fourth similarity does not satisfy the third object posture condition, adjust the current object posture and return to step S402.

[0089] In some specific embodiments, when the fourth similarity does not meet the third object posture condition, the posture of the object to be registered is adjusted by rotation or translation. Optionally, the posture of the object to be registered is adjusted using a two-stage method, that is, the low-resolution image is adjusted first, and then the high-resolution image is used, the low-resolution image is adjusted using a covariance matrix adaptive evolutionary strategy optimizer, and the high-resolution image is adjusted using a faster gradient-independent optimization algorithm. In some specific embodiments, when the fourth similarity does not meet the third object posture condition, the current object posture is adjusted, and step S402 is returned to regenerate the corresponding virtual two-dimensional image and calculate the fourth similarity.

[0090] Figure 5 Schematic diagram of the image registration device of the present application. Figure 5 As shown, the above-mentioned device includes an initial posture generation module, a target object posture generation module, a target lateral posture generation module and a registration result integration module.

[0091] exist Figure 5 In the method, the initial pose generation module generates the initial object pose of the object to be registered and the initial anteroposterior pose and initial lateral pose of the C-arm X-ray machine according to the three-dimensional image, the two-dimensional image of the frontal position, the two-dimensional image of the lateral position and the internal parameters of the C-arm X-ray machine.

[0092] In some specific embodiments, the three-dimensional image is a three-dimensional CT image taken before surgery. The three-dimensional image includes a named field of the object to be registered and corresponding three-dimensional image guidance information, wherein the coordinates of the points in the three-dimensional image adopt the CT image coordinate system. In some specific embodiments, the two-dimensional image is an X-ray image taken by a C-arm X-ray machine during surgery. The anteroposterior two-dimensional image and the lateral two-dimensional image include corresponding named fields and two-dimensional image guidance information of the anteroposterior and lateral objects to be registered, wherein the coordinates of the points in the anteroposterior two-dimensional image and the lateral two-dimensional image adopt the X-ray two-dimensional coordinate system. In some specific embodiments, the object posture of the object to be registered is a conversion matrix of the coordinates of the object to be registered from the CT image coordinate system to the patient coordinate system.

[0093] In some specific embodiments, the C-arm X-ray machine is used to take an anteroposterior two-dimensional image and a lateral two-dimensional image during surgery. The internal parameters of the C-arm X-ray machine include the focal length of the C-arm, the horizontal coordinates and the vertical coordinates of the viewpoint center of the C-arm in the taken X-ray image, wherein the horizontal coordinates and the vertical coordinates refer to the coordinates in the X-ray two-dimensional coordinate system. In some specific embodiments, the external parameters of the C-arm X-ray machine are calculated using the PNP (Perspective-n-Point) algorithm according to the three-dimensional image, the anteroposterior two-dimensional image, the lateral two-dimensional image and the internal parameters of the C-arm X-ray machine, and the external parameters of the C-arm X-ray machine include the initial anteroposterior posture and the initial lateral posture of the C-arm X-ray machine. In some specific embodiments, the conversion matrix from the CT image coordinate system to the three-dimensional coordinate system of the X-ray machine is determined according to the three-dimensional image guidance information and the two-dimensional image guidance information of the object to be registered. In some specific embodiments, the initial object posture of the object to be registered is obtained according to the conversion matrix from the CT image coordinate system to the three-dimensional coordinate system of the X-ray machine, the external parameters of the C-arm X-ray machine and the internal parameters of the C-arm X-ray machine.

[0094] exist Figure 5 In the present invention, the target object pose generation module performs registration optimization on the initial object pose according to the frontal two-dimensional image and the lateral two-dimensional image to generate the target object pose of the object to be registered.

[0095] In some specific embodiments, a first temporary object pose is generated according to the initial object pose, the anteroposterior two-dimensional image and the lateral two-dimensional image, and a virtual anteroposterior two-dimensional image and a virtual lateral two-dimensional image corresponding to the initial object pose are generated by using a DRR (Digitally Reconstructed Radiographs) method, and the DRR method is specifically implemented by using a line integral method. In some specific embodiments, a second similarity between the virtual anteroposterior two-dimensional image and the anteroposterior two-dimensional image generated by the DRR, and a third similarity between the virtual lateral two-dimensional image and the lateral two-dimensional image are calculated. In some specific embodiments, when the second similarity and the third similarity do not meet the second object pose condition, the object pose is adjusted, and a virtual anteroposterior two-dimensional image and a virtual lateral two-dimensional image corresponding to the adjusted object pose are regenerated, and the similarity is recalculated; in some specific embodiments, when the second similarity and the third similarity meet the second object pose condition, the object pose of the current object to be registered is used as the first temporary object pose. Optionally, the second object posture condition is that a difference between two adjacent iterations of the second similarity is greater than a preset second similarity threshold, and a difference between two adjacent iterations of the third similarity is greater than a preset third similarity threshold.

[0096] In some specific embodiments, the target object posture is generated according to the first temporary object posture and the orthostatic two-dimensional image. A third virtual orthostatic two-dimensional image corresponding to the first temporary object posture is generated using the DRR method, and the fourth similarity between the third virtual orthostatic two-dimensional image and the orthostatic two-dimensional image is calculated. In some specific embodiments, when the fourth similarity does not meet the third object posture condition, the first temporary object posture is adjusted, and the third virtual orthostatic two-dimensional image corresponding to the adjusted first temporary object posture is regenerated, and the fourth similarity between the adjusted third virtual orthostatic two-dimensional image and the orthostatic two-dimensional image is calculated; when the fourth similarity meets the third object posture condition, the current first temporary object posture is used as the target object posture. Optionally, the third object posture condition is that the difference between two adjacent iterations of the fourth similarity is greater than a preset fourth similarity threshold.

[0097] exist Figure 5 In the present invention, the target lateral posture generation module performs registration optimization on the initial lateral posture of the C-arm X-ray machine according to the target object posture and the lateral two-dimensional image, and generates the target lateral posture of the C-arm X-ray machine after registration.

[0098] In some specific embodiments, the PNP (Perspective-n-Point) algorithm is used to calculate the initial lateral posture of the C-arm X-ray machine according to the lateral two-dimensional image of the C-arm. A first virtual object posture is generated according to the target object posture and the initial lateral posture of the C-arm X-ray machine, a first virtual lateral two-dimensional image corresponding to the target object posture is generated using the DRR method, and a first similarity between the first virtual lateral two-dimensional image and the lateral two-dimensional image is calculated. In some specific embodiments, when the first similarity does not meet the first object posture condition, the target object posture is adjusted, the first virtual lateral two-dimensional image corresponding to the adjusted target object posture is regenerated, and the first similarity between the adjusted first virtual lateral two-dimensional image and the lateral two-dimensional image is calculated; in some specific embodiments, when the first similarity meets the first object posture condition, the current object posture is used as the first virtual object posture. Optionally, the first object posture condition is that the difference between two adjacent iterations of the first similarity is greater than a preset first similarity threshold.

[0099] In some specific embodiments, a registered target lateral posture is generated based on the relative positional relationship between the first virtual object posture and the lateral two-dimensional image, a first transition posture is generated based on the first virtual object posture, and the first virtual object posture is inverted to obtain the inverse as the first transition posture.

[0100] In some specific embodiments, the initial lateral posture of the C-arm X-ray machine is calculated using a PNP algorithm, and the first transition posture is multiplied by the initial lateral posture of the current C-arm X-ray machine to obtain the target lateral posture of the C-arm X-ray machine after registration. In some specific embodiments, there is a relative relationship between the first virtual object posture and the initial lateral posture. The lateral posture of the C-arm X-ray machine is calculated based on the relative relationship when the object posture remains as the target object posture, and the lateral posture at this time is determined as the target lateral posture of the C-arm X-ray machine.

[0101] exist Figure 5 In the registration result integration module, the target object posture of the object to be registered, the target lateral posture of the C-arm X-ray machine and the initial frontal posture of the C-arm X-ray machine are integrated into the registration optimization result.

[0102] In some specific embodiments, the target object posture of the object to be registered is the conversion matrix of the coordinates from the CT image coordinate system to the patient coordinate system after registration optimization. The target lateral posture and initial anteroposterior posture of the C-arm X-ray machine are the conversion matrices of the coordinates from the patient coordinate system to the three-dimensional coordinate system of the X-ray machine after registration optimization. In some specific embodiments, the registration result integration module integrates the target object posture of the object to be registered, the target lateral posture of the C-arm X-ray machine, and the initial anteroposterior posture of the C-arm X-ray machine to generate a registration optimization result, which is provided to the subsequent operator for operation.

[0103] Figure 6 Schematic diagram of the orthopedic surgery navigation system based on image registration of the present application. Figure 6 As shown, the orthopedic surgery navigation system based on image registration includes a three-dimensional CT device, an X-ray machine, a processing module and an operating robot.

[0104] In some specific embodiments, Figure 6 In the example, the X-ray machine is a C-arm X-ray machine. Figure 6 The system shown is described in detail.

[0105] like Figure 6 As shown, in some specific embodiments, the three-dimensional CT device is used to take a three-dimensional image of the part to be operated on before surgery; the C-arm X-ray machine is used to take a frontal two-dimensional image and a lateral two-dimensional image of the part to be operated on during surgery; the processing module is used to determine the surgical operation plan according to the three-dimensional image; in some specific embodiments, the processing module is also used to use Figure 1 The method shown generates a registration optimization result based on a three-dimensional image, an anterior two-dimensional image, a lateral two-dimensional image and the internal parameters of a C-arm X-ray machine, and the registration optimization result includes a target object posture of the object to be registered, a target lateral posture of the C-arm X-ray machine and an initial anterior posture of the C-arm X-ray machine; the processing module is also used to convert the coordinates of the surgical operation plan from the CT image coordinate system to the patient coordinate system according to the target object posture; convert the coordinates of the surgical operation plan from the patient coordinate system to the three-dimensional coordinate system of the X-ray machine according to the target lateral posture and the initial anterior posture; convert the coordinates of the surgical operation plan from the C-arm three-dimensional coordinate system to the anterior X-ray two-dimensional coordinate system and the lateral X-ray two-dimensional coordinate system according to the internal parameters of the C-arm X-ray machine; the operating robot is used to operate the part to be operated according to the surgical operation plan.

[0106] In some specific embodiments, the part to be operated is the spine, using Figure 6 The process of orthopedic surgical navigation by the system shown is as follows:

[0107] Before the operation, a three-dimensional image of the spine to be operated is taken by a three-dimensional CT device. The processing module determines the surgical operation plan to be performed on the spine to be operated based on the three-dimensional image. Optionally, the surgical operation plan includes the position of the pedicle screws for each single vertebra in the spine to be operated.

[0108] During the operation, the C-arm X-ray machine takes two-dimensional anteroposterior and lateral images of the spine to be operated. The processing module Figure 1The method shown generates a target object posture of the vertebra to be operated and a target lateral posture of the C-arm. The processing module converts the surgical operation plan from the CT coordinate system to the patient coordinate system through the target object posture. The processing module then projects the surgical operation plan in the patient coordinate system onto the X-ray for display using the internal and external references of the C-arm X-ray machine. The operating robot operates the spinal column to be operated according to the surgical operation plan displayed on the X-ray. Optionally, the operating robot fixes the pedicle screw at the position of the surgical operation plan.

[0109] Figure 7 The present application provides a structural diagram of an electronic device, including a processor and a memory. The memory stores computer instructions. When the computer instructions are executed by the processor, the processor executes the computer instructions to achieve the following Figure 1 The method and refinement scheme shown.

[0110] It should be understood that the above device embodiments are only illustrative, and the device disclosed in the present invention can also be implemented in other ways. For example, the division of units / modules described in the above embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0111] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present invention may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0112] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. If not otherwise specified, the processor or chip may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. If not otherwise specified, the on-chip cache, the off-chip memory, and the memory may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc.

[0113] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0114] The present application also provides a non-transitory computer storage medium storing a computer program. When the computer program is executed by multiple processors, the processors execute the following Figure 1 The method and refinement scheme shown.

[0115] The embodiments of the present application are described in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and its core idea of ​​the present application. At the same time, changes or deformations made by those skilled in the art based on the ideas of the present application, the specific implementation methods and the scope of application of the present application, all belong to the scope of protection of the present application. In summary, the content of this specification should not be construed as a limitation on the present application.

Claims

1. An image registration method, characterized in that: include: Generate an initial object pose of the object to be registered and an initial anteroposterior pose and an initial lateral pose of the X-ray machine according to the three-dimensional image, the two-dimensional anteroposterior image, the two-dimensional lateral image of the object to be registered and the internal parameters of the X-ray machine; Performing registration optimization on the initial object posture according to the frontal two-dimensional image and the lateral two-dimensional image to generate a target object posture of the object to be registered; Performing registration optimization on the initial lateral posture of the X-ray machine according to the target object posture and the lateral two-dimensional image to generate a target lateral posture of the X-ray machine after registration; The target object posture of the object to be registered, the target lateral posture of the X-ray machine and the initial frontal posture of the X-ray machine are integrated into a registration optimization result.

2. The method according to claim 1, characterized in that The step of performing registration optimization on the initial lateral posture of the X-ray machine according to the target object posture and the lateral two-dimensional image to generate the registered target lateral posture of the X-ray machine includes: Generate a first virtual object posture according to the target object posture and the lateral two-dimensional image; The target lateral posture of the registered X-ray machine is generated according to the relative relationship between the first virtual object posture and the lateral two-dimensional image.

3. The method according to claim 2, characterized in that The step of generating the target lateral position of the registered X-ray machine according to the relative relationship between the first virtual object position and the lateral two-dimensional image comprises: generating a first transition posture according to the first virtual object posture; The target lateral posture of the X-ray machine after the alignment is obtained according to the first transitional posture and the initial lateral posture of the current X-ray machine.

4. The method according to claim 2, characterized in that Generating a first virtual object posture according to the target object posture and the lateral two-dimensional image comprises: S201, taking the target object posture as the current object posture; S202, generating a first virtual lateral two-dimensional image corresponding to the current object posture; S203, calculating a first similarity between the lateral two-dimensional image and the first virtual lateral two-dimensional image; S204, when the first similarity satisfies a first object posture condition, taking the current object posture as the first virtual object posture; S205: When the first similarity does not satisfy the first object posture condition, adjust the current object posture and return to step S202.

5. The method according to claim 1, characterized in that The registering and optimizing the initial object posture according to the frontal two-dimensional image and the lateral two-dimensional image to generate the target object posture of the object to be registered comprises: Generate a first temporary object pose according to the initial object pose, the frontal two-dimensional image and the lateral two-dimensional image; The target object pose is generated according to the first temporary object pose and the orthogonal two-dimensional image.

6. The method according to claim 5, characterized in that Generating a first temporary object posture according to the initial object posture, the frontal two-dimensional image and the lateral two-dimensional image comprises: S301, taking the initial object posture as the current object posture; S302, generating a second virtual frontal two-dimensional image and a second virtual lateral two-dimensional image corresponding to the current object posture; S303, calculating a second similarity between the anteroposterior two-dimensional image and the second virtual anteroposterior two-dimensional image, and a third similarity between the lateral two-dimensional image and the second virtual lateral two-dimensional image; S304: When the second similarity and the third similarity satisfy a second object posture condition, use the current object posture as the first temporary object posture; S305: When the second similarity and the third similarity do not satisfy the second object posture condition, adjust the current object posture and return to step S302.

7. The method according to claim 5, characterized in that Generating the target object posture according to the first temporary object posture and the orthogonal two-dimensional image comprises: S401, using the first temporary object posture as the current object posture; S402, generating a third virtual orthotopic two-dimensional image corresponding to the current object posture; S403, calculating a fourth similarity between the orthostatic two-dimensional image and the third virtual orthostatic two-dimensional image; S404, when the fourth similarity satisfies the third object posture condition, taking the current object posture as the target object posture; S405, when the fourth similarity does not satisfy the third object posture condition, adjust the current object posture and return to step S402.

8. The method according to any one of claims 4, 6 and 7, characterized in that: The adjusting the object posture comprises: The position and posture of the object to be registered is rotated or translated.

9. An image registration device, characterized in that: It includes an initial pose generation module, a target object pose generation module, a target lateral pose generation module and a registration result integration module, among which: The initial posture generation module is used to generate the initial object posture of the object to be registered and the initial anteroposterior posture and initial lateral posture of the X-ray machine according to the three-dimensional image, the two-dimensional image of the anteroposterior position, the two-dimensional image of the lateral position and the internal parameters of the X-ray machine of the object to be registered; The target object posture generation module is used to perform registration optimization on the initial object posture according to the frontal two-dimensional image and the lateral two-dimensional image to generate the target object posture of the object to be registered; The target lateral posture generation module is used to perform registration optimization on the initial lateral posture of the X-ray machine according to the target object posture and the lateral two-dimensional image, and generate the target lateral posture of the X-ray machine after registration; The registration result integration module is used to integrate the target object posture of the object to be registered, the target lateral posture of the X-ray machine and the initial frontal posture of the X-ray machine into a registration optimization result.

10. An orthopedic surgery navigation system based on image registration, characterized in that: include: 3D CT equipment, used to take 3D images of the part to be operated on; An X-ray machine, used to take two-dimensional anteroposterior and lateral images of the part to be operated on, and provide the internal reference of the X-ray machine to the processing module; A processing module, used for determining a surgical operation plan according to the three-dimensional image; Using the method according to any one of claims 1 to 8, a registration optimization result is generated according to the three-dimensional image, the frontal two-dimensional image, the lateral two-dimensional image and the internal parameters of the X-ray machine, wherein the registration optimization result includes the target object pose of the object to be registered, the target lateral pose of the X-ray machine and the initial frontal pose of the X-ray machine; The coordinates of the surgical operation plan are converted from the CT image coordinate system to the patient coordinate system according to the target object posture; the coordinates of the surgical operation plan are converted from the patient coordinate system to the three-dimensional coordinate system of the X-ray machine according to the target lateral posture and the initial anteroposterior posture; the coordinates of the surgical operation plan are converted from the three-dimensional coordinate system of the X-ray machine to the two-dimensional coordinate system of the anteroposterior X-ray and the lateral X-ray according to the internal reference of the X-ray machine; An operating robot is used to operate the part to be operated according to the surgical operation plan.

11. An electronic device, comprising: processor; A memory storing a computer program, which, when executed by the processor, enables the processor to perform the method according to any one of claims 1 to 8.

12. A non-transitory computer-readable storage medium having computer-readable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 8.

Images