Image registration method and model training method applied to image registration

Abstract

The embodiment of the invention provides an image registration method and a model training method applied to image registration. The method comprises the following steps: acquiring to-be-processed image data under a view; wherein the to-be-processed image data comprises a projection image of a single cone in a three-dimensional image and a two-dimensional image; processing the to-be-processed image data under the plurality of views according to a registration model to obtain rotation point features of rotation points of the single cone in the to-be-processed image data; according to the registration model, processing to-be-processed image data under a plurality of views and rotation point features of rotation points of single cones in the to-be-processed image data to obtain registration parameters of the single cones; and performing registration processing on the projection image and the two-dimensional image of the single cone according to the registration parameters. The method is used for achieving the effect of improving the accuracy of registration parameters.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image registration method and a model training method applied to image registration. Background Art

[0002] During the operation, two-dimensional images are taken and the preoperative three-dimensional images are registered with the intraoperative two-dimensional images using image registration technology to obtain registration transformation parameters. Based on the registration transformation parameters, the preoperative plan can be accurately converted to the actual position of the patient during the operation, thereby guiding the surgical robot or assisting the doctor to perform high-precision surgical operations.

[0003] Currently, the registration model can be used to predict the registration parameters, and then the three-dimensional image and the two-dimensional image can be registered based on the registration parameters. However, in this method, the calculation of the registration parameters of all vertebrae mostly regards the rotation point as the same fixed value, while the changes of each vertebra are different, resulting in the registration parameters not fitting the actual vertebral bending changes, that is, the accuracy of the registration parameters is low. Summary of the invention

[0004] The embodiments of the present application provide an image registration method and a model training method applied to image registration, so as to achieve the effect of improving the accuracy of registration parameters.

[0005] In a first aspect, an embodiment of the present application provides an image registration method, comprising:

[0006] Acquire image data to be processed under the view; wherein the image data to be processed includes a projection image and a two-dimensional image of a single vertebra in a three-dimensional image;

[0007] Processing the image data to be processed under the plurality of views according to the registration model to obtain the rotation point features of the rotation point of the single vertebra in the image data to be processed;

[0008] According to the registration model, the image data to be processed under multiple views and the rotation point features of the rotation points of the single vertebra in the image data to be processed are processed to obtain the registration parameters of the single vertebra;

[0009] The projection image and the two-dimensional image of the single vertebra are registered according to the registration parameters.

[0010] In a possible implementation, the registration model includes a plurality of processing branches, wherein the processing branches are associated with the visual Figure 1 One-to-one correspondence; processing the image data to be processed under the multiple views according to the registration model to obtain the rotation point features of the rotation point of the single vertebra in the image data to be processed, including:

[0011] For each processing branch in the registration model, based on the processing branch, rotation point features are extracted from the image data to be processed in the view corresponding to the processing branch to obtain rotation point features of the rotation points of single vertebrae in the image data to be processed.

[0012] In a possible implementation, each processing branch of the registration model includes a first sub-branch and a second sub-branch, and each sub-branch includes a rotation point feature extraction module;

[0013] The step of extracting rotation point features of the image data to be processed under the view corresponding to the processing branch based on the processing branch to obtain the rotation point features of the rotation points of the single vertebra in the image data to be processed includes:

[0014] Based on the rotation point feature extraction module of the first sub-branch of the processing branch, rotation point feature extraction is performed on the projection image of the single vertebra under the view to obtain the rotation point features of the rotation point of the single vertebra in the projection image; and based on the rotation point feature extraction module of the second sub-branch of the processing branch, rotation point feature extraction is performed on the two-dimensional image of the single vertebra under the view to obtain the rotation point features of the rotation point of the single vertebra in the two-dimensional image.

[0015] In a possible implementation, the registration model includes a plurality of processing branches, wherein the processing branches are associated with the visual Figure 1 One-to-one correspondence; according to the registration model, the image data to be processed under multiple views and the rotation point features of the rotation point of the single vertebra in the image data to be processed are processed to obtain the registration parameters of the single vertebra, including:

[0016] For each processing branch in the registration model, based on the processing branch, image features of the image data to be processed in the view corresponding to the processing branch are extracted to obtain image features of the image data to be processed in the view;

[0017] Based on the processing branch, the image features of the image data to be processed in the view and the rotation point features of the rotation points of the single vertebra in the image data to be processed are spliced ​​to obtain the position transformation features of the single vertebra in the view;

[0018] The position transformation features of the single vertebra in each view are fused to obtain the registration parameters of the single vertebra.

[0019] In a possible implementation, the registration model includes a fusion module and a perceptron, and performs fusion processing on the position transformation features of a single vertebra in each view to obtain the registration parameters of the single vertebra, including:

[0020] Based on the fusion module in the registration model, the position transformation features of the single vertebra in each view are fused to obtain the fused position transformation features of the single vertebra;

[0021] Based on the perceptron in the registration model, the position transformation features of the single vertebra after the fusion are predicted and processed to obtain the registration parameters of the single vertebra.

[0022] In a second aspect, an embodiment of the present application provides a model training method for image registration, including:

[0023] Acquire the image data to be trained under the view; wherein the image data to be trained includes a projection image and a two-dimensional image of a target single vertebra in a three-dimensional image; the target single vertebra in the image data to be trained has a rotation point;

[0024] Pre-training the initial model according to the image data to be trained in multiple views and the rotation point of the target single vertebra in the image data to be trained to obtain the pre-trained initial model;

[0025] According to the image data to be trained under multiple views, the pre-trained initial model is trained to obtain a registration model; wherein the registration model is used to obtain registration parameters for registering the projection image and the two-dimensional image of a target single vertebra;

[0026] Wherein, the registration model is the registration model in any one of the methods described in the first aspect.

[0027] In a possible implementation, the pre-trained initial model is trained according to the image data to be trained under multiple views to obtain a registration model, including:

[0028] According to the pre-trained initial model, the image data to be trained under the view is processed to obtain the rotation point feature of the rotation point of the target single vertebra in the image data to be trained;

[0029] The pre-trained initial model is trained according to the image data to be trained in multiple views and the rotation point features of the rotation point of the target single vertebra in the image data to be trained to obtain a registration model.

[0030] In a possible implementation, the initial model includes a plurality of processing branches, wherein the processing branches are associated with the visual Figure 1 One by one, each processing branch includes a first sub-branch and a second sub-branch, and each sub-branch includes an initial rotation point feature extraction module;

[0031] Pre-training the initial model according to the image data to be trained in multiple views and the rotation point of the target single vertebra in the image data to be trained to obtain the pre-trained initial model, including:

[0032] For each processing branch of the initial model, based on the initial rotation point feature extraction module of the first sub-branch of the processing branch, the projection image of the target single vertebra under the view is subjected to rotation point feature extraction to obtain the predicted rotation point of the target single vertebra in the projection image; and based on the initial rotation point feature extraction module of the second sub-branch of the processing branch, the two-dimensional image of the target single vertebra under the view is subjected to rotation point feature extraction to obtain the predicted rotation point of the target single vertebra in the two-dimensional image;

[0033] According to the predicted rotation point of the target single vertebra in the projection image and the rotation point of the target single vertebra in the projection image, the initial rotation point feature extraction module of the first sub-branch is optimized to obtain the optimized rotation point feature extraction module of the first sub-branch; and according to the predicted rotation point of the target single vertebra in the two-dimensional image and the rotation point of the target single vertebra in the two-dimensional image, the initial rotation point feature extraction module of the second sub-branch is optimized to obtain the optimized rotation point feature extraction module of the second sub-branch;

[0034] The pre-trained initial model is obtained according to the rotation point feature extraction module optimized by the first sub-branch and the rotation point feature extraction module optimized by the second sub-branch.

[0035] In a possible implementation, obtaining image data to be trained under a view includes:

[0036] Acquire a three-dimensional image; and determine a target single vertebra in the three-dimensional image, and a rotation point of the target single vertebra;

[0037] According to the rotation point of the target single vertebra and the preset initial registration parameters, the preset initial position information is transformed to obtain the target position information of the target single vertebra;

[0038] Under multiple views, the projection image of the target single vertebra is obtained according to the preset initial position information and the three-dimensional image; and under multiple views, the two-dimensional image of the target single vertebra is determined according to the target position information and the three-dimensional image; wherein the projection image of the target single vertebra under the view, the two-dimensional image of the target single vertebra, the initial alignment parameters, and the rotation point of the target single vertebra constitute the image data to be trained.

[0039] In a possible implementation, determining a target single vertebra in a three-dimensional image includes:

[0040] Obtaining the centroid point of each single vertebra in the three-dimensional image, as well as the projection points corresponding to multiple views;

[0041] For each view, determining a single vertebra to be selected in each view according to the projection point corresponding to the view and the centroid point of each single vertebra;

[0042] The target single vertebra is determined according to the intersection of the candidate single vertebra in each view.

[0043] In a possible implementation, determining the single vertebra to be selected in each view according to the projection point corresponding to the view and the centroid point of each single vertebra includes:

[0044] In the Z-axis direction of the coordinate system of the three-dimensional image, according to the centroid point of each single vertebra and the projection point, determine the distance between the centroid point of each single vertebra and the projection point;

[0045] According to the distance, a single vertebra to be selected in each view is obtained by screening from the multiple single vertebrae.

[0046] In a third aspect, an embodiment of the present application provides an image registration device, comprising:

[0047] An acquisition module, used for acquiring image data to be processed under a view; wherein the image data to be processed includes a projection image and a two-dimensional image of a single vertebra in a three-dimensional image;

[0048] A first processing module is used to process the image data to be processed under the plurality of views according to the registration model to obtain the rotation point feature of the rotation point of the single vertebra in the image data to be processed;

[0049] A second processing module is used to process the image data to be processed under multiple views and the rotation point features of the rotation points of the single vertebra in the image data to be processed according to the registration model to obtain the registration parameters of the single vertebra;

[0050] The registration module is used to perform registration processing on the projection image and the two-dimensional image of the single vertebra according to the registration parameters.

[0051] In a fourth aspect, an embodiment of the present application provides a model training device for image registration, comprising:

[0052] An acquisition module is used to acquire the image data to be trained under the view; wherein the image data to be trained includes a projection image and a two-dimensional image of a target single vertebra in a three-dimensional image; the target single vertebra in the image data to be trained has a rotation point;

[0053] A first training module is used to pre-train the initial model according to the image data to be trained in multiple views and the rotation point of the target single vertebra in the image data to be trained, so as to obtain the pre-trained initial model;

[0054] The second training module is used to train the pre-trained initial model according to the image data to be trained under multiple views to obtain a registration model; wherein the registration model is used to obtain registration parameters for registering the projection image and the two-dimensional image of the target single vertebra.

[0055] In a fifth aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0056] The memory stores computer-executable instructions;

[0057] The processor executes the computer-executable instructions stored in the memory, so that the processor executes various possible implementations of the first aspect and / or the second aspect as described above.

[0058] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-readable storage medium is stored computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement various possible implementations of the first aspect and / or the second aspect as described above.

[0059] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements various possible implementations of the first aspect and / or the second aspect as described above.

[0060] The image registration method and the model training method applied to image registration provided in the embodiments of the present application utilize a registration model to process the image data to be processed under multiple views, obtain the rotation point features of the rotation points of the single vertebra in the image data to be processed under multiple views, and then obtain the registration parameters based on the rotation point features of the rotation points of the single vertebra in the image data to be processed under multiple views, and utilize the registration parameters to achieve the registration of the projected image and the two-dimensional image in the image data to be processed. In this way, during image registration, not only the rotation point features of the rotation point of each single vertebra can be considered, but also the rotation point features of the rotation point of each single vertebra under multiple views, so that the registration parameters of the single vertebra can better fit the actual bending changes of the single vertebra, improve the accuracy of the registration parameters, and thus improve the accuracy of image registration. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0062] Figure 1 Schematic diagram of the process of the model training method for image registration provided in this application Figure 1 ;

[0063] Figure 2 Schematic diagram of the process of the model training method for image registration provided in this application Figure 2 ;

[0064] Figure 3 A schematic diagram of a three-dimensional image projection provided by this application;

[0065] Figure 4 A schematic diagram of a three-dimensional image provided for this application;

[0066] Figure 5 A schematic diagram of a projection image of a target single vertebra provided in the present application;

[0067] Figure 6 A schematic diagram of obtaining image data to be trained provided in this application;

[0068] Figure 7 A schematic diagram of the structure of an initial model provided for this application;

[0069] Figure 8 Schematic diagram of the process of the model training method for image registration provided in this application Figure 3 ;

[0070] Fig. 9 A schematic diagram of the structure of an initial rotation point feature extraction module provided in this application;

[0071] Fig.10 A schematic diagram of the structure of an initial fusion module provided in this application;

[0072] Fig.11 Schematic diagram of the image registration method provided in this application Figure 1 ;

[0073] Fig.12 Schematic diagram of the image registration method provided in this application Figure 2 ;

[0074] Fig.13 A schematic diagram of the structure of a model training device for image registration provided in this application;

[0075] Fig.14 A schematic diagram of the structure of the image registration device provided in this application;

[0076] Fig.15 A schematic diagram of the structure of the electronic device provided in this application.

[0077] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0078] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0079] First, the terms involved in this application are explained:

[0080] Single vertebra: refers to a single vertebral unit in the spine.

[0081] The vertebral body top view (AP position) and lateral view (LAT position) are two commonly used photographic positions in medical imaging technology. Among them, the AP position refers to the photographic position of observing vertically downward from the top of the vertebral body; the LAT position refers to the photographic position of observing from the side of the vertebral body.

[0082] Voxel: short for Volume Pixel, is the smallest unit of digital data in three-dimensional space segmentation. Voxel can be used in three-dimensional imaging, scientific data, medical imaging and other fields.

[0083] Before surgery, preoperative planning can be performed by taking three-dimensional images, such as computed tomography (CT) images. During surgery, since it is impossible to take three-dimensional images in real time, two-dimensional images, such as X-ray images, can be taken, and image registration technology can be used to register the preoperative three-dimensional images with the intraoperative two-dimensional images to obtain registration parameters. Then, based on the registration parameters, the three-dimensional image and the two-dimensional image are registered, so that the preoperative plan can be accurately converted to the actual position of the patient during surgery, thereby guiding the surgical robot or assisting the doctor to perform high-precision surgical operations.

[0084] Currently, the registration parameters between the X-ray image and the digitally reconstructed radiographs (DRR) projected at the initial position of the CT image can be learned through the registration model, and then the registration parameters can be predicted based on the registration model.

[0085] In the above method, the CT image usually includes multiple single vertebrae, and the registration parameters predicted by the registration model are generated for all single vertebrae based on a fixed rotation point. However, in actual movement, the movement of each single vertebra is different. Using the same rotation point causes the registration parameters to not fit the actual bending changes of the vertebra, that is, the accuracy of the registration parameters is low.

[0086] In view of this, the present application provides an image registration method, in which the rotation point of each single vertebra in the three-dimensional image is considered during the registration process of the three-dimensional image and the two-dimensional image, and the registration parameters of the single vertebra are determined according to the rotation point of the single vertebra. In this way, the rotation point of the single vertebra is used as a calculation factor for the registration parameters of the single vertebra, so that the registration parameters of the single vertebra can better fit the actual bending changes of the single vertebra, thereby improving the accuracy of the registration parameters.

[0087] The executor of the present application may be an electronic device with processing capabilities, such as a computer, a server, etc., or a customized medical device, such as a surgical robot, etc., which is not limited in the present application.

[0088] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and corresponding operation entrances shall be provided for users to choose to authorize or refuse.

[0089] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0090] First, taking the execution subject as an electronic device as an example, how to train the registration model is explained. Figure 1 Schematic diagram of the process of the model training method for image registration provided in this application Figure 1 ,like Figure 1 As shown, the method includes:

[0091] S101: Obtain image data to be trained in a view.

[0092] Exemplarily, the above-mentioned image data to be trained is data for training a registration model, and the image data to be trained is obtained under a view, wherein a view can be regarded as a shooting angle or a projection angle, and the image data to be trained under each view includes a projection image and a two-dimensional image of the target single vertebra in the three-dimensional image under the view, and each target single vertebra in the image data to be trained has its own rotation point. The above-mentioned target single vertebra represents a single vertebra to be registered, which can be a preset single vertebra or a randomly determined single vertebra.

[0093] It should be noted that the projection image and the two-dimensional image include the target single vertebra. The three-dimensional image may be a computed tomography (CT) image, the projection image may be a digitally reconstructed radiograph (DRR), and the two-dimensional image may be an X-ray image. It should be noted that the present application does not limit the types of three-dimensional images, projection images, and two-dimensional images.

[0094] The rotation point of the target single vertebra can represent the axis point of the target single vertebra during rotational motion. It should be understood that different single vertebrae have differences in motion, so the rotation points of different single vertebrae are different. The rotation point of the target single vertebra can be, for example, the center of mass of the target single vertebra, or the midpoint of the upper and lower edges or other characteristic points with anatomical significance, which is not limited in this application.

[0095] In one example, the electronic device may communicate with other electronic devices to obtain the image data to be trained under the view; the electronic device may extract the image data to be trained under the view from a database; the electronic device may receive the image data to be trained under the view imported by an external device, such as a USB flash drive; or the electronic device may synthesize the image data to be trained under the view based on sample image data.

[0096] It should be noted that the number of views in the present application is related to the setting of the initial model. For example, there can be at least two views, such as an AP view and a LAT view. It can also be centered on the axis perpendicular to the horizontal plane, with each deviation from a preset angle as a view.

[0097] S102 , pre-training the initial model according to the image data to be trained in multiple views and the rotation point of the target single vertebra in the image data to be trained to obtain the pre-trained initial model.

[0098] Exemplarily, the electronic device can, for each view, use the rotation point of the target single vertebra in the image data to be trained under the view as a label or supervisory information, and use the projection image and two-dimensional image in the image data to be trained under the view to pre-train a preset initial model, so that the initial model can learn the rotation point features of the rotation point of the target single vertebra in the projection image and two-dimensional image under each view, so that the pre-trained initial model can recognize the rotation point features of the rotation point of the single vertebra based on the projection image and two-dimensional image under each view.

[0099] For example, in the pre-training stage, the electronic device can use the rotation point of the target single vertebra as a supervisory signal and use the loss function to measure the accuracy of the rotation point feature of the rotation point of the target single vertebra. Then, the parameters of the initial model can be continuously adjusted through optimization methods such as back propagation algorithm and gradient descent to minimize the loss function and improve the performance of the initial model, thereby obtaining the initial model after pre-training.

[0100] S103: training the pre-trained initial model according to the image data to be trained in multiple views to obtain a registration model.

[0101] Exemplarily, the above registration model is used to obtain the registration parameters for registering the projection image and the two-dimensional image of the target single vertebra. The above image data to be trained under multiple views has initial registration parameters, and the initial registration parameters represent the registration parameters for registering the projection image and the two-dimensional image under multiple views. It should be understood that the above initial registration parameters have a corresponding relationship with the three-dimensional image, that is, if the three-dimensional image does not change, the projection image and the two-dimensional image under any view correspond to the same initial registration parameters.

[0102] In one example, the electronic device can use the initial registration parameters under multiple views as labels or supervision information based on the pre-trained initial model, and use the image data to be trained under multiple views to retrain the pre-trained initial model. During the training process, the pre-trained initial model can fuse the rotation point features of the rotation points of the target single vertebra under multiple views, and obtain the fused image features of the target single vertebra. Based on the fused rotation point features of the rotation points of the target single vertebra and the image features of the target single vertebra, the registration parameters are predicted, and then the pre-trained initial model can be optimized according to the initial registration parameters to obtain the registration model.

[0103] In this way, the trained registration model can not only take into account the rotation point features of the rotation point of the target single vertebra in each view, but also take into account the rotation point features of the rotation point of the target single vertebra after multi-view fusion, as well as the image features of the target single vertebra after multi-view fusion. Then, the registration parameters of the target single vertebra can be predicted by fusing the image features of the target single vertebra in multiple views and the rotation point features of the rotation point of the target single vertebra, thereby improving the accuracy of the registration parameters and further improving the robustness of the model.

[0104] The model training method for image registration provided in the embodiment of the present application utilizes the image data to be trained under multiple views and the rotation point of the target single vertebra in the image data to be trained to first pre-train the initial model to obtain the pre-trained initial model, and then the pre-trained initial model can be trained according to the image data to be trained under multiple views to obtain the registration model.

[0105] In this way, the pre-trained initial model can learn the rotation point features of the rotation point of the target single vertebra in the projection image and the two-dimensional image under each view; after retraining the pre-trained initial model, the registration model can not only take into account the rotation point features of the rotation point of the target single vertebra under each view, but also take into account the rotation point features of the rotation point of the target single vertebra after multi-view fusion, as well as the image features of the target single vertebra after multi-view fusion, and then the registration parameters of the target single vertebra can be predicted by fusing the image features of the target single vertebra in multiple views and the rotation point features of the rotation point of the target single vertebra, so that the registration parameters of the single vertebra can better fit the actual bending changes of the single vertebra, improve the accuracy of the registration parameters, and further improve the robustness of the model.

[0106] Figure 2 Schematic diagram of the process of the model training method for image registration provided in this application Figure 2 ,like Figure 2 As shown, in this embodiment Figure 1 Based on the embodiment, how to obtain the image data to be trained is described in detail, and the method includes:

[0107] S201, acquiring a three-dimensional image; and determining a target single vertebra in the three-dimensional image, and a rotation point of the target single vertebra.

[0108] Exemplarily, the electronic device can acquire a three-dimensional image including multiple single vertebrae, such as a CT image, and then determine the target single vertebrae and the rotation point of the target single vertebrae from the three-dimensional image. For example, the single vertebrae closest to the projection point can be used as the target single vertebrae, and the center of mass of the single vertebra can be used as the rotation point.

[0109] In some possible implementations, the target single vertebra in the three-dimensional image may be determined by the following steps:

[0110] S2011. Obtain the centroid point of each single vertebra in the three-dimensional image, and the projection points corresponding to multiple views.

[0111] Exemplarily, the projection points corresponding to the above-mentioned multiple views represent the projection points under the view, that is, the position points at which the three-dimensional image is projected under the view. The projection point is related to the setting of the view. The projection point can be preset based on the view or randomly selected based on the view. For example, if the view is an AP position, a point can be randomly selected in a direction perpendicular to the three-dimensional image as the projection point under the view. Optionally, a projection plane under each view is determined to obtain a projected image on the projection plane, and the position of the projection plane is set based on the view. Reference Figure 3 As shown, Figure 3 A schematic diagram of a three-dimensional image projection provided in this application, a point can be selected directly above the three-dimensional image as the projection point under the view, and a projection plane can be selected directly below the three-dimensional image. It should be noted that the principle of determining the projection point under other views is similar and will not be repeated here.

[0112] In one example, the electronic device may perform mask processing on the three-dimensional image to obtain a mask image of each single vertebra, and then obtain the centroid of the single vertebra based on the average value of the coordinates of the voxels of the single vertebra in the mask image of each single vertebra.

[0113] S2012: For each view, determine a candidate single vertebra in each view according to the projection point corresponding to the view and the centroid point of each single vertebra.

[0114] Exemplarily, for each view, the single vertebra to be selected in each view may be determined according to the distance between the centroid point of each single vertebra and the projection point.

[0115] Specifically, Figure 4 A schematic diagram of a three-dimensional image provided in this application, refer to Figure 4 The coordinate system of the three-dimensional image shown in the Z-axis direction of the coordinate system of the three-dimensional image is used to determine the distance between the center of mass point of each single vertebra and the projection point according to the center of mass point of each single vertebra and the projection point; for example, the difference between the Z-axis coordinate of the center of mass point of each single vertebra and the Z-axis coordinate of the projection point is used as the distance between the center of mass point of each single vertebra and the projection point; based on the distance, the single vertebra to be selected in each view is screened from multiple single vertebrae, for example, a preset number of single vertebrae with the smallest distance, such as 4, can be used as the single vertebrae to be selected; or the single vertebrae with a distance less than or equal to a preset threshold can be used as the single vertebrae to be selected.

[0116] S2013. Determine a target single vertebra according to the intersection of the candidate single vertebrae in each view.

[0117] Illustratively, through the above steps, the candidate single vertebrae in each view can be obtained, and the target single vertebrae can be determined from the intersection of the candidate single vertebrae in each view, for example, any single vertebrae in the intersection can be used as the target single vertebrae.

[0118] Through the above steps, the spatial position relationship between the target single vertebra and the projection points of each view is ensured to be relatively close, and then a single vertebra that is relatively accurate in each view can be determined from the three-dimensional image as the target single vertebra, thereby improving the accuracy of determining the target single vertebra and reducing possible errors or misleading information in a single view.

[0119] In some possible implementations, the electronic device may use the center of mass of the target single vertebra as the rotation point of the target single vertebra.

[0120] S202: transform the preset initial position information according to the rotation point of the target single vertebra and the preset initial registration parameters to obtain the target position information of the target single vertebra.

[0121] Exemplarily, the above-mentioned preset initial position information represents the posture information of the target single vertebra in the three-dimensional image when it is projected. The initial position information can be represented by translation parameters and rotation parameters, for example, it can be represented as (r x ,r y ,r z ,t x ,t y ,t z ), wherein r represents the rotation angle based on each coordinate axis, and t represents the translation amount based on each coordinate axis. The electronic device can randomly take a value within a preset angle range, such as within the range of (-10°, 10°), to obtain the rotation parameter, and randomly take a value within a preset position range, such as within the range of (-10mm, 10mm), to obtain the translation parameter, and then obtain the preset initial position information. The above-mentioned preset initial registration parameters represent the registration parameters for aligning the projection image with the two-dimensional image, and can also be represented by the aforementioned translation parameters and rotation parameters. The method for determining the initial registration parameters is similar to the method for determining the initial position information, and will not be repeated here. The above-mentioned target position information represents the posture information of the target single vertebra when taking a two-dimensional image (X-ray), and can also be represented by the aforementioned translation parameters and rotation parameters.

[0122] In one example, the electronic device can transform the preset initial position information according to the rotation point of the target single vertebra and the preset initial alignment parameters, that is, according to the preset mapping relationship, based on the rotation point of the target single vertebra, the preset initial alignment parameters and the preset initial position information, calculate the target position information.

[0123] S203, obtaining a projection image of a target single vertebra according to preset initial position information and a three-dimensional image in multiple views; and determining a two-dimensional image of a target single vertebra according to the target position information and the three-dimensional image in multiple views.

[0124] In one example, the electronic device may project the three-dimensional image under multiple views and preset initial position information to obtain a projection image of a target single vertebra.

[0125] Specifically, Figure 5 A schematic diagram of a projection image of a target single vertebra provided in this application, referring to Figure 5 As shown, for each view, the electronic device can project the three-dimensional image under the preset initial position information to obtain a first initial projection image; wherein the first initial projection image includes multiple single vertebrae; then the first initial projection image is subjected to single vertebrae detection processing to obtain first bounding box information of multiple single vertebrae; according to the positions of the projection points and the projection plane under the view, the projection position point of the rotation point of the target vertebra in the first initial projection image is calculated, and the first bounding box information of the single vertebra to which the projection position point belongs is determined to obtain the first bounding box information of the target single vertebra; according to the first bounding box information of the target single vertebra, the first initial projection image is segmented to obtain the projection image of the target single vertebra under the view.

[0126] In one example, the electronic device may first determine a second initial projection image of a target single vertebra according to the target position information and the three-dimensional image in multiple views, and then determine a two-dimensional image of the target single vertebra according to the second initial projection image.

[0127] Specifically, for each view, the electronic device can project the three-dimensional image under the target position information to obtain a second initial projection image; wherein the second initial projection image includes multiple single vertebrae; and perform style transfer processing on the second initial projection image to obtain an initial two-dimensional image; wherein the initial two-dimensional image includes multiple single vertebrae; and then perform single vertebrae detection processing on the initial two-dimensional image to obtain second bounding box information of multiple single vertebrae; according to the position of the projection point and the projection plane under the view, calculate the projection position point of the rotation point of the target vertebra in the second initial projection image, determine the second bounding box information of the single vertebra to which the projection position point belongs, and determine it as the second bounding box information of the target single vertebra; according to the second bounding box information of the target single vertebra, segment the initial two-dimensional image to obtain a two-dimensional image of the target single vertebra under the view. Among them, the electronic device can preset a style transfer model, and perform style transfer processing on the second initial projection image based on the style transfer model to obtain the initial two-dimensional image.

[0128] Optionally, to simplify model calculations, the image to be subjected to single vertebra detection processing may be normalized before single vertebra detection processing, and the coordinates of the projection position point of the rotation point on the projection image may be normalized to reduce the amount of calculations.

[0129] Through the above steps, the projection image of the target single vertebra under the view, the two-dimensional image of the target single vertebra, the initial registration parameters, and the rotation point of the target single vertebra constitute the image data to be trained under the view.

[0130] Take the AP bit view and the LAT bit view as an example. Figure 6 A schematic diagram of obtaining image data to be trained provided in this application, refer to Figure 6 As shown, based on the aforementioned steps, the electronic device can perform AP-position projection and LAT-position projection on the three-dimensional image under the preset initial position information, respectively, to obtain the first initial projection image under the AP-position view and the first initial projection image under the LAT-position view, and obtain the projection image of the target single vertebra under the AP-position view and the projection image of the target single vertebra under the LAT-position view after segmentation processing; at the same time, the electronic device can first perform AP-position projection and LAT-position projection on the three-dimensional image under the target position information, respectively, to obtain the second initial projection image under the AP-position view and the second initial projection image under the LAT-position view, and then perform style transfer processing on the second initial projection image under the AP-position view and the second initial projection image under the LAT-position view, respectively, to obtain the initial two-dimensional image under the AP-position view and the initial two-dimensional image under the LAT-position view, and obtain the two-dimensional image of the target single vertebra under the AP-position view and the two-dimensional image of the target single vertebra under the LAT-position view after segmentation processing.

[0131] Through the above steps, the electronic device can obtain the image data to be trained of the three-dimensional images in multiple views for multiple three-dimensional images, thereby ensuring the sample volume of the training data.

[0132] The model training method for image registration provided in the embodiment of the present application determines the rotation point of the target single vertebra based on the target single vertebra in the three-dimensional image, and determines the target position information of the target single vertebra according to the rotation point of the target single vertebra, the preset initial position information and the preset initial configuration parameters, and then obtains the projection image of the target single vertebra using the initial position information, and obtains the two-dimensional image of the target single vertebra using the target position information, forming the image data to be trained. In this way, the electronic device can synthesize the image data to be trained for training the registration model according to the rotation point of the single vertebra, and ensure the accuracy of the image data to be trained. At the same time, this method does not need to manually mark the position of the single vertebra, which improves the efficiency of obtaining the image data to be trained.

[0133] In some embodiments, Figure 7 A schematic diagram of the structure of an initial model provided for this application, refer to Figure 7 As shown in Figure 2, the initial model includes an initial fusion module, an initial perceptron, and multiple processing branches. Figure 1 Correspondingly, each processing branch includes a first sub-branch, a second sub-branch and a splicing module, and each sub-branch includes an initial rotation point feature extraction module and an initial image feature extraction module. Figure 7 The example of including two processing branches is merely used for illustration.

[0134] Figure 8 Schematic diagram of the process of the model training method for image registration provided in this application Figure 3 ,refer to Figure 7 and Figure 8 In this embodiment, Figure 1 Based on the embodiment, a model training method applied to image registration is described in detail, and the method includes:

[0135] S301: Obtain image data to be trained in a view.

[0136] It should be noted that this step can refer to the description in the aforementioned embodiment and will not be repeated here.

[0137] S302. For each processing branch of the initial model, based on the initial rotation point feature extraction module of the first sub-branch of the processing branch, the projection image of the target single vertebra in the view is subjected to rotation point feature extraction to obtain the predicted rotation point of the target single vertebra in the projection image.

[0138] Exemplarily, the initial rotation point feature extraction module is used to extract the rotation point features of the rotation point of the target single vertebra in the image, and the initial rotation point feature extraction module can output the predicted rotation point of the target single vertebra.

[0139] In one example, the electronic device can perform image feature extraction on the projection image of the target single vertebra in the view based on the initial image feature extraction module of the first sub-branch of the processing branch, and then perform rotation point feature extraction on the extracted image features based on the initial rotation point feature extraction module of the first sub-branch of the processing branch to obtain the predicted rotation point of the target single vertebra in the projection image. The initial image feature extraction module can be a densely connected convolutional network (DenseNet) model or other convolutional networks, which is not limited in this application.

[0140] Optional, reference Fig. 9 As shown, Fig. 9 This is a schematic diagram of the structure of an initial rotation point feature extraction module provided by the present application. The initial rotation point feature extraction module may include a perceptron and at least one bottleneck module, each bottleneck module includes a residual module, at least one activation function and at least one convolutional layer. It should be noted that Fig. 9 It is only illustrated by including three bottleneck modules, three convolutional layers and three activation functions.

[0141] In this implementation, after the image features are input into the bottleneck module, the first 1×1 convolution layer (Conv1×1) of each bottleneck module performs channel dimension reduction to reduce the amount of parameter calculation for subsequent modules. The rotation point features are then extracted through the second 3×3 convolution layer (Conv3×3). Finally, a 1×1 convolution layer (Conv1×1) is used to expand the channel dimension back to its original size. An activation function Relu is connected after each convolution layer, and the rotation point features are added to the original image features through the residual module to obtain the rotation point features, which are then input into the perceptron to predict the rotation point of the target single vertebra.

[0142] S303: Based on the initial rotation point feature extraction module of the second sub-branch of the processing branch, the rotation point feature of the two-dimensional image of the target single vertebra in the view is extracted to obtain the predicted rotation point of the target single vertebra in the two-dimensional image.

[0143] It should be noted that the principle of this step is similar to that of the aforementioned step S302 and will not be repeated here.

[0144] S304, optimizing the initial rotation point feature extraction module of the first sub-branch according to the predicted rotation point of the target single vertebra in the projection image and the rotation point of the target single vertebra in the projection image, to obtain an optimized rotation point feature extraction module of the first sub-branch.

[0145] Exemplarily, the initial rotation point feature extraction module of the above-mentioned first sub-branch is optimized by back propagation algorithm, gradient descent and other optimization methods through the rotation point of the target single vertebra in the projection image included in the image data to be trained (for example, the projection position point of the rotation point of the target single vertebra in the projection image calculated in the aforementioned embodiment), and the predicted rotation point of the target single vertebra in the projection image, to obtain the optimized rotation point feature extraction module of the first sub-branch.

[0146] S305. Optimize the initial rotation point feature extraction module of the second sub-branch according to the predicted rotation point of the target single vertebra in the two-dimensional image and the rotation point of the target single vertebra in the two-dimensional image to obtain an optimized rotation point feature extraction module of the second sub-branch.

[0147] It should be noted that the principle of this step is similar to that of the aforementioned step S304 and will not be repeated here.

[0148] S306. Obtain a pre-trained initial model according to the rotation point feature extraction module optimized by the first sub-branch and the rotation point feature extraction module optimized by the second sub-branch.

[0149] Exemplarily, the pre-training refers to pre-training the rotation point feature extraction module to obtain the weight parameters of the rotation point feature extraction module under each sub-branch, and then obtain the pre-trained initial model.

[0150] S307 , processing the image data to be trained under the view according to the pre-trained initial model to obtain the rotation point features of the rotation point of the target single vertebra in the image data to be trained.

[0151] Exemplarily, for each view, according to the pre-trained initial model and based on the rotation point feature extraction module of each processing branch, rotation point features are extracted for the image data to be trained to obtain the rotation point features of the rotation points of the target single vertebra in the image data to be trained. The rotation point features of the rotation points of the target single vertebra in the image data to be trained include the rotation point features of the rotation points of the target single vertebra in the projection image and the rotation point features of the rotation points of the target single vertebra in the two-dimensional image.

[0152] S308 , training the pre-trained initial model according to the image data to be trained in multiple views and the rotation point features of the rotation point of the target single vertebra in the image data to be trained to obtain a registration model.

[0153] Exemplarily, for the image data to be trained in each view, after being processed by the processing branch in the pre-trained initial model, the rotation point features of the rotation points of the target single vertebra in the image data to be trained in the view are obtained, and then based on the initial splicing module of the processing branch, the image features of the image data to be trained and the rotation point features of the rotation points of the target single vertebra in the image data to be trained are spliced ​​to obtain the position transformation features of the target single vertebra in the image data to be trained in the view, and then based on the initial fusion module in the pre-trained initial model, the position transformation features of the target single vertebra in the image data to be trained in multiple views are fused and input into the initial perceptron to obtain predicted alignment parameters; the electronic device can optimize the pre-trained initial model based on the initial alignment parameters in the image data to be trained and the predicted alignment parameters to obtain the alignment model.

[0154] Optional, reference Fig.10 As shown, Fig.10 A structural schematic diagram of an initial fusion module provided for the present application, the initial fusion module may include multiple branches and an intermediate layer module, each branch corresponds to a processing branch one by one, each branch adopts four residual connection modules, each residual connection module contains two 3×3 convolution layers and two activation functions Relu, and each residual connection module is used to extract high-dimensional features layer by layer. The outputs of each layer of residual connection modules of each branch are added and then processed by the intermediate layer module for multi-view feature fusion. Among them, the intermediate layer module includes four layers of sub-modules, each sub-module includes a 1×1 convolution and a 3×3 convolution and two activation functions Relu. In order to avoid the loss of parameter information in the subsequent extraction process, the output of each layer of the intermediate layer module is added and fused with each layer corresponding to multiple branches, so as to fully improve the accuracy of the extracted features.

[0155] Through the initial fusion module, the position transformation features under multiple views can be extracted and fused layer by layer to obtain the fused position transformation features.

[0156] The model training method for image registration provided in the embodiment of the present application uses the rotation point of the target single vertebra and the image data to be trained to pre-train the initial model, so that the pre-trained initial model learns the rotation point features of the rotation point of the target single vertebra in the image data to be trained, and then trains the pre-trained initial model based on the image data to be trained to obtain a registration model. In this way, a rotation point feature extraction module for the projection image and a rotation point feature extraction module for the two-dimensional image are set based on each processing branch, so that the pre-trained initial model can accurately learn the rotation point features of the rotation point of the target single vertebra in the projection image and the two-dimensional image under each view, and ensure the accuracy of the rotation point features of the rotation point of the target single vertebra, thereby improving the accuracy of the predicted registration parameters of the target single vertebra, so that the registration parameters of the single vertebra can better fit the actual bending changes of the single vertebra, and can further improve the robustness of the model.

[0157] Next, we will explain how to implement image registration based on the registration model. Fig.11 Schematic diagram of the image registration method provided in this application Figure 1 ,like Fig.11 As shown, the method includes:

[0158] S401: Obtain image data to be processed in a view.

[0159] Exemplarily, the image data to be processed include a projection image and a two-dimensional image of a single vertebra in a three-dimensional image. The single vertebra may be a single vertebra to be surgically processed, which is not limited in the present application.

[0160] In one example, a medical device can be used to capture a three-dimensional image before surgery, and the three-dimensional image can be projected to obtain a projection image of a single vertebra, and a medical device can be used to capture a two-dimensional image of a single vertebra during surgery. Therefore, the electronic device can communicate with other electronic devices to obtain the projection image of the single vertebra, as well as the two-dimensional image of the single vertebra; it can also extract the projection image of the single vertebra, as well as the two-dimensional image of the single vertebra from a database; or it can receive the projection image of the single vertebra, as well as the two-dimensional image of the single vertebra imported from an external device, such as a USB flash drive.

[0161] It should be noted that the view setting of the image data to be processed obtained under the view is related to the setting of the registration model, and this application does not limit the view here.

[0162] S402 , processing the image data to be processed in multiple views according to the registration model to obtain the rotation point features of the rotation points of the single vertebra in the image data to be processed.

[0163] Exemplarily, the image data to be processed in multiple views are input into the registration model, and based on the registration model, the rotation point features of the rotation points of a single vertebra in the image data to be processed in each view are obtained.

[0164] S403 , according to the registration model, processing the image data to be processed under multiple views and the rotation point features of the rotation points of the single vertebra in the image data to be processed to obtain the registration parameters of the single vertebra.

[0165] Exemplarily, based on the registration model, the image data to be processed under multiple views and the rotation point features of the rotation points of the single vertebra in the image data to be processed are processed to predict the registration parameters of the single vertebra.

[0166] S404: Perform registration processing on the projection image and the two-dimensional image of a single vertebra according to the registration parameters.

[0167] For example, the electronic device can register each pixel point or preset feature point in the projection image of a single vertebra to the two-dimensional image based on the registration parameters; or each pixel point or preset feature point in the two-dimensional image can be registered to the projection image based on the registration parameters, and this application does not limit how to register. Among them, the feature point can be a landmark point, an edge point, etc. of the anatomical structure.

[0168] In some possible situations, doctors may need to register preoperative CT images with intraoperative X-ray images so that the preoperative planned trajectory can be displayed on the intraoperative image to guide the surgical operation. At this time, by transforming the CT image into an X-ray image based on the registration parameters, the doctor can understand the relative position relationship between the surgical instrument and the preoperative planned trajectory in real time, thereby improving the accuracy and safety of the surgery.

[0169] In some possible cases, it may be necessary to register intraoperative X-ray images with preoperative CT images in order to monitor and evaluate the surgical site in real time. At this time, by transforming X-ray images into CT images based on the registration parameters, doctors can more accurately understand the anatomical structure and pathological changes of the surgical site, providing strong support for surgical decision-making.

[0170] The image registration method provided in the embodiment of the present application utilizes a registration model to process the image data to be processed under multiple views, obtains the rotation point features of the rotation points of the single vertebra in the image data to be processed under multiple views, and then obtains the registration parameters based on the rotation point features of the rotation points of the single vertebra in the image data to be processed under multiple views, and utilizes the registration parameters to realize the registration of the projected image and the two-dimensional image in the image data to be processed. In this way, during image registration, not only the rotation point features of the rotation point of each single vertebra can be considered, but also the rotation point features of the rotation point of each single vertebra under multiple views can be considered, so that the registration parameters of the single vertebra can better fit the actual bending changes of the single vertebra, improve the accuracy of the registration parameters, and thus improve the accuracy of image registration.

[0171] Fig.12 Schematic diagram of the image registration method provided in this application Figure 2 ,like Fig.12 As shown, in this embodiment Fig.11 Based on the embodiment, the image registration method is described in detail, and the method includes:

[0172] S501: Obtain image data to be processed in a view.

[0173] It should be noted that this step can refer to the aforementioned step S401 and will not be repeated here.

[0174] S502 , for each processing branch in the registration model, based on the processing branch, extract rotation point features of the image data to be processed in the view corresponding to the processing branch to obtain rotation point features of the rotation points of single vertebrae in the image data to be processed.

[0175] Exemplarily, the above registration model includes multiple processing branches, and the processing branches are connected with the visual Figure 1 One-to-one correspondence. It should be understood that the structure of the initial model is consistent with that of the registration model, which can be referred to Figure 7 According to the model structure shown, the electronic device can extract the rotation point features of the projection image of the single vertebra under the view based on the processing branch and utilize the rotation point feature extraction module included in each sub-branch of the processing branch to obtain the rotation point features of the rotation point of the single vertebra in the image data to be processed.

[0176] In some possible implementations, based on the rotation point feature extraction module of the first sub-branch of the processing branch, rotation point feature extraction is performed on the projection image of the single vertebra under the view to obtain the rotation point features of the rotation points of the single vertebra in the projection image; and based on the rotation point feature extraction module of the second sub-branch of the processing branch, rotation point feature extraction is performed on the two-dimensional image of the single vertebra under the view to obtain the rotation point features of the rotation points of the single vertebra in the two-dimensional image.

[0177] S503 . For each processing branch in the registration model, based on the processing branch, image features of the image data to be processed in the view corresponding to the processing branch are extracted to obtain image features of the image data to be processed in the view.

[0178] For example, reference Figure 7 As shown, the electronic device can extract image features from the image data to be processed based on the image feature extraction module included in each sub-branch of the processing branch to obtain image features of the image data to be processed under the view.

[0179] S504: Based on the processing branch, the image features of the image data to be processed under the view and the rotation point features of the rotation points of the single vertebra in the image data to be processed are spliced ​​to obtain the position transformation features of the single vertebra under the view.

[0180] For example, reference Figure 7 As shown, the electronic device can perform stitching processing on the image features of the image data to be processed under the view and the rotation point features of the rotation point of the single vertebra in the image data to be processed based on the stitching module included in the processing branch to obtain the position transformation features of the single vertebra under the view.

[0181] S505 , fusing the position transformation features of the single vertebra in each view to obtain the registration parameters of the single vertebra.

[0182] For example, reference Figure 7 As shown, the electronic device can fuse the position transformation features of the single vertebra in each view based on the fusion module to obtain the position transformation features of the single vertebra, and predict the position transformation features of the single vertebra based on the perception machine to obtain the alignment parameters of the single vertebra.

[0183] Specifically, based on the fusion module in the registration model, the position transformation features of the single vertebra in each view are fused to obtain the position transformation features of the fused single vertebra; based on the perceptron in the registration model, the position transformation features of the single vertebra after fusion are predicted to obtain the registration parameters of the single vertebra.

[0184] S506: Perform registration processing on the projection image and the two-dimensional image of the single vertebra according to the registration parameters.

[0185] It should be noted that this step can refer to the aforementioned step S404 and will not be repeated here.

[0186] The image registration method provided in the embodiment of the present application, for the image data to be processed under multiple views, uses multiple processing branches in the registration model to process the image data to be processed under different views respectively, obtains the rotation point features and image features of the rotation points of the single vertebra in the image data to be processed under each view, and then obtains the position transformation features of the single vertebra under each view after splicing the rotation point features and image features of the rotation points of the single vertebra in the image data to be processed, obtains the registration parameters after fusing the position transformation features of the single vertebra under each view, and uses the registration parameters to realize the registration of the projection image and the two-dimensional image in the image data to be processed. In this way, during image registration, the rotation point features of the rotation points of each single vertebra under multiple views and the position transformation features of the single vertebra under multiple views can be taken into account, so that the registration parameters of each single vertebra can fuse the position transformation features of the single vertebra under multiple views, better fit the actual bending changes of the single vertebra, improve the accuracy of the registration parameters, and thus improve the accuracy of image registration.

[0187] Fig.13 A schematic diagram of the structure of a model training device for image registration provided in this application, such as Fig.13 As shown, the model training device 600 for image registration provided in this embodiment includes:

[0188] An acquisition module 601 is used to acquire image data to be processed under a view; wherein the image data to be processed includes a projection image and a two-dimensional image of a single vertebra in a three-dimensional image;

[0189] A first processing module 602 is used to process the image data to be processed under the multiple views according to the registration model to obtain the rotation point features of the rotation point of the single vertebra in the image data to be processed;

[0190] The second processing module 603 is used to process the image data to be processed under multiple views and the rotation point features of the rotation points of the single vertebra in the image data to be processed according to the registration model to obtain the registration parameters of the single vertebra;

[0191] The registration module 604 is used to perform registration processing on the projection image and the two-dimensional image of the single vertebra according to the registration parameters.

[0192] In a possible implementation, the registration model includes a plurality of processing branches, wherein the processing branches are associated with the visual Figure 1 One-to-one correspondence; the first processing module 602 is specifically used for:

[0193] For each processing branch in the registration model, based on the processing branch, rotation point features are extracted from the image data to be processed in the view corresponding to the processing branch to obtain rotation point features of the rotation points of single vertebrae in the image data to be processed.

[0194] In a possible implementation, each processing branch of the registration model includes a first sub-branch and a second sub-branch, and each sub-branch includes a rotation point feature extraction module;

[0195] The first processing module 602 is specifically used for:

[0196] Based on the rotation point feature extraction module of the first sub-branch of the processing branch, rotation point feature extraction is performed on the projection image of the single vertebra under the view to obtain the rotation point features of the rotation point of the single vertebra in the projection image; and based on the rotation point feature extraction module of the second sub-branch of the processing branch, rotation point feature extraction is performed on the two-dimensional image of the single vertebra under the view to obtain the rotation point features of the rotation point of the single vertebra in the two-dimensional image.

[0197] In a possible implementation, the registration model includes a plurality of processing branches, wherein the processing branches are associated with the visual Figure 1 One-to-one correspondence; the second processing module 603 is specifically used for:

[0198] For each processing branch in the registration model, based on the processing branch, image features of the image data to be processed in the view corresponding to the processing branch are extracted to obtain image features of the image data to be processed in the view;

[0199] Based on the processing branch, the image features of the image data to be processed in the view and the rotation point features of the rotation points of the single vertebra in the image data to be processed are spliced ​​to obtain the position transformation features of the single vertebra in the view;

[0200] The position transformation features of the single vertebra in each view are fused to obtain the registration parameters of the single vertebra.

[0201] In a possible implementation, the registration model includes a fusion module and a perceptron, and the second processing module 603 is specifically configured to:

[0202] Based on the fusion module in the registration model, the position transformation features of the single vertebra in each view are fused to obtain the fused position transformation features of the single vertebra;

[0203] Based on the perceptron in the registration model, the position transformation features of the single vertebra after the fusion are predicted and processed to obtain the registration parameters of the single vertebra.

[0204] The model training device for image registration provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and are not described in detail in this embodiment.

[0205] Fig.14 A schematic diagram of the structure of the image registration device provided in this application, such as Fig.14 As shown, the image registration device 700 provided in this embodiment includes:

[0206] The acquisition module 701 is used to acquire the image data to be trained under the view; wherein the image data to be trained includes the projection image and the two-dimensional image of the target single vertebra in the three-dimensional image; the target single vertebra in the image data to be trained has a rotation point;

[0207] A first training module 702 is used to pre-train the initial model according to the image data to be trained in multiple views and the rotation point of the target single vertebra in the image data to be trained, so as to obtain the pre-trained initial model;

[0208] The second training module 703 is used to train the pre-trained initial model according to the image data to be trained under multiple views to obtain a registration model; wherein the registration model is used to obtain registration parameters for registering the projection image and the two-dimensional image of the target single vertebra.

[0209] In a possible implementation manner, the second training module 703 is specifically configured to:

[0210] According to the pre-trained initial model, the image data to be trained under the view is processed to obtain the rotation point feature of the rotation point of the target single vertebra in the image data to be trained;

[0211] The pre-trained initial model is trained according to the image data to be trained in multiple views and the rotation point features of the rotation point of the target single vertebra in the image data to be trained to obtain a registration model.

[0212] In a possible implementation, the initial model includes a plurality of processing branches, wherein the processing branches are associated with the visual Figure 1 One by one, each processing branch includes a first sub-branch and a second sub-branch, and each sub-branch includes an initial rotation point feature extraction module;

[0213] The first training module 702 is specifically used for:

[0214] For each processing branch of the initial model, based on the initial rotation point feature extraction module of the first sub-branch of the processing branch, the projection image of the target single vertebra under the view is subjected to rotation point feature extraction to obtain the predicted rotation point of the target single vertebra in the projection image; and based on the initial rotation point feature extraction module of the second sub-branch of the processing branch, the two-dimensional image of the target single vertebra under the view is subjected to rotation point feature extraction to obtain the predicted rotation point of the target single vertebra in the two-dimensional image;

[0215] According to the predicted rotation point of the target single vertebra in the projection image and the rotation point of the target single vertebra in the projection image, the initial rotation point feature extraction module of the first sub-branch is optimized to obtain the optimized rotation point feature extraction module of the first sub-branch; and according to the predicted rotation point of the target single vertebra in the two-dimensional image and the rotation point of the target single vertebra in the two-dimensional image, the initial rotation point feature extraction module of the second sub-branch is optimized to obtain the optimized rotation point feature extraction module of the second sub-branch;

[0216] The pre-trained initial model is obtained according to the rotation point feature extraction module optimized by the first sub-branch and the rotation point feature extraction module optimized by the second sub-branch.

[0217] In a possible implementation, the acquisition module 701 is specifically configured to:

[0218] Acquire a three-dimensional image; and determine a target single vertebra in the three-dimensional image, and a rotation point of the target single vertebra;

[0219] According to the rotation point of the target single vertebra and the preset initial registration parameters, the preset initial position information is transformed to obtain the target position information of the target single vertebra;

[0220] Under multiple views, the projection image of the target single vertebra is obtained according to the preset initial position information and the three-dimensional image; and under multiple views, the two-dimensional image of the target single vertebra is determined according to the target position information and the three-dimensional image; wherein the projection image of the target single vertebra under the view, the two-dimensional image of the target single vertebra, the initial alignment parameters, and the rotation point of the target single vertebra constitute the image data to be trained.

[0221] In a possible implementation, the acquisition module 701 is specifically configured to:

[0222] Obtaining the centroid point of each single vertebra in the three-dimensional image, as well as the projection points corresponding to multiple views;

[0223] For each view, determining a single vertebra to be selected in each view according to the projection point corresponding to the view and the centroid point of each single vertebra;

[0224] The target single vertebra is determined according to the intersection of the candidate single vertebra in each view.

[0225] In a possible implementation, the acquisition module 701 is specifically configured to:

[0226] In the Z-axis direction of the coordinate system of the three-dimensional image, according to the centroid point of each single vertebra and the projection point, determine the distance between the centroid point of each single vertebra and the projection point;

[0227] According to the distance, a single vertebra to be selected in each view is obtained by screening from the multiple single vertebrae.

[0228] The image registration device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0229] Fig.15 This is a schematic diagram of the structure of the electronic device provided in this application. Fig.15 As shown, the electronic device 800 provided in this embodiment includes: at least two processors 801 and a memory 802. Optionally, the device 80 further includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0230] In a specific implementation process, at least two processors 801 execute computer execution instructions stored in the memory 802, so that at least two processors 801 execute the above method.

[0231] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0232] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0233] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least two disk memories.

[0234] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0235] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0236] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0237] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0238] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0239] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0240] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0241] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0242] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0243] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0244] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image registration method, characterized in that: include: Acquire image data to be processed under the view; wherein the image data to be processed includes a projection image and a two-dimensional image of a single vertebra in a three-dimensional image; Processing the image data to be processed under the plurality of views according to the registration model to obtain the rotation point features of the rotation point of the single vertebra in the image data to be processed; According to the registration model, the image data to be processed under multiple views and the rotation point features of the rotation points of the single vertebra in the image data to be processed are processed to obtain the registration parameters of the single vertebra; The projection image and the two-dimensional image of the single vertebra are registered according to the registration parameters.

2. The method according to claim 1, characterized in that The registration model includes a plurality of processing branches, and the processing branches correspond to the views one by one; the image data to be processed under the plurality of views are processed according to the registration model to obtain the rotation point features of the rotation points of the single vertebra in the image data to be processed, including: For each processing branch in the registration model, based on the processing branch, rotation point features are extracted from the image data to be processed in the view corresponding to the processing branch to obtain rotation point features of the rotation points of single vertebrae in the image data to be processed.

3. The method according to claim 2, characterized in that Each processing branch of the registration model includes a first sub-branch and a second sub-branch, and each sub-branch includes a rotation point feature extraction module; The step of extracting rotation point features of the image data to be processed under the view corresponding to the processing branch based on the processing branch to obtain the rotation point features of the rotation points of the single vertebra in the image data to be processed includes: Based on the rotation point feature extraction module of the first sub-branch of the processing branch, rotation point feature extraction is performed on the projection image of the single vertebra under the view to obtain the rotation point features of the rotation point of the single vertebra in the projection image; and based on the rotation point feature extraction module of the second sub-branch of the processing branch, rotation point feature extraction is performed on the two-dimensional image of the single vertebra under the view to obtain the rotation point features of the rotation point of the single vertebra in the two-dimensional image.

4. The method according to any one of claims 1 to 3, characterized in that: The registration model includes a plurality of processing branches, and the processing branches correspond to the views one by one. According to the registration model, the image data to be processed under the plurality of views and the rotation point features of the rotation points of the single vertebra in the image data to be processed are processed to obtain the registration parameters of the single vertebra, including: For each processing branch in the registration model, based on the processing branch, image features of the image data to be processed in the view corresponding to the processing branch are extracted to obtain image features of the image data to be processed in the view; Based on the processing branch, the image features of the image data to be processed in the view and the rotation point features of the rotation points of the single vertebra in the image data to be processed are spliced ​​to obtain the position transformation features of the single vertebra in the view; The position transformation features of the single vertebra in each view are fused to obtain the registration parameters of the single vertebra.

5. The method according to claim 4, characterized in that The registration model includes a fusion module and a perceptron, which performs fusion processing on the position transformation features of a single vertebra in each view to obtain the registration parameters of the single vertebra, including: Based on the fusion module in the registration model, the position transformation features of the single vertebra in each view are fused to obtain the fused position transformation features of the single vertebra; Based on the perceptron in the registration model, the position transformation features of the single vertebra after the fusion are predicted and processed to obtain the registration parameters of the single vertebra.

6. A model training method for image registration, characterized in that: include: Acquire the image data to be trained under the view; wherein the image data to be trained includes a projection image and a two-dimensional image of a target single vertebra in a three-dimensional image; the target single vertebra in the image data to be trained has a rotation point; Pre-training the initial model according to the image data to be trained in multiple views and the rotation point of the target single vertebra in the image data to be trained to obtain the pre-trained initial model; According to the image data to be trained under multiple views, the pre-trained initial model is trained to obtain a registration model; wherein the registration model is used to obtain registration parameters for registering the projection image and the two-dimensional image of a target single vertebra; Wherein, the registration model is the registration model in the method described in any one of claims 1-5.

7. The method according to claim 6, characterized in that According to the image data to be trained under multiple views, the pre-trained initial model is trained to obtain a registration model, including: According to the pre-trained initial model, the image data to be trained under the view is processed to obtain the rotation point feature of the rotation point of the target single vertebra in the image data to be trained; The pre-trained initial model is trained according to the image data to be trained in multiple views and the rotation point features of the rotation point of the target single vertebra in the image data to be trained to obtain a registration model.

8. The method according to claim 6, characterized in that The initial model includes a plurality of processing branches, each of which corresponds to the views one by one, each of which includes a first sub-branch and a second sub-branch, and each of which includes an initial rotation point feature extraction module; Pre-training the initial model according to the image data to be trained in multiple views and the rotation point of the target single vertebra in the image data to be trained to obtain the pre-trained initial model, including: For each processing branch of the initial model, based on the initial rotation point feature extraction module of the first sub-branch of the processing branch, the projection image of the target single vertebra under the view is subjected to rotation point feature extraction to obtain the predicted rotation point of the target single vertebra in the projection image; and based on the initial rotation point feature extraction module of the second sub-branch of the processing branch, the two-dimensional image of the target single vertebra under the view is subjected to rotation point feature extraction to obtain the predicted rotation point of the target single vertebra in the two-dimensional image; According to the predicted rotation point of the target single vertebra in the projection image and the rotation point of the target single vertebra in the projection image, the initial rotation point feature extraction module of the first sub-branch is optimized to obtain the optimized rotation point feature extraction module of the first sub-branch; and according to the predicted rotation point of the target single vertebra in the two-dimensional image and the rotation point of the target single vertebra in the two-dimensional image, the initial rotation point feature extraction module of the second sub-branch is optimized to obtain the optimized rotation point feature extraction module of the second sub-branch; The pre-trained initial model is obtained according to the rotation point feature extraction module optimized by the first sub-branch and the rotation point feature extraction module optimized by the second sub-branch.

9. The method according to any one of claims 6 to 8, characterized in that: Get the image data to be trained under the view, including: Acquire a three-dimensional image; and determine a target single vertebra in the three-dimensional image, and a rotation point of the target single vertebra; According to the rotation point of the target single vertebra and the preset initial registration parameters, the preset initial position information is transformed to obtain the target position information of the target single vertebra; Under multiple views, the projection image of the target single vertebra is obtained according to the preset initial position information and the three-dimensional image; and under multiple views, the two-dimensional image of the target single vertebra is determined according to the target position information and the three-dimensional image; wherein the projection image of the target single vertebra under the view, the two-dimensional image of the target single vertebra, the initial alignment parameters, and the rotation point of the target single vertebra constitute the image data to be trained.

10. The method according to claim 9, characterized in that Determine the target single vertebra in the three-dimensional image, including: Obtaining the centroid point of each single vertebra in the three-dimensional image, as well as the projection points corresponding to multiple views; For each view, determining a single vertebra to be selected in each view according to the projection point corresponding to the view and the centroid point of each single vertebra; The target single vertebra is determined according to the intersection of the candidate single vertebra in each view.

11. The method according to claim 10, characterized in that Determining a single vertebra to be selected in each view according to the projection point corresponding to the view and the centroid point of each single vertebra, comprising: In the Z-axis direction of the coordinate system of the three-dimensional image, according to the centroid point of each single vertebra and the projection point, determine the distance between the centroid point of each single vertebra and the projection point; According to the distance, a single vertebra to be selected in each view is obtained by screening from the multiple single vertebrae.

12. An image registration device, characterized in that: include: An acquisition module, used for acquiring image data to be processed under a view; wherein the image data to be processed includes a projection image and a two-dimensional image of a single vertebra in a three-dimensional image; A first processing module is used to process the image data to be processed under the plurality of views according to the registration model to obtain the rotation point feature of the rotation point of the single vertebra in the image data to be processed; A second processing module is used to process the image data to be processed under multiple views and the rotation point features of the rotation points of the single vertebra in the image data to be processed according to the registration model to obtain the registration parameters of the single vertebra; The registration module is used to perform registration processing on the projection image and the two-dimensional image of the single vertebra according to the registration parameters.

13. A model training device for image registration, characterized in that: include: An acquisition module is used to acquire the image data to be trained under the view; wherein the image data to be trained includes a projection image and a two-dimensional image of a target single vertebra in a three-dimensional image; the target single vertebra in the image data to be trained has a rotation point; A first training module is used to pre-train the initial model according to the image data to be trained in multiple views and the rotation point of the target single vertebra in the image data to be trained, so as to obtain the pre-trained initial model; The second training module is used to train the pre-trained initial model according to the image data to be trained under multiple views to obtain a registration model; wherein the registration model is used to obtain registration parameters for registering the projection image and the two-dimensional image of the target single vertebra.

14. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the image registration method as described in any one of claims 1-5, or the model training method applied to image registration as described in any one of claims 6-11.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the image registration method as described in any one of claims 1 to 5, or the model training method for image registration as described in any one of claims 6 to 11.

16. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the image registration method as described in any one of claims 1 to 5, or the model training method for image registration as described in any one of claims 6 to 11.

Images