Artifact image registration method and device, terminal equipment and storage medium
By deartective image input generation network with artifacts, and register the image after artifact removal with the preoperative image input registration network, combined with the joint training of the discriminative network, the problem of artifact affecting registration accuracy in interventional puncture surgery is solved, and a higher image registration accuracy is achieved.
Patent Information
- Application Number
- CN202311156288.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-07
- Publication Date
- 2025-03-11
AI Technical Summary
During interventional puncture surgery, artifacts appear in intraoperative images due to the intraoperative image not perpendicular to the image plane or the needle vibrates, affecting the registration accuracy of intraoperative images and preoperative images.
By obtaining the intraoperative image with artifacts, inputting it into the pre-trained generative network for artifact removal, obtaining the target intraoperative image, and then inputting the target intraoperative image and the preoperative image for registration network for registration processing, obtaining the registration relationship between the two. At the same time, a discriminating network is designed for joint training to ensure that the generation network, registration network and discriminating network promote each other and supervise each other, and improve the registration accuracy.
By removing the registration of intraoperative images and preoperative images after artifacts, the accuracy of image registration is significantly improved and the impact of artifacts on registration results is reduced.
Smart Images
Figure CN119600066A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to an artifact image registration method, device, terminal equipment and storage medium. Background Art
[0002] With the development of medical technology, interventional puncture has been widely used in the fields of cardiac catheterization, lung cancer embolization, and lung biopsy. During interventional puncture surgery, doctors generally plan the needle entry point and target point in the preoperative image, and then use the interventional surgery robot to perform needle puncture. The robot aligns the intraoperative image with the preoperative image in real time, thereby mapping the preoperatively planned needle entry point and target point to the intraoperative image and guiding the robot's puncture action. However, during the needle puncture process, due to reasons such as the needle not being perpendicular to the image plane or the needle vibrating, artifacts will appear in the collected intraoperative image, which will affect the registration accuracy of the intraoperative image and the preoperative image. Summary of the invention
[0003] In view of this, the embodiments of the present application provide an artifact image registration method, apparatus, terminal device and storage medium, which can improve the registration accuracy of intraoperative artifact images and preoperative images.
[0004] A first aspect of an embodiment of the present application provides an artifact image registration method, comprising:
[0005] Acquire intraoperative and preoperative images with artifacts;
[0006] The intraoperative image with artifacts is input into the pre-trained generative network for artifact removal to obtain the target intraoperative image;
[0007] The preoperative image and the target intraoperative image are input into a pre-trained registration network for registration processing to obtain the registration relationship between the target intraoperative image and the preoperative image;
[0008] Among them, the generation network, the registration network and the discriminant network are jointly trained together. The discriminant network is used to discriminate whether the input image is a deformed image or a preoperative image. The deformed image is obtained by deforming the image output by the generation network using the registration relationship output by the registration network.
[0009] In an embodiment of the present application, when an intraoperative image with artifacts is obtained, it will be input into the generation network for artifact removal processing, so as to obtain the target intraoperative image after the artifacts are removed. Then, the target intraoperative image and the preoperative image are input into the registration network for registration processing, so as to obtain the registration relationship between the target intraoperative image and the preoperative image. Since the registration of the intraoperative image and the preoperative image after the artifacts are removed is performed here, there is no influence of artifacts, so the image registration accuracy can be improved. In addition, the embodiment of the present application also designs a discriminant network, which is mainly used to discriminate whether the input image is a preoperative image or a deformed image obtained by deforming the image output by the generation network using the registration relationship, so that the generation network, the registration network and the discriminant network are jointly trained together, so that each network can promote and supervise each other, and finally achieve common optimization. For example, the generative network can provide clearer and more accurate image data for the registration network, thereby improving the registration effect of the registration network; as the quality of the input image data improves, the registration accuracy of the registration network will also improve. Therefore, the quality of the deformed image obtained after deforming the target intraoperative image using the registration relationship output by the registration network will also be higher. When the deformed image and the preoperative image are input into the discriminative network, it will be more difficult for the discriminative network to distinguish between the deformed image and the preoperative image, thereby promoting the convergence of each network; the generative network and the discriminative network are in an adversarial relationship and can supervise each other to optimize network performance, and so on.
[0010] In one implementation of the embodiment of the present application, the joint training process of the generation network, the registration network and the discrimination network includes:
[0011] Alternately executing the first optimization training process and the second optimization training process until a training end condition is reached;
[0012] In the first optimization training process, the parameters of the generation network and the registration network are optimized and updated; in the second optimization training process, the parameters of the discrimination network and the registration network are optimized and updated.
[0013] In one implementation of the embodiment of the present application, the first loss function used in the first optimization training process is determined according to the loss function of the generation network and the loss function of the registration network.
[0014] Specifically, the loss function of the generation network is determined according to the discrimination result of the discriminant network on the deformed image of the image output by the generation network, and the loss function of the registration network is determined according to the registration result of the registration network on the preoperative image and the image output by the generation network.
[0015] In one implementation of the embodiment of the present application, the second loss function used in the second optimization training process is determined based on the loss function of the discriminant network and the loss function of the registration network.
[0016] Specifically, the loss function of the discriminant network is determined according to the discriminant network's discrimination results on the preoperative image and the deformed image, and the loss function of the registration network is determined according to the registration results of the registration network on the preoperative image and the image output by the generation network.
[0017] In one implementation of the embodiment of the present application, the intraoperative image with artifacts also includes a metal needle track; the intraoperative image with artifacts is input into a generation network for artifact removal processing to obtain a target intraoperative image, including:
[0018] The intraoperative image with artifacts is input into the generative network for artifact removal and metal needle track processing to obtain the target intraoperative image.
[0019] A second aspect of an embodiment of the present application provides an artifact image registration device, comprising:
[0020] An image acquisition module, used for acquiring intraoperative images and preoperative images with artifacts;
[0021] An image artifact removal module is used to input the intraoperative image with artifacts into a pre-trained generation network for artifact removal processing to obtain a target intraoperative image;
[0022] An image registration module is used to input the preoperative image and the target intraoperative image into a pre-trained registration network for registration processing to obtain a registration relationship between the target intraoperative image and the preoperative image;
[0023] Among them, the generation network, the registration network and the discriminant network are jointly trained together. The discriminant network is used to discriminate whether the input image is a deformed image or a preoperative image. The deformed image is obtained by deforming the image output by the generation network using the registration relationship output by the registration network.
[0024] A third aspect of an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the artifact image registration method provided in the first aspect of the embodiment of the present application is implemented.
[0025] A fourth aspect of the embodiments of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the artifact image registration method provided in the first aspect of the embodiments of the present application is implemented.
[0026] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device executes the artifact image registration method provided in the first aspect of the embodiments of the present application.
[0027] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] Figure 1 is a flow chart of an artifact image registration method provided by an embodiment of the present application;
[0029] Figure 2 It is a schematic diagram of the operation flow of the artifact image registration method provided by an embodiment of the present application in an actual application scenario;
[0030] Figure 3 is a flowchart of an image artifact removal and registration operation provided by an embodiment of the present application;
[0031] Figure 4 is a schematic structural diagram of an artifact image registration device provided by an embodiment of the present application;
[0032] Figure 5 It is a schematic diagram of a terminal device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0033] In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structures, technologies, etc. are proposed, so as to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, the detailed description of well-known systems, devices, circuits and methods is omitted to prevent unnecessary details from hindering the description of the present application. In addition, in the description of the present application specification and the attached claims, the terms "first", "second", "third" etc. are only used to distinguish the description, and cannot be interpreted as indicating or suggesting relative importance.
[0034] Interventional puncture is a medical method for diagnosis and treatment by inserting a needle or catheter into the human body. It has been widely used in the fields of cardiac catheterization, lung cancer embolization, and lung biopsy. During the interventional puncture, the doctor will plan the needle entry point and target point in the preoperative image, and then use the interventional surgical robot to puncture the needle. The robot will align the intraoperative image with the preoperative image in real time, thereby mapping the preoperatively planned needle entry point and target point to the intraoperative image to guide the robot's puncture action. However, due to reasons such as the needle not being perpendicular to the image plane or the needle vibrating, artifacts will appear in the collected intraoperative image, which will affect the registration accuracy of the intraoperative image and the preoperative image.
[0035] In view of this, the embodiments of the present application provide an artifact image registration method, apparatus, terminal device and storage medium, which can improve the registration accuracy of intraoperative images and preoperative images. For more specific technical implementation details of the embodiments of the present application, please refer to the method embodiments described below.
[0036] It should be understood that the execution subjects of the various method embodiments of the present application are various types of terminal devices or servers, such as mobile phones, tablet computers, wearable devices, medical devices, augmented reality (AR) / virtual reality (VR) devices, desktop computers, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), large-screen TVs, etc. The embodiments of the present application do not impose any restrictions on the specific types of the terminal devices or servers.
[0037] See also Figure 1 , shows an artifact image registration method provided by an embodiment of the present application, comprising:
[0038] 101. Obtain intraoperative images and preoperative images with artifacts;
[0039] First, obtain the intraoperative image and preoperative image that need to be registered. Among them, the preoperative image is an artifact-free image collected before the operation. The doctor can plan the needle entry point and target point in the preoperative image, and map the needle entry point and target point of the preoperative image to the intraoperative image through image registration, so as to guide the robot's puncture action. The intraoperative image is a real-time image collected during the operation. Due to reasons such as the needle is not perpendicular to the image plane or the needle vibrates, the real-time image collected during the operation will have artifacts, so it is called an intraoperative image with artifacts.
[0040] 102. Inputting the intraoperative image with artifacts into a pre-trained generative network for artifact removal to obtain a target intraoperative image;
[0041] After acquiring the intraoperative image with artifacts, it is input into a pre-trained generative network for artifact removal, thereby obtaining an intraoperative image with artifacts removed, which is represented by a target intraoperative image. The working principle of the generative network is roughly as follows: input an image with artifacts, and after a series of operations such as convolution, pooling and loss function, it can output an image with artifacts removed. In actual operation, the generative network can adopt a generative adversarial network, and is obtained by training a neural network using a training data set. On the one hand, the above-mentioned training data set can use sample images with artifacts and their corresponding sample images with artifacts removed, and on the other hand, it can also use unpaired sample images, for example, the collected preoperative images and intraoperative images with artifacts are used as the above-mentioned training data sets.
[0042] In one embodiment of the present application, an intraoperative image with artifacts is input into a pre-trained generation network for artifact removal processing to obtain a target intraoperative image, including:
[0043] The intraoperative image with artifacts is input into the generative network for artifact removal and metal needle track processing to obtain the target intraoperative image.
[0044] Since the metal needle track is highlighted in the intraoperative image, but there is no needle track information in the preoperative image, in order to prevent the information asymmetry of the two images from affecting the registration accuracy, if the intraoperative image contains the metal needle track in addition to the artifact, it is necessary to remove the artifact and the metal needle track together. To meet this requirement, a generative network with the function of removing both artifacts and metal needle tracks can be trained. Its network training method is similar to the above-mentioned generative network for artifact removal. The training data set used can use sample images with artifacts and metal needle tracks, and their corresponding sample images without artifacts and metal needle tracks. On the other hand, it can also use unpaired sample images, such as the collected preoperative images without artifacts and metal needle tracks, and intraoperative images with artifacts and metal needle tracks as the above-mentioned training data sets. After the intraoperative image with artifacts and metal needle tracks is input into the generative network, the artifacts and metal needle tracks in the image can be removed simultaneously through convolution, pooling and loss function processing performed by the generative network, thereby outputting the intraoperative image without artifacts and metal needle tracks as the target intraoperative image.
[0045] It should be noted that the various embodiments of the present application do not limit the type and network structure of the generative network. Generally speaking, it is only necessary to define an end-to-end neural network and ensure that the size of the input image matches the size of the output image. For example, the generative network can adopt a CycleGAN network, a DCGAN network, or a DualGAN network in Generative Adversarial Networks (GAN), etc.
[0046] 103. The preoperative image and the target intraoperative image are input into a pre-trained registration network for registration processing to obtain a registration relationship between the target intraoperative image and the preoperative image.
[0047] After obtaining the target intraoperative image, the preoperative image and the target intraoperative image are input into the pre-trained registration network for image registration processing, thereby obtaining the registration relationship between the target intraoperative image and the preoperative image. Since the registration here is performed between the intraoperative image after removing artifacts and the preoperative image, or the registration is performed between the intraoperative image after removing artifacts and metal needle tracks and the preoperative image, there is no influence of artifacts and metal needle tracks, so the image registration accuracy can be improved.
[0048] To understand the above steps 101-103, please refer to Figure 2 Flowchart shown. Figure 2 FIG. 1 is a schematic diagram of the operation flow of an artifact image registration method provided by an embodiment of the present application in an actual application scenario. Figure 2 After obtaining the intraoperative image with artifacts and metal needle tracts, the intraoperative image with artifacts and metal needle tracts is first input into the generation network for artifact and metal needle tract removal processing, and the target intraoperative image is output; then, the target intraoperative image and the preoperative image are input into the registration network for registration processing to obtain the registration relationship between the target intraoperative image and the preoperative image; then, using the registration relationship, the preoperatively planned needle entry point and target are mapped from the preoperative image to the target intraoperative image; finally, the needle entry point and target mapped to the target intraoperative image are used to guide the surgical robot to complete the interventional puncture surgery. Since the intraoperative image obtained here is after the artifacts and metal needle tracts are removed, there is no influence of artifacts and metal needle tracts when registering it with the preoperative image, so the image registration accuracy can be improved.
[0049] The working principle of the above-mentioned registration network is roughly as follows: the preoperative image and the intraoperative image with artifact removal are input simultaneously, or the preoperative image and the intraoperative image with artifact removal and metal needle track are input simultaneously, and after a series of operations such as convolution, pooling and loss function, the registration relationship between the two images can be output. Similarly, the various embodiments of the present application do not limit the type and network structure of the registration network. In general, it is only necessary to define an end-to-end neural network. Considering that the images processed are generally three-dimensional images, the size of the deformation field output by the registration network is set to be three times the fixed image size. For example, the registration network can adopt networks such as VoxelMorph and TransMorph.
[0050] The embodiment of the present application also designs a discriminant network, which is mainly used to discriminate whether the input image is a deformed image or a preoperative image. The deformed image refers to an image obtained by deforming the image output by the generation network using the registration relationship output by the registration network. In general, the working principle of the discriminant network is roughly as follows: input an image, and after a series of operations such as convolution, pooling and loss function, it can output the probability of the image being true or false, where true means that the input image is a preoperative image, and false means that the input image is a deformed image. The various embodiments of the present application do not limit the type and network structure of the discriminant network. For example, a GAN network such as a CycleGAN network, a DCGAN network or a DualGAN network can be used. The last layer activation function of the discriminant network can be a function such as sigmoid or softmax, which is used to output a probability value between 0 and 1, indicating the probability that the discriminant network believes that the input image is true. The closer the output probability value is to 1, the more likely the discriminant network believes that the input image is a preoperative image. The closer the output probability value is to 0, the more likely the discriminant network believes that the input image is a deformed image, that is, a de-artifacted image after deformation, or a de-artifacted and metal needle track image after deformation.
[0051] In some embodiments of the present application, the generation network, the registration network and the discrimination network are jointly trained together. Please refer to the following text for the specific training process.
[0052] To understand the joint working principle of the generation network, registration network and discrimination network, please refer to Figure 3 Flowchart shown. Figure 3 is a flowchart of an image artifact removal and registration operation provided by an embodiment of the present application. Figure 3 In the method, the image with artifacts and metal needle tracks collected during surgery is first input into the generative network for artifact and metal needle track removal, and the target intraoperative image with artifacts and metal needle tracks removed is output; then, the target intraoperative image and the preoperative image are input into the registration network for registration to obtain the registration relationship, and the registration relationship is used to deform the target intraoperative image to obtain the deformed image; in addition, the preoperative image and the deformed image can be input into the discriminant network, and the discriminant network distinguishes the two inputs. Using the joint training process of the generative network, the registration network, and the discriminant network described later, each network is continuously iterated and optimized, and finally the discriminant network cannot distinguish the deformed image from the preoperative image. The generative network provides clearer and more accurate image data for the registration network, and the registration network obtains better image registration effect.
[0053] In one embodiment of the present application, the joint training process of the generation network, the registration network and the discrimination network includes:
[0054] The first optimization training process and the second optimization training process are performed alternately until the training end condition is reached.
[0055] The joint training process of the generative network, the registration network and the discriminative network includes two alternately executed optimization training processes, which are respectively recorded as the first optimization training process and the second optimization training process. In the first optimization training process, the parameters of the generative network and the registration network are optimized and updated, and the parameters of the discriminative network are not updated at this time; while in the second optimization training process, the parameters of the discriminative network and the registration network are optimized and updated, and the parameters of the generative network are not updated at this time. When the first optimization training process and the second optimization training process are alternately executed, the parameters and performance of the generative network, the registration network and the discriminative network are continuously optimized until the preset training end conditions are reached. The training end conditions here include: the generative network, the registration network and the discriminative network all reach a balanced state. The actual manifestation of the equilibrium state is usually: the loss function curves of the three networks, the generative network, the registration network and the discriminative network, tend to be stable, with a small fluctuation range. In the equilibrium state, the generative network can generate sufficiently realistic sample images, which are almost indistinguishable from the real images. The discriminative network cannot reliably distinguish between the sample images generated by the generative network and the real images, and the judgment accuracy is close to random guessing, that is, the output probability of the discriminative network is close to 0.5.
[0056] In some embodiments of the present application, the sample image is a de-artifacted image output by the generating network, or a de-artifacted and metal needle tract image output by the generating network, and the real image is a preoperative image without artifacts and metal needle tracts.
[0057] In some embodiments of the present application, the first optimization training process is to generate a network input of a series of intraoperative images with artifacts, output corresponding de-artifacted images, and then input the de-artifacted images and pre-operative images collected in advance into the registration network for registration, and the registration network outputs a deformation field, and the de-artifacted images are deformed using the deformation field to output a deformed image; in addition, the deformed image and the pre-operative image will be input into the discriminant network for discrimination, and the discriminant network will determine whether the input image is a pre-operative image or a deformed image.
[0058] In some other embodiments of the present application, the first optimization training process is to generate a network input of a series of intraoperative images with artifacts and metal needle tracts, output corresponding de-artifacted and metal needle tract images, and then use the de-artifacted and metal needle tract images and the pre-acquired preoperative images to input into the registration network for registration, and the registration network outputs a deformation field, and the deformation field is used to deform the de-artifacted and metal needle tract images to output a deformed image; in addition, the deformed image and the preoperative image will be input into the discriminant network for discrimination, and the discriminant network determines whether the input image is a preoperative image or a deformed image.
[0059] In some embodiments of the present application, the first loss function used in the first optimization training process is determined according to the loss function of the generation network and the loss function of the registration network.
[0060] Considering that the first optimization training process only performs parameter optimization and update on the generation network and the registration network, its loss function is determined according to the loss function of the generation network and the loss function of the registration network, and is represented by the first loss function. For example, the loss function of the generation network and the loss function of the registration network are designed separately, and then the loss functions of the two networks are added together to obtain the first loss function. Exemplarily, the first loss function can be represented by the following formula:
[0061] L Total1 (G, R, D) = L Reg (G, R)+loss_G
[0062] Among them, L Total1 (G, R, D) represents the first loss function. Reg (G, R) represents the loss function of the registration network, loss_G represents the loss function of the generation network, G represents the generation network, R represents the registration network, and D represents the discrimination network.
[0063] Specifically, the loss function of the generation network is determined according to the discrimination result of the discriminant network on the deformed image of the image output by the generation network, and the loss function of the registration network is determined according to the registration result of the registration network on the preoperative image and the image output by the generation network.
[0064] In some embodiments of the present application, the loss function of the generated network is expressed by the following formula:
[0065]
[0066] Where loss_G represents the loss function of the generative network, x represents the intraoperative image with artifacts, y represents the preoperative image, G represents the generative network, R represents the registration network, D represents the discriminative network, G(x) represents the output of the generative network, R(G(x), y) represents the output of the registration network, It indicates deformation processing. It can be seen from the loss function of the generated network that the generated network and the discriminant network are in an adversarial relationship.
[0067] In some embodiments of the present application, the loss function of the registration network is expressed by the following formula:
[0068]
[0069] Among them, L Reg(G, R) represents the loss function of the registration network, G represents the generation network, R represents the registration network, x represents the intraoperative image with artifacts, y represents the preoperative image, G(x) represents the output of the generation network, R(G(x), y) represents the output of the registration network, Indicates deformation processing, L Corr It is a similarity measure for calculating images, including but not limited to mutual information, normalized cross-correlation and mean square error, etc. Smooth It is the regularization of the calculated deformation field, which can make the generated deformation field smooth and reversible.
[0070] As for the second optimization training process, its basic processing is the same as the first optimization training process, that is, the generation network inputs a series of intraoperative images with artifacts, outputs the corresponding de-artifacted images, and then inputs the de-artifacted images and the pre-collected preoperative images into the registration network for registration, and the registration network outputs the deformation field, which is used to deform the de-artifacted images and output the deformed images; in addition, the deformed images and preoperative images are input into the discriminant network for discrimination, and the discriminant network determines whether the input image is a preoperative image or a deformed image. The main difference is that the first optimization training process updates the parameters of the generation network and the registration network, while the second optimization training process updates the parameters of the discriminant network and the registration network.
[0071] In some embodiments of the present application, the second loss function used in the second optimization training process is determined based on the loss function of the discriminant network and the loss function of the registration network.
[0072] Considering that the second optimization training process only optimizes and updates the parameters of the discriminant network and the registration network, its loss function is determined according to the loss function of the discriminant network and the loss function of the registration network, and is expressed by the second loss function. For example, the loss function of the discriminant network and the loss function of the registration network are designed separately, and then the loss functions of the two networks are added together to obtain the second loss function. Exemplarily, the second loss function can be expressed by the following formula:
[0073] L Total2 (G, R, D) = L Reg (G, R)+loss_D
[0074] Among them, L Total2 (G, R, D) represents the second loss function. Reg (G, R) represents the loss function of the registration network, loss_D represents the loss function of the discriminant network, G represents the generation network, R represents the registration network, and D represents the discriminant network.
[0075] Specifically, the loss function of the discriminant network is determined according to the discriminant network's discrimination results on the preoperative image and the deformed image, and the loss function of the registration network is determined according to the registration results of the registration network on the preoperative image and the image output by the generation network.
[0076] In some embodiments of the present application, for the second optimization training process, the loss function of the registration network can be the same as the loss function of the registration network used in the first optimization training process. As for the loss function of the discriminant network, it can be expressed by the following formula:
[0077]
[0078] Where loss_D represents the loss function of the discriminant network, x represents the intraoperative image with artifacts, y represents the preoperative image, G represents the generative network, D represents the discriminant network, R represents the registration network, G(x) represents the output of the generative network, and R(G(x), y) represents the output of the registration network. Indicates deformation processing.
[0079] According to the description above, in the joint training process of the generation network, the registration network and the discriminant network, the optimization and update process of the generation network and the discriminant network is carried out alternately, because the generation network and the discriminant network are in an adversarial relationship. The generation network continuously improves its ability to generate real samples by minimizing the loss function loss_G to achieve the purpose of deceiving the discriminant network. The discriminant network continuously improves its accuracy in distinguishing true and false samples by minimizing the loss function loss_D to prevent being deceived by the generation network. If the generation network and the discriminant network are updated at the same time, the first case is that the update speed of the discriminant network is faster, which will cause the loss function loss_G of the generation network to be in a high value state for a long time, resulting in instability in the training process; the second case is that the update speed of the generation network is faster, which will cause the loss function loss_D of the discriminant network to be in a high value state for a long time, which also causes instability in the training process. Therefore, the embodiment of the present application allows the generation network and the discriminant network to be optimized and updated alternately, so that the optimization and update process of the two can be balanced, and the rate of decline of the loss function will also be relatively stable, so that the training process of the two networks is relatively stable. In general, one network updates its own capabilities to achieve a stronger challenge to the other network, and the other network improves its capabilities based on the updated opponent in the next update. This cycle repeats itself, so that the improvement of the capabilities of the two networks reaches a dynamic balance, which is conducive to the convergence of generative adversarial networks.
[0080] In some embodiments of the present application, the parameters of the discriminant network may be updated once the parameters of the generating network are updated, or the parameters of the discriminant network may be updated once the parameters of the generating network are updated several times. The specific setting method should take into account actual needs, such as which update method can better keep the training process of each network stable, so as to reach the equilibrium state of each network faster.
[0081] The embodiment of the present application allows the generation network, the registration network and the discriminant network to be jointly trained, so that each network can promote and supervise each other, and finally achieve common optimization. For example, the generation network can provide clearer and more accurate image data for the registration network, thereby improving the registration effect of the registration network; as the quality of the input image data improves, the registration accuracy of the registration network will also improve, so the quality of the deformed image obtained after deforming the target intraoperative image using the registration relationship output by the registration network will also be higher. When the deformed image and the preoperative image are input into the discriminant network, it can make it more difficult for the discriminant network to distinguish between the deformed image and the preoperative image, thereby promoting the convergence of each network; the generation network and the discriminant network are in an adversarial relationship and can supervise each other to optimize network performance, and so on.
[0082] In summary, in the embodiments of the present application, when an intraoperative image with artifacts and metal needle tracts is obtained, it will be input into the generation network for artifact and metal needle tract removal processing, thereby obtaining a target intraoperative image after removing the artifacts and metal needle tracts. Then, the target intraoperative image and the preoperative image are input into the registration network for registration processing, thereby obtaining a registration relationship between the target intraoperative image and the preoperative image. Since what is performed here is the intraoperative image after the artifacts and metal needle tracts are removed, the registration with the preoperative image is not affected by the artifacts and metal needle tracts, so the image registration accuracy can be improved.
[0083] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0084] An artifact image registration method is mainly described above, and an artifact image registration device will be described below.
[0085] See also Figure 4 , an embodiment of an artifact image registration device in the embodiment of the present application includes:
[0086] An image acquisition module 401 is used to acquire intraoperative images and preoperative images with artifacts;
[0087] An image artifact removal module 402 is used to input the intraoperative image with artifacts into a pre-trained generation network for artifact removal processing to obtain a target intraoperative image;
[0088] An image registration module 403 is used to input the preoperative image and the target intraoperative image into a pre-trained registration network for registration processing to obtain a registration relationship between the target intraoperative image and the preoperative image;
[0089] Among them, the generation network, the registration network and the discriminant network are jointly trained together. The discriminant network is used to discriminate whether the input image is a deformed image or a preoperative image. The deformed image is obtained by deforming the image output by the generation network using the registration relationship output by the registration network.
[0090] In an implementation of the embodiment of the present application, the artifact image registration device further includes:
[0091] The network training module is used to alternately execute the first optimization training process and the second optimization training process until the training end condition is reached; wherein, in the first optimization training process, the parameters of the generation network and the registration network are optimized and updated; and in the second optimization training process, the parameters of the discrimination network and the registration network are optimized and updated.
[0092] In one implementation of the embodiment of the present application, the first loss function used in the first optimization training process is determined according to the loss function of the generation network and the loss function of the registration network.
[0093] Specifically, the loss function of the generation network is determined according to the discrimination result of the discriminant network on the deformed image of the image output by the generation network, and the loss function of the registration network is determined according to the registration result of the registration network on the preoperative image and the image output by the generation network.
[0094] In one implementation of the embodiment of the present application, the second loss function used in the second optimization training process is determined based on the loss function of the discriminant network and the loss function of the registration network.
[0095] Specifically, the loss function of the discriminant network is determined according to the discriminant network's discrimination results on the preoperative image and the deformed image, and the loss function of the registration network is determined according to the registration results of the registration network on the preoperative image and the image output by the generation network.
[0096] In one implementation of the embodiment of the present application, the intraoperative image with artifacts also includes a metal needle track; the image artifact removal module is specifically used to: input the intraoperative image with artifacts into the generation network for artifact removal and metal needle track processing to obtain a target intraoperative image.
[0097] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the artifact image registration method represented by any of the above embodiments is implemented.
[0098] An embodiment of the present application further provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the artifact image registration method represented by any of the above embodiments.
[0099] Figure 5 Schematic diagram of a terminal device provided by an embodiment of the present application. Figure 5 As shown, the terminal device 5 of this embodiment includes: a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50. When the processor 50 executes the computer program 52, the steps in the embodiments of the above-mentioned artifact image registration method are implemented, for example Figure 1 Alternatively, when the processor 50 executes the computer program 52, the functions of each module / unit in the above-mentioned device embodiments are realized, for example, Figure 4 Functions of modules 401 to 403 are shown.
[0100] The computer program 52 may be divided into one or more modules / units, which are stored in the memory 51 and executed by the processor 50 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 52 in the terminal device 5.
[0101] The processor 50 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0102] The memory 51 may be an internal storage unit of the terminal device 5, such as a hard disk or memory of the terminal device 5. The memory 51 may also be an external storage device of the terminal device 5, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 5. Further, the memory 51 may also include both an internal storage unit and an external storage device of the terminal device 5. The memory 51 is used to store the computer program and other programs and data required by the terminal device. The memory 51 may also be used to temporarily store data that has been output or is to be output.
[0103] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0105] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0106] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0107] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0108] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.
[0109] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0110] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0111] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for artifact image registration, characterized in that: include: Acquire intraoperative and preoperative images with artifacts; Inputting the artifact-bearing intraoperative image into a pre-trained generation network for artifact removal processing to obtain a target intraoperative image; Inputting the preoperative image and the target intraoperative image into a pre-trained registration network for registration processing to obtain a registration relationship between the target intraoperative image and the preoperative image; The generation network, the registration network and the discrimination network are jointly trained together. The discrimination network is used to discriminate whether the input image is a deformed image or a preoperative image. The deformed image is obtained by deforming the image output by the generation network using the registration relationship output by the registration network.
2. The method according to claim 1, characterized in that The joint training process of the generation network, the registration network and the discrimination network includes: Alternately executing the first optimization training process and the second optimization training process until a training end condition is reached; In which, in the first optimization training process, the parameters of the generation network and the registration network are optimized and updated; in the second optimization training process, the parameters of the discrimination network and the registration network are optimized and updated.
3. The method according to claim 2, characterized in that The first loss function used in the first optimization training process is determined according to the loss function of the generation network and the loss function of the registration network.
4. The method according to claim 3, characterized in that The loss function of the generation network is determined according to the discrimination result of the discriminant network on the deformed image of the image output by the generation network, and the loss function of the registration network is determined according to the registration result of the registration network on the preoperative image and the image output by the generation network.
5. The method according to claim 2, characterized in that The second loss function used in the second optimization training process is determined according to the loss function of the discriminant network and the loss function of the registration network.
6. The method according to claim 5, characterized in that The loss function of the discriminant network is determined according to the discriminant network's discriminant results on the preoperative image and the deformed image, and the loss function of the registration network is determined according to the registration results of the registration network on the preoperative image and the image output by the generation network.
7. The method according to any one of claims 1 to 6, characterized in that: The intraoperative image with artifacts also includes a metal needle track; the intraoperative image with artifacts is input into a generation network for artifact removal processing to obtain a target intraoperative image, including: The intraoperative image with artifacts is input into the generation network for artifact removal and metal needle track processing to obtain the target intraoperative image.
8. An artifact image registration device, characterized in that: include: An image acquisition module, used for acquiring intraoperative images and preoperative images with artifacts; An image artifact removal module, used for inputting the intraoperative image with artifacts into a pre-trained generation network for artifact removal processing to obtain a target intraoperative image; An image registration module, used for inputting the preoperative image and the target intraoperative image into a pre-trained registration network for registration processing to obtain a registration relationship between the target intraoperative image and the preoperative image; The generation network, the registration network and the discrimination network are jointly trained together. The discrimination network is used to discriminate whether the input image is a deformed image or a preoperative image. The deformed image is obtained by deforming the image output by the generation network using the registration relationship output by the registration network.
9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the artifact image registration method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the artifact image registration method according to any one of claims 1 to 7 is implemented.