Medical image registration method based on dynamic network generation and related device

Abstract

The invention discloses a medical image registration method based on dynamic network generation and a related device. The method comprises the following steps: acquiring an intra-operative image, and inputting the intra-operative image into a meta-generation network with a shared feature encoder and a weight prediction head to extract multi-scale features and dynamically predict a registration weight; instantiating an implicit neural representation network as an intraoperative registration network by using the weight; registering the intra-operative image and the pre-operative image through the network to obtain a deformation field; and finally, calculating image similarity and deformation field regularization measurement according to a registration result, and feeding back to the meta-generation network to optimize the weight. According to the method, real-time and adaptive network generation of intra-operative images of different cases is realized, challenges of intra-operative image degradation, shielding, deformation mutation and the like are effectively overcome, the robustness and precision of registration are improved, and reliable support is provided for surgical navigation and artificial intelligence biological feature recognition.

Description

Technical Field

[0001] This invention relates to the fields of medicine, artificial intelligence and robotics, specifically a medical image registration method and related apparatus based on dynamic network generation. Background Technology

[0002] In image-guided surgery, real-time and accurate registration of preoperative high-resolution images (such as CT / MRI) with intraoperative images (such as ultrasound / endoscopy) is crucial for navigation and visualization. Traditional methods, such as biometric recognition software, are typically based on pre-trained uniform models, which struggle to adapt to highly personalized scenarios involving different patient anatomy, intraoperative image quality degradation, and abrupt tissue deformation. Existing technologies, such as learning-based registration networks, while possessing some generalization ability, have fixed model parameters, leading to a significant decrease in registration accuracy when faced with intraoperative ultrasound speckle noise, optical occlusion, or instrument artifacts. Furthermore, while registration methods based on implicit neural representations can perform continuous coordinate mapping, they are usually designed for single image pairs, making it difficult to achieve rapid adaptation under complex and variable intraoperative conditions. Therefore, how to construct an artificial intelligence system capable of generating customized registration networks in real time based on the features of individual intraoperative images, achieving robust and accurate intraoperative dynamic registration, remains a pressing technical challenge in this field. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention relates to a medical image registration method and related apparatus based on dynamic network generation, including but not limited to a medical image registration device, electronic device, computer-readable storage medium, and computer program product based on dynamic network generation.

[0004] Firstly, a medical image registration method based on dynamic network generation is provided, including the following steps: S1. Acquire intraoperative images and input the intraoperative images into a meta-generative network to obtain the registration weights of the intraoperative registration network; the meta-generative network has a shared feature encoder and a weight prediction head; the feature encoder extracts multi-scale feature maps from the intraoperative images; the weight prediction head is connected to the feature encoder, receives the multi-scale feature maps, and outputs the registration weights; S2. Instantiate the intraoperative registration network based on the registration weights; the intraoperative registration network is a hidden neural representation network; S3. The intraoperative image and the preoperative image are registered using the instantiated intraoperative registration network to obtain the registration result; S4. Based on the registration result, calculate the image similarity metric and the deformation field regularization metric; feed the image similarity metric and the deformation field regularization metric back to the meta-generation network to adjust the registration weights.

[0005] In conjunction with any embodiment of this application, the training process of the meta-generative network includes the following steps: S01. During the forward propagation of the registration network, simulated in-surgery degraded noise perturbation is injected into the input or intermediate features of the registration network; S02. Calculate the loss between the output of the registration network after noise injection and the actual deformation field; S03. Update the parameters of the meta-generator network based on the loss.

[0006] In any embodiment of this application, the noise disturbance includes ultrasound speckle noise, partial occlusion of optical markers, abrupt changes in tissue deformation, or artifacts introduced by instruments.

[0007] In any embodiment of this application, the weight prediction head uses the global feature vector obtained by aggregating the multi-scale feature maps as a condition to generate a set of modulation parameters through a linear layer sequence; the modulation parameters dynamically scale and shift the predefined convolutional kernel weights in the intraoperative registration network.

[0008] In any embodiment of this application, the registration of the intraoperative image with the preoperative image includes: Obtain the spatial coordinates of the preoperative image; The spatial coordinate points and the multi-scale feature map are input into the intraoperative registration network; The intraoperative registration network processes the spatial coordinate points and the multi-scale feature map to output the corresponding displacement vector; Based on the spatial coordinates and the displacement vector, a deformation field is generated from the preoperative image to the intraoperative image.

[0009] Secondly, a medical image registration device based on dynamic network generation, characterized in that it includes: Data input unit: used to acquire intraoperative images, input the intraoperative images into the meta-generative network, and obtain the registration weights of the intraoperative registration network; the meta-generative network has: Shared feature encoder: used to extract multi-scale feature maps from the intraoperative images; Weight prediction head: used to connect to the feature encoder, receive the multi-scale feature map and output the registration weights; Network generation unit: used to instantiate the intraoperative registration network based on the registration weights; the intraoperative registration network is a hidden neural representation network; Registration working unit: used to register the intraoperative image with the preoperative image through the instantiated intraoperative registration network to obtain the registration result; Meta-network update unit: used to calculate image similarity metric and deformation field regularization metric based on the registration result; also used to feed the image similarity metric and deformation field regularization metric back to the meta-generation network to adjust the registration weights.

[0010] Thirdly, an electronic device is provided, comprising: a processor, a communication module, a sensor, a user interface, and a storage unit, wherein the storage unit is used to store computer program code, the program code including computer instructions. When the processor executes these instructions, the electronic device performs the methods described in the second aspect above and any of its embodiments.

[0011] Fourthly, another electronic device is provided, comprising: a processor, a wireless communication module, a touchscreen, a speaker, and a storage unit, wherein the storage unit is used to store computer program code, the program code including computer instructions. When the processor executes these instructions, the electronic device performs the methods described in the second aspect above and any of its embodiments.

[0012] Fifthly, a computer-readable storage medium is provided, wherein a computer program is stored, the program comprising program instructions. When these instructions are executed by a processor, the processor performs the methods described in the second aspect above and any of its embodiments.

[0013] In a sixth aspect, a computer program product is provided, the computer program product comprising a computer program or instructions. When the computer program or instructions are executed on a computer, the computer will perform the methods described in the second aspect above and any of its embodiments.

[0014] This application presents a medical image registration method and related apparatus based on dynamic network generation, compared to existing technologies. Addressing the limitations of existing pre-trained registration models in adapting to dynamic scenarios such as intraoperative image degradation, occlusion, and tissue deformation, this invention introduces a meta-generative network to dynamically predict and generate the weight parameters of the implicit neural representation registration network based on the input intraoperative image, achieving adaptive network instantiation for single images. During the registration process, the network weights are further optimized online through feedback from image similarity and deformation field regularization metrics. This method overcomes the bottleneck of fixed models' insufficient adaptability to complex intraoperative conditions, significantly improving the robustness, accuracy, and real-time performance of registration, and providing more reliable artificial intelligence and biometric recognition technology support for surgical navigation. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to illustrate the technical solutions of this application.

[0017] Figure 1 A flowchart of a medical image registration method based on dynamic network generation is provided for an embodiment of this application.

[0018] Figure 2 This is a schematic diagram of a medical image registration device based on dynamic network generation, provided as an embodiment of this application.

[0019] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To enable those skilled in the art to more fully understand the technical solutions of this application, the technical solutions of this application will be explained in detail and clearly with reference to the accompanying drawings. It should be particularly noted that the described embodiments are only some examples of this application and do not represent all of them. Based on these embodiments, those skilled in the art can directly deduce all other possible implementation schemes without creative thinking, and these are also included within the protection scope of this application.

[0021] In the specification, claims, and related drawings of this application, the terms "first," "second," etc., are used only to distinguish different elements and do not imply any specific order. Furthermore, the use of "comprising" and "having," and their variations, indicates non-exclusive inclusion. This means that if a process, method, system, product, or device comprises a series of steps or components, it indicates that the process, method, system, product, or device is not limited to the listed steps or components and may also include other steps or components not listed, or other inherent steps or units thereof.

[0022] The term "embodiment" as used herein refers to any instance combining a particular feature, structure, or characteristic, which may be at least one embodiment of this application. The "embodiments" mentioned herein do not necessarily refer to the same specific case, nor do they imply that they are independent or exclusive alternatives. Those skilled in the art will understand that the embodiments described herein can be used with other embodiments. It should be clarified that in this application, "at least one" includes one or more instances, "multiple" means two or more instances, and "at least two" refers to two or more instances.

[0023] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The embodiments of this application will now be described with reference to the accompanying drawings.

[0024] Please see Figure 1 , Figure 1 A flowchart of a medical image registration method based on dynamic network generation is provided for an embodiment of this application.

[0025] 101. Data Acquisition: Acquire intraoperative images and input them into the meta-generative network to obtain the registration weights of the intraoperative registration network.

[0026] In this embodiment, the sources of the intraoperative images include, but are not limited to, intraoperative ultrasound, endoscopy, or magnetic resonance imaging modalities.

[0027] In this embodiment, the meta-generation network has: Shared feature encoder: used to extract multi-scale feature maps from the intraoperative images; Weight prediction head: used to connect to the feature encoder, receive the multi-scale feature map and output the registration weights.

[0028] In this embodiment, the training process of the meta-generative network includes the following steps: S01. During the forward propagation of the registration network, simulated in-surgery degraded noise perturbation is injected into the input or intermediate features of the registration network; S02. Calculate the loss between the output of the registration network after noise injection and the actual deformation field; S03. Update the parameters of the meta-generator network based on the loss.

[0029] In this embodiment, the noise disturbance includes ultrasound speckle noise, partial occlusion of optical markers, abrupt changes in tissue deformation, or artifacts introduced by instruments.

[0030] In this embodiment, the weight prediction head uses the global feature vector obtained by aggregating the multi-scale feature maps as a condition to generate a set of modulation parameters through a linear layer sequence.

[0031] In this embodiment, the modulation parameters dynamically scale and shift the predefined convolutional kernel weights in the intraoperative registration network.

[0032] In another possible implementation, the intraoperative image may undergo a preprocessing process, including noise reduction, contrast enhancement, or artifact suppression, before being input into the meta-generator network, in order to improve the robustness of subsequent feature extraction.

[0033] In another possible implementation, the feature encoder may employ a multi-branch structure with skip connections to extract and fuse features with different receptive fields in parallel from the intraoperative images in order to construct the multi-scale feature map more comprehensively.

[0034] 102. Network generation: Based on the registration weights, instantiate the intraoperative registration network; the intraoperative registration network is an implicit neural representation network.

[0035] In this embodiment, the implicit neural representation network uses a multilayer perceptron as its main structure, and the weights of each layer are dynamically generated by the corresponding modulation parameters in the registration weights.

[0036] In another possible implementation, attention-based networks or graph neural networks are used as the main structure of the implicit neural representation network, and its network parameters are also dynamically generated by the registration weights.

[0037] In another possible implementation, the registration weights are combined with predefined base weights to achieve the instantiation.

[0038] 103. Image registration: The intraoperative image and the preoperative image are registered using the instantiated intraoperative registration network to obtain the registration result.

[0039] In this embodiment, the registration of the intraoperative image with the preoperative image includes: Obtain the spatial coordinates of the preoperative image; The spatial coordinate points and the multi-scale feature map are input into the intraoperative registration network; The intraoperative registration network processes the spatial coordinate points and the multi-scale feature map to output the corresponding displacement vector; Based on the spatial coordinates and the displacement vector, a deformation field is generated from the preoperative image to the intraoperative image.

[0040] In this embodiment, the intraoperative registration network uses the coordinate point as the query location and combines it with the contextual information provided by the multi-scale feature map to perform feature query and fusion.

[0041] In this embodiment, the intraoperative registration network processes the query information in a feedforward manner and finally outputs a corresponding three-dimensional or two-dimensional displacement vector for each spatial coordinate point.

[0042] In this embodiment, the intraoperative registration network can be used in the end-to-end modular network of the patent "A Dynamic Compensation Method and Related Device for Surgical Robots Based on Multimodal Real-time 4D Digital Twin" (Publication No.: CN120899400A).

[0043] In another possible implementation, the contextual information input to the intraoperative registration network can also be fused with corresponding features extracted from the preoperative images to provide richer cross-modal matching cues.

[0044] In another possible implementation, the deformation field can be generated by post-processing methods such as spline interpolation or optical flow propagation to densify and smooth the sparse displacement field composed of the displacement vectors.

[0045] 104. Meta-network update: Based on the registration results, calculate the image similarity metric and the deformation field regularization metric; feed the image similarity metric and the deformation field regularization metric back to the meta-generation network to adjust the registration weights.

[0046] In this embodiment, the image similarity metric is used to evaluate the alignment degree between the registered images. The metric methods include, but are not limited to, normalized cross-correlation, mutual information, or feature-based similarity metrics.

[0047] In this embodiment, the deformation field regularization metric is used to constrain the smoothness and rationality of the deformation field. The metric methods include, but are not limited to, diffusion regularization, bending energy, or Jacobian determinant constraints.

[0048] In this embodiment, the image similarity metric and the deformation field regularization metric are weighted and summed to form the total loss. Based on this total loss, the parameters in the meta-generator network are fine-tuned online through gradient backpropagation or meta-learning strategies, thereby achieving dynamic adjustment and optimization of the registration weights.

[0049] In another possible implementation, the composition of the total loss may incorporate additional clinical constraint measures, such as a measure based on anatomical landmark alignment error, to make the registration results more clinically relevant.

[0050] In another possible implementation, the online fine-tuning may also employ strategies based on reinforcement learning or Bayesian optimization to explore and adjust the registration weights more efficiently.

[0051] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0052] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0053] Please see Figure 2 , Figure 2 This application provides a schematic diagram of a medical image registration device based on dynamic network generation. The image registration device 1 includes: a data input unit 11, a network generation unit 12, a registration working unit 13, and a meta-network update unit 14. Specifically: Data input unit 11: used to acquire intraoperative images, input the intraoperative images into the meta-generative network, and obtain the registration weights of the intraoperative registration network; the meta-generative network has: Shared feature encoder: used to extract multi-scale feature maps from the intraoperative images; Weight prediction head: used to connect to the feature encoder, receive the multi-scale feature map and output the registration weights; Network generation unit 12: used to instantiate the intraoperative registration network based on the registration weights; the intraoperative registration network is an implicit neural representation network; Registration working unit 13: used to register the intraoperative image with the preoperative image through the instantiated intraoperative registration network to obtain the registration result; Meta-network update unit 14: is used to calculate image similarity metric and deformation field regularization metric based on the registration result; and is also used to feed the image similarity metric and deformation field regularization metric back to the meta-generation network to adjust the registration weights.

[0054] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0055] Please see Figure 3 , Figure 3 A schematic diagram of the hardware architecture of an electronic device according to an embodiment of this application is shown. The electronic device 2 mainly consists of a processor 21 and a memory 22. In addition, the device may also include an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are interconnected via connecting components, which can be various interfaces, data lines, or communication buses, etc., and are not specifically specified in this embodiment.

[0056] Processor 21 may be one or more graphics processing units (GPUs). If processor 21 is a GPU, the GPU may be single-core or multi-core. Alternatively, processor 21 may also be a processor group consisting of multiple GPUs interconnected via one or more buses. Furthermore, the processor may be other types of processors, which are not specifically limited in this embodiment.

[0057] The memory 22 is designed to store the instructions of a computer program and various program codes required to execute the present application. Optionally, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or optical disc read-only memory (CD-ROM), which are used to store related instructions and data.

[0058] Input device 23 is used to input data and / or signals, and output device 24 is used to output data and / or signals. Input device 23 and output device 24 can be independent devices or an integrated device.

[0059] It should be understood that, in the embodiments of this application, the memory 22 can store not only related instructions but also related data. The embodiments of this application do not specify the specific data content stored in the memory.

[0060] It should be understood that Figure 3 This illustration only shows a simplified design of an electronic device. In actual use, the electronic device may also include other necessary components, such as different numbers of input / output devices, processors, memory, etc. All electronic devices capable of implementing the embodiments of this application are within the protection scope of this application.

[0061] Those skilled in the art will recognize that the components and algorithm steps of the various examples described in the embodiments disclosed herein can be implemented by electronic hardware or by a combination of computer software and electronic hardware. Whether these functions are implemented through hardware or software will be determined based on the specific application requirements and design constraints of the technical solution. Those skilled in the art can adopt different implementation methods according to the needs of each specific application, but such implementation should not be considered as exceeding the scope of protection of this application.

[0062] Those skilled in the art should understand that, for ease of description and simplification, the specific operational procedures of the aforementioned systems, devices, and components can be referred to the corresponding steps in the preceding method embodiments, and will not be repeated here. Furthermore, those skilled in the art should also understand that each embodiment in this application has its own focus, and for ease of description and simplification, the same or similar content may not be repeated in different embodiments. Therefore, if a part is not mentioned or not described in detail in a certain embodiment, it can be referred to the relevant description in other embodiments.

[0063] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can also be implemented through other means. For example, the described apparatus embodiments are merely exemplary, and the division of the units therein is only a logical functional division; different division methods may exist in actual implementation. For example, multiple units or components may be merged or integrated into another system, or certain features may be omitted, or certain steps may not be performed. Furthermore, the interconnections shown or discussed, whether direct or indirect, whether coupling or communication connections, may be implemented electrically, mechanically, or otherwise through interfaces, devices, or units.

[0064] Units described as independent components may or may not actually be physically separate; parts presented as units may or may not be physical entities, meaning they may be concentrated in one location or distributed across multiple network nodes. Depending on the actual needs, some or all of these units can be selected to achieve the objectives of this embodiment.

[0065] Furthermore, in the various embodiments of this application, each functional unit can be integrated into a single processing unit, exist independently, or two or more units can be merged into one unit. In the foregoing embodiments, the relevant functions can be fully or partially implemented through software, hardware, firmware, or any combination thereof. If software implementation is chosen, it can be implemented entirely or partially in the form of a computer program product. This computer program product contains one or more computer instructions. When these instructions are loaded and executed on a computer, they will produce all or part of the processes or functions described in the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. These computer instructions can be stored in computer-readable storage media or transmitted through such media. Computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, DSL) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any computer-accessible available medium, or a data storage facility such as a server or data center that integrates one or more available media. These available media may include magnetic media (such as floppy disks, hard disks, and magnetic tapes), optical media (such as DVDs), semiconductor media (such as SSDs), etc. Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by computer program instructions and related hardware, and these programs can be stored in computer-readable storage media. When these programs are executed, they will contain the processes of the above method embodiments. The aforementioned storage media include, but are not limited to, various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A medical image registration method based on dynamic network generation, characterized in that, Includes the following steps: S1. Acquire intraoperative images and input the intraoperative images into a meta-generative network to obtain the registration weights of the intraoperative registration network; the meta-generative network has a shared feature encoder and a weight prediction head; the feature encoder extracts multi-scale feature maps from the intraoperative images; the weight prediction head is connected to the feature encoder, receives the multi-scale feature maps, and outputs the registration weights; S2. Instantiate the intraoperative registration network based on the registration weights; the intraoperative registration network is a hidden neural representation network; S3. The intraoperative image and the preoperative image are registered using the instantiated intraoperative registration network to obtain the registration result; S4. Based on the registration results, calculate the image similarity metric and the deformation field regularization metric; The image similarity metric and the deformation field regularization metric are fed back to the meta-generation network to adjust the registration weights.

2. The method according to claim 1, characterized in that, The training process of the meta-generative network includes the following steps: S01. During the forward propagation of the registration network, simulated in-surgery degraded noise perturbation is injected into the input or intermediate features of the registration network; S02. Calculate the loss between the output of the registration network after noise injection and the actual deformation field; S03. Update the parameters of the meta-generator network based on the loss.

3. The method according to claim 2, characterized in that, The noise disturbances include ultrasound speckle noise, partial obstruction of optical markers, abrupt changes in tissue deformation, or artifacts introduced by instruments.

4. The method according to claim 1, characterized in that, The weight prediction head uses the global feature vector obtained by aggregating the multi-scale feature maps as a condition to generate a set of modulation parameters through a linear layer sequence; the modulation parameters dynamically scale and shift the predefined convolutional kernel weights in the intraoperative registration network.

5. The method according to claim 1, characterized in that, The registration of the intraoperative images with the preoperative images includes: Obtain the spatial coordinates of the preoperative image; The spatial coordinate points and the multi-scale feature map are input into the intraoperative registration network; The intraoperative registration network processes the spatial coordinate points and the multi-scale feature map to output the corresponding displacement vector; Based on the spatial coordinates and the displacement vector, a deformation field is generated from the preoperative image to the intraoperative image.

6. A medical image registration device based on dynamic network generation, characterized in that, include: Data input unit: used to acquire intraoperative images, input the intraoperative images into the meta-generative network, and obtain the registration weights of the intraoperative registration network; The meta-generating network has: Shared feature encoder: used to extract multi-scale feature maps from the intraoperative images; Weight prediction head: used to connect to the feature encoder, receive the multi-scale feature map and output the registration weights; Network generation unit: used to instantiate the intraoperative registration network based on the registration weights; the intraoperative registration network is a hidden neural representation network; Registration working unit: used to register the intraoperative image with the preoperative image through the instantiated intraoperative registration network to obtain the registration result; Metanet update unit: used to calculate image similarity metric and deformation field regularization metric based on the registration result; It is also used to feed back the image similarity metric and the deformation field regularization metric to the meta-generation network to adjust the registration weights.

7. An electronic device, characterized in that, include: A processor and a storage unit for storing computer program code, the code containing computer instructions, wherein when the processor executes the computer instructions, the electronic device performs the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program containing program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1 to 5.

9. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method described in any one of claims 1 to 5.

Images