Image registration methods, registration network training methods, devices, equipment and media

Abstract

This application discloses an image registration method, a registration network training method, apparatus, device, and medium, belonging to the field of artificial intelligence technology. In the embodiments of this application, after registering images using an image registration network, a method is provided to further optimize the initial registration result. This allows for more accurate registration results when the initial result is inaccurate, overcoming the problem of insufficient generalization ability of the image registration network on new datasets. During this optimization process, transformation parameters are continuously adjusted to increase the similarity between the reference image and the second image, thereby obtaining a more accurate registration result.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image registration method, a registration network training method, apparatus, device and medium. Background Technology

[0002] Image registration refers to the process of matching and superimposing two or more images acquired at different times, by different sensors (imaging devices), or under different conditions (weather, illumination, camera position and angle, etc.).

[0003] Currently, image registration methods typically involve pre-training an image registration network, inputting the image to be registered and a reference image into the network, which then performs the image registration and outputs the registration result.

[0004] In the methods described above, the image registration network is typically trained on a specific image set, which results in poor generalization ability on new datasets. Furthermore, when using these methods, the input images to be registered and the reference images are not from the training image set, thus the obtained registration results may be inaccurate. Summary of the Invention

[0005] This application provides an image registration method, a registration network training method, an apparatus, a device, and a medium, which improves the accuracy of image registration. The technical solution is as follows:

[0006] On the one hand, an image registration method is provided, the method comprising:

[0007] Based on the image registration network, the reference image and the first image are registered to obtain the initial registration result;

[0008] In response to the initial registration result satisfying the condition, the initial transformation parameters corresponding to the initial registration result are updated;

[0009] The first image is transformed based on the updated transformation parameters to obtain the second image;

[0010] In response to the similarity between the reference image and the second image meeting the condition, the transformation parameters are updated, and transformation processing and transformation parameters are continued based on the updated transformation parameters until the target condition is met, and the target registration result is obtained.

[0011] On the one hand, an image registration network training method is provided, the method comprising:

[0012] Obtain sample image pairs, wherein the sample image pair includes a sample reference image and a sample first image, and the sample image pair carries image type information;

[0013] Based on the image type information carried by the sample image pair, image registration is performed on the sample image pair based on the branch corresponding to the image type information in the image registration network to obtain the transformation parameters;

[0014] Based on the transformation parameters, the network parameters of the image registration network are trained.

[0015] On one hand, an image registration device is provided, the device comprising:

[0016] The registration module is used to perform image registration between the reference image and the first image based on the image registration network to obtain the initial registration result.

[0017] The update module is used to update the initial transformation parameters corresponding to the initial registration result in response to the initial registration result meeting the conditions.

[0018] A transformation module is used to transform the first image based on the updated transformation parameters to obtain a second image;

[0019] The update module and the transformation module are further configured to update the transformation parameters in response to the similarity condition between the reference image and the second image, and continue to perform transformation processing and update the transformation parameters based on the updated transformation parameters until the target condition is met, thereby obtaining the target registration result.

[0020] In some embodiments, the update module is configured to perform any of the following:

[0021] In response to receiving an optimization instruction for the initial registration result, the step of updating the initial transformation parameters corresponding to the initial registration result is executed;

[0022] In response to the fact that the similarity between the initial second image registered with the first image and the reference image in the initial registration result is less than a first similarity threshold, the step of updating the initial transformation parameters corresponding to the initial registration result is performed.

[0023] In some embodiments, the image registration network includes a first branch and a second branch, wherein the first branch is used for rigid body registration and the second branch is used for non-rigid body registration;

[0024] The registration module is used for:

[0025] In response to the rigid body registration command, image registration is performed on the reference image and the first image based on the first branch of the image registration network to obtain an initial registration result. The initial transformation parameter corresponding to the initial registration result is the initial affine matrix.

[0026] In response to the non-rigid registration instruction, image registration is performed on the reference image and the first image based on the second branch of the image registration network to obtain an initial registration result. The initial transformation parameters corresponding to the initial registration result are the initial vector field.

[0027] In some embodiments, the update module is configured to perform any of the following:

[0028] In response to the initial registration result satisfying the condition, the initial affine matrix corresponding to the initial registration result is updated;

[0029] In response to the initial registration result satisfying the condition, the initial vector field corresponding to the initial registration result is updated;

[0030] The transformation module is used to perform any of the following:

[0031] The second image is obtained by performing an affine transformation on the first image based on the updated affine matrix.

[0032] The first image is subjected to strain transformation based on the updated vector field to obtain the second image.

[0033] In some embodiments, the update module is configured to update the transformation parameters in response to the similarity between the reference image and the second image being less than a second similarity threshold.

[0034] In some embodiments, the process of obtaining the similarity between the reference image and the second image includes:

[0035] Based on the modal information of the reference image and the second image, the similarity between the reference image and the second image is obtained using the similarity acquisition method corresponding to the modal information.

[0036] In some embodiments, obtaining the similarity between the reference image and the second image based on the modal information of the reference image and the second image, using a similarity acquisition method corresponding to the modal information, includes:

[0037] In response to the fact that the modal information of the reference image and the modal information of the second image are the same, the normalized cross-correlation coefficient between the sub-image set of the reference image and the second image is obtained, and the normalized cross-correlation coefficient is the similarity between the reference image and the second image;

[0038] In response to the difference between the modal information of the reference image and the modal information of the second image, normalized mutual information between the sub-image set of the reference image and the second image is obtained, and the similarity between the reference image and the second image is obtained based on the normalized mutual information.

[0039] In some embodiments, obtaining the similarity between the reference image and the second image based on the normalized mutual information includes:

[0040] In response to the transformation parameters being a vector field, the transformation processing of the first image is a strain transformation, and a smoothing constraint value is obtained based on the vector field;

[0041] The similarity between the reference image and the second image is obtained based on the smoothing constraint value and the normalized mutual information.

[0042] On one hand, an image registration device is provided, the device comprising:

[0043] The acquisition module is used to acquire sample image pairs, wherein the sample image pair includes a sample reference image and a sample first image, and the sample image pair carries image type information;

[0044] The registration module is used to perform image registration on the sample image pair based on the image type information carried by the sample image pair and the branch corresponding to the image type information in the image registration network to obtain transformation parameters;

[0045] The training module is used to train the network parameters of the image registration network based on the transformation parameters.

[0046] In some embodiments, the training module is configured to perform any of the following:

[0047] The sample first image is processed based on the transformation parameters to obtain the target second image. The network parameters of the image registration network are trained based on the similarity between the sample reference image and the target second image.

[0048] Based on the transformation parameters and the target transformation parameters carried by the sample images, the network parameters of the image registration network are trained.

[0049] In some embodiments, the image type information includes a first image type and a second image type;

[0050] The registration module is used for:

[0051] In response to the image type information carried by the sample image pair being a first image type, image registration is performed on the sample image pair based on the first branch in the image registration network to obtain an affine matrix, wherein the affine matrix is ​​a transformation parameter, and the first branch is used for rigid body registration.

[0052] In response to the image type information carried by the sample image pair being a second image type, image registration is performed on the sample image pair based on the second branch in the image registration network to obtain a vector field, wherein the vector field is a transformation parameter, and the second branch is used for non-rigid body registration.

[0053] In some embodiments, the acquisition module is further configured to acquire batch data in response to an update instruction from the image registration network. The batch data includes target image pairs and sample image pairs. The target image pairs are image pairs optimized from the initial registration result output by the image registration network during its use.

[0054] The training module is also used to train the trained image registration network based on the batch data to obtain an updated image registration network.

[0055] On one hand, an electronic device is provided, comprising one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement various optional implementations of the above-described image registration method or image registration network training method.

[0056] On the one hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the storage medium, and the at least one computer program is loaded and executed by a processor to implement various optional implementations of the above-described image registration method or image registration network training method.

[0057] In one aspect, a computer program product or computer program is provided, the computer program product or computer program comprising one or more lines of program code stored in a computer-readable storage medium. One or more processors of an electronic device read the one or more lines of program code from the computer-readable storage medium, and the one or more processors execute the one or more lines of program code, causing the electronic device to perform an image registration method or an image registration network training method according to any of the above possible embodiments.

[0058] This application provides a method to further optimize the initial registration result after registering the images using an image registration network. This allows for more accurate registration results when the initial registration is inaccurate, overcoming the problem of insufficient generalization ability of the image registration network on new datasets. During this optimization process, the transformation parameters are continuously adjusted to increase the similarity between the reference image and the second image, thereby obtaining a more accurate registration result. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a schematic diagram of the implementation environment of an image registration method provided in an embodiment of this application;

[0061] Figure 2 This is a flowchart of an image registration method provided in an embodiment of this application;

[0062] Figure 3 This is a flowchart of an image registration network training method provided in an embodiment of this application;

[0063] Figure 4 This is a schematic diagram of the structure of an image registration device provided in an embodiment of this application;

[0064] Figure 5 This is a schematic diagram of an image registration network provided in an embodiment of this application;

[0065] Figure 6 This is a flowchart of an image registration network training method provided in an embodiment of this application;

[0066] Figure 7 This is a flowchart of an image registration method provided in an embodiment of this application;

[0067] Figure 8 This is a schematic diagram of the structure of an image registration device provided in an embodiment of this application;

[0068] Figure 9 This is a schematic diagram of the structure of an image registration network training device provided in an embodiment of this application;

[0069] Figure 10 This is a structural block diagram of an electronic device provided in an embodiment of this application;

[0070] Figure 11 This is a structural block diagram of a terminal provided in an embodiment of this application;

[0071] Figure 12 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0073] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have substantially the same function and purpose. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the various examples described, a first image is referred to as a second image, and similarly, a second image is referred to as a first image. Both the first image and the second image are images, and in some cases, they are separate and distinct images.

[0074] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple data packets means two or more data packets.

[0075] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0076] It should also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" describes an association between related objects, indicating the existence of three relationships; for example, A and / or B means: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects are in an "or" relationship.

[0077] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0078] It should also be understood that determining B based on A does not mean determining B solely based on A, but also based on A and / or other information.

[0079] It should also be understood that the term “comprising” (also referred to as “inCludes”, “inCluding”, “Comprises”, and / or “Comprising”) as used in this specification specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0080] It should also be understood that the term "if" can be interpreted as meaning "when" or "upon" or "in response to determination" or "in response to detection." Similarly, depending on the context, the phrases "if determination..." or "if detection [the stated condition or event]" can be interpreted as meaning "when determination..." or "in response to determination..." or "when detection [the stated condition or event]" or "in response to detection [the stated condition or event]."

[0081] This application relates to artificial intelligence technology, specifically image registration based on artificial intelligence technology. A brief introduction to artificial intelligence is given below.

[0082] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0083] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0084] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0085] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0086] The solutions provided in this application relate to technologies such as image processing in computer vision of artificial intelligence, and are specifically illustrated through the following embodiments.

[0087] The implementation environment of this application is described below.

[0088] Figure 1 This is a schematic diagram of an implementation environment for an image registration method provided in this application embodiment. The implementation environment includes a terminal 101, or it includes a terminal 101 and an image registration platform 102. The terminal 101 is connected to the image registration platform 102 via a wireless network or a wired network.

[0089] Terminal 101 is at least one of a smartphone, game console, desktop computer, tablet computer, e-book reader, MP3 (Moving Picture Experts Group Audio Layer III) player or MP4 (Moving Picture Experts Group Audio Layer IV) player, and laptop computer. Terminal 101 has an application installed and running that supports image registration, such as a system application, instant messaging application, news push application, shopping application, online video application, or social application.

[0090] For example, the terminal 101 has image acquisition and image processing functions, processes the acquired images, and executes corresponding functions based on the processing results. Optionally, the terminal 101 may have image processing functions to process images acquired by other devices. The terminal 101 performs this work independently and also provides data services to it through the image registration platform 102. This application embodiment does not limit this aspect.

[0091] Image registration platform 102 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. Image registration platform 102 provides background services for applications supporting image registration. Optionally, image registration platform 102 undertakes the primary processing task, and terminal 101 undertakes secondary processing task; or, image registration platform 102 undertakes secondary processing task, and terminal 101 undertakes primary processing task; or, image registration platform 102 or terminal 101 each undertakes processing task independently. Alternatively, image registration platform 102 and terminal 101 collaborate using a distributed computing architecture.

[0092] Optionally, the image registration platform 102 includes at least one server 1021 and a database 1022. The database 1022 is used to store data. In this embodiment, the database 1022 stores sample image pairs or image pairs to be processed, providing data services to at least one server 1021.

[0093] A server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. A terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these.

[0094] Those skilled in the art will know that the number of terminals 101 and servers 1021 may be more or less. For example, there may be only one terminal 101 or server 1021, or there may be dozens or hundreds of terminals 101 or servers 1021, or even more. The embodiments of this application do not limit the number or type of terminals or servers.

[0095] Figure 2 This is a flowchart of an image registration method provided in an embodiment of this application. The method is applied in an electronic device, which may be a terminal or a server. See also... Figure 2 Taking the application of this method to a terminal as an example, the method includes the following steps.

[0096] 201. The electronic device performs image registration on a reference image and a first image based on an image registration network to obtain an initial registration result.

[0097] Image registration networks are used to register input image pairs. The specific process of image registration can be understood as follows: for an image pair, a spatial transformation is found to map one image (the moving image, the image to be registered) onto the other image (the fixed image), so that points corresponding to the same spatial location in the two images correspond one-to-one, thereby achieving the purpose of information fusion.

[0098] The image pair includes a reference image and a first image. The first image is the image to be registered. During registration, the reference image should be used as the standard, and the relevant information of the first image should be transformed into the coordinate system of the reference image. This ensures that identical or similar image content in the first and reference images is located in the same position through image registration. This makes subsequent processing of the first and reference images more comparative and allows for a more intuitive understanding of the identical or similar image content between the two images.

[0099] In some embodiments, the electronic device can input a reference image and a first image into an image registration network, which then performs image registration on the reference image and the first image based on network parameters, and outputs an initial second image after the first image is registered. In some embodiments, the electronic device can also output a reference image. In other embodiments, the electronic device can also output initial transformation parameters for transforming the first image during the image registration process, that is, the initial transformation parameters corresponding to the initial registration result. These initial transformation parameters are obtained by the image registration network processing the reference image and the first image.

[0100] 202. In response to the initial registration result meeting the conditions, the electronic device updates the initial transformation parameters corresponding to the initial registration result.

[0101] After obtaining the initial registration result through the image registration network, it can be further determined whether the initial registration result is accurate. If it is inaccurate, the electronic device can further optimize the initial registration result to improve the accuracy of the registration result.

[0102] The condition that the initial registration result meets indicates that the initial registration result is not accurate enough. In this case, the electronic device optimizes the initial registration result. During the optimization, the initial transformation parameters corresponding to the initial registration result can be updated. Further analysis is then conducted through subsequent steps to determine whether the updated initial transformation parameters will make the registered second image more similar to the reference image.

[0103] Of course, the purpose of optimization is to update the transformation parameters so that the second image obtained by transforming the first image based on the transformation parameters is more similar to the reference image. The more similar the second image is to the reference image, the more accurate the image registration result will be.

[0104] 203. The electronic device performs transformation processing on the first image based on the updated transformation parameters to obtain the second image.

[0105] After the electronic device updates the transformation parameters, it can then transform the first image based on those parameters to obtain the second image. It should be noted that this transformation process essentially transforms the content of the first image into the coordinate system of the reference image. This ensures that identical or similar image content in both the reference and second images has the same position, size, and shape, allowing for effective analysis of targets within the images by combining the two images.

[0106] For example, if the reference image and the first image are images of the human brain taken from different angles, the above transformation process can adjust the position, size, and shape of the same brain tissue in the two images to be consistent. In this way, when analyzing the brain tissue by combining the two images, there is no need to artificially imagine what the brain tissue in the first image would look like from the angle of the reference image.

[0107] 204. In response to the similarity between the reference image and the second image meeting the condition, the electronic device updates the transformation parameters, continues to perform transformation processing and update the transformation parameters based on the updated transformation parameters, and stops when the target condition is met, thus obtaining the target registration result.

[0108] Steps 202 and 203 described above are the steps required to be executed in one iteration of the optimization process. In this iteration, the electronic device can obtain the similarity between the reference image and the second image, and use this similarity as a metric to determine whether the registration result is accurate. If it is not accurate enough, the transformation parameters can be updated, and then steps 202 and 203 described above can be repeated to improve the similarity between the reference image and the second image, thereby achieving the optimization effect on the initial registration result.

[0109] This application provides a method to further optimize the initial registration result after registering the images using an image registration network. This allows for more accurate registration results when the initial registration is inaccurate, overcoming the problem of insufficient generalization ability of the image registration network on new datasets. During this optimization process, the transformation parameters are continuously adjusted to increase the similarity between the reference image and the second image, thereby obtaining a more accurate registration result.

[0110] Figure 3 This is a flowchart of an image registration network training method provided in an embodiment of this application. The method is applied in an electronic device, which may be a terminal or a server. (See also...) Figure 3 Taking the application of this method to a terminal as an example, the method includes the following steps.

[0111] 301. An electronic device acquires a sample image pair, the sample image pair including a sample reference image and a sample first image, the sample image pair carrying image type information.

[0112] Image registration includes rigid body registration and non-rigid body registration. Image type information is used to indicate the image registration method used for the sample reference image and the sample first image.

[0113] In some embodiments, the image type information includes a first image type and a second image type. The first image type indicates rigid body registration of the sample reference image and the sample first image. The second image type indicates non-rigid body registration of the sample reference image and the sample first image.

[0114] In this embodiment, the image registration network has rigid body registration and non-rigid body registration functions. When training the image registration network, the image type information can be set for the sample image pairs, so that the image registration network can freely switch between rigid body registration and non-rigid body registration according to the image type information, so as to train the two registration functions of the image registration network at the same time, which effectively improves the intelligence and efficiency of training.

[0115] 302. The electronic device performs image registration on the sample image pair based on the image type information carried by the sample image pair and the branch corresponding to the image type information in the image registration network to obtain transformation parameters.

[0116] Different registration functions yield different transformation parameters, and the methods for image registration differ. Therefore, different registration functions of the image registration network are implemented through different branches. For example, rigid body registration is implemented through the first branch, while non-rigid body registration is implemented through the second branch.

[0117] During training, electronic devices can freely switch branches of the image registration network to achieve image registration by reading the image type information carried by sample image pairs.

[0118] 303. The electronic device trains the network parameters of the image registration network based on the transformation parameters.

[0119] After obtaining the transformation parameters based on the image registration network, the electronic device can use these transformation parameters to determine whether the transformation parameters obtained from the current network parameters are accurate.

[0120] The accuracy of transformation parameters can be measured in different ways. For example, with unsupervised training, the first image can be transformed using the transformation parameters, and the accuracy can be measured by the similarity between the transformed second image and the reference image. With supervised training, the sample image pairs can carry target transformation parameters, and the electronic device can measure its accuracy by comparing the transformation parameters output by the image registration network with the target transformation parameters.

[0121] The image registration network trained in this embodiment supports multiple image registration methods. When training the image registration network, image type information can be set for sample image pairs, so that the image registration network can automatically adapt the image registration method according to the image type information, thereby training the network parameters of this image registration method. In this way, multiple image registration methods are trained simultaneously during the training process, which improves the training efficiency and applicability of the image registration network.

[0122] Figure 4 This is a flowchart of an image registration network training method provided in an embodiment of this application. See also... Figure 4 The method includes the following steps.

[0123] 401. An electronic device acquires a sample image pair, the sample image pair including a sample reference image and a sample first image, the sample image pair carrying image type information.

[0124] The image registration network training method can be applied to two-dimensional image registration as well as three-dimensional image registration. That is, the images in the sample image pair can be two-dimensional images or three-dimensional images. This application does not limit this.

[0125] The sample image pair may be stored in different locations, and correspondingly, electronic devices can acquire the sample image pair in different ways.

[0126] In some embodiments, the sample image pair may be stored in an electronic device, and the electronic device may retrieve the sample image pair from the stored data.

[0127] In other embodiments, the sample image pair may be stored in an image database. Accordingly, the electronic device can retrieve the sample image pair from the image database.

[0128] The above provides two possible methods for obtaining sample image pairs, and this application embodiment does not limit these methods.

[0129] Image type information is used to indicate the image registration method used for the image pair.

[0130] In some embodiments, image registration may include rigid registration and non-rigid registration. The image type information indicates whether rigid or non-rigid registration is performed on the image pair. Rigid registration refers to performing a rigid transformation on the first image so that the first image is mapped into the coordinate system of the reference image. Non-rigid registration refers to performing a non-rigid transformation on the first image so that the first image is mapped into the coordinate system of the reference image.

[0131] A rigid body is an object whose shape and size remain unchanged during motion and after being subjected to forces, and whose internal relative positions remain unchanged. Rigid body transformations refer to rotations, translations, mirroring, etc., of a geometric object, that is, maintaining the invariance of length, angles, area, gradient, divergence, and curl during the motion. Non-rigid transformations are more complex than rigid body transformations; examples include scaling, affine transformations, transmission transformations, and polynomial transformations. Non-rigid body transformations describe changes in the size, rather than the shape, of a geometric object.

[0132] In some embodiments, the image type information can be set by a person skilled in the art for the sample image pair. This image type information can be understood as annotation data or labels for the sample image pair. By setting this image type information, the electronic device can determine the image registration method used for the sample image pair.

[0133] In other embodiments, the image type information is stored in association with sample image pairs, and the electronic device acquires the image type information synchronously when acquiring sample image pairs.

[0134] 402. In response to the image type information carried by the sample image pair being a first image type, the electronic device performs image registration on the sample image pair based on the first branch in the image registration network to obtain an affine matrix, wherein the affine matrix is ​​a transformation parameter, and the first branch is used for rigid body registration.

[0135] When the image type information is different, the image registration method used by the electronic device may be different. In some embodiments, when the image type information is a first image type, the image registration network performs rigid body registration on the image pair. When the image type information is a second image type, the image registration network performs non-rigid body registration on the image pair.

[0136] In some embodiments, the image type information is represented by 0 or 1, where 0 represents a first image type and 1 represents a second image type.

[0137] After the electronic device acquires a sample image pair, it can input the sample image pair into an image registration network. The image registration network can read the image type information carried by the sample image pair. When the image type information is the first image type, it can be input into the first branch, and the first branch will perform the image registration step.

[0138] In some embodiments, the image registration network can extract features from sample image pairs, input the extracted feature maps into a first branch, and process the feature maps by the first branch to obtain an affine matrix.

[0139] Optionally, the first branch can process the feature map by performing global pooling on the feature map to obtain a vector, and then performing dimensionality reduction and reshaping on the vector to obtain an affine matrix.

[0140] In some embodiments, the image registration network includes a feature extraction module and a registration module. The registration module includes a first branch and a second branch. After the electronic device inputs a sample image pair into the image registration network, the feature extraction module of the image registration network can extract features from the sample image pair to obtain feature maps of the sample reference image and the sample first image. Then, the feature maps are input into the first branch of the registration module, where the first branch processes the feature maps to obtain an affine matrix.

[0141] In some embodiments, the feature extraction module may employ a feature extraction network, which can be understood as an encoder. The encoder encodes sample image pairs to obtain feature maps.

[0142] For example, the feature extraction network or encoder can use a ResNet backbone network. The registration module can be understood as a decoder. Different registration functions correspond to different decoders. That is, rigid body registration corresponds to one type of decoder, and non-rigid body registration corresponds to another type of decoder.

[0143] For this first branch, the feature map can be further processed to obtain an affine matrix, which is the transformation parameter for the first image. In other words, by analyzing the feature map, it determines how to process the first image to transform it into the coordinate system of the reference image.

[0144] In some embodiments, the first branch may include a Global Average Pooling (GAP) layer, a linear layer, and a reshaping layer. The global pooling layer performs global pooling on the feature map, compressing it into a vector to reduce data dimensionality and subsequent computation. The linear layer performs dimensionality reduction on the vector. The reshaping layer reshapes the vector, converting it into an affine matrix.

[0145] In a specific example, a sample image undergoes feature extraction to obtain one or more feature maps. For each feature map, a gap-averaging process (GAP) can be used to convert it into a numerical value. This GAP process involves averaging the pixel values ​​of all pixels in the feature map to obtain an average value. Therefore, the multiple average values ​​corresponding to multiple feature maps form a vector. In a specific example, there are 256 feature maps; through GAP, a vector of length 256 can be obtained. The first branch then continues to process the vector of length 256 through a linear layer, obtaining a vector of length 12. The first branch then uses a reshaping layer to transform this 12-length vector into matrix form, thus obtaining an affine matrix. For example, this affine matrix might be a 3x4 matrix.

[0146] For example, this image registration network can be like Figure 5 As shown, for the image to be registered (Mov Image) 501 and the reference image (Fixed Image) 502, after inputting into the feature extraction module 503, the feature map of the image can be obtained. Then, through GAP 504, it can be converted into a vector. Through the linear layer 505 and the reshaping process 506, the affine matrix 507 can be obtained.

[0147] 403. In response to the image type information carried by the sample image pair being a second image type, the electronic device performs image registration on the sample image pair based on the second branch in the image registration network to obtain a vector field, wherein the vector field is a transformation parameter, and the second branch is used for non-rigid body registration.

[0148] After the electronic device acquires a sample image pair, it can input the sample image pair into an image registration network. The image registration network can read the image type information carried by the sample image pair. When the image type information is the second image type, it can be input into the second branch, and the second branch will perform the image registration step.

[0149] Similarly, in some embodiments, the image registration network can extract features from sample image pairs, input the extracted feature maps into a second branch, and process the feature maps to obtain a vector field. The vector field is a function of one vector corresponding to another.

[0150] Optionally, the first branch can process the feature map by performing convolution on the feature map to obtain a vector field.

[0151] In some embodiments, the image registration network includes a feature extraction module and a registration module. The registration module includes a first branch and a second branch. After the electronic device inputs a sample image pair into the image registration network, the feature extraction module of the image registration network can extract features from the sample image pair to obtain feature maps of the sample reference image and the sample first image. Then, the feature maps are input into the second branch of the registration module, where the second branch processes the feature maps to obtain a vector field.

[0152] The second branch further processes the feature map to obtain a vector field, which serves as the transformation parameters for the first image. In other words, by analyzing the feature map, it determines how to process the first image to transform it into the coordinate system of the reference image.

[0153] In some embodiments, the second branch may include multiple convolutional layers, which can be understood as vector field decoders used to transform feature maps into vector fields. A vector field is a function of one vector corresponding to another.

[0154] In some embodiments, the multiple convolutional layers of the second branch and the convolutional layers in the feature extraction module are connected by skip connections, allowing the decoder and encoder to form a Unet structure. Thus, when each convolutional layer decodes, its input includes not only the feature map output by the feature extraction module but also the output of a certain convolutional layer within the feature extraction module. This enables the second branch to be independent of the output of the feature extraction module and to obtain a more accurate vector field based on more raw data.

[0155] In some embodiments, the sample image is subjected to feature extraction to obtain one or more feature maps, and the second branch performs convolution processing on one or more feature maps to obtain a vector field.

[0156] For example, this image registration network can be like Figure 5 As shown, for the image to be registered (Mov Image) 501 and the reference image (Fixed Image) 502, after inputting into the feature extraction module 503, the feature map of the image can be obtained, and then through multiple convolutional layers 508 of the second branch, it can be transformed into a vector field 509.

[0157] Steps 402 and 403 above are processes for performing image registration on the sample image pair based on the image type information carried by the sample image pair and the branch corresponding to the image type information in the image registration network to obtain transformation parameters. The above two steps are rigid body registration process and non-rigid body registration process, respectively.

[0158] 404. The electronic device trains the network parameters of the image registration network based on the transformation parameters.

[0159] After obtaining the transformation parameters based on the image registration network, the electronic device can use these transformation parameters to determine whether the transformation parameters obtained from the current network parameters are accurate.

[0160] The accuracy of transformation parameters can be measured in different ways. For example, with unsupervised training, the first image can be transformed using the transformation parameters, and the accuracy can be measured by the similarity between the transformed second image and the reference image. With supervised training, the sample image pairs can carry target transformation parameters, and the electronic device can measure its accuracy by comparing the transformation parameters output by the image registration network with the target transformation parameters.

[0161] That is, step 404 can be implemented in the following two ways:

[0162] Method 1: The electronic device can process the sample first image based on the transformation parameters to obtain the target second image, and train the network parameters of the image registration network based on the similarity between the sample reference image and the target second image.

[0163] In this method, the first sample image is transformed using the obtained transformation parameters, and the accuracy of the transformation parameters is measured by the similarity between the second target image obtained after the transformation and the reference image.

[0164] Understandably, after obtaining the affine matrix and vector field in the first and second branches respectively, the first branch can perform an affine transformation on the first image based on the affine matrix to obtain the target second image. The second branch can perform a strain transformation on the first image based on the vector field to obtain the target second image. If the target second image is similar to the reference image, it indicates that the transformation parameters are relatively accurate. If the target second image is not similar to the reference image, it indicates that the transformation parameters are not very accurate, and the network parameters need to be updated to obtain more accurate transformation parameters for either the first or second branch.

[0165] For example, such as Figure 5 As shown, after obtaining the affine matrix 507, the first branch can perform an affine transformation 510 on the image 501 to be registered using the affine matrix 507. For the second branch, after obtaining the vector field 509, it can perform a deformable transformation 511 on the image 501 to be registered using the vector field 509. The transformed image is referred to here as the target second image.

[0166] Method 2: The electronic device trains the network parameters of the image registration network based on the transformation parameters and the target transformation parameters carried by the sample image.

[0167] In Method Two, supervised training is used. The sample images carry target transformation parameters, which are correct and realistic transformation parameters. That is, by processing the first sample image using the target transformation parameters, it can be accurately transformed into the coordinate system of the sample reference image. In this way, through the transformation parameters estimated by the network and the target transformation parameters, the electronic device can determine whether the transformation parameters are accurate.

[0168] The target transformation parameters can be obtained through open-source registration tools (such as Elastix) or from historical registration records.

[0169] The training process described above essentially involves updating the network parameters and repeating the registration iterative process. During training, a training objective can be set to determine when to end the training.

[0170] In some embodiments, the electronic device may obtain the value of an objective function based on transformation parameters, the value of which is used to indicate the similarity between the sample reference image and the target second image, or to indicate the similarity between the transformation parameters and the target transformation parameters.

[0171] Taking the value of the objective function as an indicator of the similarity between the sample reference image and the target second image as an example, the objective function can be the NCC (Normalized Cross Correlation) function or the NMI (Normalized Mutual Information) function.

[0172] In some embodiments, the image registration network can perform image registration on unimodal image pairs or multimodal image pairs. A unimodal image pair refers to an image pair in which the images have the same or similar pixel distribution, or that the imaging devices are the same. A multimodal image pair refers to an image pair in which the images have different pixel distributions, or that the imaging devices are different. For example, two images obtained from CT and MRI are multimodal image pairs.

[0173] In some embodiments, different objective functions can be used for training image pairs with different modalities. That is, the electronic device can obtain the similarity between the sample reference image and the sample second image based on the modal information of the sample reference image and the sample first image, using the similarity acquisition method corresponding to the modal information.

[0174] Method 1: In response to the fact that the modal information of the reference image is the same as that of the first image, obtain the normalized cross-correlation coefficient between the sub-image set of the reference image and the second image, wherein the normalized cross-correlation coefficient is the similarity between the reference image and the second image.

[0175] This method refers to using the NCC function to obtain the similarity between the sample reference image and the sample second image for a single-modal image pair.

[0176] Method 2: In response to the difference between the modal information of the reference image and the modal information of the first image, obtain the normalized mutual information between the sub-image set of the reference image and the second image, and obtain the similarity between the reference image and the second image based on the normalized mutual information.

[0177] Method 2 refers to using the NMI function to obtain the similarity between the sample reference image and the sample second image for single-modal image pairs.

[0178] For example, such as Figure 5 As shown, after transforming the image to be registered to obtain the target second image, the loss value can be calculated based on the target second image and the reference image 502 using the NCC or NMI loss function 512. This loss value is also the value of the objective function (NCC or NMI loss function).

[0179] The end of training can be determined by the convergence of the similarity between the sample reference image and the sample second image, or by reaching a preset number of iterations. This application does not limit this to any particular outcome.

[0180] In some embodiments, for non-rigid transformations, i.e., registration results obtained through the second branch, when determining the similarity between the sample reference image and the sample second image, the electronic device not only considers the above objective function, but also the smoothness of the vector field transformation to measure the accuracy of the vector field.

[0181] Specifically, the electronic device can respond to the transformation parameter being a vector field by performing a strain transformation on the sample first image, obtain a smoothing constraint value based on the vector field, and then obtain the similarity between the sample reference image and the target second image based on the smoothing constraint value and the normalized mutual information.

[0182] In a specific example, the smoothing constraint value can be obtained using the following formula.

[0183] + Formula 1

[0184] in, To smooth out constraint values. It is a vector field (transformation field).

[0185] Therefore, considering the above smoothing constraint values, the overall objective function can be obtained as shown in Formula 2 below.

[0186] Formula 2

[0187] in, To smooth out constraint values. The value of the overall objective function. The value of the objective function obtained based on NCC or NMI.

[0188] The training process described above is an iterative process. In each iteration, the image registration network processes the input image pairs and obtains the value of the objective function. If the value of the objective function does not meet the target conditions, the electronic device can optimize the parameters of the image registration network based on the value of the objective function. If the value of the objective function meets the target conditions, the electronic device can determine that the training has ended and use the network parameters used in this iteration as the final network parameters of the image registration network.

[0189] This optimization process can be implemented in various ways. Taking gradient descent as an example, the electronic device can obtain the gradient of the network parameters based on the value of the objective function, and update the network parameters accordingly. In the next iteration, the image registration network can use the updated network parameters when processing the input image pairs.

[0190] After the image registration network is trained, if the electronic device requires image registration, it can call the trained image registration network to perform image registration. Specifically, in response to an image registration command, the electronic device acquires a reference image and a first image, and performs image registration on the reference image and the first image based on the trained image registration network to obtain the image registration result. The process of image registration performed by this image registration network is similar to the feature extraction, transformation parameter determination, and transformation processing processes in steps 402-404 above, and will not be elaborated further here.

[0191] For details on the usage of this image registration network, please refer to the following: Figure 6 The illustrated embodiment also allows for online learning during use. Based on the target image pairs generated during use, combined with the aforementioned original sample image pairs, the image registration network is updated and trained online, enabling continuous optimization during operation. This can be further explained in the following sections. Figure 6The content shown will not be elaborated upon here.

[0192] The image registration network trained in this embodiment supports multiple image registration methods. When training the image registration network, image type information can be set for sample image pairs, so that the image registration network can automatically adapt the image registration method according to the image type information, thereby training the network parameters of this image registration method. In this way, multiple image registration methods are trained simultaneously during the training process, which improves the training efficiency and applicability of the image registration network.

[0193] The above method describes the training process of the image registration network. The following describes the usage process of this image registration network.

[0194] Figure 6 This is a flowchart of an image registration method provided in an embodiment of this application. See also... Figure 6 The method includes the following steps.

[0195] 601. The electronic device acquires a reference image and a first image.

[0196] In some embodiments, the reference image and the first image can be captured by other electronic devices and sent to those electronic devices. That is, when other electronic devices have image registration requirements, the image pair to be registered can be sent to those electronic devices.

[0197] In other embodiments, the reference image and the first image can be captured by the electronic device. For example, they can be captured by photographing an object at different times, under different lighting conditions, or from different angles.

[0198] Given two captured images, the electronic device can respond to a setting command for one of the images, setting it as the reference image and the other image as the first image.

[0199] This image registration network can perform image registration on both unimodal and multimodal image pairs. A unimodal image pair refers to images with identical or similar pixel distributions, or images acquired using the same imaging device. A multimodal image pair refers to images with different pixel distributions, or images acquired using different imaging devices. For example, two images obtained from CT and MRI are multimodal image pairs. Accordingly, the reference image and the first image can be obtained from the same imaging device or from different imaging devices.

[0200] 602. The electronic device performs image registration on a reference image and a first image based on an image registration network to obtain an initial registration result.

[0201] When electronic devices require image registration, they can call upon a pre-trained image registration network to perform image registration. The training process for this image registration network can be found above. Figure 4 The illustrated embodiment.

[0202] The image registration process for this image registration network can be the same as the process of obtaining the target second image in steps 402-404 above.

[0203] Similarly, after acquiring the reference image and the first image, the electronic device can input the reference image and the first image into an image registration network. This image registration network can extract features from the reference image and the first image, then process the extracted feature maps to obtain initial transformation parameters. Based on these initial transformation parameters, the first image is transformed to obtain an initial registration result. This initial registration result may include an initial second image registered with the first image. The initial registration result may also include the aforementioned initial transformation parameters.

[0204] In some embodiments, similarly, the image registration network includes a first branch and a second branch, whereby the first branch is used for rigid body registration and the second branch is used for non-rigid body registration. In response to a rigid body registration command, the electronic device can perform image registration on the reference image and the first image based on the first branch of the image registration network to obtain an initial registration result, wherein the initial transformation parameter corresponding to the initial registration result is an initial affine matrix. In response to a non-rigid body registration command, the electronic device can perform image registration on the reference image and the first image based on the second branch of the image registration network to obtain an initial registration result, wherein the initial transformation parameter corresponding to the initial registration result is an initial vector field.

[0205] The rigid body registration command and the non-rigid body registration command can be triggered by the user's registration operation. The user can choose to perform rigid body registration or non-rigid body registration, and accordingly, the image registration network can perform image registration steps based on the corresponding branch.

[0206] Similarly, after the electronic device inputs the reference image and the first image into the image registration network, the image registration network can extract features from the reference image and the first image to obtain a feature map. In response to the rigid body registration command, the image registration network can input the feature map into the first branch, and the first branch processes the feature map to obtain an initial affine matrix. Based on the initial affine matrix, the first image is subjected to an affine transformation to obtain the initial registration result.

[0207] Alternatively, after the electronic device inputs the reference image and the first image into the image registration network, the image registration network can extract features from the reference image and the first image to obtain a feature map. In response to the non-rigid body registration command, the image registration network can input the feature map into the second branch, and the second branch processes the feature map to obtain an initial vector field. Based on the initial vector field, the first image is subjected to strain transformation to obtain the initial registration result.

[0208] In some embodiments, the image registration network includes a feature extraction module and a registration module. The registration module includes the first branch and the second branch described above. The feature extraction module is used to extract features from the reference image and the first image to obtain a feature map.

[0209] Similarly, in some embodiments, if rigid body registration is performed, the first branch may include a Global Average Pooling (GAP) layer, a linear layer, and a reshaping layer. In a specific example, the reference image and the first image are processed to obtain one or more feature maps. For each feature map, it can be converted into a numerical value using GAP. Multiple feature maps are processed by GAP to obtain a vector. The first branch then continues to process the vector based on the linear layer to obtain a shorter vector, and then uses the reshaping layer to convert the vector into a matrix form, thereby obtaining an initial affine matrix. The first branch can then perform an affine transformation on the first image based on the initial affine matrix to obtain an initial second image.

[0210] In other embodiments, if non-rigid registration is performed, the second branch may include multiple convolutional layers, which can be understood as vector field decoders used to transform feature maps into vector fields. In a specific example, the reference image and the first image are subjected to feature extraction to obtain one or more feature maps. For each feature map, one or more feature maps can be convolved by multiple convolutional layers to obtain an initial vector field. The second branch can then perform a strain transformation on the first image based on the initial vector field to obtain an initial second image.

[0211] 603. In response to the initial registration result meeting the conditions, the electronic device updates the initial transformation parameters corresponding to the initial registration result.

[0212] The initial registration result meeting the conditions can be triggered by user operation or by electronic equipment analyzing the initial registration result. Specifically, it can include the following two methods.

[0213] Method 1: In response to receiving an optimization instruction for the initial registration result, the step of updating the initial transformation parameters corresponding to the initial registration result is executed.

[0214] In Method 1, after the electronic device obtains the initial registration result, it can display the initial second image after initial registration on the screen, and the user can determine whether it is accurate enough. If the user feels that it is not accurate enough, an optimization operation can be performed to trigger the electronic device to perform the optimization step.

[0215] Method 2: In response to the fact that the similarity between the initial second image registered with the first image and the reference image in the initial registration result is less than a first similarity threshold, the step of updating the initial transformation parameters corresponding to the initial registration result is executed.

[0216] In Method 2, after the electronic device obtains the initial registration result, it can analyze the initial registration result. The accuracy of the initial registration result can be measured by comparing the similarity between the registered initial second image and the reference image.

[0217] The first similarity threshold can be set by relevant technical personnel according to their needs, and this embodiment does not limit this setting. The method for obtaining the similarity between the two images can be similar to that shown in step 404 above, or other methods can be used, and this embodiment does not limit this method.

[0218] With the above Figure 4 Similarly, in the illustrated embodiment, during rigid body registration, the transformation parameter corresponding to the initial registration result is the initial affine matrix. The electronic device updates the initial affine matrix corresponding to the initial registration result in response to the initial registration result satisfying a condition. During non-rigid body registration, the transformation parameter corresponding to the initial registration result is the initial vector field. The electronic device can update the initial vector field corresponding to the initial registration result in response to the initial registration result satisfying a condition.

[0219] It should be noted that, in this embodiment, after updating the initial transformation parameters, the first image is transformed again to obtain the second image. The electronic device then determines the similarity between the second image and the reference image to determine whether further optimization of the transformation parameters is needed. If further optimization is required, the optimization process is the same as in step 603, and the electronic device will repeat step 604 and the process of determining whether further optimization is needed based on similarity. The above process is an iterative optimization process. By updating the transformation parameters through one or more iterations, the second image obtained based on the transformation parameters becomes more similar to the reference image, thus resulting in a more accurate registration result.

[0220] 604. The electronic device performs transformation processing on the first image based on the updated transformation parameters to obtain the second image.

[0221] With the above Figure 4Similarly, in the illustrated embodiment, during rigid body registration, the electronic device can perform an affine transformation on the first image based on the updated affine matrix to obtain the second image. During non-rigid body registration, the electronic device can perform a strain transformation on the first image based on the updated vector field to obtain the second image.

[0222] With the above Figure 4 Similarly, in the illustrated embodiment, the electronic device can respond to the transformation parameter being a vector field, perform a strain transformation on the first image, obtain a smoothing constraint value based on the vector field, and then obtain the similarity between the reference image and the second image based on the smoothing constraint value and the normalized mutual information.

[0223] 605. In response to the similarity between the reference image and the second image meeting the condition, the electronic device updates the transformation parameters, continues to perform transformation processing and update the transformation parameters based on the updated transformation parameters, and stops when the target condition is met, thereby obtaining the target registration result.

[0224] The similarity threshold can be set by relevant technical personnel according to their needs. In some embodiments, the electronic device updates the transformation parameters in response to the similarity between the reference image and the second image being less than a second similarity threshold. The second similarity threshold can be set by relevant technical personnel according to their needs, and this application embodiment does not limit this setting.

[0225] The process of obtaining this similarity and Figure 4 The content shown in step 404 of the illustrated embodiment is the same, and will not be repeated here.

[0226] The processes of updating the initial transformation parameters in step 603 and step 605 are similar to those of updating the network parameters in step 404, except that the updates are for the transformation parameters. When processing the initial transformation parameters, they can be updated based on the similarity between the initial second image and the reference image. In subsequent iterations, when updating the transformation parameters, the electronic device can also update them based on the similarity between the second image and the reference image.

[0227] Similarly to step 404 above, the similarity between the second image and the reference image can be represented by the value of the objective function. That is, the similarity between the second image and the reference image can be obtained through the objective function.

[0228] With the above Figure 4Similarly, in some embodiments, different objective functions can be used for the similarity acquisition steps for image pairs with different modalities. The process of acquiring the similarity between the reference image and the second image can be determined with the modal information of the two images. Specifically, the electronic device can acquire the similarity between the reference image and the second image based on the modal information of the reference image and the first image, using the similarity acquisition method corresponding to the modal information.

[0229] In some embodiments, in response to the modal information of the reference image being identical to that of the first image, the electronic device obtains a normalized cross-correlation coefficient between the sub-image set of the reference image and the second image, wherein the normalized cross-correlation coefficient represents the similarity between the reference image and the second image. That is, if the reference image and the first image are a unimodal image pair, the similarity can be determined using the NCC function.

[0230] In response to the difference between the modal information of the reference image and the modal information of the first image, the electronic device obtains the normalized mutual information between the sub-image set of the reference image and the second image, and obtains the similarity between the reference image and the second image based on the normalized mutual information. That is, since the reference image and the first image are a multimodal image pair, the similarity can be determined using the NMI function.

[0231] After obtaining the value of the objective function, similar to step 404, when updating the initial transformation parameters and updating the transformation parameters, the electronic device can obtain the gradient of the transformation parameters based on the value of the objective function, and update the transformation parameters based on this gradient to obtain the updated transformation parameters. In the next iteration, the first image can be transformed based on the updated transformation parameters.

[0232] Steps 603 and 604 above are iterative processes. When it is determined that another iteration is needed, the parameter update process can be performed in the same way as step 603, and then step 604 can be repeated to determine whether the iteration has ended.

[0233] The conditions for ending the iteration, also known as the target conditions, include either reaching a target number of iterations or achieving similarity convergence. These conditions can be set by relevant technical personnel according to their needs. The target number of iterations can be set by technical personnel or obtained based on user-defined settings.

[0234] For example, such as Figure 7As shown, the reference image and the first image can be initially registered using an image registration network to obtain an initial solution, which is the initial registration result. Then, its quality is judged. If the initial registration result is good, no optimization is needed, and the registration can be terminated directly. If the initial registration result is poor, iterative optimization can be performed to obtain the target registration result.

[0235] Steps 603 to 605 constitute an iterative optimization process. In some embodiments, the update of the aforementioned transformation parameters can also be achieved by updating the network parameters of the image registration network. That is, the electronic device can also use the aforementioned image registration network for optimization. For example, in step 603, the electronic device can re-input the reference image and the initial second image from the initial registration result into the image registration network. The image registration network processes the input image pairs, and based on the registration result output by the network, obtains the similarity through the objective function. Then, it updates the network parameters based on the similarity, and re-registers the reference image and the first image using the updated network parameters. This process can be repeated multiple times to obtain better registration results. This iterative process is similar to the iterative process of the image registration network training process, and will not be elaborated further here.

[0236] Through iterative optimization in this way, the network parameters of the image registration network change. The electronic device can then reacquire the sample image pairs from step 601, combine them with the target image pair composed of the reference image and the first image, and train the image registration network to obtain an updated image registration network. Through this online learning method, the image registration network can be trained using new image pairs and the original training image pairs during use, continuously optimizing the network parameters and improving its performance and generalization ability.

[0237] In other embodiments, the iterative optimization process described above can be implemented without the image registration network. During use, many new image pairs can be collected, referred to herein as target image pairs. For example, the target image pairs are the image pairs input to the image registration network during its use, namely the aforementioned reference image and the first image. By using the image pairs generated during the process, the image registration network can be further trained in conjunction with the original sample image pairs. This allows the image registration network to inherit the registration capabilities from the original sample image pairs while also possessing the ability to register image pairs generated during use.

[0238] Specifically, in response to an update instruction from the image registration network, the electronic device acquires batch data, which includes target image pairs and sample image pairs. Then, based on the batch data, it trains the trained image registration network to obtain an updated image registration network.

[0239] The image registration network update command can be triggered periodically or when the number of times the image registration network is used reaches a threshold. For example, an update cycle can be set for the image registration network, with an update training performed once every update cycle. The electronic device can acquire the image pairs input to the image registration network within the update cycle as target image pairs, as well as the sample image pairs used in the previous training, to train the image registration network. Batch data refers to the data composed of the image pairs input to the image registration network within the update cycle and the sample image pairs used in the previous training. As a specific example, the image registration network can be set to be updated once a day. The electronic device can acquire the image pairs input to the image registration network from the previous day at a fixed time each day, combining them with the sample image pairs used in the previous training for training. Batch data refers to the data composed of the image pairs input to the image registration network from the previous day and the sample image pairs used in the previous training.

[0240] For example, the image registration method described above can be applied to surgical navigation applications. Patients typically undergo image data scanning from multiple modalities, such as CT (Computed Tomography) and MRI (Magnetic Resonance Imaging). In preoperative planning systems, multiple scanned images need to be registered. The accuracy of registration is crucial, indirectly affecting navigation accuracy. By registering images to approximate each other, a more accurate understanding of the patient's condition can be achieved. Due to the variety of diseases and the variability of images, to ensure algorithm speed, inference-based registration is initially employed. If the initial solution is good enough, registration ends; if the initialization is not accurate enough, further iterative optimization of the initial solution is required.

[0241] The above method combines inference and iterative registration techniques, achieving a balance between speed and accuracy, and fully utilizing various devices to accelerate the algorithm. Furthermore, the algorithm's versatility is improved, making it applicable to single-modal, multi-modal, rigid, and non-rigid body registration tasks. The registration network employs a dynamic structure, switching between multiple tasks. Moreover, the images described are not limited to a single dimension; that is, the method can be used for both 2D and 3D image registration tasks.

[0242] This application provides a method to further optimize the initial registration result after registering the images using an image registration network. This allows for more accurate registration results when the initial registration is inaccurate, overcoming the problem of insufficient generalization ability of the image registration network on new datasets. During this optimization process, the transformation parameters are continuously adjusted to increase the similarity between the reference image and the second image, thereby obtaining a more accurate registration result.

[0243] All the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this application, and will not be described in detail here.

[0244] Figure 8 This is a schematic diagram of the structure of an image registration device provided in an embodiment of this application. See also... Figure 8 The device includes:

[0245] The registration module 801 is used to perform image registration between the reference image and the first image based on the image registration network to obtain the initial registration result;

[0246] Update module 802 is used to update the initial transformation parameters corresponding to the initial registration result in response to the initial registration result meeting the conditions;

[0247] Transformation module 803 is used to transform the first image based on the updated transformation parameters to obtain a second image;

[0248] The update module 802 and the transformation module 803 are further configured to update the transformation parameters in response to the similarity condition between the reference image and the second image, and continue to perform transformation processing and update the transformation parameters based on the updated transformation parameters until the target condition is met, thereby obtaining the target registration result.

[0249] In some embodiments, the update module 802 is configured to perform any of the following:

[0250] In response to receiving an optimization instruction for the initial registration result, the step of updating the initial transformation parameters corresponding to the initial registration result is executed;

[0251] In response to the fact that the similarity between the initial second image registered with the first image and the reference image in the initial registration result is less than a first similarity threshold, the step of updating the initial transformation parameters corresponding to the initial registration result is performed.

[0252] In some embodiments, the image registration network includes a first branch and a second branch, wherein the first branch is used for rigid body registration and the second branch is used for non-rigid body registration;

[0253] The registration module 801 is used for:

[0254] In response to the rigid body registration command, image registration is performed on the reference image and the first image based on the first branch of the image registration network to obtain an initial registration result. The initial transformation parameter corresponding to the initial registration result is the initial affine matrix.

[0255] In response to the non-rigid registration instruction, image registration is performed on the reference image and the first image based on the second branch of the image registration network to obtain an initial registration result. The initial transformation parameters corresponding to the initial registration result are the initial vector field.

[0256] In some embodiments, the update module 802 is configured to perform any of the following:

[0257] In response to the initial registration result satisfying the condition, the initial affine matrix corresponding to the initial registration result is updated;

[0258] In response to the initial registration result satisfying the condition, the initial vector field corresponding to the initial registration result is updated;

[0259] The transformation module 803 is used to perform any of the following:

[0260] The second image is obtained by performing an affine transformation on the first image based on the updated affine matrix.

[0261] The first image is subjected to strain transformation based on the updated vector field to obtain the second image.

[0262] In some embodiments, the update module 802 is used to update the transformation parameters in response to the similarity between the reference image and the second image being less than a second similarity threshold.

[0263] In some embodiments, the process of obtaining the similarity between the reference image and the second image includes:

[0264] Based on the modal information of the reference image and the second image, the similarity between the reference image and the second image is obtained using the similarity acquisition method corresponding to the modal information.

[0265] In some embodiments, obtaining the similarity between the reference image and the second image based on the modal information of the reference image and the second image, using a similarity acquisition method corresponding to the modal information, includes:

[0266] In response to the fact that the modal information of the reference image and the modal information of the second image are the same, the normalized cross-correlation coefficient between the sub-image set of the reference image and the second image is obtained, and the normalized cross-correlation coefficient is the similarity between the reference image and the second image;

[0267] In response to the difference between the modal information of the reference image and the modal information of the second image, normalized mutual information between the sub-image set of the reference image and the second image is obtained, and the similarity between the reference image and the second image is obtained based on the normalized mutual information.

[0268] In some embodiments, obtaining the similarity between the reference image and the second image based on the normalized mutual information includes:

[0269] In response to the transformation parameters being a vector field, the transformation processing of the first image is a strain transformation, and a smoothing constraint value is obtained based on the vector field;

[0270] The similarity between the reference image and the second image is obtained based on the smoothing constraint value and the normalized mutual information.

[0271] This application provides a method to further optimize the initial registration result after registering the images using an image registration network. This allows for more accurate registration results when the initial registration is inaccurate, overcoming the problem of insufficient generalization ability of the image registration network on new datasets. During this optimization process, the transformation parameters are continuously adjusted to increase the similarity between the reference image and the second image, thereby obtaining a more accurate registration result.

[0272] It should be noted that the image registration device provided in the above embodiments is only illustrated by the division of the above functional modules during image registration. In actual applications, the above functions may be assigned to different functional modules as needed, that is, the internal structure of the image registration device may be divided into different functional modules to complete all or part of the functions described above. In addition, the image registration device and the image registration method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0273] Figure 9 This is a schematic diagram of the structure of an image registration network training device provided in an embodiment of this application. See also... Figure 9 The device includes:

[0274] The acquisition module 901 is used to acquire sample image pairs, the sample image pairs including a sample reference image and a sample first image, and the sample image pairs carry image type information;

[0275] Registration module 902 is used to perform image registration on the sample image pair based on the image type information carried by the sample image pair and the branch corresponding to the image type information in the image registration network to obtain transformation parameters;

[0276] Training module 903 is used to train the network parameters of the image registration network based on the transformation parameters.

[0277] In some embodiments, the training module 903 is configured to perform any of the following:

[0278] The sample first image is processed based on the transformation parameters to obtain the target second image. The network parameters of the image registration network are trained based on the similarity between the sample reference image and the target second image.

[0279] Based on the transformation parameters and the target transformation parameters carried by the sample images, the network parameters of the image registration network are trained.

[0280] In some embodiments, the image type information includes a first image type and a second image type;

[0281] The registration module 902 is used for:

[0282] In response to the image type information carried by the sample image pair being a first image type, image registration is performed on the sample image pair based on the first branch in the image registration network to obtain an affine matrix, wherein the affine matrix is ​​a transformation parameter, and the first branch is used for rigid body registration.

[0283] In response to the image type information carried by the sample image pair being a second image type, image registration is performed on the sample image pair based on the second branch in the image registration network to obtain a vector field, wherein the vector field is a transformation parameter, and the second branch is used for non-rigid body registration.

[0284] In some embodiments, the acquisition module 901 is further configured to acquire batch data in response to an update instruction from the image registration network. The batch data includes target image pairs and sample image pairs. The target image pairs are image pairs optimized from the initial registration results output by the image registration network during its use.

[0285] The training module 903 is also used to train the trained image registration network based on the batch data to obtain an updated image registration network.

[0286] The image registration network trained in this embodiment supports multiple image registration methods. When training the image registration network, image type information can be set for sample image pairs, so that the image registration network can automatically adapt the image registration method according to the image type information, thereby training the network parameters of this image registration method. In this way, multiple image registration methods are trained simultaneously during the training process, which improves the training efficiency and applicability of the image registration network.

[0287] It should be noted that the image registration network training device provided in the above embodiments is only illustrated by the division of the above functional modules when training the image registration network. In actual applications, the above functions may be assigned to different functional modules as needed, that is, the internal structure of the image registration network training device may be divided into different functional modules to complete all or part of the functions described above. In addition, the image registration network training device and the image registration network training method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0288] Figure 10 This is a schematic diagram of the structure of an electronic device 1000 provided in an embodiment of this application. The electronic device 1000 can vary significantly due to differences in configuration or performance. It includes one or more Central Processing Units (CPUs) 1001 and one or more memories 1002. The memories 1002 store at least one computer program, which is loaded and executed by the processor 1001 to implement the image registration method or image registration network training method provided in the various method embodiments described above. The electronic device also includes other components for implementing device functions. For example, the electronic device also has wired or wireless network interfaces and input / output interfaces for input and output. Details of these components are not elaborated upon here.

[0289] The electronic device in the above method embodiments is implemented as a terminal. For example, Figure 11 This is a structural block diagram of a terminal provided in an embodiment of this application. The terminal 1100 can be a portable mobile terminal, such as a smartphone, tablet computer, MP3 (Moving Picture Experts Group Audio Layer III) player, MP4 (Moving Picture Experts Group Audio Layer IV) player, laptop computer, or desktop computer. The terminal 1100 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0290] Typically, terminal 1100 includes a processor 1101 and a memory 1102.

[0291] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0292] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 are used to store at least one instruction, which is executed by the processor 1101 to implement the image registration method or image registration network training method provided in the method embodiments of this application.

[0293] In some embodiments, the terminal 1100 may also optionally include a peripheral device interface 1103 and at least one peripheral device. The processor 1101, memory 1102, and peripheral device interface 1103 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1103 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107, and a power supply 1109.

[0294] Peripheral device interface 1103 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1101 and memory 1102. In some embodiments, processor 1101, memory 1102 and peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1101, memory 1102 and peripheral device interface 1103 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0295] The radio frequency (RF) circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1104 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1104 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1104 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0296] Display screen 1105 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1105 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1101 for processing. In this case, display screen 1105 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1105, disposed on the front panel of terminal 1100; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1100 or in a folded design; in still other embodiments, display screen 1105 may be a flexible display screen, disposed on a curved or folded surface of terminal 1100. Furthermore, display screen 1105 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1105 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0297] The camera assembly 1106 is used to acquire images or videos. Optionally, the camera assembly 1106 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1106 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0298] The audio circuit 1107 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1101 for processing, or input to the radio frequency circuit 1104 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1100. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1101 or the radio frequency circuit 1104 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1107 may also include a headphone jack.

[0299] Power supply 1109 is used to power the various components in terminal 1100. Power supply 1109 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1109 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0300] In some embodiments, the terminal 1100 further includes one or more sensors 1110. The one or more sensors 1110 include, but are not limited to: an acceleration sensor 1111, a gyroscope sensor 1112, a pressure sensor 1113, an optical sensor 1115, and a proximity sensor 1116.

[0301] Accelerometer 1111 can detect the magnitude of acceleration along the three axes of a coordinate system established with terminal 1100. For example, accelerometer 1111 can be used to detect the components of gravitational acceleration along the three axes. Processor 1101 can control display screen 1105 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1111. Accelerometer 1111 can also be used for collecting game or user motion data.

[0302] The gyroscope sensor 1112 can detect the orientation and rotation angle of the terminal 1100. The gyroscope sensor 1112 can work in conjunction with the accelerometer sensor 1111 to collect the user's 3D movements on the terminal 1100. Based on the data collected by the gyroscope sensor 1112, the processor 1101 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0303] The pressure sensor 1113 can be disposed on the side bezel of the terminal 1100 and / or on the lower layer of the display screen 1105. When the pressure sensor 1113 is disposed on the side bezel of the terminal 1100, it can detect the user's grip signal on the terminal 1100, and the processor 1101 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1113. When the pressure sensor 1113 is disposed on the lower layer of the display screen 1105, the processor 1101 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1105. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0304] An optical sensor 1115 is used to collect ambient light intensity. In one embodiment, the processor 1101 can control the display brightness of the display screen 1105 based on the ambient light intensity collected by the optical sensor 1115. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1105 is increased; when the ambient light intensity is low, the display brightness of the display screen 1105 is decreased. In another embodiment, the processor 1101 can also dynamically adjust the shooting parameters of the camera assembly 1106 based on the ambient light intensity collected by the optical sensor 1115.

[0305] The proximity sensor 1116, also known as a distance sensor, is typically mounted on the front panel of the terminal 1100. The proximity sensor 1116 is used to detect the distance between the user and the front of the terminal 1100. In one embodiment, when the proximity sensor 1116 detects that the distance between the user and the front of the terminal 1100 is gradually decreasing, the processor 1101 controls the display screen 1105 to switch from a screen-on state to a screen-off state; when the proximity sensor 1116 detects that the distance between the user and the front of the terminal 1100 is gradually increasing, the processor 1101 controls the display screen 1105 to switch from a screen-off state to a screen-on state.

[0306] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on terminal 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0307] The electronic device in the above method embodiments is implemented as a server. For example, Figure 12This is a schematic diagram of a server structure provided in an embodiment of this application. The server 1200 can vary significantly due to different configurations or performance. It includes one or more Central Processing Units (CPUs) 1201 and one or more memories 1202. The memories 1202 store at least one computer program, which is loaded and executed by the processor 1201 to implement the image registration method or image registration network training method provided in the above-described method embodiments. Of course, the server also has wired or wireless network interfaces and input / output interfaces for input and output. The server also includes other components for implementing device functions, which will not be elaborated here.

[0308] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including at least one computer program, which is executable by a processor to perform the image registration method or image registration network training method in the above embodiments. For example, the computer-readable storage medium is a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0309] In an exemplary embodiment, a computer program product or computer program is also provided, the computer program product or computer program comprising one or more lines of program code stored in a computer-readable storage medium. One or more processors of an electronic device read the one or more lines of program code from the computer-readable storage medium, and the one or more processors execute the one or more lines of program code, causing the electronic device to perform the image registration method or image registration network training method described above.

[0310] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0311] Those skilled in the art will understand that all or part of the steps of the above embodiments are implemented by hardware, or by a program instructing related hardware to implement them. The program is stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0312] The above description is only an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image registration method, characterized in that, The method includes: Based on the image registration network, the reference image and the first image are registered to obtain the initial registration result; The initial second image registered with the first image in the initial registration result is displayed on the screen. In response to receiving an optimization instruction for the initial registration result, the initial transformation parameters corresponding to the initial registration result are updated; or, in response to the similarity between the initial second image registered with the first image in the initial registration result and the reference image being less than a first similarity threshold, the initial transformation parameters corresponding to the initial registration result are updated. The first image is transformed based on the updated transformation parameters to obtain the second image; In response to the similarity between the reference image and the second image being less than a second similarity threshold, the transformation parameters are updated, and transformation processing and further updates are performed based on the updated transformation parameters until the target condition is met, at which point the target registration result is obtained. The target condition includes the number of iterations reaching a target number or the similarity converging. Based on the target image pair and the target registration result, the network parameters of the image registration network are trained. The target image pair includes the reference image and the first image.

2. The method according to claim 1, characterized in that, The image registration network includes a first branch and a second branch, wherein the first branch is used for rigid body registration and the second branch is used for non-rigid body registration. The image registration network performs image registration on the reference image and the first image to obtain an initial registration result, including: In response to the rigid body registration command, image registration is performed on the reference image and the first image based on the first branch of the image registration network to obtain an initial registration result. The initial transformation parameter corresponding to the initial registration result is the initial affine matrix. In response to the non-rigid registration instruction, image registration is performed on the reference image and the first image based on the second branch of the image registration network to obtain an initial registration result. The initial transformation parameters corresponding to the initial registration result are the initial vector field.

3. The method according to claim 2, characterized in that, The update of the initial transformation parameters corresponding to the initial registration result in response to the initial registration result satisfying the condition includes any one of the following: In response to the initial registration result satisfying the condition, the initial affine matrix corresponding to the initial registration result is updated; In response to the initial registration result satisfying the condition, the initial vector field corresponding to the initial registration result is updated; The transformation process performed on the first image based on the updated transformation parameters to obtain the second image includes any of the following: The second image is obtained by performing an affine transformation on the first image based on the updated affine matrix. The first image is subjected to strain transformation based on the updated vector field to obtain the second image.

4. The method according to claim 1, characterized in that, The process of obtaining the similarity between the reference image and the second image includes: Based on the modal information of the reference image and the first image, the similarity between the reference image and the second image is obtained using the similarity acquisition method corresponding to the modal information.

5. The method according to claim 4, characterized in that, The step of obtaining the similarity between the reference image and the second image based on the modal information of the reference image and the first image, using a similarity acquisition method corresponding to the modal information, includes: In response to the fact that the modal information of the reference image is the same as that of the first image, the normalized cross-correlation coefficient between the sub-image set of the reference image and the second image is obtained, and the normalized cross-correlation coefficient is the similarity between the reference image and the second image; In response to the difference between the modal information of the reference image and the modal information of the first image, normalized mutual information between the sub-image set of the reference image and the second image is obtained, and the similarity between the reference image and the second image is obtained based on the normalized mutual information.

6. The method according to claim 5, characterized in that, The step of obtaining the similarity between the reference image and the second image based on the normalized mutual information includes: In response to the transformation parameters being a vector field, the transformation processing of the first image is a strain transformation, and a smoothing constraint value is obtained based on the vector field; The similarity between the reference image and the second image is obtained based on the smoothing constraint value and the normalized mutual information.

7. A method for training an image registration network, characterized in that, The method includes: Obtain sample image pairs, wherein the sample image pair includes a sample reference image and a sample first image, and the sample image pair carries image type information; Based on the image type information carried by the sample image pair, image registration is performed on the sample image pair based on the branch corresponding to the image type information in the image registration network to obtain transformation parameters. Different branches have different image registration functions. Based on the transformation parameters, the network parameters of the image registration network are trained; Based on the image registration network, image registration is performed on the reference image and the first image to obtain the initial registration result; The initial second image registered with the first image in the initial registration result is displayed on the screen. In response to receiving an optimization instruction for the initial registration result, the initial transformation parameters corresponding to the initial registration result are updated; or, in response to the similarity between the initial second image registered with the first image in the initial registration result and the reference image being less than a first similarity threshold, the initial transformation parameters corresponding to the initial registration result are updated. The first image is transformed based on the updated transformation parameters to obtain the second image; In response to the similarity between the reference image and the second image being less than a second similarity threshold, the transformation parameters are updated, and transformation processing and further updates are performed based on the updated transformation parameters until the target condition is met, at which point the target registration result is obtained. The target condition includes the number of iterations reaching a target number or the similarity converging. Based on the target image pair and the target registration result, the network parameters of the image registration network are trained. The target image pair includes the reference image and the first image.

8. The method according to claim 7, characterized in that, The image type information includes a first image type and a second image type; The step of performing image registration on the sample image pair based on the image type information carried by the sample image pair and the branch corresponding to the image type information in the image registration network to obtain transformation parameters includes: In response to the image type information carried by the sample image pair being a first image type, image registration is performed on the sample image pair based on the first branch in the image registration network to obtain an affine matrix, wherein the affine matrix is ​​a transformation parameter, and the first branch is used for rigid body registration. In response to the image type information carried by the sample image pair being a second image type, image registration is performed on the sample image pair based on the second branch in the image registration network to obtain a vector field, wherein the vector field is a transformation parameter, and the second branch is used for non-rigid body registration.

9. The method according to claim 7, characterized in that, The method further includes: In response to the update instruction of the image registration network, batch data is acquired, the batch data including the target image pair and the sample image pair; Based on the batch data, the trained image registration network is trained to obtain an updated image registration network.

10. An image registration device, characterized in that, The device includes: The registration module is used to perform image registration between the reference image and the first image based on the image registration network to obtain the initial registration result. An update module is configured to display the initial second image registered with the first image in the initial registration result on the screen, and update the initial transformation parameters corresponding to the initial registration result in response to receiving an optimization instruction for the initial registration result; or, update the initial transformation parameters corresponding to the initial registration result in response to the similarity between the initial second image registered with the first image in the initial registration result and the reference image being less than a first similarity threshold. A transformation module is used to transform the first image based on the updated transformation parameters to obtain a second image; The update module and the transformation module are further configured to update the transformation parameters in response to the similarity between the reference image and the second image being less than a second similarity threshold, and continue to perform transformation processing and update the transformation parameters based on the updated transformation parameters until the target condition is met, thereby obtaining the target registration result; The training module is used to train the network parameters of the image registration network based on the target image pair and the target registration result, wherein the target image pair includes the reference image and the first image.

11. An image registration device, characterized in that, The device includes: The acquisition module is used to acquire sample image pairs, wherein the sample image pair includes a sample reference image and a sample first image, and the sample image pair carries image type information; The registration module is used to perform image registration on the sample image pair based on the image type information carried by the sample image pair and the branch corresponding to the image type information in the image registration network to obtain transformation parameters; The training module is used to train the network parameters of the image registration network based on the transformation parameters; The registration module is further configured to perform image registration on the reference image and the first image based on the image registration network to obtain an initial registration result; An update module is configured to display the initial second image registered with the first image in the initial registration result on the screen, and update the initial transformation parameters corresponding to the initial registration result in response to receiving an optimization instruction for the initial registration result; or, update the initial transformation parameters corresponding to the initial registration result in response to the similarity between the initial second image registered with the first image in the initial registration result and the reference image being less than a first similarity threshold. A transformation module is used to transform the first image based on the updated transformation parameters to obtain a second image; The update module and the transformation module are further configured to update the transformation parameters in response to the similarity between the reference image and the second image being less than a second similarity threshold, and continue to perform transformation processing and update the transformation parameters based on the updated transformation parameters until the target condition is met, thereby obtaining the target registration result. The target condition includes the number of iterations reaching a target number or the similarity converging. The training module is further configured to train the network parameters of the image registration network based on the target image pair and the target registration result, wherein the target image pair includes the reference image and the first image.

12. An electronic device, characterized in that, The electronic device includes one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories, and the at least one computer program is loaded and executed by the one or more processors to implement the image registration method as described in any one of claims 1 to 6, or to implement the image registration network training method as described in any one of claims 7 to 9.

13. A computer-readable storage medium, characterized in that, The storage medium stores at least one computer program, which is loaded and executed by a processor to implement the image registration method as described in any one of claims 1 to 6, or to implement the image registration network training method as described in any one of claims 7 to 9.

14. A computer program product, characterized in that, The computer program product includes one or more lines of program code stored in a computer-readable storage medium. One or more processors of the electronic device read the one or more lines of program code from the computer-readable storage medium, and the one or more processors execute the one or more lines of program code, causing the electronic device to perform the image registration method as described in any one of claims 1 to 6, or to implement the image registration network training method as described in any one of claims 7 to 9.

Images