Image registration method, model training method, electronic device, and program product

Abstract

The invention is suitable for the technical field of image processing, and provides an image registration method, a model training method, electronic equipment and a program product. The image registration method comprises the following steps: acquiring a reference image and a moving image; determining a registration field based on the reference image, the moving image and a first activation function, the first activation function being associated with a first basis function, the first basis function representing a non-linear position transformation relationship of image units between the reference image and the moving image through a combination of unary functions; and registering the moving image based on the registration field to obtain a registered image. In the embodiment of the invention, the activation function associated with the first primary function is introduced, so that nonlinear complex deformation in the image can be effectively dealt with, and the accuracy of image registration is improved.

Description

Technical Field

[0001] This application belongs to the technical field of image processing, and particularly relates to an image registration method, a training method of a model, an electronic device, and a program product. Background Art

[0002] Image registration is a computer-aided technology that focuses on precisely spatially aligning two or more images acquired at different time points, different perspectives, or by different imaging devices. The purpose of this technology is to make the corresponding structures and features in the images correspond in spatial positions for effective comparison and comprehensive analysis. Image registration has extremely important application values in clinical and research fields. At the technical level, image registration usually involves complex image processing algorithms, however, traditional methods are insufficient in dealing with non-linear complex deformations. Summary of the Invention

[0003] Embodiments of this application provide an image registration method, an electronic device, and a computer program product. By introducing an activation function associated with a first basis function, it can effectively deal with non-linear complex deformations in images and improve the accuracy of image registration.

[0004] A first aspect of the embodiments of this application provides an image registration method, including: obtaining a reference image and a moving image; determining a registration field based on the reference image, the moving image, and a first activation function, where the first activation function is associated with a first basis function, and the first basis function represents the non-linear position transformation relationship of image units between the reference image and the moving image through a combination of unary functions; registering the moving image based on the registration field to obtain a registered image.

[0005] In some embodiments of the first aspect, the determining a registration field based on the reference image, the moving image, and a first activation function includes: splicing the reference image and the moving image, and performing feature extraction on the splicing result of the reference image and the moving image to obtain a first feature map; determining a spline output based on the first activation function and the first feature map; determining the registration field based on the spline output.

[0006] In some embodiments of the first aspect, after splicing the reference image and the moving image, and performing feature extraction on the splicing result of the reference image and the moving image to obtain a first feature map, it further includes: dividing the first feature map into multiple blocks; performing a linear transformation operation on the multiple blocks.

[0007] In some embodiments of the first aspect, the image registration method further includes: determining a base output based on a second activation function and the first feature map, where the second activation function is used to represent the position transformation relationship of image units between the reference image and the moving image through an exponential function; the determining the registration field based on the spline output includes: splicing the spline output with the base output to obtain a second feature map; obtaining an adjustment weight; based on the adjustment weight, adjusting the weights of each channel of the second feature map to obtain a third feature map; and determining the registration field based on the third feature map.

[0008] In some embodiments of the first aspect, the obtaining the adjustment weight includes: compressing the second feature map and processing the compressed feature map through a third activation function to obtain a fourth feature map; multiplying the difference between the second feature map and the fourth feature map by a scaling factor to obtain the adjustment weight.

[0009] In some embodiments of the first aspect, the determining the registration field based on the third feature map includes: performing an upsampling operation and a convolution operation on the third feature map to obtain a fifth feature map; and processing the fifth feature map through a fourth activation function to obtain the registration field, where the fourth activation function is associated with a second basis function, and the second basis function is used to represent the non-linear position transformation relationship of image units between the reference image and the moving image through a combination of unary functions.

[0010] The second aspect of the embodiments of the present application provides a training method for an image registration model, including: obtaining training samples, where the training samples include sample reference images and sample moving images; inputting the sample reference images and the sample moving images into a model to be trained, and performing iterative training with the minimization of a loss function as the objective to obtain an image registration model, where the loss function is associated with the similarity between the sample reference image and the sample registration image, and the sample registration image is obtained by registering the sample moving image based on the sample registration field output by the image registration model; the image registration model is used to determine a registration field based on the reference image, the moving image, and a first activation function, where the first activation function is associated with a first basis function, and the first basis function represents the non-linear position transformation relationship of image units between the reference image and the moving image through a combination of unary functions.

[0011] In some embodiments of the second aspect, the loss function is further associated with a regularization constraint of the sample registration field output by the image registration model, and the regularization constraint is used to constrain the position change amount between each image unit and its corresponding neighboring image unit.

[0012] An image registration device provided in the third aspect of the embodiments of the present application includes: an image acquisition unit configured to acquire a reference image and a moving image; a registration field determination unit configured to determine a registration field based on the reference image, the moving image, and a first activation function, the first activation function being associated with a first basis function, the first basis function representing a non-linear position transformation relationship of image units between the reference image and the moving image through a combination of unary functions; and an image registration unit configured to register the moving image based on the registration field to obtain a registered image.

[0013] A training device for an image registration model provided in the fourth aspect of the embodiments of the present application includes: a sample acquisition unit configured to acquire training samples, the training samples including a sample reference image and a sample moving image; and a model training unit configured to input the sample reference image and the sample moving image into a model to be trained, and perform iterative training with the goal of minimizing a loss function to obtain an image registration model, the loss function being associated with the similarity between the sample reference image and a sample registered image, the sample registered image being obtained by registering the sample moving image based on a sample registration field output by the image registration model; the image registration model is configured to determine a registration field based on the reference image, the moving image, and a first activation function, the first activation function being associated with a first basis function, the first basis function representing a non-linear position transformation relationship of image units between the reference image and the moving image through a combination of unary functions.

[0014] An electronic device provided in the fifth aspect of the embodiments of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the image registration method according to any one of the first aspect are implemented, or when the processor executes the computer program, the steps of the training method of the image registration model according to any one of the second aspect are implemented.

[0015] A computer-readable storage medium provided in the sixth aspect of the embodiments of the present application stores a computer program. When the computer program is executed by a processor, the steps of the above image registration method are implemented, or the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the above image registration model training method are implemented.

[0016] A computer program product provided in the seventh aspect of the embodiments of the present application, when running on an electronic device, causes the electronic device to execute the steps of the above image registration method, or when running on an electronic device, causes the electronic device to execute the steps of the above image registration model training method.

[0017] In an embodiment of the present application, by obtaining a reference image and a moving image, a registration field is determined based on the reference image, the moving image, and a first activation function, so as to register the moving image based on the registration field to obtain a registered image. Since the first activation function is associated with a first basis function, and the first basis function represents the non-linear position transformation relationship of image units between the reference image and the moving image through a combination of unary functions, by introducing an activation function associated with the first basis function, the non-linear complex deformation in the image can be effectively dealt with, and the accuracy of image registration is improved. Description of the Drawings

[0018] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 It is a schematic implementation flowchart of an image registration method provided by an embodiment of the present application;

[0020] Figure 2 It is a schematic specific implementation flowchart of determining a registration field provided by an embodiment of the present application;

[0021] Figure 3 It is a schematic structural diagram of an image registration model provided by an embodiment of the present application;

[0022] Figure 4 It is a schematic structural diagram of a KAN module provided by an embodiment of the present application;

[0023] Figure 5 It is a schematic structural diagram of a KAN linear layer provided by an embodiment of the present application;

[0024] Figure 6 It is a schematic structural diagram of a channel aggregation layer provided by an embodiment of the present application;

[0025] Figure 7 It is a schematic implementation flowchart of a training method of an image registration model provided by an embodiment of the present application;

[0026] Figure 8 It is a schematic structural diagram of an image registration device provided by an embodiment of the present application;

[0027] Figure 9 It is a schematic structural diagram of a training device of an image registration model provided by an embodiment of the present application;

[0028] Figure 10 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0029] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts belong to the protection scope of the present application.

[0030] It should be understood that when used in the specification of the present application and the appended claims, the term "comprising" indicates the presence of the described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their combinations.

[0031] In the description of the specification of the present application and the appended claims, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance.

[0032] The reference to "one embodiment" or "some embodiments" etc. in the specification of the present application means that a specific feature, structure or characteristic described in combination with the embodiment is included in one or more embodiments of the present application. Thus, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "comprising", "including", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0033] Image registration is a computer-aided technology that focuses on precisely spatially aligning two or more images acquired at different time points, from different perspectives, or by different imaging devices. The purpose of this technology is to make the corresponding structures and features between images correspond in spatial positions for effective comparison and comprehensive analysis. Image registration has extremely important application values in the clinical and research fields. At the technical level, image registration usually involves complex image processing algorithms, and traditional methods are insufficient in dealing with non-linear complex deformations.

[0034] In view of this, the present application proposes an image registration method. By introducing an activation function associated with a first basis function, it can effectively handle non-linear complex deformations in images and improve the accuracy of image registration.

[0035] In order to illustrate the technical solution of the present application, it will be described below through specific embodiments.

[0036] Please refer to Figure 1 , Figure 1 A schematic diagram of an implementation flow of an image registration method provided in an embodiment of the present application is shown, and the method can be applied to electronic devices.

[0037] The above-mentioned electronic device may refer to an ultrasonic scanning device, a computed tomography (CT) device, a magnetic resonance (MR) device, a positron emission tomography (PET), an X-ray imaging device or other types of medical equipment, or may be a computer, a tablet computer or other smart device for image processing, which is not limited in this application.

[0038] Specifically, the above image registration method may include the following steps S101 to S103.

[0039] Step S101, obtaining a reference image and a moving image.

[0040] The reference image is an image used as a benchmark, and the moving image is an image that needs to be aligned with the reference image.

[0041] In the implementation manner of the present application, the reference image and the moving image may be selected according to the application scenario.

[0042] In some application scenarios, the reference image and the moving image can be medical images of different modalities and / or different imaging times (for example, the reference image and the moving image are preoperative images and postoperative images, respectively). By registering the reference image and the moving image, doctors can more accurately observe and compare changes in patient lesions, evaluate surgical effects, monitor complications, and develop follow-up treatment plans.

[0043] In other application scenarios, the reference image and the moving image can be the functional image and the structural image respectively. Therefore, by aligning the reference image and the moving image, the doctor can plan the surgical incision and surgical path before the operation to avoid damage to the key functional areas of the human body as much as possible, improve the success rate of the operation, and reduce the risk of postoperative sequelae.

[0044] Step S102: determining a registration field based on the reference image, the moving image and the first activation function.

[0045] In an embodiment of the present application, the first activation function is associated with the first basis function, and the first basis function can represent the non-linear position transformation relationship of image units between the reference image and the moving image through the combination of unary functions. Herein, the image unit may refer to the pixel of a two-dimensional image or the voxel of a three-dimensional image. The registration field refers to the set of transformation parameters used to align the moving image to the reference image through spatial transformation.

[0046] Specifically, the Kolmogorov-Arnold representation theorem states that any continuous function of n variables can be represented by the combination of a series of continuous unary functions. That is:

[0047]

[0048] where φ q and ψ p,q are continuous unary functions.

[0049] In other words, a complex multi-variable function can be decomposed into the combination of several simple unary functions. Therefore, by stacking and combining unary functions to form the first basis function, the first basis function can be used to model complex non-linear functions. Applying this principle to the image registration algorithm, the non-linear position transformation relationship of image units between the reference image and the moving image can be modeled using the first basis function.

[0050] In an embodiment of the present application, the first basis function can be selected according to the application scenario and effect of registration. Exemplarily, the first basis function can be any one of B-spline basis function, Radial Basis Function (RBF), Fourier basis function, and Chebyshev polynomial. Among them, the B-spline basis function represents the complex mapping relationship between the input and the registration field through piecewise polynomials, and can take into account both global and local deformations. The radial basis function takes the Euclidean distance between the input and the center point as the core, and can be used to generate a registration field with strong local response and weak global influence, which is suitable for describing significant local deformations. The Fourier basis function describes periodic characteristics through the decomposition of sine and cosine waveforms, has global approximation ability, can capture periodic patterns in dynamic medical images (such as deformations caused by heartbeat and breathing), is suitable for the registration of time-series images, and supports accurate registration field modeling of dynamic deformations. The Chebyshev polynomial is a kind of orthogonal polynomial, which can reduce the computational complexity of the high-dimensional registration field and provide stable registration field modeling in global deformation tasks (such as the overall registration of chest CT or X-ray images).

[0051] Step S103: Register the moving image based on the registration field to obtain a registered image.

[0052] In an embodiment of the present application, the registered image is the result of aligning the moving image to the space of the reference image. Based on a set of transformation parameters in the registration field, each image unit of the moving image can be resampled, and each image unit is transformed into the space of the reference image to obtain the registered image.

[0053] In an embodiment of the present application, by acquiring the reference image and the moving image, the registration field is determined based on the reference image, the moving image, and the first activation function, so as to register the moving image based on the registration field to obtain the registered image. Since the first activation function is associated with the first basis function, and the first basis function represents the non-linear position transformation relationship of image units between the reference image and the moving image through the combination of unary functions, by introducing the activation function associated with the first basis function, the non-linear complex deformation in the image can be effectively dealt with, and the accuracy of image registration is improved.

[0054] In some embodiments of the present application, as Figure 2 shown, determining the registration field based on the reference image, the moving image, and the first activation function may include: step S201 to step S203.

[0055] Step S201: Stitch the reference image and the moving image, and extract features from the stitching result of the reference image and the moving image to obtain the first feature map.

[0056] Specifically, the first feature map can characterize the image features such as textures and contours in the reference image and the moving image.

[0057] In some embodiments of the present application, the reference image and the moving image may be stitched in the channel dimension, and the stitching result is input into an encoder for feature extraction to obtain the first feature map.

[0058] Specifically, the reference image and the moving image can be stitched in the channel dimension, and the stitched result obtained by stitching is input into an encoder for downsampling and feature extraction. The image shape of the stitched result is (B, C, D, H, W), where B, C, D, H, and W respectively represent the batch size (Batch Size, B), the number of channels (Channels, C), the depth / slice number (Depth, D), the height (Heigth, H), and the width (Width, W). The encoder is a module that converts input data into a specific representation form. The encoder can achieve downsampling and feature extraction through consecutive convolutional layers to reduce the spatial resolution while increasing the number of channels. Exemplarily, the output channels of the first three convolutional layers of the encoder are 8, 32, and 32 respectively. The convolutional kernel used is a three-dimensional convolutional kernel with a size of 3 and a stride of 2. After each convolutional operation, it is processed through a LeakyReLU layer with a parameter of 0.2. In some embodiments, the above convolutional operation can be a strided convolutional operation so that the spatial resolution is reduced by half in each convolutional operation.

[0059] In some embodiments of the present application, the number of the above-mentioned encoders can be one or more. By connecting multiple encoders, the spatial resolution can be gradually reduced while the number of channels is increased to obtain a more refined feature map.

[0060] Step S202: Determine a spline output based on the first activation function and the first feature map.

[0061] Among them, the spline output (Spline Output) is a feature image generated based on the first basis function in the first activation function.

[0062] In some embodiments of the present application, after stitching the reference image and the moving image and performing feature extraction on the stitched result of the reference image and the moving image to obtain the first feature map, it may further include: dividing the first feature map into multiple blocks; performing a linear transformation operation on the multiple blocks.

[0063] Specifically, by dividing the first feature map into blocks (patches) of a fixed size and then performing a linear projection on each block, an embedded feature tensor of a fixed dimension can be generated. The shape of the embedded feature tensor is (B, num_patches, embed_dim). This method can retain the local spatial features in the first feature map while reducing the subsequent computational complexity through the change in shape.

[0064] In some embodiments of the present application, the embedded feature tensor can be encoded, and the encoded embedded feature tensor can be processed successively through a fifth activation function and a first activation function. Subsequently, a spline output is obtained through a linear combination operation. The fifth activation function can be selected as the GELU (Gaussian Error Linear Unit) activation function or other existing activation functions, and the present application does not limit this.

[0065] To improve the optimization performance, in some embodiments of the present application, the above-mentioned first activation function can also be associated with a basis function b(x) (such as the SiLU activation function). Specifically, the first activation function φ(x) can be expressed as:

[0066] φ(x) = w b b(x) + w s spline (x);

[0067] where spline(x) is the first basis function. Taking the first basis function as a B-spline function as an example, then spline(x) =

[0068] ∑ i c i B i (x), w b and w s are weights respectively.

[0069] Step S203, determining a registration field based on the spline output.

[0070] In some embodiments of the present application, the registration field can be obtained by restoring the resolution of the spline output, so that the registration field can effectively handle the non-linear complex deformation in the image and improve the accuracy of image registration.

[0071] To further improve the effect of image registration, in some embodiments of the present application, the image registration method may further include: determining a base output based on a second activation function and a first feature map, and the second activation function represents the position transformation relationship of image units between the reference image and the moving image through an exponential function.

[0072] where the base output is a feature image generated based on the second activation function and is used to provide a basic non-linear transformation. In some embodiments of the present application, the above-mentioned second activation function can be the SiLU (Sigmoid Linear Unit) activation function.

[0073] In some embodiments of the present application, the foregoing embedded feature tensor can be encoded, and the encoded embedded feature tensor is processed sequentially through a fifth activation function and a second activation function. Subsequently, a basic output is obtained through a linear combination operation.

[0074] Correspondingly, determining the registration field based on the spline output may include: splicing the spline output and the basic output to obtain a second feature map; obtaining adjustment weights; based on the adjustment weights, adjusting the weights of each channel of the second feature map to obtain a third feature map; and determining the registration field based on the third feature map.

[0075] Specifically, the basic output and the spline output can be spliced along the channel dimension to form a larger feature space, thereby obtaining a second feature map.

[0076] In some embodiments of the present application, obtaining the adjustment weights may include: compressing the second feature map and processing the compressed feature map through a third activation function to obtain a fourth feature map. Multiplying the difference between the second feature map and the fourth feature map by a scaling factor to obtain the adjustment weights.

[0077] Correspondingly, adjusting the weights of each channel of the second feature map based on the adjustment weights to obtain a third feature map may include: adding the adjustment weights to the second feature map to obtain a third feature map.

[0078] Specifically, the third feature map can be expressed as:

[0079] x ′ = x + σ(x - GELU(Decompose(x)));

[0080] where σ is an adaptive scaling factor (ElementScale), x represents the second feature map, and x ′ represents the third feature map. Decompose(x) represents compressing x, which can be implemented by performing a convolution operation with a 1*1 convolution kernel. GELU() represents processing through the GELU activation function (the third activation function). In this way, each channel can be adaptively adjusted according to the input features, highlighting important information, suppressing irrelevant or redundant features, and achieving adaptive feature fusion. Moreover, the adaptive scaling factor realizes a reasonable distribution of information between different channels, avoiding errors caused by unbalanced features.

[0081] In some embodiments of the present application, after obtaining the third feature map, the three steps of re-determining the basic output and the spline output based on the third feature map, splicing the basic output and the spline output, and adjusting the weights of each channel can be repeatedly executed one or more times.

[0082] Taking one repetition as an example, at this time, based on the first activation function and the third feature map, the spline output can be determined, and based on the second activation function and the third feature map, the basic output can be determined. After concatenating the basic output and the spline output, weight adjustment is performed on each channel to obtain a new feature map, and the new feature map is used as the updated third feature map to determine the registration field.

[0083] Moreover, after obtaining the third feature map, the registration field can be determined through upsampling and convolution operations.

[0084] Specifically, the resolution can be restored through one or more up-convolution (UP Conv) operations. In some embodiments of the present application, before each execution of the up-convolution operation, it may further include: determining the spline output based on the first activation function and determining the basic output based on the second activation function. After concatenating the basic output and the spline output, weight adjustment is performed on each channel to obtain a new feature map and obtain refined image features. Subsequently, an up-convolution operation is performed on the new feature map.

[0085] In some embodiments of the present application, the upsampling operation and the convolution operation performed on the third feature map can be implemented based on a decoder. Specifically, the decoder corresponds to the aforementioned encoder one by one. The decoder may include a first convolutional layer and a second convolutional layer. The first convolutional layer is responsible for the upsampling operation, and the input of each layer is the concatenation of the output of the previous layer and the feature map output by the corresponding encoder. Each decoder can restore the spatial resolution to the input resolution of the corresponding encoder. The second convolutional layer is an additional convolutional layer for refining the upsampled feature map to generate the final high-resolution output.

[0086] Through the upsampling operation and the convolution operation, the multi-scale features of the encoder can be fully utilized, and accurate feature reconstruction can be achieved through skip connections and gradual upsampling.

[0087] In some embodiments of the present application, the feature map output by the encoder can also be subjected to a three-dimensional convolution operation to obtain more feature details.

[0088] To further improve the adaptability of the registration field to non-linear complex deformations, in some embodiments of the present application, determining the registration field based on the third feature map may include: performing an upsampling operation and a convolution operation on the third feature map to obtain a fifth feature map; processing the fifth feature map with a fourth activation function to obtain the registration field. Among them, the fourth activation function is associated with the second basis function, and the second basis function represents the non-linear position transformation relationship between image units of the reference image and the moving image through the combination of unary functions.

[0089] Among them, the first basis function and the second basis function may be the same or different, and the fourth activation function and the first activation function may also be the same or different. Similar to the aforementioned first activation function, the fourth activation function is used to process the fifth feature map, and the non-linear position transformation relationship of image units between the reference image and the moving image can be further learned.

[0090] In some embodiments of the present application, the process of determining the registration field based on the reference image, the moving image, and the first activation function can be implemented by an image registration model.

[0091] That is, step S102 may include: inputting the reference image and the moving image into the image registration model to obtain the registration field output by the image registration model, where the activation function used in the image registration model includes the first activation function.

[0092] Specifically, the structure of the image registration model is as Figure 3 shown. The image registration model may include an encoder module, a PatchEmbed3D module, a KAN (Kolmogorov–Arnold Network) module, a transposed convolution module, a decoder module, a convolution module, and a KAN Reghead module.

[0093] Among them, the encoder module can be used to extract features from the splicing result obtained by splicing the reference image and the moving image. In Figure 3 , three encoder modules are connected in sequence. In practical applications, the number of encoder modules can be more or less, and the present application does not limit this.

[0094] The PatchEmbed3D module is connected to the encoder module and is used to divide the first feature map output by the encoder module into multiple blocks and perform a linear transformation on each block to generate an embedded feature tensor with a fixed dimension.

[0095] The KAN module is connected to the PatchEmbed3D module. As Figure 4 shown, the KAN module may include an encoding layer, a KAN Linear layer, a ChannelAggregationConv layer, a DepthwiseSeparable Convolution (DWConv) layer, and a LayerNormalization (Layernorm) layer. Among them, the encoding layer is used to encode the embedded feature tensor. As Figure 5The KAN linear layer shown is used to process the encoded embedded feature tensor through the fifth activation function (illustrated as the GELU activation function), and then process it through the first activation function and the second activation function respectively, and then process it through a linear combination operation to obtain a spline output and a base output. As Figure 6 The channel aggregation layer shown is used to concatenate the base output and the spline output along the channel dimension to obtain a second feature map, compress the second feature map through a convolutional layer, and process the compressed feature map through a third activation function to obtain a fourth feature map. Multiply the difference between the second feature map and the fourth feature map by a scaling factor to obtain an adjusted weight, and add the adjusted weight to the second feature map to obtain a third feature map, and then perform convolutional processing on the third feature map. Figure 3 "×2" in

[0096] means that there are 2 KAN modules connected after the PatchEmbed3D module. In practical applications, the number of KAN modules can be more or less, and this application does not limit this. Figure 3 After the KAN module, a convolutional processing module (shown as 31 in Figure 3 can be connected. The convolutional processing module can include a KAN module and a transposed convolutional module connected to the KAN module. The structure and function of the KAN module can refer to the previous description. The transposed convolutional module can be used to perform a transposed convolutional operation. The transposed convolutional processing module 31 can be used to perform more refined feature extraction on the third feature map and restore the resolution. Similarly,

[0097] The decoder module is connected to the transposed convolutional processing module and is used to perform upsampling operations and convolutional operations on the third feature map that has undergone feature extraction. In Figure 3 three decoder modules are connected in sequence. In practical applications, the number of decoder modules can be more or less, and this application does not limit this. And the decoder modules correspond one by one to the encoder modules.

[0098] The convolutional module is used to further perform feature extraction on the result output by the decoder module to generate a fifth feature map.

[0099] The KAN Reghead module is used to process the fifth feature map using the fourth activation function to obtain a registration field. Among them, the fourth activation function is associated with the second basis function to learn more accurate and complex non-linear relationships.

[0100] In the embodiments of the present application, through the efficient feature extraction and channel adjustment strategy based on the KAN module, multi-scale non-linear feature extraction and optimized representation of images are achieved, thereby improving the accuracy and robustness of image registration, and effectively addressing the problems of anatomical deformation and imaging noise. By extracting features at different scales through the encoder, the adaptability of the image registration model in processing multi-modal and highly individualized data can be enhanced. Such a combination significantly improves the stability of the model on cross-patient data. The image registration model can generate a registration field end-to-end, while maintaining high accuracy, significantly reducing the computational overhead, making it have significant advantages in scenarios with high real-time requirements (such as clinical scenarios).

[0101] Correspondingly, please refer to Figure 7 , Figure 7 which shows a schematic diagram of the implementation process of a method for training an image registration model provided by an embodiment of the present application. This method can be applied to an electronic device. Figure 7 The electronic device to which the method shown is applied and Figure 1 the electronic device to which the method shown is applied can be the same or different, and the present application does not limit this.

[0102] Specifically, the above image registration method may include the following steps S701 to step S702.

[0103] Step S701, obtain training samples.

[0104] In the implementation manner of the present application, the training samples include a sample reference image and a sample moving image. The sample reference image and the sample moving image are respectively the reference image and the moving image for training the model.

[0105] Step S702, input the sample reference image and the sample moving image into the model to be trained, and perform iterative training with the goal of minimizing the loss function to obtain an image registration model.

[0106] Among them, the loss function is associated with the similarity between the sample reference image and the sample registered image. The sample registered image is obtained by registering the sample moving image based on the sample registration field output by the image registration model.

[0107] The image registration model is used to determine a registration field based on a reference image, a moving image, and a first activation function. The first activation function is associated with a first basis function, and the first basis function represents the non-linear position transformation relationship of image units between the reference image and the moving image through a combination of unary functions. The specific structure and working process of the image registration model can refer to the description above Figures 1 to 6 and the present application will not elaborate on this.

[0108] Specifically, by inputting the sample reference image and the sample moving image into the model to be trained, a sample registration field output by the image registration model can be obtained. By registering the sample moving image based on the sample registration field, a sample registered image can be obtained. By calculating the similarity between the sample reference image and the sample registered image, the loss value of the loss function can be obtained. The loss value can be negatively correlated with the similarity. By adjusting the model weights of the image registration model by means of gradient descent, etc., the similarity between the sample reference image and the sample registered image can be gradually increased, and the loss value of the loss function can be decreased, and then the model converges to obtain the image registration model.

[0109] In the embodiments of the present application, model training is performed through a loss function associated with similarity. Without high-quality labeled data, the preparation cost of training samples can be greatly reduced, and the applicability of the algorithm can be improved.

[0110] In some embodiments of the present application, corresponding image registration models can be trained for different registration modes. The division and loading methods of the reference image and the moving image are different between different registration modes.

[0111] The division and loading method refers to the way of classifying images. The classification process can be to divide the images into preoperative images and postoperative images, or to divide the images according to modalities, or to divide the images according to patients, etc.

[0112] In some embodiments of the present application, after loading the medical image dataset, preprocessing such as skull stripping and rigid registration can be performed using the integrated toolbox FSL (FMRIB Software Library) or Freesufer, so that the network can better perform non-linear registration mapping learning, and then the registered images can be trained according to the division and loading methods of registering the anatomical atlas with the patient's image data (atlastopatient) or registering the patient's image data with each other (patient to patient).

[0113] In some embodiments of the present application, the above similarity can be obtained based on a similarity metric function, and the similarity metric function can be a normalized cross-correlation function or a mean square error function.

[0114] Specifically, when the above similarity metric function is a normalized cross-correlation function, the similarity LNCC(I f ,I m ,φ) can be expressed as:

[0115]

[0116] where, I f 、I m, φ represent the sample reference image, the sample moving image, and the sample registration field respectively, and represents the average voxel value within a local window of size n 3 centered on the image unit p, and Ω represents the number of image units within the local window. represents image registration of the sample moving image based on the sample registration field.

[0117] To improve the registration accuracy, in some embodiments of the present application, the loss function is further associated with a regularization constraint of the sample registration field output by the image registration model, and the regularization constraint is used to constrain the position change amount between each image unit and its corresponding neighboring image unit.

[0118] Specifically, the above loss function can be expressed as:

[0119]

[0120] where, is the loss value, represents the similarity, is the regularization constraint, and λ is a preset parameter.

[0121] The regularization constraint can be expressed as:

[0122]

[0123] where, is the spatial gradient of the sample field u. The spatial gradient can be approximated using forward differences, i.e.:

[0124]

[0125] x, y, z represent the directions of the gradient.

[0126] By constraining the position change amount between each image unit and its corresponding neighboring image unit through the regularization constraint, it is possible to prevent the image unit and its corresponding neighboring image unit from being mapped to the same position after registration by the registration field, thereby improving the accuracy of image registration.

[0127] In the embodiments of the present invention, based on the efficient encoder of the KAN module and the dynamic feature fusion strategy, multi-scale non-linear feature extraction and optimized characterization of medical image data are achieved, thereby improving the accuracy and robustness of medical image registration. By directly generating the registration field through end-to-end training of the image registration model, while maintaining high accuracy, the computational overhead is significantly reduced, giving it significant advantages in clinical scenarios with high real-time requirements.

[0128] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences.

[0129] As Figure 8 shown is a schematic structural diagram of an image registration device 800 provided by an embodiment of the present application. The image registration device 800 is configured on an electronic device.

[0130] Specifically, the image registration device 800 may include:

[0131] An image acquisition unit 801, configured to acquire a reference image and a moving image;

[0132] A registration field determination unit 802, configured to determine a registration field based on the reference image, the moving image, and a first activation function. The first activation function is associated with a first basis function, and the first basis function represents a non-linear position transformation relationship between image units of the reference image and the moving image through a combination of unary functions;

[0133] An image registration unit 803, configured to register the moving image based on the registration field to obtain a registered image.

[0134] In some embodiments of the present application, the registration field determination unit 802 may specifically be configured to: splice the reference image and the moving image, and perform feature extraction on the splicing result of the reference image and the moving image to obtain a first feature map; determine a spline output based on the first activation function and the first feature map; determine the registration field based on the spline output.

[0135] In some embodiments of the present application, the registration field determination unit 802 may specifically be configured to: after performing feature extraction on the reference image and the moving image to obtain a first feature map, divide the first feature map into multiple blocks; perform a linear transformation operation on the multiple blocks.

[0136] In some embodiments of the present application, the registration field determination unit 802 may specifically be configured to: obtain a basic output based on the second activation function and the first feature map. The second activation function represents a position transformation relationship between image units of the reference image and the moving image through an exponential function; splice the spline output with the basic output to obtain a second feature map; obtain an adjustment weight; perform weight adjustment on each channel of the second feature map based on the adjustment weight to obtain a third feature map; determine the registration field based on the third feature map.

[0137] In some embodiments of the present application, the registration field determination unit 802 may be specifically configured to: compress the second feature map, and process the compressed feature map through a third activation function to obtain a fourth feature map; multiply the difference between the second feature map and the fourth feature map by a scaling factor to obtain an adjustment weight.

[0138] In some embodiments of the present application, the registration field determination unit 802 may be specifically configured to: perform an upsampling operation and a convolution operation on the third feature map to obtain a fifth feature map; process the fifth feature map by using a fourth activation function to obtain the registration field, where the fourth activation function is associated with a second basis function, and the second basis function represents a non-linear position transformation relationship between image units of the reference image and the moving image through a combination of unary functions.

[0139] It should be noted that for the convenience and conciseness of description, the specific working process of the above image registration device 800 may refer to Figures 1 to 6 the corresponding process of the method, which will not be elaborated here.

[0140] As Figure 9 shown is a schematic structural diagram of a training device 900 for an image registration model provided by an embodiment of the present application. The training device 900 for the image registration model is configured on an electronic device.

[0141] Specifically, the training device 900 for the image registration model may include:

[0142] A sample acquisition unit 901, configured to acquire training samples, where the training samples include a sample reference image and a sample moving image;

[0143] A model training unit 902, configured to input the sample reference image and the sample moving image into a model to be trained, and perform iterative training with the goal of minimizing a loss function to obtain an image registration model. The loss function is associated with the similarity between the sample reference image and a sample registered image, and the sample registered image is obtained by registering the sample moving image based on a sample registration field output by the image registration model. The image registration model is used to determine a registration field based on the reference image, the moving image, and a first activation function, where the first activation function is associated with a first basis function, and the first basis function represents a non-linear position transformation relationship between image units of the reference image and the moving image through a combination of unary functions.

[0144] In some embodiments of the present application, the loss function is further associated with a regularization constraint of the sample registration field output by the image registration model, and the regularization constraint is used to constrain the position change amount between each image unit and its corresponding neighboring image unit.

[0145] It should be noted that, for the convenience and conciseness of description, the specific working process of the above-mentioned training device 900 for the image registration model can be referred to Figure 7 the corresponding process of the method described above, which will not be elaborated here.

[0146] As Figure 10 shown, it is a schematic diagram of an electronic device 10 provided by an embodiment of the present application. Specifically, the electronic device 10 may include: a processor 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the processor 100, such as an image registration program. When the processor 100 executes the computer program 102, the steps in the above-mentioned various embodiments of the image registration method are implemented, such as Figure 1 the steps S101 to S103 shown. Or, when the processor 100 executes the computer program 102, the steps in the above-mentioned various embodiments of the training method of the image registration model are implemented, such as Figure 7 the steps S701 to S702 shown.

[0147] Or, when the processor 100 executes the computer program 102, the functions of each module / unit in the above-mentioned various device embodiments are implemented, such as Figure 8 the functions of the image acquisition unit 801, the registration field determination unit 802, and the image registration unit 803 shown. Or, when the processor 100 executes the computer program 102, the functions of each module / unit in the above-mentioned various device embodiments are implemented, such as Figure 9 the functions of the configuration information sample acquisition unit 901 and the model training unit 902 shown.

[0148] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory 101 and executed by the processor 100 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, and these instruction segments are used to describe the execution process of the computer program in the electronic device 10.

[0149] The electronic device 10 may include, but is not limited to, a processor 100 and a memory 101. Those skilled in the art can understand that Figure 10 this is only an example of the electronic device 10, and does not constitute a limitation on the electronic device 10. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the electronic device 10 may further include input / output devices, network access devices, buses, etc.

[0150] The so-called processor 100 may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), off-the-shelf programmable gate arrays, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0151] The memory 101 may be an internal storage unit of the electronic device 10, such as the hard disk or memory of the electronic device 10. The memory 101 may also be an external storage device of the electronic device 10, such as a plug-in hard disk equipped on the electronic device 10, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 101 may also include both the internal storage unit and the external storage device of the electronic device 10. The memory 101 is used to store the computer program and other programs and data required by the electronic device 10. The memory 101 may also be used to temporarily store data that has been output or will be output.

[0152] It should be noted that for the convenience and simplicity of description, the structure of the above-mentioned electronic device 10 may also refer to the specific description of the structure in the method embodiments, which will not be elaborated here.

[0153] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the above division of each functional unit and module is used as an example. In actual applications, the above functions can be allocated to different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of this application. The specific working processes of the units and modules in the above system can refer to the corresponding processes in the foregoing method embodiments, which will not be elaborated here.

[0154] In the above embodiments, the descriptions of the respective embodiments each have their own emphasis. For parts not detailed or recorded in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0155] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0156] In the embodiments provided in this application, it should be understood that the disclosed devices / apparatuses and methods can be implemented in other ways. For example, the device / apparatus embodiments described above are merely illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of the devices or units can be in electrical, mechanical or other forms.

[0157] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0159] When the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, to implement all or part of the processes in the above-described embodiment methods of this application, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0160] The above-described embodiments are only used to illustrate the technical solutions of this application, rather than to limit them; although this application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of this application, and should all be included in the protection scope of this application.

Claims

1. An image registration method, characterized in that: include: Acquire reference images and moving images; determining a registration field based on the reference image, the moving image and a first activation function, wherein the first activation function is associated with a first basis function, and the first basis function represents a nonlinear position transformation relationship of image units between the reference image and the moving image through a combination of unary functions; The moving image is registered based on the registration field to obtain a registered image.

2. The image registration method according to claim 1, characterized in that: The determining of the registration field based on the reference image, the moving image and the first activation function comprises: splicing the reference image and the moving image, and performing feature extraction on a splicing result of the reference image and the moving image to obtain a first feature map; Determining a spline output based on the first activation function and the first feature map; The registration field is determined based on the spline output.

3. The image registration method according to claim 2, characterized in that: After the reference image and the moving image are spliced ​​and feature extraction is performed on the splicing result of the reference image and the moving image to obtain a first feature map, the method further includes: Splitting the first feature map into a plurality of blocks; A linear transform operation is performed on the plurality of blocks.

4. The image registration method according to claim 2, characterized in that: The image registration method further comprises: Determining a basic output based on a second activation function and the first feature map, wherein the second activation function represents a position transformation relationship between image units of the reference image and the moving image through an exponential function; The determining the registration field based on the spline output comprises: Splicing the spline output with the basic output to obtain a second feature map; Get the adjustment weight; Based on the adjustment weight, weight adjustment is performed on each channel of the second feature map to obtain a third feature map; Based on the third feature map, the registration field is determined.

5. The image registration method according to claim 4, characterized in that: The obtaining of the adjustment weight comprises: Compressing the second feature map, and processing the compressed feature map through a third activation function to obtain a fourth feature map; The difference between the second feature map and the fourth feature map is multiplied by a scaling factor to obtain an adjustment weight.

6. The image registration method according to claim 4, characterized in that: The step of determining the registration field based on the third feature map comprises: Performing an upsampling operation and a convolution operation on the third feature map to obtain a fifth feature map; The fifth feature map is processed by using a fourth activation function to obtain the registration field, the fourth activation function is associated with a second basis function, and the second basis function represents the nonlinear position transformation relationship between the image units of the reference image and the moving image through a combination of unary functions.

7. A training method for an image registration model, characterized in that: include: Acquire a training sample, wherein the training sample includes a sample reference image and a sample moving image; The sample reference image and the sample moving image are input into a model to be trained, and iterative training is performed with the goal of minimizing a loss function to obtain an image registration model, wherein the loss function is associated with the similarity between the sample reference image and the sample registration image, and the sample registration image is obtained by registering the sample moving image based on a sample registration field output by the image registration model; the image registration model is used to determine a registration field based on a reference image, a moving image and a first activation function, wherein the first activation function is associated with a first basis function, and the first basis function represents a nonlinear position transformation relationship between image units between the reference image and the moving image through a combination of unary functions.

8. The method for training an image registration model according to claim 7, wherein: The loss function is also associated with a regularization constraint of a sample registration field output by the image registration model, wherein the regularization constraint is used to constrain a position change between each image unit and a corresponding neighborhood image unit.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the image registration method as described in any one of claims 1 to 6 are implemented, or when the processor executes the computer program, the steps of the image registration model registration method as described in any one of claims 7 to 8 are implemented.

10. A computer program product, characterized in that It comprises a computer program, which, when executed by a processor, implements the steps of the image registration method as described in any one of claims 1 to 6, or, when executed by a processor, implements the steps of the image registration model training method as described in any one of claims 7 to 8.

Images