Medical image registration method based on image segmentation and convolutional neural network

Abstract

The invention discloses a medical image registration method based on image segmentation and a convolutional neural network, and relates to the field of medical image registration, and the method specifically comprises the steps: obtaining image data, carrying out the segmentation and preprocessing based on the medical image data, constructing a multi-dimensional space-time memory network based on the segmented and preprocessed data obtained in S100, and carrying out the segmentation and preprocessing based on the multi-dimensional space-time memory network. S200, training and optimizing the multi-dimensional space-time memory network, performing weight adjustment and optimization based on the multi-dimensional space-time memory network obtained in the S200, constructing and optimizing a deformation model based on a result output by the multi-dimensional space-time memory network after weight adjustment and optimization in the S300, and performing adjustment and continuous optimization based on a result output by the deformation model in the S400. By constructing a multi-dimensional space-time memory network and combining spatial features extracted by a convolutional neural network and a dynamic deformation model, the precision of dynamic image registration is improved, and through an adaptive weight adjustment mechanism, registration weights among frames in an image sequence are dynamically adjusted, so that the continuity of space-time embedded features and the precision of a registration result are ensured.

Description

Technical Field

[0001] The present invention relates to the field of medical image registration, and in particular to a medical image registration method based on image segmentation and convolutional neural network. Background Art

[0002] With the rapid development of medical technology, medical imaging has become one of the important means of modern clinical diagnosis and treatment. Medical imaging technologies, such as computed tomography, magnetic resonance imaging, and ultrasound imaging, can provide doctors with accurate anatomical and functional information to help diagnose a variety of diseases. Especially in the field of dynamic imaging, medical equipment can continuously capture the movement changes of organs in the patient's body, such as the beating of the heart and the expansion of the lungs during breathing. Such dynamic images provide rich time series data, which helps doctors locate lesions, monitor treatment effects, and track disease progression. In order to ensure that image data acquired from different time points are clinically usable, image registration technology is widely used in the field of image processing. Its purpose is to accurately align images acquired at different times or angles, thereby eliminating errors between images and improving image coherence and diagnostic accuracy.

[0003] Although image registration technology has been widely used in the field of medical image processing, the existing technology still has some shortcomings. Most of the existing registration methods mainly process single-frame images and lack the analysis of the global spatiotemporal correlation of dynamic image sequences, resulting in incoherent registration results between previous and next frames, which cannot effectively reflect the dynamic changes of organs or lesions.

[0004] In addition, existing technologies perform poorly in dealing with complex non-rigid deformations, especially for deformations of organs over time, such as changes in the lungs caused by breathing or changes in the shape of the heart when beating. Existing registration algorithms often have difficulty capturing these complex movements and deformations, resulting in insufficient registration accuracy, which affects the accuracy of diagnosis and the effectiveness of subsequent treatment.

[0005] Finally, traditional image registration methods usually rely on feature matching of static images and cannot fully utilize the temporal correlation between previous and next frames in dynamic images, which makes the consistency of dynamic image sequence registration in the temporal dimension poor. Summary of the invention

[0006] The purpose of the present invention is to provide a medical image registration method based on image segmentation and convolutional neural network in view of the shortcomings of the prior art, aiming to solve the problem of spatiotemporal coherence of the prior art in processing dynamic images by constructing a multidimensional spatiotemporal memory network and combining spatiotemporal embedding features. First, based on each frame in the dynamic image sequence, a convolutional neural network is used to extract the spatial features of each frame image, and a spatiotemporal embedding feature is generated to capture the dynamic association between each frame image and the previous and next frames. By introducing a spatiotemporal consistency loss function, the feature difference between adjacent frames is reduced, and the overall registration accuracy of the image sequence is improved. In addition, a dynamic deformation model is introduced to accurately capture the deformation of non-rigid organs, making the deformation capture process smoother and improving the registration consistency of the previous and next frame images. Finally, through an adaptive weight adjustment mechanism, the spatiotemporal features of the current frame and the previous and next frames are optimized and adjusted to ensure the consistency of the dynamic image sequence in the time dimension, significantly improve the accuracy and coherence of the registration, and overcome the problem of insufficient ability of traditional methods to handle complex deformations.

[0007] To this end, the present application provides a medical image registration method based on image segmentation and convolutional neural network, comprising the following steps:

[0008] Step S100, acquiring image data, and performing segmentation and preprocessing based on the medical image data;

[0009] Step S200: constructing a multidimensional spatiotemporal memory network based on the segmented and preprocessed data obtained in step S100, and training and optimizing the multidimensional spatiotemporal memory network;

[0010] Step S300, adjusting and optimizing weights based on the multidimensional spatiotemporal memory network obtained in step S200;

[0011] Step S400, based on the result output by the multidimensional spatiotemporal memory network after weight adjustment and optimization in step S300, construct and optimize the deformation model;

[0012] Step S500, adjusting and continuously optimizing based on the output result of the deformation model in step S400;

[0013] Step S600: Based on the multi-dimensional spatiotemporal memory network obtained in step S200 and the deformation model obtained in step S400, a final registration parameter result is generated and analyzed.

[0014] In some specific implementations, obtaining medical image data, and performing segmentation and preprocessing based on the medical image data specifically include:

[0015] Step S100.1, acquiring dynamic image sequence data based on ultrasound imaging;

[0016] The dynamic image sequence data includes: heart data and lung data.

[0017] Step S100.1.1: label the obtained dynamic image sequence data based on the image labeling tool, and obtain a segmentation labeling file.

[0018] Step S100.2, preprocessing based on the obtained dynamic image sequence data;

[0019] Among them, preprocessing includes: image denoising, data enhancement, standardization, normalization and time frame alignment of dynamic image sequences.

[0020] It should be noted here that pretreatment is an existing technology well known to those skilled in the art and will not be described in detail here.

[0021] In some specific embodiments, based on the obtained segmented and preprocessed data, a multidimensional spatiotemporal memory network is constructed, and the multidimensional spatiotemporal memory network is trained and optimized, specifically including:

[0022] Step S200.1, using a convolutional neural network to extract spatial features from each frame of the dynamic image sequence data, and obtain spatial features;

[0023] Among them, the spatial features of different levels are extracted layer by layer from the dynamic image sequence data through the convolution kernel in the convolutional neural network;

[0024] The feature information includes edge, shape and texture.

[0025] Step S200.2: construct a multidimensional spatiotemporal memory network based on the obtained spatial features.

[0026] Step S200.2.1: Introduce a spatiotemporal memory unit, take each frame of dynamic image sequence data as input, and generate the corresponding spatiotemporal embedding feature H through the spatiotemporal memory unit t , construct a multidimensional spatiotemporal memory network, specifically:

[0027] H t =α t ·X t +β t ·H t-1 +γ t ·M t-1

[0028] Where: H t is the spatiotemporal embedding feature of the tth frame, which is the result of combining the features of the current frame and its previous and next frames. t is the spatial feature of the tth frame, extracted by the convolutional neural network, and contains the spatial information of the frame;

[0029] H t-1It is the spatiotemporal embedding feature of the previous frame t-1, which means the feature generated after the previous frame image is processed by the spatiotemporal memory unit, and it will be transferred to the current frame as memory;

[0030] M t-1 It is the memory unit state of the previous frame, storing the cumulative memory of the spatiotemporal embedding features of the frame, which is used to regulate the impact of spatiotemporal changes on the current frame;

[0031] α t , β t , γ t They are dynamic factors that control the weights of the current frame features, the previous frame embedding features, and the previous frame memory state. These factors are dynamically adjusted according to the changes in the time series;

[0032] Step S200.2.2: Dynamically adjust based on the position of the current frame in the dynamic image sequence data and its change.

[0033] Introducing dynamic adjustment factor: α t , β t , γ t , the introduced dynamic adjustment factor will be dynamically adjusted according to the position and change of the current frame in the dynamic image sequence data; specifically:

[0034]

[0035] β t =1-α t

[0036]

[0037] Where α t Is to control the current frame feature X t Embedding H t The influence of, with the change of time step t, it changes adaptively according to the center time T / 2 of the dynamic image sequence data, with an initial value of 1, and gradually decreases in the middle of the sequence;

[0038] β t is to control the embedding feature H of the previous frame t-1 The weight of t Adaptive adjustment of changes in γ t It controls the previous frame state M t-1 The impact on the current frame embedding, usually set to α t to keep the memory of the previous and next frames balanced.

[0039] Step S200.2.3: Introduce memory unit M t , used to accumulate the features of all previous frames, specifically:

[0040] Mt =f t ·M t-1 +i t ·X t

[0041] Where M t is the current state of the memory unit, which is used to store the cumulative memory of the dynamic image sequence, f t is the forget gate factor, which controls the influence of the memory state of the previous frame on the current frame. t It is the input gate factor, which controls the update amount of the memory unit by the current frame feature.

[0042] Step S200.3.1, introduce the spatiotemporal consistency loss function to train the multidimensional spatiotemporal memory network, specifically:

[0043]

[0044] Where: is a spatiotemporal consistency loss function, which is used to constrain the spatiotemporal embedding features Z between adjacent time frames. t and Z t+1 Differences between,smoothing lines and consistency optimization by minimizing the differences between adjacent frames;

[0045] It represents the accumulation operation of all time steps t from 1 to T-1, where T is the total number of frames in the time series,

[0046] Z t -Z t+1 2 is the t-th spatiotemporal embedding feature Z t and the spatiotemporal embedding feature Z of the t+1th frame t+1 The square of the distance in Euclidean L2 between them is used to embed features in space and time by minimizing the distance.

[0047] Step S200.3: The spatiotemporal embedding feature H obtained based on the training result of the multidimensional spatiotemporal memory network t , by weighted fusion of the spatial features X of the current frame t and the memory state M of the previous frame t-1 , and perform adaptive weight adjustment optimization, specifically:

[0048] Z t =λ1·H t +λ2·X t

[0049] Where: Z t It represents the final optimized spatiotemporal embedding features, which are used for subsequent image segmentation and registration, combining the spatial features and spatiotemporal embedding features of the current frame;

[0050] λ1 and λ2 are the control space-time embedding features H t and spatial feature X t In the final fusion feature Z t The weight factor in is initially set to be greater than 0, indicating that the spatiotemporal embedding features occupy a large proportion in the fusion. Big weight.

[0051] In some specific implementations, adaptive weight adjustment is used to adjust the weight of the multidimensional spatiotemporal memory network output, specifically including:

[0052] Step S300.1, using adaptive weight adjustment, combining the spatial features and the previous and next frame association features of the spatiotemporal memory unit, and adaptively adjusting the weights of the spatial features and the previous and next frame association features;

[0053] Fusion of spatial features X of the tth frame t , a convolutional neural network is used to extract static information of anatomical structures;

[0054] Fusion of the spatiotemporal embedding feature H of the previous frame t-1 t-1 , obtain the correlation information between the previous and next frames in the time series.

[0055] Using dynamic adjustment factor α t and β t Control the spatial features X of the tth frame respectively t and the previous frame spatiotemporal embedding H t-1 The weight ratio is adjusted based on the changing trend of the dynamic image sequence data, and the weight is adjusted adaptively. Specifically:

[0056] H t =α t ·X t +β t ·H t-1

[0057] In the formula, H t It is the spatiotemporal embedding feature corresponding to the current time step t, which is the result of this calculation. The spatiotemporal embedding feature integrates the spatial features of the current frame image and the spatiotemporal embedding features of the previous one;

[0058] α t is a dynamic adjustment factor used to control the spatial feature X of the current time step t t In the final space-time embedding H t The weight in X t It is the spatial feature of the current time step t, which is extracted from the dynamic image sequence data through a convolutional neural network and represents the spatial information of the current frame;

[0059] β tis a dynamic adjustment factor used to control the previous spatiotemporal embedding feature H t-1 The weight, H t-1 It is the spatiotemporal embedding feature corresponding to the previous time step t-1, representing the spatiotemporal embedding feature of the previous frame of image;

[0060] Dynamic regulatory factor alpha t and β t , based on the position of the current image frame t in the dynamic image sequence data and the change trend of the image data, adaptive adjustment is performed;

[0061] The spatial feature weight of the current frame is high in the initial stage of the sequence, while in the middle and late stages of the sequence, the spatiotemporal correlation feature weights of the previous and next frames will gradually increase.

[0062] Step S300.2: Based on the adaptive weight adjustment, the spatiotemporal embedding feature H after weight adjustment is finally obtained. t ;

[0063] Spatiotemporal embedding feature H t It will contain the spatial information of the current frame and the dynamic change information of the previous and next frames;

[0064] Among them, the output spatiotemporal embedding feature H t Combined with the spatial features X of the current frame t and the spatiotemporal embedding features H of the previous and next frames of the spatiotemporal memory unit t-1 And by adaptively adjusting the weights, the smoothness and consistency of spatiotemporal embedding features are ensured.

[0065] In some specific implementations, based on the output result of the multidimensional spatiotemporal memory network after weight adjustment in step S300, a deformation model is constructed and optimized, specifically including:

[0066] Step S400.1: Based on spatial feature X t And the weighted spatiotemporal embedding feature H t Construct a dynamic deformation model, specifically:

[0067] ΔX t =X t+1 -X t

[0068] ΔH t =H t+1 -H t

[0069] Where ΔX t It represents the change of spatial features between the tth frame and the t+1th frame, reflecting the deformation of organs or tissues. ΔH t It represents the change of spatiotemporal embedding features and shows the dynamic changes between previous and next frames.

[0070] Step S400.1.1: Capture the deformation trend of each frame of image based on the constructed deformation model and derive the deformation trend.

[0071] Step S400.2: derive the deformation trend result based on the deformation model, register the deformation information of each frame, and dynamically adjust the registration parameters of the dynamic image sequence data, specifically:

[0072] P t =P t-1 +γ(ΔX t +ΔH t

[0073] Where P t is the tth registration parameter, which is used to complete the precise registration on this image. t-1 is the registration parameter of the previous frame, γ is the optimization step size, which is used to control the amplitude of the adjustment of the registration parameter, ΔX t and ΔH t are the changes of spatial features and spatiotemporal embedding features respectively.

[0074] Step S400.2, evaluating and correcting based on the obtained registration parameters;

[0075] The evaluation and correction are: the evaluation is performed by comparing the registration results of the current frame with those of the previous and next frames. If there is an error in the registration, correction will be performed based on the registration results. Specifically:

[0076] P t =P t +λ(F t -P t )

[0077] Where P t is the corrected registration parameter, F t It is the target registration parameter evaluated by the feedback mechanism based on the registration results of the current frame and the previous and next frames, and λ is the feedback adjustment step size, which is used to control the correction amplitude.

[0078] In some specific implementations, adjusting and continuously optimizing based on the output result of the deformation model in step S400 specifically includes:

[0079] Step S500.1: Based on the evaluated and corrected registration parameters, detect the deformation difference between the current frame and the previous and next frames. If the deformation difference is large, it is considered that there is an error in the registration parameters of the current frame.

[0080] Step S500.1.1: Dynamically adjust the registration parameter error by dynamic adjustment, specifically:

[0081] P t =P t-λE t

[0082] P t is the corrected registration parameter, which is dynamically adjusted to ensure that the registration error between image frames is corrected. t is the current frame registration error detected by the feedback mechanism, and λ is the inverse adjustment step size, which is used to control the correction amplitude.

[0083] Step S500.2, repeatedly performing iterative optimization based on the registration parameter error of each frame;

[0084] It should be noted here that iterative optimization is an existing technology well known to those skilled in the art and will not be described in detail here.

[0085] Step S500.2.1: Based on the registration parameter error after iterative optimization, the registration parameters are continuously adjusted, the error is detected, and correction is performed according to the deformation difference. When the registration parameter result is consistent with the previous and next frames, the iterative optimization is stopped.

[0086] In some specific implementations, based on the multidimensional spatiotemporal memory network obtained in step S200 and the deformation model obtained in step S400, a final registration parameter result is generated and analyzed, specifically including:

[0087] Step S600.1: Perform final registration based on the optimized registration parameters.

[0088] Step S600.1.1: Combine spatial feature X t and spatiotemporal embedding features H t , adaptive weight adjustment is used to optimize the final registration parameter result Z t .

[0089] Step S600.2, performing segmentation based on the generated final registration parameter result, extracting the lesion area, and performing further analysis;

[0090] The analysis includes: comparing the changes in features, dynamic trends, texture features and spatiotemporal embedding features in different frames.

[0091] In summary, the present application provides a medical image registration method based on image segmentation and convolutional neural network. First, image data is acquired, and segmentation and preprocessing are performed based on the medical image data. Based on the segmented and preprocessed data obtained in S100, a multidimensional space-time memory network is constructed, and the multidimensional space-time memory network is trained and optimized. Based on the multidimensional space-time memory network obtained in S200, weight adjustment and optimization are performed. Based on the output result of the multidimensional space-time memory network after weight adjustment and optimization in S300, a deformation model is constructed and optimized. Based on the output result of the deformation model in S400, adjustment and continuous optimization are performed. Based on the multidimensional space-time memory network obtained in S200, the weight adjustment and optimization are performed. The final registration parameter results are generated and analyzed by combining the deformation model obtained by the memory network and S400. By constructing a multidimensional spatiotemporal memory network and combining the spatial features extracted by the convolutional neural network and the dynamic deformation model, the accuracy of dynamic image registration is improved. The registration weights between frames in the image sequence are dynamically adjusted through the adaptive weight adjustment mechanism, thereby ensuring the coherence of the spatiotemporal embedding features and the accuracy of the registration results. This method solves the problems of insufficient complex deformation processing and weak spatiotemporal correlation in the prior art, ensures a smooth transition of deformation between the previous and next frames, improves the overall registration effect of dynamic image sequence data, and is suitable for complex medical image analysis and processing scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 This is an overall flow chart of a medical image registration method based on image segmentation and convolutional neural network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] Please refer to Figure 1 , which shows the process of an embodiment of a medical image registration method based on image segmentation and convolutional neural network according to the present disclosure

[0094] like Figure 1 As shown, a medical image registration method based on image segmentation and convolutional neural network includes the following steps:

[0095] Step S100, acquiring image data, and performing segmentation and preprocessing based on the medical image data;

[0096] Step S200: constructing a multidimensional spatiotemporal memory network based on the segmented and preprocessed data obtained in step S100, and training and optimizing the multidimensional spatiotemporal memory network;

[0097] Step S300, adjusting and optimizing weights based on the multidimensional spatiotemporal memory network obtained in step S200;

[0098] Step S400, based on the result output by the multidimensional spatiotemporal memory network after weight adjustment and optimization in step S300, construct and optimize the deformation model;

[0099] Step S500, adjusting and continuously optimizing based on the output result of the deformation model in step S400;

[0100] Step S600: Based on the multi-dimensional spatiotemporal memory network obtained in step S200 and the deformation model obtained in step S400, a final registration parameter result is generated and analyzed.

[0101] In this embodiment, medical image data is obtained, and segmentation and preprocessing are performed based on the medical image data, specifically including:

[0102] Step S100.1, acquiring dynamic image sequence data based on ultrasound imaging;

[0103] The dynamic image sequence data includes: heart data and lung data.

[0104] Step S100.1.1: label the obtained dynamic image sequence data based on the image labeling tool, and obtain a segmentation labeling file.

[0105] Step S100.2, preprocessing based on the obtained dynamic image sequence data;

[0106] Among them, preprocessing includes: image denoising, data enhancement, standardization, normalization and time frame alignment of dynamic image sequences.

[0107] It should be noted here that pretreatment is an existing technology well known to those skilled in the art and will not be described in detail here.

[0108] In this embodiment, based on the obtained segmented and preprocessed data, a multidimensional spatiotemporal memory network is constructed, and the multidimensional spatiotemporal memory network is trained and optimized, specifically including:

[0109] Step S200.1, using a convolutional neural network to extract spatial features from each frame of the dynamic image sequence data, and obtain spatial features;

[0110] Among them, the spatial features of different levels are extracted layer by layer from the dynamic image sequence data through the convolution kernel in the convolutional neural network;

[0111] The feature information includes edge, shape and texture.

[0112] Step S200.2: construct a multidimensional spatiotemporal memory network based on the obtained spatial features.

[0113] Step S200.2.1: Introduce a spatiotemporal memory unit, take each frame of dynamic image sequence data as input, and generate the corresponding spatiotemporal embedding feature H through the spatiotemporal memory unit t , construct a multidimensional spatiotemporal memory network, specifically:

[0114] H t =α t ·X t +β t ·H t-1 +γ t ·M t-1

[0115] Where: H t is the spatiotemporal embedding feature of the tth frame, which is the result of combining the features of the current frame and its previous and next frames. t is the spatial feature of the tth frame, extracted by the convolutional neural network, and contains the spatial information of the frame;

[0116] H t-1 It is the spatiotemporal embedding feature of the previous frame t-1, which means the feature generated after the previous frame image is processed by the spatiotemporal memory unit, and it will be transferred to the current frame as memory;

[0117] M t-1 It is the memory unit state of the previous frame, storing the cumulative memory of the spatiotemporal embedding features of the frame, which is used to regulate the impact of spatiotemporal changes on the current frame;

[0118] α t , β t , γ t They are dynamic factors that control the weights of the current frame features, the previous frame embedded features, and the previous frame memory state. These factors are dynamically adjusted according to the changes in the time series.

[0119] Step S200.2.2, dynamically adjusting based on the position of the current frame in the dynamic image sequence data and its change;

[0120] Introducing dynamic adjustment factor: α t , β t , γ t , the introduced dynamic adjustment factor will be dynamically adjusted according to the position and change of the current frame in the dynamic image sequence data; specifically:

[0121]

[0122] β t =1-α t

[0123]

[0124] Where α t Is to control the current frame feature X t Embedding H tThe influence of, with the change of time step t, it changes adaptively according to the center time T / 2 of the dynamic image sequence data, with an initial value of 1, and gradually decreases in the middle of the sequence;

[0125] β t is to control the embedding feature H of the previous frame t-1 The weight of t Adaptive adjustment of changes in γ t It controls the previous frame state M t-1 The impact on the current frame embedding, usually set to α t to keep the memory of the previous and next frames balanced.

[0126] Step S200.2.3: Introduce memory unit M t , used to accumulate the features of all previous frames, specifically:

[0127] M t =f t ·M t-1 +i t ·X t

[0128] Where M t is the current state of the memory unit, which is used to store the cumulative memory of the dynamic image sequence, f t is the forget gate factor, which controls the influence of the memory state of the previous frame on the current frame. t It is the input gate factor, which controls the update amount of the memory unit by the current frame feature.

[0129] Step S200.3.1, introduce the spatiotemporal consistency loss function to train the multidimensional spatiotemporal memory network, specifically:

[0130]

[0131] Where: is a spatiotemporal consistency loss function, which is used to constrain the spatiotemporal embedding features Z between adjacent time frames. t and Z t+1 Differences between,smoothing lines and consistency optimization by minimizing the differences between adjacent frames;

[0132] It represents the accumulation operation of all time steps t from 1 to T-1, where T is the total number of frames in the time series,

[0133] Z t -Z t+1 2 is the t-th spatiotemporal embedding feature Z t and the spatiotemporal embedding feature Z of the t+1th frame t+1The square of the distance in Euclidean L2 between them is used to embed features in space and time by minimizing the distance.

[0134] Step S200.3: The spatiotemporal embedding feature H obtained based on the training result of the multidimensional spatiotemporal memory network t , by weighted fusion of the spatial features X of the current frame t and the memory state M of the previous frame t-1 , and perform adaptive weight adjustment optimization, specifically:

[0135] Z t =λ1·H t +λ2·X t

[0136] Where: Z t It represents the final optimized spatiotemporal embedding features, which are used for subsequent image segmentation and registration, combining the spatial features and spatiotemporal embedding features of the current frame;

[0137] λ1 and λ2 are the control space-time embedding features H t and spatial feature X t In the final fusion feature Z t The weight factors in are initially set to: λ1 = 0.7, λ2 = 0.3, indicating that the spatiotemporal embedding features occupy a large proportion in the fusion. Big weight.

[0138] In this embodiment, adaptive weight adjustment is used to adjust the weight of the multidimensional spatiotemporal memory network output, specifically including:

[0139] Step S300.1, using adaptive weight adjustment, combining the spatial features and the previous and next frame association features of the spatiotemporal memory unit, and adaptively adjusting the weights of the spatial features and the previous and next frame association features;

[0140] Fusion of spatial features X of the tth frame t , a convolutional neural network is used to extract static information of anatomical structures;

[0141] Fusion of the spatiotemporal embedding feature H of the previous frame t-1 t-1 , obtain the correlation information between the previous and next frames in the time series.

[0142] Using dynamic adjustment factor α t and β t Control the spatial features X of the tth frame respectively t and the previous frame spatiotemporal embedding H t-1 The weight ratio is adjusted based on the changing trend of the dynamic image sequence data, and the weight is adjusted adaptively. Specifically:

[0143] H t =α t ·Xt +β t ·H t-1

[0144] In the formula, H t It is the spatiotemporal embedding feature corresponding to the current time step t, which is the result of this calculation. The spatiotemporal embedding feature integrates the spatial features of the current frame image and the spatiotemporal embedding features of the previous one;

[0145] α t is a dynamic adjustment factor used to control the spatial feature X of the current time step t t In the final space-time embedding H t The weight in X t It is the spatial feature of the current time step t, which is extracted from the dynamic image sequence data through a convolutional neural network and represents the spatial information of the current frame;

[0146] β t is a dynamic adjustment factor used to control the previous spatiotemporal embedding feature H t-1 The weight, H t-1 It is the spatiotemporal embedding feature corresponding to the previous time step t-1, representing the spatiotemporal embedding feature of the previous frame of image;

[0147] Dynamic regulatory factor alpha t and β t , based on the position of the current image frame t in the dynamic image sequence data and the change trend of the image data, adaptive adjustment is performed;

[0148] The spatial feature weight of the current frame is high in the initial stage of the sequence, while in the middle and late stages of the sequence, the spatiotemporal correlation feature weights of the previous and next frames will gradually increase.

[0149] Step S300.2: Based on the adaptive weight adjustment, the spatiotemporal embedding feature H after weight adjustment is finally obtained. t ;

[0150] Spatiotemporal embedding feature H t It will contain the spatial information of the current frame and the dynamic change information of the previous and next frames;

[0151] Among them, the output spatiotemporal embedding feature H t Combined with the spatial features X of the current frame t and the spatiotemporal embedding features H of the previous and next frames of the spatiotemporal memory unit t-1 And by adaptively adjusting the weights, the smoothness and consistency of spatiotemporal embedding features are ensured.

[0152] In this embodiment, based on the output result of the multidimensional spatiotemporal memory network after weight adjustment in step S300, a deformation model is constructed and optimized, specifically including:

[0153] Step S400.1: Based on spatial feature X t And the weighted spatiotemporal embedding feature H t Construct a dynamic deformation model, specifically:

[0154] ΔX t =X t+1 -X t

[0155] ΔH t =H t+1 -H t

[0156] Where ΔX t It represents the change of spatial features between the tth frame and the t+1th frame, reflecting the deformation of organs or tissues. ΔH t It represents the change of spatiotemporal embedding features and shows the dynamic changes between previous and next frames.

[0157] Step S400.1.1: Capture the deformation trend of each frame of image based on the constructed deformation model and derive the deformation trend.

[0158] Step S400.2: derive the deformation trend result based on the deformation model, register the deformation information of each frame, and dynamically adjust the registration parameters of the dynamic image sequence data, specifically:

[0159] P t =P t-1 +γ(ΔX t +ΔH t

[0160] Where P t is the tth registration parameter, which is used to complete the precise registration on this image. t-1 is the registration parameter of the previous frame, γ is the optimization step size, which is used to control the amplitude of the adjustment of the registration parameter, ΔX t and ΔH t are the changes of spatial features and spatiotemporal embedding features respectively.

[0161] Step S400.2, evaluating and correcting based on the obtained registration parameters;

[0162] The evaluation and correction are: the evaluation is performed by comparing the registration results of the current frame with those of the previous and next frames. If there is an error in the registration, correction will be performed based on the registration results. Specifically:

[0163] P t =P t +λ(F t -P t )

[0164] Where P tis the corrected registration parameter, F t is the target registration parameter evaluated by the feedback mechanism based on the registration results of the current frame and the previous and next frames, and λ is the feedback adjustment step size, which is used to control the correction amplitude;

[0165] In this embodiment, adjustment and continuous optimization are performed based on the output result of the deformation model in step S400, specifically including:

[0166] Step S500.1: Based on the evaluated and corrected registration parameters, detect the deformation difference between the current frame and the previous and next frames. If the deformation difference is large, it is considered that there is an error in the registration parameters of the current frame.

[0167] Step S500.1.1: Dynamically adjust the registration parameter error by dynamic adjustment, specifically:

[0168] P t =P t -λE t

[0169] P t is the corrected registration parameter, which is dynamically adjusted to ensure that the registration error between image frames is corrected. t is the current frame registration error detected by the feedback mechanism, and λ is the inverse adjustment step size, which is used to control the correction amplitude.

[0170] Step S500.2, repeatedly performing iterative optimization based on the registration parameter error of each frame;

[0171] It should be noted that iterative optimization is an existing technology well known to those skilled in the art and will not be described in detail here.

[0172] Step S500.2.1: Based on the registration parameter error after iterative optimization, the registration parameters are continuously adjusted, the error is detected, and correction is performed according to the deformation difference. When the registration parameter result is consistent with the previous and next frames, the iterative optimization is stopped.

[0173] In this embodiment, based on the multidimensional spatiotemporal memory network obtained in step S200 and the deformation model obtained in step S400, the final registration parameter result is generated and analyzed, specifically including:

[0174] Step S600.1: Perform final registration based on the optimized registration parameters.

[0175] Step S600.1.1: Combine spatial feature X t and spatiotemporal embedding features H t , adaptive weight adjustment is used to optimize the final registration parameter result Z t .

[0176] Step S600.2, performing segmentation based on the generated final registration parameter result, extracting the lesion area, and performing further analysis;

[0177] The analysis includes: comparing the changes in features, dynamic trends, texture features and spatiotemporal embedding features in different frames.

[0178] In the above content, in practical application, first of all, the preprocessing of dynamic image sequence data is the starting step of dynamic image sequence registration. The reason why the image data needs to be denoised, standardized and frame aligned is that these processes can improve the quality of image data, eliminate noise and unnecessary errors, and thus improve the accuracy of subsequent feature extraction. In the preprocessing, the filter parameters used in the denoising process are adjusted based on experimental results, and are usually set to a median filter size of 3×3, because this size can well balance noise removal and detail retention. The standardization process scales the image data to the range of 0-1 to ensure that image data of different amplitudes are input consistently in the network, and to prevent training instability caused by excessive numerical differences.

[0179] Secondly, a multi-dimensional spatiotemporal memory network is constructed. The reason for using convolutional neural networks to extract spatial features is that convolutional neural networks can extract local features in images layer by layer. The choice of convolution kernel is based on the image resolution requirement. The smaller convolution kernel 3×3 is used to extract low-level features, while the larger convolution kernel 7×7 is used to extract higher-level shape and texture information. This design is to capture the complex structural features in dynamic image sequences.

[0180] When constructing a multidimensional spatiotemporal memory network, the spatiotemporal memory unit is selected to ensure that the correlation between the previous and next frames is preserved, especially for the dynamic changes of organs and lesions, specifically:

[0181] H t =α t ·X t +β t ·H t-1 +γ t ·M t-1

[0182] Among them, α t , β t and γ t The selection of these parameters is based on the temporal correlation between the previous and next frames in the dynamic image sequence. In the initial training, α t and β t The weight of is set to 0.5 to balance the influence of the current frame and the previous and next frames, and γ t=0.2 is smaller because the memory unit has a weaker influence on the current frame. By adjusting these parameters experimentally, the effective fusion of spatiotemporal information can be ensured.

[0183] Secondly, based on the optimization of weight adjustment, in the registration process, the spatiotemporal consistency loss function is used to optimize the smoothness between the previous and next frames. The weight factors λ1 and λ2 in the loss function are used to balance the contribution of the spatiotemporal embedding features and the spatial features of the current frame, respectively. The basis for setting λ1=0.7 and λ2=0.3 is that the spatiotemporal features of dynamic images are more important in the registration process because they can reflect the changes of the previous and next frames, while the spatial features of the current frame only account for a certain proportion to ensure the overall consistency of the registration results. The selection of the weight parameter is the result of experimental adjustment on different data sets.

[0184] Secondly, a deformation model is constructed and optimized. The dynamic deformation model is used to capture the non-rigid deformation of organs or lesions in the image sequence. The reason for choosing a deformation model is that in dynamic images, organs such as the heart and lungs will undergo obvious morphological changes with breathing or blood flow. Traditional rigid models are difficult to accurately align these dynamic changes. Specifically:

[0185] ΔX t =X t+1 -X t

[0186] ΔH t =H t+1 -H t

[0187] Among them, the dynamic deformation model captures the dynamic deformation information of the previous and next frames. The core of this formula is to use the embedded feature H t To characterize the spatiotemporal dynamics of each frame, the difference value ΔX t Used to reflect the size of the deformation between frames.

[0188] Secondly, the deformation model is adjusted and optimized, and the step size parameter η = 0.01 is used to control the adjustment amount of the registration parameters. The choice of the step size is obtained through multiple experiments to ensure that the changes in the registration parameters are smooth enough to avoid registration instability caused by abrupt adjustments. The step size is selected based on the frequency of deformation changes in each frame of the image. A smaller step size can capture complex non-rigid changes more finely, which is especially suitable for dynamic morphological changes of organs such as the heart and lungs.

[0189] Secondly, the final registration results are generated and analyzed. In the process of spatiotemporal consistency optimization, the difference between adjacent frames is reduced by the loss function. Initially, the weights λ1 and λ2 are set to 0.7 and 0.3 to ensure the dominant role of spatiotemporal features in the entire registration process. Such weight selection is the result of multiple experiments on different organs and dynamic image data sets to ensure that better registration effects can be achieved in different dynamic environments. The final registration results are as follows:

[0190] Z t =λ1·H t +λ2·X t

[0191] Among them, λ1 represents the weight of spatiotemporal embedding features and λ2 represents the spatial features of the current frame. According to experimental data, such weight distribution can ensure the registration accuracy in dynamic image sequences of different organs, especially the ability to capture the dynamic deformation of non-rigid organs.

Claims

1. A medical image registration method based on image segmentation and convolutional neural network, characterized in that: The steps include: S100, acquiring image data, and performing segmentation and preprocessing based on the medical image data; S200, constructing a multidimensional spatiotemporal memory network based on the segmented and preprocessed data obtained in S100, and training and optimizing the multidimensional spatiotemporal memory network; S300, weight adjustment and optimization based on the multi-dimensional spatiotemporal memory network obtained in S200; S400, based on the output result of the multi-dimensional spatiotemporal memory network after weight adjustment and optimization in S300, construct and optimize the deformation model; S500, adjust and continuously optimize based on the output results of the S400 deformation model; S600: Based on the multi-dimensional spatiotemporal memory network obtained in S200 and the deformation model obtained in S400, the final registration parameter results are generated and analyzed.

2. The medical image registration method based on image segmentation and convolutional neural network according to claim 1, characterized in that: Obtain medical imaging data, and perform segmentation and preprocessing based on the medical imaging data, including: S100.

1. Acquiring dynamic image sequence data based on ultrasound imaging; S100.1.

1. Label the obtained dynamic image sequence data based on an image annotation tool, and obtain a segmentation annotation file; S100.

2. Preprocessing is performed based on the obtained dynamic image sequence data.

3. The medical image registration method based on image segmentation and convolutional neural network according to claim 1, characterized in that: Based on the segmented and preprocessed data, a multidimensional spatiotemporal memory network is constructed, and the multidimensional spatiotemporal memory network is trained and optimized, including: S200.

1. Using a convolutional neural network to extract spatial features from each frame of the dynamic image sequence data, and obtain spatial features; S200.

2. Based on the obtained spatial features, a multidimensional spatiotemporal memory network is constructed; S200.2.

1. Introduce the spatiotemporal memory unit, take each frame of dynamic image sequence data as input, and generate the corresponding spatiotemporal embedding feature H through the spatiotemporal memory unit. t , construct a multidimensional spatiotemporal memory network, specifically: H t =a t ·X t +b t ·H t-1 +g t ·M t-1 Where: H t is the spatiotemporal embedding feature of the tth frame, which is the result of combining the features of the current frame and its previous and next frames. t is the spatial feature of the tth frame, extracted by the convolutional neural network, and contains the spatial information of the frame; H t-1 It is the spatiotemporal embedding feature of the previous frame t-1, which means the feature generated after the previous frame image is processed by the spatiotemporal memory unit, and it will be transferred to the current frame as memory; M t-1 It is the memory unit state of the previous frame, storing the cumulative memory of the spatiotemporal embedding features of the frame, which is used to regulate the impact of spatiotemporal changes on the current frame; α t , β t , γ t They are dynamic factors that control the weights of the current frame features, the previous frame embedding features, and the previous frame memory state. These factors are dynamically adjusted according to the changes in the time series; S200.2.2, dynamically adjusting based on the position of the current frame in the dynamic image sequence data and its changes; Introducing dynamic adjustment factor: α t , β t , γ t , the introduced dynamic adjustment factor will be dynamically adjusted according to the position and change of the current frame in the dynamic image sequence data; S200.2.3, Introducing the memory unit M t , used to accumulate features of all previous frames; S200.3.

1. Introduce the spatiotemporal consistency loss function to train the multidimensional spatiotemporal memory network, specifically: Where: is a spatiotemporal consistency loss function, which is used to constrain the spatiotemporal embedding features Z between adjacent time frames. t and Z t+1 Differences between,smoothing lines and consistency optimization by minimizing the differences between adjacent frames; It represents the accumulation operation of all time steps t from 1 to T-1, where T is the total number of frames in the time series, Z t -Z t+1 2 is the t-th spatiotemporal embedding feature Z t and the spatiotemporal embedding feature Z of the t+1th frame t+1 The square of the distance in Euclidean L2 between them is used to embed features in space and time by minimizing the distance; S200.

3. Spatiotemporal embedding feature H obtained based on the training results of multidimensional spatiotemporal memory network t , by weighted fusion of the spatial features X of the current frame t and the memory state M of the previous frame t-1 , and perform adaptive weight adjustment optimization, specifically: Z t =λ1·H t +λ2·X t Where: Z t It represents the final optimized spatiotemporal embedding features, which are used for subsequent image segmentation and registration, combining the spatial features and spatiotemporal embedding features of the current frame; λ1 and λ2 are the control space-time embedding features H t and spatial feature X t In the final fusion feature Z t The weight factor in is initially set to be greater than 0, indicating that the spatiotemporal embedding features occupy a large proportion in the fusion. Big weight.

4. The medical image registration method based on image segmentation and convolutional neural network according to claim 1, characterized in that: Adaptive weight adjustment is used to adjust the weight of the multidimensional spatiotemporal memory network output, including: S300.1, using adaptive weight adjustment, combining spatial features and the previous and next frame association features of the spatiotemporal memory unit, and adaptively adjusting the weights of the spatial features and the previous and next frame association features; Fusion of spatial features X of the tth frame t , a convolutional neural network is used to extract static information of anatomical structures; Fusion of the spatiotemporal embedding feature H of the previous frame t-1 t-1 , obtain the correlation information between the previous and next frames in the time series; Using dynamic adjustment factor α t and β t Control the spatial features X of the tth frame respectively t and the previous frame spatiotemporal embedding H t-1 The weight ratio is adjusted based on the changing trend of the dynamic image sequence data, and the weight is adjusted adaptively. Specifically: H t =a t ·X t +b t ·H t-1 In the formula, H t It is the spatiotemporal embedding feature corresponding to the current time step t, which is the result of this calculation. The spatiotemporal embedding feature integrates the spatial features of the current frame image and the spatiotemporal embedding features of the previous one; α t is a dynamic adjustment factor used to control the spatial feature X of the current time step t t In the final space-time embedding H t The weight in X t It is the spatial feature of the current time step t, which is extracted from the dynamic image sequence data through a convolutional neural network and represents the spatial information of the current frame; β t is a dynamic adjustment factor used to control the previous spatiotemporal embedding feature H t-1 The weight, H t-1 It is the spatiotemporal embedding feature corresponding to the previous time step t-1, representing the spatiotemporal embedding feature of the previous frame of image; S300.2, based on adaptive weight adjustment, the spatiotemporal embedding feature H after weight adjustment is finally obtained t .

5. The medical image registration method based on image segmentation and convolutional neural network according to claim 1, characterized in that: Based on the output results of the multi-dimensional spatiotemporal memory network after S300 weight adjustment, the deformation model is constructed and optimized, including: S400.1, based on spatial feature X t And the weighted spatiotemporal embedding feature H t Construct a dynamic deformation model, specifically: ΔX t =X t+1 -X t ΔH t =H t+1 -H t Where ΔX t It represents the change of spatial features between the tth frame and the t+1th frame, reflecting the deformation of organs or tissues. ΔH t It represents the change of spatiotemporal embedding features, showing the dynamic changes between previous and next frames; S400.1.

1. Capture the deformation trend of each frame of image based on the constructed deformation model and derive the deformation trend; S400.

2. Determine the deformation trend result based on the deformation model, register the deformation information of each frame, and dynamically adjust the registration parameters of the dynamic image sequence data, specifically: P t =P t-1 +γ(ΔX t +ΔH t ) Where P t is the tth registration parameter, which is used to complete accurate registration on this image, P t-1 is the registration parameter of the previous frame, γ is the optimization step size, which is used to control the amplitude of the adjustment of the registration parameter, ΔX t and ΔH t are the changes of spatial features and spatiotemporal embedding features respectively; S400.

2. Evaluate and correct based on the obtained registration parameters.

6. The medical image registration method based on image segmentation and convolutional neural network according to claim 1, characterized in that: Adjustment and continuous optimization based on the output results of the S400 deformation model, including: S500.

1. Based on the evaluated and corrected registration parameters, detect the deformation difference between the current frame and the previous and next frames. If the deformation difference is large, it is considered that there is an error in the registration parameters of the current frame. S500.1.

1. Dynamically adjust the registration parameter error by dynamic adjustment, specifically: P t =P t -λE t P t is the corrected registration parameter, which is dynamically adjusted to ensure that the registration error between image frames is corrected. t is the current frame registration error detected by the feedback mechanism, λ is the inverse adjustment step size, which is used to control the correction amplitude; S500.2, based on the registration parameter error of each frame, iterative optimization is repeatedly performed; S500.2.

1. Based on the registration parameter error after iterative optimization, the registration parameters are continuously adjusted, the error is detected, and correction is performed according to the deformation difference. When the registration parameter result is consistent with the previous and next frames, the iterative optimization is stopped.

7. The medical image registration method based on image segmentation and convolutional neural network according to claim 1, characterized in that: Based on the multidimensional spatiotemporal memory network obtained by S200 and the deformation model obtained by S400, the final registration parameter results are generated and analyzed, including: S600.

1. Perform final registration based on the optimized registration parameters; S600.1.1, combined with spatial features X t and spatiotemporal embedding features H t , adaptive weight adjustment is used to optimize the final registration parameter result Z t ; S600.

2. Perform segmentation based on the generated final registration parameter result, extract the lesion area, and perform further analysis.

Images