Image registration segmentation joint model training method and device

Abstract

The application discloses a kind of image registration segmentation joint model training method, device, electronic equipment and storage medium.The method comprises the following steps: obtaining a plurality of blood vessel sample images and the label information corresponding to each blood vessel sample image;Multiple blood vessel sample images are input into neural network model, and the blood vessel segmentation result of each blood vessel sample image is obtained;The blood vessel segmentation result of each blood vessel sample image is registered, and the registration result of each blood vessel sample image is obtained;Determine the joint model loss based on the blood vessel segmentation result, registration result and label information of each blood vessel sample image, adjust the parameters of neural network model based on the joint model loss, until the iteration stop condition is met to obtain the registration segmentation joint model of training completion.The registration segmentation joint model trained by the above technical solution can effectively improve the segmentation accuracy of image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and storage medium for training a joint image registration and segmentation model. Background Technology

[0002] With the development of image processing technology, its application in the medical field is becoming increasingly widespread.

[0003] Currently, the technical approach of segmenting computed tomography (CT) images using neural network models suffers from low accuracy in segmentation results. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and storage medium for training a joint image registration and segmentation model, in order to improve the accuracy of the model segmentation results.

[0005] According to one aspect of the present invention, a method for training a joint image registration and segmentation model is provided, comprising:

[0006] Acquire multiple blood vessel sample images and the corresponding label information for each blood vessel sample image;

[0007] The multiple blood vessel sample images are input into a neural network model to obtain the blood vessel segmentation results for each blood vessel sample image;

[0008] The blood vessel segmentation results of each of the blood vessel sample images are registered to obtain the registration results of each of the blood vessel sample images;

[0009] Based on the blood vessel segmentation results, registration results, and label information of each blood vessel sample image, the joint model loss is determined. The parameters of the neural network model are adjusted based on the joint model loss until the iteration stopping condition is met to obtain the trained joint registration and segmentation model.

[0010] According to another aspect of the present invention, an image registration and segmentation joint model training apparatus is provided, comprising:

[0011] The sample image acquisition module is used to acquire multiple blood vessel sample images and the label information corresponding to each blood vessel sample image;

[0012] The blood vessel image segmentation module is used to input the multiple blood vessel sample images into a neural network model to obtain the blood vessel segmentation results of each blood vessel sample image;

[0013] The blood vessel image registration module is used to register the blood vessel segmentation results of each of the blood vessel sample images to obtain the registration results of each of the blood vessel sample images;

[0014] The joint loss modeling module is used to determine the joint model loss based on the blood vessel segmentation results, registration results, and label information of each blood vessel sample image, and to adjust the parameters of the neural network model based on the joint model loss until the iteration stopping condition is met to obtain the trained registration and segmentation joint model.

[0015] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0016] At least one processor;

[0017] and a memory communicatively connected to the at least one processor;

[0018] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the image registration and segmentation joint model training method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the image registration and segmentation joint model training method according to any embodiment of the present invention.

[0020] The technical solution of this invention acquires multiple vascular sample images and corresponding label information for each vascular sample image, then inputs the multiple vascular sample images into a neural network model to obtain vascular segmentation results for each vascular sample image. The vascular segmentation results for each vascular sample image are then registered to obtain registration results. Based on the vascular segmentation results, registration results, and corresponding label information of the multiple vascular sample images, a joint model loss is determined. The parameters of the neural network model are then adjusted based on the joint model loss until the iteration stopping condition is met, resulting in a trained joint registration and segmentation model. Compared with existing technologies, the joint model loss in this embodiment considers both vascular segmentation and registration results, enabling the model to learn the correlation features between multiple vascular sample images, thereby improving the segmentation accuracy of the joint registration and segmentation model for vascular images.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of an image registration and segmentation joint model training method provided in Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of an image registration and segmentation joint model training method provided in Embodiment 2 of the present invention;

[0025] Figure 3 This is a flowchart of an image registration and segmentation joint model training method provided in Embodiment 3 of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of a neural network model provided in Embodiment 3 of the present invention;

[0027] Figure 5 This is a flowchart of an image registration and segmentation joint model training method provided in Embodiment 4 of the present invention;

[0028] Figure 6 This is a flowchart of an image registration and segmentation joint model training method provided in Embodiment 5 of the present invention;

[0029] Figure 7 This is a schematic diagram of the structure of an image registration and segmentation joint model training device according to Embodiment Six of the present invention;

[0030] Figure 8 This is a schematic diagram of the structure of an electronic device that implements the image registration and segmentation joint model training method according to embodiments of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Example 1

[0034] Figure 1 This is a flowchart of an image registration and segmentation joint model training method provided in Embodiment 1 of the present invention. This embodiment is applicable to the training of blood vessel image segmentation models. The method can be executed by an image registration and segmentation joint model training device, which can be implemented in hardware and / or software and can be configured in a computer terminal. Figure 1 As shown, the method includes:

[0035] S110. Acquire multiple blood vessel sample images and the label information corresponding to each blood vessel sample image.

[0036] In this embodiment, the vascular sample image refers to the image to be used for model training. For example, the vascular sample image can be medical imaging data, such as computed tomographic arteriography (CTA) or magnetic resonance imaging (MRI) data. Optionally, the vascular sample image can be an image containing the aorta and valve regions.

[0037] It should be noted that the label information corresponding to multiple vascular sample images can be that each vascular sample image has label information, or that some vascular sample images have label information; there is no limitation here.

[0038] Specifically, multiple vascular sample images and corresponding tag information can be obtained from the preset storage location of the electronic device, or multiple vascular sample images and corresponding tag information can be obtained from other devices connected to the electronic device or the cloud, without limitation.

[0039] S120. Input the multiple blood vessel sample images into a neural network model to obtain the blood vessel segmentation results of each blood vessel sample image.

[0040] In this embodiment, the neural network model is an untrained network model. The neural network model can be a network consisting of an encoder and a decoder or a network architecture of other types, such as U-net.

[0041] Specifically, each blood vessel sample image can be used as a channel, and the blood vessel sample images of each channel are segmented using the same neural network model to obtain the blood vessel segmentation results of each blood vessel sample image.

[0042] S130. Register the blood vessel segmentation results of each of the blood vessel sample images to obtain the registration results of each of the blood vessel sample images.

[0043] In this embodiment, the method for registering the blood vessel segmentation results of each blood vessel sample image may include non-rigid registration method and rigid registration method, and is not limited here.

[0044] Specifically, the blood vessel segmentation results of each blood vessel sample image are registered according to either a non-rigid registration method or a rigid registration method to obtain the registration results of each blood vessel sample image.

[0045] S140. Based on the blood vessel segmentation results, registration results, and label information of each blood vessel sample image, determine the joint model loss, and adjust the parameters of the neural network model based on the joint model loss until the iteration stopping condition is met to obtain the trained registration and segmentation joint model.

[0046] In this embodiment, the joint model loss refers to the loss used to train the neural network model, which can be calculated using a pre-built loss function. The pre-built loss function can consist of one or more sub-loss functions, such as the cross-entropy loss function, the mean squared error loss function, etc.

[0047] It should be noted that the joint model loss in this embodiment takes into account both the blood vessel segmentation result and the registration result, enabling the model to learn the correlation features between multiple blood vessel sample images, thereby improving the segmentation accuracy of the joint registration and segmentation model for blood vessel images.

[0048] In some optional embodiments, after obtaining the trained joint registration and segmentation model, the method further includes: acquiring a blood vessel image to be segmented; inputting the blood vessel image to be segmented into the pre-trained joint registration and segmentation model to obtain the blood vessel segmentation result.

[0049] For example, multiple images of blood vessels to be segmented within a cardiac cycle are received from a computed tomography (CT) scanner at preset time intervals. These multiple images are then used as input data to a pre-trained joint registration and segmentation model to obtain the blood vessel segmentation results, which are then output.

[0050] The technical solution of this invention acquires multiple vascular sample images and corresponding label information for each vascular sample image, then inputs the multiple vascular sample images into a neural network model to obtain vascular segmentation results for each vascular sample image. The vascular segmentation results for each vascular sample image are then registered to obtain registration results. Based on the vascular segmentation results, registration results, and corresponding label information of the multiple vascular sample images, a joint model loss is determined. The parameters of the neural network model are then adjusted based on the joint model loss until the iteration stopping condition is met, resulting in a trained joint registration and segmentation model. Compared with existing technologies, the joint model loss in this embodiment considers both vascular segmentation and registration results, enabling the model to learn the correlation features between multiple vascular sample images, thereby improving the segmentation accuracy of the joint registration and segmentation model for vascular images.

[0051] Example 2

[0052] Figure 2 This is a flowchart of an image registration and segmentation joint model training method provided in Embodiment 2 of the present invention. The method of this embodiment can be combined with various optional schemes in the image registration and segmentation joint model training method provided in the above embodiments. The image registration and segmentation joint model training method provided in this embodiment has been further optimized. Optionally, acquiring multiple vascular sample images includes: acquiring multiple vascular sample images collected based on a preset time interval within the cardiac cycle.

[0053] like Figure 2 As shown, the method includes:

[0054] S210. Acquire multiple vascular sample images collected within a preset time interval during the cardiac cycle, as well as the label information corresponding to each vascular sample image.

[0055] S220. Input the multiple blood vessel sample images into a neural network model to obtain the blood vessel segmentation results of each blood vessel sample image.

[0056] S230. Register the blood vessel segmentation results of each of the blood vessel sample images to obtain the registration results of each of the blood vessel sample images.

[0057] S240. Based on the blood vessel segmentation results, registration results, and label information of each blood vessel sample image, determine the joint model loss, and adjust the parameters of the neural network model based on the joint model loss until the iteration stopping condition is met to obtain the trained registration and segmentation joint model.

[0058] Understandably, during diastole, the internal pressure of the heart decreases, allowing blood to flow back into the heart through the vena cava; during systole, the internal pressure increases, pumping blood into the arteries. Each systole and diastole constitutes a cardiac cycle. In this embodiment, the vascular sample images can be vascular images within one or more cardiac cycles. Furthermore, within each cardiac cycle, multiple vascular sample images can be acquired at preset time intervals to obtain multi-phase vascular sample images, enriching the model training samples and improving model accuracy.

[0059] For example, the vascular sample image can be a CT angiography image. Specifically, when the preset time interval is set to 10% of the cardiac cycle, the vascular sample image can be a CT angiography image corresponding to 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, and 90% of the cardiac cycle.

[0060] The technical solution of this invention obtains multiple vascular sample images collected at preset time intervals within the cardiac cycle, as well as the label information corresponding to the multiple vascular sample images, to obtain vascular sample images of multiple phases, thereby enriching the model training samples and improving the model training accuracy.

[0061] Example 3

[0062] Figure 3 This is a flowchart of an image registration and segmentation joint model training method provided in Embodiment 3 of the present invention. The method of this embodiment can be combined with various optional schemes in the image registration and segmentation joint model training methods provided in the above embodiments. The image registration and segmentation joint model training method provided in this embodiment has been further optimized. Optionally, the neural network model includes: an encoder, used to obtain the vascular coding features of each vascular sample image after inputting each vascular sample image; and a decoder, used to obtain the vascular segmentation result of each vascular sample image after inputting the vascular coding features of each vascular sample image.

[0063] like Figure 3 As shown, the method includes:

[0064] S310. Acquire multiple blood vessel sample images and the label information corresponding to each blood vessel sample image.

[0065] S320. Input the multiple blood vessel sample images into the encoder to obtain the blood vessel coding features of each blood vessel sample image.

[0066] S330. Input the vascular coding features of each of the vascular sample images into the decoder to obtain the vascular segmentation results of each of the vascular sample images.

[0067] S340. Register the blood vessel segmentation results of each of the blood vessel sample images to obtain the registration results of each of the blood vessel sample images.

[0068] S350. Based on the blood vessel segmentation results, registration results, and label information of each blood vessel sample image, determine the joint model loss, and adjust the parameters of the neural network model based on the joint model loss until the iteration stopping condition is met to obtain the trained registration and segmentation joint model.

[0069] In this embodiment, the neural network model includes an encoder and a decoder, which work together to segment blood vessel sample images.

[0070] For example, Figure 4 This is a schematic diagram of the structure of a neural network model provided in this embodiment. The vascular sample image can be multiple CT angiography images within the cardiac cycle, including target vascular sample images and motion vascular sample images. Specifically, multiple CT angiography images within the cardiac cycle can be input into an encoder to obtain the vascular coding features of each CT angiography image. Then, the vascular coding features of each CT angiography image are input into a decoder to obtain the vascular segmentation results of each CT angiography image. The vascular segmentation results of the CT angiography image include target vascular segmentation results and motion vascular segmentation results.

[0071] The technical solution of this invention involves inputting multiple vascular sample images into an encoder to obtain the vascular coding features of each vascular sample image, enabling the encoder to learn the compressed representation of each vascular sample image, and then inputting the vascular coding features of each vascular sample image into a decoder, enabling the decoder to learn how to reconstruct the image using the compressed representation of the vascular sample image, thereby obtaining the vascular segmentation result of each vascular sample image.

[0072] Example 4

[0073] Figure 5 This is a flowchart of an image registration and segmentation joint model training method provided in Embodiment 4 of the present invention. The method of this embodiment can be combined with various optional schemes in the image registration and segmentation joint model training method provided in the above embodiments. The image registration and segmentation joint model training method provided in this embodiment has been further optimized. Optionally, registering the blood vessel segmentation results of each blood vessel sample image to obtain the registration result of each blood vessel sample image includes: registering the blood vessel segmentation results of each blood vessel sample image based on a rigid registration transformation matrix to obtain the registration result of each blood vessel sample image.

[0074] like Figure 5 As shown, the method includes:

[0075] S410. Acquire multiple blood vessel sample images and the label information corresponding to each blood vessel sample image.

[0076] S420. Input the multiple blood vessel sample images into a neural network model to obtain the blood vessel segmentation results of each blood vessel sample image.

[0077] S430. Based on the rigid registration transformation matrix, the blood vessel segmentation results of each blood vessel sample image are registered to obtain the registration results of each blood vessel sample image.

[0078] S440. Based on the blood vessel segmentation results, registration results, and label information of each blood vessel sample image, determine the joint model loss, and adjust the parameters of the neural network model based on the joint model loss until the iteration stopping condition is met to obtain the trained registration and segmentation joint model.

[0079] To preserve radial deformation in the vascular sample images, this embodiment employs a rigid registration method to register the vascular segmentation results of each vascular sample image. This method corrects for displacement caused by equipment and / or heartbeats during vascular sample image acquisition. Specifically, rigid registration can be performed on the vascular segmentation results of each vascular sample image based on the rigid registration transformation matrix, thereby obtaining the vascular registration result.

[0080] It should be noted that at the beginning of model training, the rigid registration transformation matrix is ​​a pre-set transformation matrix; during model training, the rigid registration transformation matrix is ​​continuously adjusted and optimized through joint model loss to obtain the optimal transformation matrix registration parameters for image registration.

[0081] The technical solution of this invention, by rigidly registering the blood vessel segmentation results of each blood vessel sample image according to the rigid registration transformation matrix, can correct the displacement caused by the device and / or heartbeat during the acquisition of blood vessel sample images.

[0082] Example 5

[0083] Figure 6This is a flowchart of an image registration and segmentation joint model training method provided in Embodiment 5 of the present invention. The method of this embodiment can be combined with various optional schemes in the image registration and segmentation joint model training methods provided in the above embodiments. The image registration and segmentation joint model training method provided in this embodiment has been further optimized. Optionally, the joint model loss includes registration result similarity loss and segmentation result loss; correspondingly, determining the joint model loss based on the blood vessel segmentation results of each blood vessel sample image, the registration results of each blood vessel sample image, and the label information corresponding to the multiple blood vessel sample images includes: determining the registration result similarity loss based on the blood vessel segmentation results, registration results, and label information of each blood vessel sample image; determining the segmentation result loss based on the blood vessel segmentation results and label information of each blood vessel sample image; and determining the joint model loss based on the registration result similarity loss and the segmentation result loss.

[0084] like Figure 6 As shown, the method includes:

[0085] S510. Acquire multiple blood vessel sample images and the label information corresponding to each blood vessel sample image.

[0086] S520. Input the multiple blood vessel sample images into a neural network model to obtain the blood vessel segmentation results of each blood vessel sample image.

[0087] S530. Register the blood vessel segmentation results of each of the blood vessel sample images to obtain the registration results of each of the blood vessel sample images.

[0088] S540. Determine the similarity loss of the registration result based on the blood vessel segmentation result, registration result, and label information of each of the blood vessel sample images.

[0089] S550. Determine the segmentation result loss based on the blood vessel segmentation results and label information of each of the blood vessel sample images.

[0090] S560. Determine the joint model loss based on the registration result similarity loss and the segmentation result loss.

[0091] S570. Adjust the parameters of the neural network model based on the joint model loss until the iteration stopping condition is met to obtain the trained registration and segmentation joint model.

[0092] In this embodiment, the joint model loss may include registration result similarity loss and segmentation result loss. The registration result similarity loss is determined based on the similarity between registration results. The segmentation result loss is determined based on the differences between segmentation results.

[0093] Specifically, the vessel segmentation results of each vessel sample image, the registration results of each vessel sample image, and the label information corresponding to multiple vessel sample images can be substituted into the registration result similarity loss function to calculate the registration result similarity loss; and the vessel segmentation results of each vessel sample image and the label information corresponding to multiple vessel sample images can be substituted into the segmentation loss function to calculate the segmentation result loss; then, the registration result similarity loss and the segmentation result loss are weighted and summed to obtain the joint model loss. The similarity loss function and the segmentation loss function can be the cross-entropy loss function, the mean squared error loss function, etc.

[0094] Based on the above embodiments, optionally, the blood vessel sample image includes a target blood vessel sample image and a moving blood vessel sample image; correspondingly, the registration result similarity loss is determined based on the blood vessel segmentation result, registration result, and label information of each blood vessel sample image, including: if the label information includes the label corresponding to the target blood vessel sample image and the label corresponding to the moving blood vessel sample image, then the registration result similarity loss is determined based on the registration result of the label corresponding to the moving blood vessel sample image and the label corresponding to the target blood vessel sample image; or, if the label information includes the label corresponding to the target blood vessel sample image, then the registration result similarity loss is determined based on the registration result of the moving blood vessel sample image and the label corresponding to the target blood vessel sample image; or, if the label information includes the label corresponding to the moving blood vessel sample image, then the registration result similarity loss is determined based on the registration result of the label corresponding to the moving blood vessel sample image and the blood vessel segmentation result corresponding to the target blood vessel sample image.

[0095] For example, the loss function for determining the similarity loss of the registration results is:

[0096]

[0097] Among them, L a I represents the loss function that determines the similarity loss of the registration results. t I represents the target blood vessel sample image. m S represents a sample image of a moving blood vessel. t S represents the label corresponding to the target blood vessel sample image. m The label represents the image of the moving blood vessel sample, T represents the rigid registration transformation matrix, and pred() represents the blood vessel segmentation result predicted by the neural network model.

[0098] Understandably, S m *T represents the registration result of the label corresponding to the motion vessel sample image. pred(I m *T represents the registration result of the motion vessel sample image. pred(I t ) represents the blood vessel segmentation result corresponding to the target blood vessel sample image.

[0099] Based on the above embodiments, optionally, the blood vessel segmentation result of the blood vessel sample image includes the blood vessel segmentation result corresponding to the target blood vessel sample image and the blood vessel segmentation result corresponding to the moving blood vessel sample image; correspondingly, determining the segmentation result loss based on the blood vessel segmentation result and label information of each blood vessel sample image includes: if the label information includes the label corresponding to the target blood vessel sample image and the label corresponding to the moving blood vessel sample image, then determining the segmentation result loss based on the blood vessel segmentation result corresponding to the target blood vessel sample image, the blood vessel segmentation result corresponding to the moving blood vessel sample image, the label corresponding to the target blood vessel sample image, and the label corresponding to the moving blood vessel sample image; or, if the label information includes the label corresponding to the target blood vessel sample image, then determining the segmentation result loss based on the blood vessel segmentation result corresponding to the target blood vessel sample image and the label corresponding to the target blood vessel sample image; or, if the label information includes the label corresponding to the moving blood vessel sample image, then determining the segmentation result loss based on the blood vessel segmentation result corresponding to the moving blood vessel sample image and the label corresponding to the moving blood vessel sample image.

[0100] For example, the loss function used to determine the loss of the segmentation result is:

[0101]

[0102] Among them, L S I represents the loss function that determines the loss of the segmentation result. t I represents the target blood vessel sample image. m S represents a sample image of a moving blood vessel. t S represents the label corresponding to the target blood vessel sample image. m The label represents the image of the moving blood vessel sample, and pred() represents the blood vessel segmentation result predicted by the neural network model.

[0103] It is understandable that pred(I) t ) represents the blood vessel segmentation result corresponding to the target blood vessel sample image, pred(I m ) represents the blood vessel segmentation result corresponding to the motion blood vessel sample image.

[0104] For example, the loss function used to determine the joint model loss can be:

[0105] L=λ s L s +λ a L a

[0106] Where L represents the loss function that determines the joint model loss, L s λ represents the loss function that determines the loss of the segmentation result. sL represents the weights of the loss function that determine the loss of the segmentation result. a λ represents the loss function used to determine the similarity loss of the registration results. a The weights represent the weights of the loss function used to determine the similarity loss of the registration results.

[0107] The technical solution of this invention determines the registration result similarity loss based on the blood vessel segmentation results of each blood vessel sample image, the registration results of each blood vessel sample image, and the label information corresponding to multiple blood vessel sample images; determines the segmentation result loss based on the blood vessel segmentation results of each blood vessel sample image and the label information corresponding to multiple blood vessel sample images; and determines the joint model loss based on the registration result similarity loss and the segmentation result loss. Compared with the prior art, the joint model loss of this embodiment considers both the registration result similarity loss and the segmentation result loss, enabling the model to learn the correlation features between multiple blood vessel sample images, thereby improving the segmentation accuracy of the registration and segmentation joint model for blood vessel images.

[0108] Example 6

[0109] Figure 7 This is a schematic diagram of the structure of an image registration and segmentation joint model training device provided in Embodiment Six of the present invention. Figure 7 As shown, the device includes:

[0110] The sample image acquisition module 610 is used to acquire multiple blood vessel sample images and label information corresponding to each blood vessel sample image;

[0111] The blood vessel image segmentation module 620 is used to input the plurality of blood vessel sample images into a neural network model to obtain the blood vessel segmentation results of each blood vessel sample image;

[0112] The blood vessel image registration module 630 is used to register the blood vessel segmentation results of each of the blood vessel sample images to obtain the registration results of each of the blood vessel sample images.

[0113] The joint loss modeling module 640 is used to determine the joint model loss based on the blood vessel segmentation results, registration results and label information of each blood vessel sample image, and adjust the parameters of the neural network model based on the joint model loss until the iteration stopping condition is met to obtain the trained registration and segmentation joint model.

[0114] The technical solution of this invention acquires multiple vascular sample images and corresponding label information for each vascular sample image, then inputs the multiple vascular sample images into a neural network model to obtain vascular segmentation results for each vascular sample image. The vascular segmentation results for each vascular sample image are then registered to obtain registration results. Based on the vascular segmentation results, registration results, and corresponding label information of the multiple vascular sample images, a joint model loss is determined. The parameters of the neural network model are then adjusted based on the joint model loss until the iteration stopping condition is met, resulting in a trained joint registration and segmentation model. Compared with existing technologies, the joint model loss in this embodiment considers both vascular segmentation and registration results, enabling the model to learn the correlation features between multiple vascular sample images, thereby improving the segmentation accuracy of the joint registration and segmentation model for vascular images.

[0115] In some optional implementations, the sample image acquisition module 610 is further configured to:

[0116] Acquire multiple vascular sample images within a preset time interval during the cardiac cycle.

[0117] In some alternative implementations, the neural network model includes:

[0118] The encoder is used to obtain the vascular coding features of each vascular sample image after inputting each vascular sample image;

[0119] A decoder is used to obtain the blood vessel segmentation result of each blood vessel sample image after inputting the blood vessel coding features of each blood vessel sample image.

[0120] In some alternative implementations, the vascular image registration module 630 is further configured to:

[0121] Based on the rigid registration transformation matrix, the blood vessel segmentation results of each blood vessel sample image are registered to obtain the registration results of each blood vessel sample image.

[0122] In some optional implementations, the joint model loss includes registration result similarity loss and segmentation result loss; the joint loss modeling module 640 includes:

[0123] The registration result similarity loss determination unit is used to determine the registration result similarity loss based on the blood vessel segmentation result, registration result, and label information of each of the blood vessel sample images;

[0124] The segmentation result loss determination unit is used to determine the segmentation result loss based on the blood vessel segmentation result and label information of each of the blood vessel sample images;

[0125] The joint model loss determination unit is used to determine the joint model loss based on the registration result similarity loss and the segmentation result loss.

[0126] In some optional implementations, the vascular sample image includes a target vascular sample image and a moving vascular sample image;

[0127] Correspondingly, the registration result similarity loss determination unit is specifically used for:

[0128] If the label information includes the label corresponding to the target blood vessel sample image and the label corresponding to the motion blood vessel sample image, then the similarity loss of the registration result is determined based on the registration result of the label corresponding to the motion blood vessel sample image and the label corresponding to the target blood vessel sample image.

[0129] Alternatively, if the label information includes a label corresponding to the target blood vessel sample image, then the similarity loss of the registration result is determined based on the registration result of the motion blood vessel sample image and the label corresponding to the target blood vessel sample image.

[0130] Alternatively, if the label information includes labels corresponding to motion vessel sample images, then the registration result similarity loss is determined based on the registration result corresponding to the labels of motion vessel sample images and the vessel segmentation result corresponding to the target vessel sample image.

[0131] In some optional implementations, the blood vessel segmentation result of the blood vessel sample image includes the blood vessel segmentation result corresponding to the target blood vessel sample image and the blood vessel segmentation result corresponding to the moving blood vessel sample image. The segmentation result loss determination unit is specifically used for:

[0132] If the label information includes the label corresponding to the target blood vessel sample image and the label corresponding to the moving blood vessel sample image, then the segmentation result loss is determined based on the blood vessel segmentation result corresponding to the target blood vessel sample image, the blood vessel segmentation result corresponding to the moving blood vessel sample image, the label corresponding to the target blood vessel sample image, and the label corresponding to the moving blood vessel sample image.

[0133] Alternatively, if the label information includes a label corresponding to the target blood vessel sample image, then the segmentation result loss is determined based on the blood vessel segmentation result corresponding to the target blood vessel sample image and the label corresponding to the target blood vessel sample image.

[0134] Alternatively, if the label information includes labels corresponding to motion vessel sample images, then the segmentation result loss is determined based on the vessel segmentation result corresponding to the motion vessel sample image and the labels corresponding to the motion vessel sample image.

[0135] The image registration and segmentation joint model training device provided in this embodiment of the invention can execute the image registration and segmentation joint model training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0136] Example 7

[0137] Figure 8 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0138] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An I / O interface 15 is also connected to the bus 14.

[0139] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0140] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as an image registration and segmentation joint model training method, which includes:

[0141] Acquire multiple blood vessel sample images and the corresponding label information for each blood vessel sample image;

[0142] The multiple blood vessel sample images are input into a neural network model to obtain the blood vessel segmentation results for each blood vessel sample image;

[0143] The blood vessel segmentation results of each of the blood vessel sample images are registered to obtain the registration results of each of the blood vessel sample images;

[0144] Based on the blood vessel segmentation results, registration results, and label information of each blood vessel sample image, the joint model loss is determined. The parameters of the neural network model are adjusted based on the joint model loss until the iteration stopping condition is met to obtain the trained joint registration and segmentation model.

[0145] In some embodiments, the image registration and segmentation joint model training method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the image registration and segmentation joint model training method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the image registration and segmentation joint model training method by any other suitable means (e.g., by means of firmware).

[0146] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0148] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0149] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0150] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0151] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0152] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0153] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for training an image registration segmentation joint model, characterized in that, The method comprises: obtaining a plurality of blood vessel sample images collected based on a preset time interval within a cardiac cycle, and label information corresponding to each of the blood vessel sample images; inputting the plurality of blood vessel sample images into a neural network model to obtain a blood vessel segmentation result of each of the blood vessel sample images; registering the blood vessel segmentation result of each of the blood vessel sample images to obtain a registration result of each of the blood vessel sample images; determining a registration result similarity loss based on the blood vessel segmentation result, the registration result and the label information of each of the blood vessel sample images, and determining a segmentation result loss based on the blood vessel segmentation result and the label information of each of the blood vessel sample images; determining a joint model loss based on the registration result similarity loss and the segmentation result loss, adjusting parameters of the neural network model based on the joint model loss until an iteration stop condition is met to obtain a trained registration and segmentation joint model.

2. The method of claim 1, wherein, The neural network model comprises: an encoder configured to obtain blood vessel encoding features of each blood vessel sample image after inputting each blood vessel sample image; a decoder configured to obtain a blood vessel segmentation result of each blood vessel sample image after inputting the blood vessel encoding features of each of the blood vessel sample images.

3. The method of claim 1, wherein, The registration of the blood vessel segmentation result of each of the blood vessel sample images to obtain the registration result of each of the blood vessel sample images comprises: registering the blood vessel segmentation result of each of the blood vessel sample images based on a rigid registration transformation matrix to obtain the registration result of each of the blood vessel sample images.

4. The method of claim 1, wherein, The blood vessel sample images comprise a target blood vessel sample image and a motion blood vessel sample image. Correspondingly, the determination of the registration result similarity loss based on the blood vessel segmentation result, the registration result and the label information of each of the blood vessel sample images comprises: if the label information comprises a label corresponding to the target blood vessel sample image and a label corresponding to the motion blood vessel sample image, determining the registration result similarity loss based on the registration result of the motion blood vessel sample image corresponding to the label and the label corresponding to the target blood vessel sample image; or, if the label information comprises a label corresponding to the target blood vessel sample image, determining the registration result similarity loss based on the registration result of the motion blood vessel sample image and the label corresponding to the target blood vessel sample image; or, if the label information comprises a label corresponding to the motion blood vessel sample image, determining the registration result similarity loss based on the registration result of the motion blood vessel sample image corresponding to the label and the blood vessel segmentation result corresponding to the target blood vessel sample image.

5. The method of claim 4, wherein, The blood vessel segmentation result of the blood vessel sample image comprises a blood vessel segmentation result corresponding to the target blood vessel sample image and a blood vessel segmentation result corresponding to the motion blood vessel sample image. Correspondingly, the determination of the segmentation result loss based on the blood vessel segmentation result and the label information of each of the blood vessel sample images comprises: if the label information comprises a label corresponding to the target blood vessel sample image and a label corresponding to the motion blood vessel sample image, determining the segmentation result loss based on the blood vessel segmentation result corresponding to the target blood vessel sample image, the blood vessel segmentation result corresponding to the motion blood vessel sample image, the label corresponding to the target blood vessel sample image and the label corresponding to the motion blood vessel sample image; Or, if the label information includes a label corresponding to the target blood vessel sample image, determining the segmentation result loss based on the blood vessel segmentation result corresponding to the target blood vessel sample image and the label corresponding to the target blood vessel sample image. Or, if the label information includes a label corresponding to the moving blood vessel sample image, determining the segmentation result loss based on the blood vessel segmentation result corresponding to the moving blood vessel sample image and the label corresponding to the moving blood vessel sample image.

6. An image registration segmentation joint model training apparatus, characterized in that, Comprising: a sample image acquisition module configured to acquire a plurality of blood vessel sample images collected based on a preset time interval within a cardiac cycle, and label information corresponding to each of the blood vessel sample images; a blood vessel image segmentation module configured to input the plurality of blood vessel sample images into a neural network model to obtain blood vessel segmentation results of the blood vessel sample images; a blood vessel image registration module configured to register the blood vessel segmentation results of the blood vessel sample images to obtain registration results of the blood vessel sample images; a joint loss modeling module configured to determine a registration result similarity loss based on the blood vessel segmentation results, the registration results, and the label information of the blood vessel sample images, and determine a segmentation result loss based on the blood vessel segmentation results and the label information of the blood vessel sample images; determine a joint model loss based on the registration result similarity loss and the segmentation result loss, adjust parameters of the neural network model based on the joint model loss until an iteration stop condition is met to obtain a trained registration and segmentation joint model.

7. An electronic device, comprising: The electronic device comprises: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the image registration and segmentation joint model training method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the processor to execute when executed to implement the image registration and segmentation joint model training method of any one of claims 1-5. The computer readable storage medium stores computer instructions for causing the processor to execute when executed to implement the image registration and segmentation joint model training method of any one of claims 1-5.

Images