Registration method and device of dynamic X-ray image, program product and medical equipment

Abstract

The invention relates to a dynamic X-ray image registration method and device, a program product and medical equipment, and relates to the technical field of X-ray image registration. The method comprises the following steps: acquiring a dynamic X-ray image sequence to be registered, a plurality of dynamic / static X-ray images for training and a first preset convolutional neural network corresponding to the dynamic / static X-ray images and used for limiting a region of interest; performing initial training on a second preset convolutional neural network used for registration by using the plurality of dynamic / static X-ray images and the first preset convolutional neural network; carrying out enhancement training on a second preset convolutional neural network after initial training by utilizing a plurality of dynamic / static X-ray image pairs corresponding to the plurality of dynamic / static X-ray images and the region-of-interest images and the first preset convolutional neural network, wherein the plurality of dynamic / static X-ray image pairs correspond to the plurality of dynamic / static X-ray images and the region-of-interest images of the plurality of dynamic / static X-ray images; and on the basis of the second preset convolutional neural network after enhancement training, registering any two region-of-interest images in the dynamic X-ray image sequence to be registered. And the registration of the dynamic X-ray image can be realized.

Description

Technical Field

[0001] The present disclosure relates to the technical field of dynamic X-ray image registration, and in particular to a dynamic X-ray image registration method and device, a program product and a medical device. Background Art

[0002] Since the AC RegNet (Mansilla L, Milone DH, Ferrante E. Learning deformable registration of medical images with anatomical constraints [J]. Neural Networks, 2020, 124: 269-279.) architecture was proposed, this technology has been considered the most advanced registration method. However, in clinical practice, accurate region of interest image registration results may be more favored and exciting than registration results for the entire X-ray image.

[0003] Based on the above, a technical solution for registering any two region-of-interest images in a dynamic X-ray image sequence needs to be provided. Summary of the invention

[0004] The present invention provides a technical solution for a dynamic X-ray image registration method and device, a program product and medical equipment.

[0005] According to one aspect of the present disclosure, a dynamic X-ray image registration method is provided, comprising:

[0006] A dynamic X-ray image sequence to be registered, a plurality of dynamic / static X-ray images for training and a corresponding first preset convolutional neural network for limiting the region of interest are obtained; using the plurality of dynamic / static X-ray images and the first preset convolutional neural network, an initial training is performed on a second preset convolutional neural network for registration; using the plurality of dynamic / static X-ray images and a plurality of dynamic / static X-ray image pairs corresponding to the plurality of dynamic / static X-ray images and the region of interest images and the first preset convolutional neural network, an enhanced training is performed on the second preset convolutional neural network after the initial training; based on the enhanced trained second preset convolutional neural network, any two region of interest images in the dynamic X-ray image sequence to be registered are registered.

[0007] Preferably, the method of performing enhanced training on the second preset convolutional neural network after initial training using the multiple dynamic / static X-ray images and the multiple dynamic / static X-ray image pairs corresponding to the multiple dynamic / static X-ray images and the images of the region of interest and the first preset convolutional neural network comprises: transforming the multiple dynamic / static X-ray images respectively to obtain a first transformed X-ray image corresponding to each dynamic / static X-ray image; constructing multiple first dynamic / static X-ray image pairs based on each dynamic / static X-ray image and its corresponding first transformed X-ray image respectively; performing a first enhanced training on the second preset convolutional neural network after initial training based on the multiple first dynamic / static X-ray image pairs and the first preset convolutional neural network. ; According to the method of transforming the multiple dynamic / static X-ray images respectively, the region of interest images corresponding to the multiple dynamic / static X-ray images are transformed respectively to obtain a second transformed X-ray image corresponding to each region of interest image; based on each region of interest image and its corresponding second transformed X-ray image, a plurality of second dynamic / static X-ray image pairs are constructed respectively; based on the plurality of second dynamic / static X-ray image pairs and the first preset convolutional neural network, a second enhancement training is performed on the second preset convolutional neural network after the first enhancement training; and further, based on the second preset convolutional neural network after the second enhancement training, any two region of interest images in the dynamic X-ray image sequence to be registered are registered.

[0008] Preferably, the method for performing a first enhanced training on the second preset convolutional neural network after the initial training based on the multiple first dynamic / static X-ray image pairs and the first preset convolutional neural network comprises: acquiring the multiple first dynamic / static X-ray image pairs and their corresponding multiple region of interest mask image pairs; determining the corresponding first training deformation field based on the fixed image and the floating image of each pair of images in the multiple first dynamic / static X-ray image pairs and the second preset convolutional neural network after the initial training; determining the corresponding first deformed image based on the floating image in each pair of images and the corresponding first training deformation field; calculating the first loss based on the first deformed image corresponding to the floating image of each pair of images and the fixed image; and calculating the first loss based on the first deformed image corresponding to the floating image of each pair of images and the fixed image. The invention discloses a method for enhancing the image quality of a mobile phone by using a mask image of a region of interest corresponding to a floating image in an image and a corresponding first training deformation field to determine a corresponding second deformed image; calculating a second loss based on the second deformed image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; encoding the second deformed image and the mask image of the region of interest corresponding to the fixed image by using the first preset convolutional neural network to obtain a first encoding feature map and a second encoding feature map; and calculating a third loss based on the first encoding feature map and the second encoding feature map; adjusting the parameters of the second preset convolutional neural network after the initial training based on the first loss, the second loss and the third loss to complete the first enhancement training of the second preset convolutional neural network after the initial training.

[0009] Preferably, before acquiring the multiple region of interest mask image pairs corresponding to the multiple first dynamic / static X-ray image pairs, the region of interest segmentation is performed on the multiple first dynamic / static X-ray image pairs respectively using a third preset convolutional neural network or a preset segmentation model for the region of interest segmentation to obtain the multiple region of interest mask image pairs corresponding to the multiple dynamic / static X-ray images.

[0010] Preferably, before transforming the region of interest images corresponding to the multiple dynamic / static X-ray images respectively to obtain the second transformed X-ray image corresponding to each region of interest image, the method of respectively determining the region of interest images corresponding to the multiple dynamic / static X-ray images includes: respectively acquiring the region of interest mask images corresponding to the multiple dynamic / static X-ray images, and configuring the mask value corresponding to the region of interest mask image as a set value; based on the configured region of interest mask image and the corresponding multiple dynamic / static X-ray images, determining the region of interest images corresponding to the multiple dynamic / static X-ray images.

[0011] Preferably, the method for performing a second enhanced training on the second preset convolutional neural network after the first enhanced training based on the multiple second dynamic / static X-ray image pairs and the first preset convolutional neural network comprises: obtaining the multiple second dynamic / static X-ray image pairs and their corresponding multiple region of interest mask image pairs; determining the corresponding second training deformation field based on the fixed image and the floating image of each pair of images in the multiple second dynamic / static X-ray image pairs and the second preset convolutional neural network of the first enhanced training; determining the corresponding third deformation image based on the floating image in each pair of images and the corresponding second training deformation field; calculating the fourth loss based on the third deformation image corresponding to the floating image of each pair of images and the fixed image; and calculating the fourth loss based on the fixed image of each pair of images. The mask image of the region of interest corresponding to the floating image in the image and the corresponding second training deformation field determine the corresponding fourth deformed image; the fifth loss is calculated based on the fourth deformed image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; the fourth deformed image and the mask image of the region of interest corresponding to the fixed image are encoded using the first preset convolutional neural network to obtain a third encoding feature map and a fourth encoding feature map; and a sixth loss is calculated based on the third encoding feature map and the fourth encoding feature map; based on the fourth loss, the fifth loss and the sixth loss, the parameters of the second preset convolutional neural network after the first enhanced training are adjusted to complete the second enhanced training of the second preset convolutional neural network after the first training.

[0012] Preferably, before obtaining the first preset convolutional neural network for limiting the region of interest corresponding to the multiple dynamic / static X-ray images for training, the region of interest is segmented on the multiple dynamic / static X-ray images respectively using a third preset convolutional neural network or a preset segmentation model for segmenting the region of interest to obtain a plurality of region of interest mask images corresponding to the multiple dynamic / static X-ray images; and the first preset convolutional neural network for limiting the region of interest is trained using the plurality of region of interest mask images.

[0013] Preferably, the method of using the multiple dynamic / static X-ray images and the first preset convolutional neural network to initially train the second preset convolutional neural network used for registration includes: using the multiple dynamic / static X-ray images to construct a dynamic / static X-ray image pair set; using the dynamic / static X-ray image pair set and the first preset convolutional neural network to initially train the second preset convolutional neural network used for registration.

[0014] Preferably, the method of constructing a dynamic / static X-ray image pair set using the multiple dynamic / static X-ray images includes: respectively extracting multiple name / ID information corresponding to the multiple dynamic / static X-ray images; respectively pairing the multiple dynamic / static X-ray images with different name / ID information to construct a dynamic / static X-ray image pair set.

[0015] Preferably, the method for initially training the second preset convolutional neural network for registration using the dynamic / static X-ray image pair set and the first preset convolutional neural network comprises: obtaining a plurality of mask images of the region of interest corresponding to the dynamic / static X-ray image pair set; determining a corresponding third training deformation field based on the fixed image and the floating image of each pair of images in the dynamic / static X-ray image pair set and the second preset convolutional neural network to be initially trained; determining a corresponding fifth deformation image based on the floating image in each pair of images in the dynamic / static X-ray image pair set and the corresponding third training deformation field; calculating a seventh loss based on the fifth deformation image corresponding to the floating image of each pair of images and the fixed image; Determine the corresponding sixth deformed image based on the mask image of the region of interest corresponding to the floating image in each pair of images and the corresponding third training deformation field; calculate the eighth loss based on the sixth deformed image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; encode the sixth deformed image and the mask image of the region of interest corresponding to the fixed image using the first preset convolutional neural network to obtain a fifth encoded feature map and a sixth encoded feature map; and calculate the eighth loss based on the fifth encoded feature map and the sixth encoded feature map; adjust the parameters of the second preset convolutional neural network for the initial training based on the sixth loss, the seventh loss and the eighth loss to complete the initial training of the second preset convolutional neural network.

[0016] Preferably, the method for registering any two region of interest images in the dynamic X-ray image sequence to be registered based on the second preset convolutional neural network after the enhanced training includes: using the third preset convolutional neural network or the preset segmentation model for the region of interest segmentation to perform the region of interest segmentation on the dynamic X-ray image sequence to be registered, respectively, to obtain a plurality of region of interest mask image sequences corresponding to the dynamic X-ray image sequence to be registered; and configuring the mask value corresponding to the region of interest mask image sequence as a set value; based on the configured region of interest mask image sequence and its corresponding dynamic X-ray image sequence to be registered, determining the region of interest image sequence corresponding to the dynamic X-ray image sequence to be registered; based on the second preset convolutional neural network after the enhanced training, registering any two region of interest images in the region of interest image sequence.

[0017] Preferably, the region of interest is configured as one or more of the left lung field, the right lung field, the heart, the diaphragm, the gastrointestinal tract, and the breast.

[0018] Preferably, the second preset convolutional neural network based on the enhanced training is used to align any two region of interest images in the dynamic X-ray image sequence to be aligned, and the corresponding two region of interest images are respectively configured as image pairs corresponding to the maximum region of interest area and the minimum region of interest area in the region of interest images.

[0019] According to one aspect of the present disclosure, a dynamic X-ray image registration device is provided, comprising:

[0020] An acquisition unit, used for acquiring a dynamic X-ray image sequence to be registered, a plurality of dynamic / static X-ray images for training, and a first preset convolutional neural network corresponding thereto for limiting a region of interest;

[0021] An initial training unit, configured to perform initial training on a second preset convolutional neural network for registration using the plurality of dynamic / static X-ray images and the first preset convolutional neural network;

[0022] An enhanced training unit, configured to perform enhanced training on the second preset convolutional neural network after initial training by using the plurality of dynamic / static X-ray image pairs corresponding to the plurality of dynamic / static X-ray images and the images of the region of interest thereof and the first preset convolutional neural network;

[0023] A registration unit, configured to register any two region of interest images in the dynamic X-ray image sequence to be registered based on the second preset convolutional neural network after the enhanced training; or,

[0024] The method comprises: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above-mentioned measurement method; or,

[0025] The invention comprises: a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above-mentioned registration method when executed by a processor.

[0026] Preferably, the registration device further comprises: a segmentation unit; wherein the segmentation unit is used to segment the region of interest in the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images to obtain a corresponding region of interest mask image.

[0027] Preferably, the registration device further comprises: a region of interest extraction unit; wherein the region of interest extraction unit is used to extract the region of interest image sequence and / or region of interest image in the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images based on the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images and their corresponding region of interest mask image sequence and / or region of interest mask image; wherein, based on the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images and their corresponding region of interest A region mask image sequence and / or a region mask image of interest is used to extract the region image sequence and / or the region image of interest in the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images, including: pixel-level multiplication of the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images with the corresponding region mask image sequence and / or the region mask image of interest, and extracting the region image sequence and / or the region image of interest in the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images.

[0028] Preferably, the registration device further includes: an area determination unit; wherein the area determination unit is used to determine a region of interest area sequence corresponding to all region of interest images in the dynamic X-ray image sequence to be registered, determine an image pair corresponding to the maximum region of interest area and the minimum region of interest area based on the region of interest area sequence, and use the registration unit to register the image pair.

[0029] According to one aspect of the present disclosure, a computer program product is provided, including a computer program / instruction, characterized in that the computer program / instruction implements the above-mentioned registration method when executed by a processor.

[0030] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to: execute the above-mentioned registration method.

[0031] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-mentioned registration method is implemented.

[0032] According to one aspect of the present disclosure, a medical device is provided, which applies the above-mentioned registration method and / or includes the above-mentioned registration device and / or includes the above-mentioned computer program product.

[0033] In the embodiments of the present disclosure, a dynamic X-ray image registration method and apparatus, a program product and a technical solution for medical equipment are proposed to solve the problem that the existing registration technology cannot accurately register the image of the region of interest in the dynamic X-ray image.

[0034] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0035] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0037] Figure 1 A flow chart showing a registration method for dynamic X-ray images according to an embodiment of the present disclosure;

[0038] Figure 2 is a block diagram of an electronic device 800 according to an exemplary embodiment;

[0039] Figure 3 is a block diagram of an electronic device 1900 according to an exemplary embodiment. DETAILED DESCRIPTION

[0040] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0041] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0042] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0043] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0044] It can be understood that the above-mentioned embodiments of the dynamic X-ray image registration method mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not go into details.

[0045] In addition, the present disclosure also provides a dynamic X-ray image registration device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any dynamic X-ray image registration method provided by the present disclosure. The corresponding technical solutions and descriptions can be found in the corresponding records in the dynamic X-ray image registration method section and will not be repeated here.

[0046] Figure 1 A flow chart of a dynamic X-ray image registration method according to an embodiment of the present disclosure is shown. Figure 1As shown, the registration method of dynamic X-ray images includes: step S101: obtaining a dynamic X-ray image sequence to be registered, multiple dynamic / static X-ray images for training and their corresponding first preset convolutional neural network for limiting the region of interest; step S102: using the multiple dynamic / static X-ray images and the first preset convolutional neural network, initially training the second preset convolutional neural network for registration; step S103: using the multiple dynamic / static X-ray images and the multiple dynamic / static X-ray image pairs corresponding to the region of interest images and the first preset convolutional neural network, enhancing the training of the second preset convolutional neural network after initial training; step S104: based on the second preset convolutional neural network after enhanced training, registering any two region of interest images in the dynamic X-ray image sequence to be registered. This solves the problem that the existing registration technology cannot accurately register the region of interest images in dynamic X-ray images.

[0047] In the embodiments of the present disclosure and other possible embodiments, the dynamic X-ray image sequence to be registered and the multiple dynamic / static X-ray images used for training can be configured as two-dimensional X-ray images taken by a DR device; wherein the dynamic X-ray image sequence comes from the same human body; similarly, the multiple dynamic X-ray images used for training come from the same human body; and the multiple dynamic X-ray images used for training come from different human bodies.

[0048] In the embodiments of the present disclosure and other possible embodiments, the dynamic X-ray image sequence to be registered can be configured as dynamic X-ray images corresponding to the same human body at multiple different moments; the multiple dynamic / static X-ray images used for training can be configured as static X-ray images corresponding to different human bodies at a certain moment. For example, the dynamic X-ray image sequence to be registered can be configured as dynamic X-ray images corresponding to multiple different moments during breathing / breath-holding state; the multiple dynamic / static X-ray images used for training can be configured as static X-ray images corresponding to a certain moment during breathing / breath-holding state.

[0049] In the embodiment of the present disclosure, the region of interest may be configured as one or more of the left lung field, the right lung field, the heart, the diaphragm, the gastrointestinal tract, the breast, etc. For example, in the embodiment of the present disclosure and other possible embodiments, the region of interest may be configured as the left lung field, the right lung field, or the entire lung field including the left lung field and the right lung field.

[0050] In an embodiment of the present disclosure, the second preset convolutional neural network after the enhanced training is used to align any two region of interest images in the dynamic X-ray image sequence to be aligned, and the corresponding two region of interest images are respectively configured as an image pair (a pair of images) corresponding to the maximum region of interest area and the minimum region of interest area in the region of interest images. For example, in the embodiment of the present disclosure and other possible embodiments, the region of interest can be configured as the left lung field, the right lung field, or the entire lung field including the lung field and the right lung field; therefore, the second preset convolutional neural network after the enhanced training is used to align any two region of interest images in the dynamic X-ray image sequence to be aligned, and the corresponding two region of interest images are respectively configured as an image pair corresponding to the maximum left lung field area / maximum right lung field area / maximum area of ​​the entire lung field including the lung field and the right lung field, and the minimum left lung field area, the minimum area of ​​the right lung field, or the minimum area of ​​the entire lung field including the lung field and the right lung field in the region of interest images.

[0051] Step S101: Acquire a dynamic X-ray image sequence to be registered, a plurality of dynamic / static X-ray images for training, and their corresponding first preset convolutional neural network for limiting the region of interest.

[0052] In an embodiment of the present disclosure, before obtaining a first preset convolutional neural network for limiting the region of interest corresponding to a plurality of dynamic / static X-ray images for training, a third preset convolutional neural network or a preset segmentation model for segmenting the region of interest is used to perform the region of interest segmentation on the plurality of dynamic / static X-ray images, respectively, to obtain a plurality of region of interest mask images corresponding to the plurality of dynamic / static X-ray images; and the first preset convolutional neural network for limiting the region of interest is trained using the plurality of region of interest mask images.

[0053] In the embodiments of the present disclosure and other possible embodiments, the third preset convolutional neural network used for segmenting the region of interest can be configured as one or more of FCN, Unet, Unet++, SegNet, RefineNet, nn Unet or their improved convolutional neural networks.

[0054] For example, in the embodiments of the present disclosure and other possible embodiments, the region of interest may be configured as the left lung field, the right lung field, or the entire lung field including the left lung field and the right lung field. Furthermore, the third preset convolutional neural network or preset segmentation model used for segmenting the region of interest may adopt the technical solutions corresponding to the lung area segmentation, lung disease assessment method and device, electronic device and storage medium (2023108100929), lung field image segmentation method, device and storage medium (2023106887446) previously disclosed by our company, or other existing lung field / lung area segmentation models.

[0055] In the embodiments of the present disclosure and other possible embodiments, before obtaining the first preset convolutional neural network for limiting the region of interest corresponding to the multiple dynamic / static X-ray images for training, the region of interest is segmented on the multiple dynamic / static X-ray images using a third preset convolutional neural network or a preset segmentation model for segmenting the region of interest, so as to obtain multiple region of interest mask images corresponding to the multiple dynamic / static X-ray images; and the first preset convolutional neural network is trained using the multiple region of interest mask images corresponding to the dynamic / static X-ray images.

[0056] Among them, in the embodiments of the present disclosure and other possible embodiments, the first preset convolutional neural network for limiting the region of interest can be configured as an Autoencoder corresponding to AC RegNet or its improvement, or a network structure with Unet or improved Unet as the backbone, or a nn Unet network architecture.

[0057] At the same time, in the embodiments of the present disclosure and other possible embodiments, the present disclosure proposes a deep multi-scale attention encoder (first preset convolutional neural network) for limiting the region of interest, in which a deep contextual attention block (DCA Block) is configured to better encode and represent the multiple region of interest mask images. Specifically, the deep multi-scale attention encoder (first preset convolutional neural network) includes: multiple parallel first convolution modules corresponding to convolution kernels of different sizes, a second convolution module connected to the output end of one of the multiple parallel first convolution modules, a splicing module connected to the output ends of other convolution modules in the multiple parallel first convolution modules, a deep contextual attention module connected to the output end of the splicing module, and a pixel-level addition module connected to the output end of the deep contextual attention module and the output end of the second convolution module. Among them, the corresponding convolution kernels of different sizes are cardinality multiples of 1×1.

[0058] Among them, in the embodiments of the present disclosure and other possible embodiments, further, the output end of the pixel-level addition model is also provided with a first normalization module (BN) and a first nonlinear module (ReLu) connected to its output end. At the same time, the output ends of the multiple parallel first convolution modules are respectively configured with multiple second normalization modules (BN) and second nonlinear modules (ReLu) connected to their output ends. Specifically, a second normalization module (BN) and a second nonlinear module (ReLu) connected to its output end are provided between the output ends of the multiple parallel first convolution modules and the second convolution modules; a second normalization module (BN) and a second nonlinear module (ReLu) connected to its output end are respectively provided between the output ends of other convolution modules in the multiple parallel first convolution modules and the splicing module.

[0059] In the embodiments of the present disclosure and other possible embodiments, multiple parallel first convolution modules corresponding to convolution kernels of different sizes are used to perform multi-scale changes on the multiple region of interest mask images respectively to obtain corresponding multiple multi-scale mask images; a second convolution module is used to expand the number of channels of a multi-scale mask image among the multiple multi-scale mask images according to a set number of channels to obtain a corresponding set channel multi-scale mask image; the deep context attention module is used to perform deep text attention processing on the other multiple multi-scale mask images (except the multi-scale mask image corresponding to the channel expansion) to obtain a corresponding deep text attention mask image; the pixel-level addition module is used to superimpose the set channel multi-scale mask image and the deep text attention mask image to obtain a final encoding feature map.

[0060] Among them, in the embodiment of the present disclosure and other possible embodiments, the method for determining the set number of channels includes: obtaining the number corresponding to the multiple parallel first convolution modules; subtracting 1 from the number to obtain the set number of channels.

[0061] In the embodiments of the present disclosure and other possible embodiments, those skilled in the art may configure the number of the plurality of parallel first convolution modules and the corresponding convolution kernels of different sizes, and the convolution kernels corresponding to the second convolution module as needed. For example, the number of the plurality of parallel first convolution modules may be configured as 4, and the corresponding convolution kernels of different sizes are 1×1, 3×3, 5×5, and 7×7 respectively; the convolution kernel corresponding to the second convolution module is configured as 1×1.

[0062] Step S102: using the plurality of dynamic / static X-ray images and the first preset convolutional neural network, initially training a second preset convolutional neural network for registration.

[0063] In the embodiments of the present disclosure and other possible embodiments, the second preset convolutional neural network used for registration can be configured as one or more of VectorCNN or its improvement corresponding to AC RegNet, or a convolutional neural network framework or its improvement corresponding to the VoxelMorph model, or a deep alignment network (DeepAlignmentNetwork) or its improvement, etc.

[0064] In an embodiment of the present disclosure, the method of initially training the second preset convolutional neural network for registration using the multiple dynamic / static X-ray images and the first preset convolutional neural network includes: constructing a dynamic / static X-ray image pair set using the multiple dynamic / static X-ray images; initially training the second preset convolutional neural network for registration using the dynamic / static X-ray image pair set and the first preset convolutional neural network; wherein, the method of constructing a dynamic / static X-ray image pair set using the multiple dynamic / static X-ray images includes: respectively extracting multiple name / ID information corresponding to the multiple dynamic / static X-ray images; respectively pairing the multiple dynamic / static X-ray images with different name / ID information to construct a dynamic / static X-ray image pair set.

[0065] In an embodiment of the present disclosure, the method for initially training the second preset convolutional neural network for registration using the dynamic / static X-ray image pair set and the first preset convolutional neural network includes: obtaining a plurality of region of interest mask images corresponding to the dynamic / static X-ray image pair set; determining a corresponding third training deformation field based on the fixed image and the floating image of each pair of images in the dynamic / static X-ray image pair set and the second preset convolutional neural network to be initially trained; determining a corresponding fifth deformation image based on the floating image in each pair of images in the dynamic / static X-ray image pair set and the corresponding third training deformation field; calculating a seventh deformation image based on the fifth deformation image corresponding to the floating image of each pair of images and the fixed image; loss; determining the corresponding sixth deformed image based on the mask image of the region of interest corresponding to the floating image in each pair of images and the corresponding third training deformation field; calculating the eighth loss based on the sixth deformed image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; encoding the sixth deformed image and the mask image of the region of interest corresponding to the fixed image using the first preset convolutional neural network respectively to obtain a fifth encoded feature map and a sixth encoded feature map; and calculating the eighth loss based on the fifth encoded feature map and the sixth encoded feature map; adjusting the parameters of the second preset convolutional neural network for the initial training based on the sixth loss, the seventh loss and the eighth loss to complete the initial training of the second preset convolutional neural network.

[0066] In the embodiments of the present disclosure and other possible embodiments, the loss function used to calculate the loss can be configured as one or more of the existing loss functions such as the cross entropy loss function, the mean squared voxel differece loss function of the image grayscale, the cross-correlation loss function, the mutual information loss function, and the normalized cross correlation loss function.

[0067] Step S103: using the multiple dynamic / static X-ray image pairs corresponding to the multiple dynamic / static X-ray images and the images of the region of interest and the first preset convolutional neural network, perform enhanced training on the second preset convolutional neural network after the initial training.

[0068] In an embodiment of the present disclosure, the method of using the multiple dynamic / static X-ray images and the multiple dynamic / static X-ray image pairs corresponding to the multiple dynamic / static X-ray images and the region of interest images thereof and the first preset convolutional neural network to perform enhanced training on the second preset convolutional neural network after initial training includes: transforming the multiple dynamic / static X-ray images respectively to obtain a first transformed X-ray image corresponding to each dynamic / static X-ray image; constructing multiple first dynamic / static X-ray image pairs based on each dynamic / static X-ray image and its corresponding first transformed X-ray image respectively; and performing a first enhanced training on the second preset convolutional neural network after initial training based on the multiple first dynamic / static X-ray image pairs and the first preset convolutional neural network. Strengthen training; transform the region of interest images corresponding to the multiple dynamic / static X-ray images respectively according to the method of transforming the multiple dynamic / static X-ray images respectively, and obtain a second transformed X-ray image corresponding to each region of interest image; construct a plurality of second dynamic / static X-ray image pairs based on each region of interest image and its corresponding second transformed X-ray image respectively; based on the plurality of second dynamic / static X-ray image pairs and the first preset convolutional neural network, perform a second enhanced training on the second preset convolutional neural network after the first enhanced training; and further, based on the second preset convolutional neural network after the second enhanced training, align any two region of interest images in the dynamic X-ray image sequence to be aligned.

[0069] In the embodiments of the present disclosure and other possible embodiments, the transformation of the multiple dynamic / static X-ray images respectively or the transformation of the region of interest images corresponding to the multiple dynamic / static X-ray images respectively according to the method of transforming the multiple dynamic / static X-ray images respectively or the transformation used by the following registration method or registration device at least includes: affine transformation. The affine transformation may include: one or more of scaling, reflection, shear mapping, etc.; the scaling may include one or more of scale, translation, rotation, etc. For example, the getRotationMatrix2D function in opencv can be used to implement the above affine transformation.

[0070] In an embodiment of the present disclosure, the plurality of dynamic / static X-ray images are transformed respectively to obtain a first transformed X-ray image corresponding to each dynamic / static X-ray image; a plurality of first dynamic / static X-ray image pairs are constructed based on each dynamic / static X-ray image and its corresponding first transformed X-ray image; based on the plurality of first dynamic / static X-ray image pairs and the first preset convolutional neural network, the second preset convolutional neural network after the initial training is subjected to a first enhancement training; according to the method of transforming the plurality of dynamic / static X-ray images respectively, the plurality of dynamic / static X-ray images are respectively subjected to a first enhancement training. The region of interest image corresponding to the X-ray image is transformed to obtain a second transformed X-ray image corresponding to each region of interest image; based on each region of interest image and its corresponding second transformed X-ray image, a plurality of second dynamic / static X-ray image pairs are constructed; based on the plurality of second dynamic / static X-ray image pairs and the first preset convolutional neural network, a second enhancement training is performed on the second preset convolutional neural network after the first enhancement training; and further, based on the second preset convolutional neural network after the second enhancement training, any two region of interest images in the dynamic X-ray image sequence to be registered are registered.

[0071] In an embodiment of the present disclosure, the method for performing a first enhanced training on the second preset convolutional neural network after the initial training based on the multiple first dynamic / static X-ray image pairs and the first preset convolutional neural network includes: acquiring the multiple first dynamic / static X-ray image pairs and their corresponding multiple region of interest mask image pairs; determining the corresponding first training deformation field based on the fixed image and the floating image of each pair of images in the multiple first dynamic / static X-ray image pairs and the second preset convolutional neural network after the initial training; determining the corresponding first deformed image based on the floating image in each pair of images and the corresponding first training deformation field; calculating the first loss based on the first deformed image corresponding to the floating image of each pair of images and the fixed image; and calculating the first loss based on the first deformed image corresponding to the floating image of each pair of images and the fixed image. The corresponding second deformed image is determined based on the mask image of the region of interest corresponding to the floating image in each pair of images and the corresponding first training deformation field; the second loss is calculated based on the second deformed image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; the second deformed image and the mask image of the region of interest corresponding to the fixed image are encoded using the first preset convolutional neural network to obtain a first encoding feature map and a second encoding feature map; and a third loss is calculated based on the first encoding feature map and the second encoding feature map; based on the first loss, the second loss and the third loss, the parameters of the second preset convolutional neural network after the initial training are adjusted to complete the first enhanced training of the second preset convolutional neural network after the initial training.

[0072] In an embodiment of the present disclosure, before transforming the region of interest images corresponding to the multiple dynamic / static X-ray images respectively to obtain a second transformed X-ray image corresponding to each region of interest image, a method for respectively determining the region of interest images corresponding to the multiple dynamic / static X-ray images includes: respectively acquiring region of interest mask images corresponding to the multiple dynamic / static X-ray images, and configuring the mask value corresponding to the region of interest mask image as a set value; determining the region of interest images corresponding to the multiple dynamic / static X-ray images based on the configured region of interest mask image and the corresponding multiple dynamic / static X-ray images; wherein the set value can be configured to 1 or other values.

[0073] In the embodiments of the present disclosure and other possible embodiments, the method for determining the region of interest image corresponding to the multiple dynamic / static X-ray images based on the configured region of interest mask image and the corresponding multiple dynamic / static X-ray images includes: multiplying (pixel level multiplication) the configured region of interest mask image with the corresponding multiple dynamic / static X-ray images to determine the region of interest image corresponding to the multiple dynamic / static X-ray images.

[0074] In an embodiment of the present disclosure, the method for performing a second enhanced training on the second preset convolutional neural network after the first enhanced training based on the multiple second dynamic / static X-ray image pairs and the first preset convolutional neural network includes: obtaining the multiple second dynamic / static X-ray image pairs and their corresponding multiple region of interest mask image pairs; determining the corresponding second training deformation field based on the fixed image and the floating image of each pair of images in the multiple second dynamic / static X-ray image pairs and the second preset convolutional neural network of the first enhanced training; determining the corresponding third deformation image based on the floating image in each pair of images and the corresponding second training deformation field; calculating the fourth loss based on the third deformation image corresponding to the floating image of each pair of images and the fixed image; and calculating the fourth loss based on the fixed image. A corresponding fourth deformed image is determined based on the mask image of the region of interest corresponding to the floating image in each pair of images and the corresponding second training deformation field; a fifth loss is calculated based on the fourth deformed image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; the fourth deformed image and the mask image of the region of interest corresponding to the fixed image are encoded using the first preset convolutional neural network to obtain a third encoding feature map and a fourth encoding feature map; and a sixth loss is calculated based on the third encoding feature map and the fourth encoding feature map; based on the fourth loss, the fifth loss and the sixth loss, the parameters of the second preset convolutional neural network after the first enhanced training are adjusted to complete the second enhanced training of the second preset convolutional neural network after the first training.

[0075] In an embodiment of the present disclosure, before acquiring a plurality of region of interest mask image pairs corresponding to the plurality of first dynamic / static X-ray image pairs, a third preset convolutional neural network or a preset segmentation model for the region of interest segmentation is used to perform the region of interest segmentation on the plurality of first dynamic / static X-ray image pairs, respectively, to obtain a plurality of region of interest mask image pairs corresponding to the plurality of dynamic / static X-ray images.

[0076] In the embodiments of the present disclosure and other possible embodiments, the multiple dynamic / static X-ray images are transformed respectively to obtain a first transformed X-ray image corresponding to each dynamic / static X-ray image; multiple first dynamic / static X-ray image pairs are constructed based on each dynamic / static X-ray image and its corresponding first transformed X-ray image; based on the multiple first dynamic / static X-ray image pairs and the first preset convolutional neural network, before the first enhanced training of the second preset convolutional neural network after the initial training, it also includes: determining that the multiple dynamic / static X-ray images for training are configured as multiple dynamic X-ray images or as dynamic X-ray images or as multiple dynamic X-ray images and dynamic X-ray images.

[0077] In the embodiments of the present disclosure and other possible embodiments, if the multiple dynamic / static X-ray images used for training are configured as static X-ray images, the multiple static X-ray images are transformed respectively to obtain a first transformed X-ray image corresponding to each static X-ray image; based on each static X-ray image and its corresponding first transformed X-ray image, a plurality of first static X-ray image pairs are constructed; based on the plurality of first static X-ray image pairs and the first preset convolutional neural network, the second preset convolutional neural network after the initial training is subjected to a first enhancement training; and then, according to the transformation of the multiple static X-ray images respectively, the first enhanced training is performed on the second preset convolutional neural network after the initial training. The invention discloses a method for transforming the region of interest images corresponding to the multiple static X-ray images, respectively, to obtain a second static transformed X-ray image corresponding to each region of interest image; respectively based on each region of interest image and its corresponding second static transformed X-ray image, construct a plurality of second static X-ray image pairs; based on the plurality of second static X-ray image pairs and the first preset convolutional neural network, perform a second enhancement training on the second preset convolutional neural network after the first enhancement training; further, based on the second preset convolutional neural network after the second enhancement training, perform registration on any two region of interest images in the dynamic X-ray image sequence to be registered.

[0078] In the embodiments of the present disclosure and other possible embodiments, the method for performing a first enhanced training on the second preset convolutional neural network after the initial training based on the multiple first static X-ray image pairs and the first preset convolutional neural network includes: obtaining the multiple first static X-ray image pairs and their corresponding multiple region of interest mask image pairs; determining the corresponding first training deformation field based on the fixed image and the floating image of each pair of images in the multiple first static X-ray image pairs and the second preset convolutional neural network after the initial training; determining the corresponding first deformed image based on the floating image in each pair of images and the corresponding first training deformation field; calculating the first loss based on the first deformed image corresponding to the floating image of each pair of images and the fixed image; and calculating the first loss based on the first deformed image corresponding to the floating image of each pair of images and the fixed image. The corresponding second deformed image is determined based on the mask image of the region of interest corresponding to the floating image in each pair of images and the corresponding first training deformation field; the second loss is calculated based on the second deformed image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; the second deformed image and the mask image of the region of interest corresponding to the fixed image are encoded using the first preset convolutional neural network to obtain a first encoding feature map and a second encoding feature map; and a third loss is calculated based on the first encoding feature map and the second encoding feature map; based on the first loss, the second loss and the third loss, the parameters of the second preset convolutional neural network after the initial training are adjusted to complete the first enhanced training of the second preset convolutional neural network after the initial training.

[0079] In the embodiments of the present disclosure and other possible embodiments, if the multiple dynamic / static X-ray images used for training are configured as dynamic X-ray images, the first enhanced training further includes: first enhanced training and second enhanced training; specifically, the multiple dynamic X-ray images are transformed respectively to obtain a first transformed X-ray image corresponding to each dynamic X-ray image; based on each dynamic X-ray image and its corresponding first transformed X-ray image, a plurality of first dynamic X-ray image pairs are constructed; based on the plurality of first dynamic X-ray image pairs and the first preset convolutional neural network, the second preset convolutional neural network after the initial training is subjected to first enhanced training; thereafter, a set of dynamic X-ray image pairs is constructed using the multiple dynamic X-ray images; the second preset convolutional neural network used for the first enhanced training is subjected to second enhanced training using the dynamic X-ray image pair set and the first preset convolutional neural network, thereby completing the first enhanced training of the second preset convolutional neural network. Among them, the method of constructing a dynamic X-ray image pair set using the multiple dynamic X-ray images includes: respectively extracting multiple name / ID information corresponding to the multiple dynamic X-ray images; respectively pairing the multiple dynamic X-ray images with different name / ID information to construct a dynamic X-ray image pair set.

[0080] In the embodiments of the present disclosure and other possible embodiments, the method for performing a first enhanced training on the second preset convolutional neural network after the initial training based on the multiple first dynamic X-ray image pairs and the first preset convolutional neural network includes: acquiring the multiple first dynamic X-ray image pairs and their corresponding multiple region of interest mask image pairs; determining the corresponding first deformation field based on the fixed image and the floating image of each pair of images in the multiple first dynamic X-ray image pairs and the second preset convolutional neural network after the initial training; determining the corresponding first training deformation image based on the floating image in each pair of images and the corresponding first deformation field; calculating the first training loss based on the first training deformation image corresponding to the floating image of each pair of images and the fixed image; and calculating the first training loss based on each pair of images. The invention relates to a method for enhancing the training image quality of a second training image. The method comprises the following steps: determining a corresponding second training deformed image based on a mask image of a region of interest corresponding to a floating image in the training image and a corresponding first training deformation field; calculating a second training loss based on the second training deformed image corresponding to the floating image of each pair of images and the mask image of a region of interest corresponding to the fixed image; encoding the second training deformed image and the mask image of a region of interest corresponding to the fixed image using the first preset convolutional neural network respectively to obtain a first encoding image and a second encoding image; and calculating a third training loss based on the first encoding image and the second encoding image; adjusting parameters of the second preset convolutional neural network after the initial training based on the first training loss, the second training loss and the third training loss to complete the first enhanced training of the second preset convolutional neural network after the initial training.

[0081] In the embodiments of the present disclosure and other possible embodiments, the method of performing second enhanced training on the second preset convolutional neural network used for the first enhanced training using the dynamic X-ray image pair set and the first preset convolutional neural network includes: obtaining the dynamic X-ray image pair set and its corresponding multiple region of interest mask image pairs; determining the corresponding second deformation field based on the fixed image and floating image of each pair of images in the dynamic X-ray image pair set and the second preset convolutional neural network after the first enhanced training; determining the corresponding third training deformed image based on the floating image in each pair of images and the corresponding second deformation field; calculating the fourth training loss based on the third training deformed image corresponding to the floating image of each pair of images and the fixed image; and calculating the fourth training loss based on each pair of images. The invention relates to a method for obtaining a plurality of image regions of interest (MRO) by using a plurality of convolutional neural networks (CNNs). The method comprises the following steps: determining a corresponding fourth training deformed image based on a mask image of a region of interest corresponding to a floating image and a corresponding second training deformation field; calculating a fifth training loss based on the fourth training deformed image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; encoding the fourth training deformed image and the mask image of the region of interest corresponding to the fixed image using the first preset convolutional neural network respectively to obtain a third encoding map and a fourth encoding map; and calculating a sixth training loss based on the third encoding map and the fourth encoding map; adjusting parameters of the second preset convolutional neural network after the first enhanced training based on the fourth training loss, the fifth training loss and the sixth training loss, and completing a second enhanced training of the second preset convolutional neural network after the first enhanced training.

[0082] In the embodiments of the present disclosure and other possible embodiments, further, the second enhanced training includes: third enhanced training and fourth enhanced training; specifically, according to the method of transforming the multiple dynamic X-ray images respectively, the region of interest images corresponding to the multiple dynamic X-ray images are transformed respectively to obtain a second dynamic transformed X-ray image corresponding to each region of interest image; based on each region of interest image and its corresponding second transformed X-ray image, a plurality of second dynamic X-ray image pairs are constructed respectively; based on the plurality of second dynamic X-ray image pairs and the first preset convolutional neural network, the second preset convolutional neural network after the first enhanced training is subjected to third enhanced training; using the region of interest images corresponding to the multiple dynamic X-ray images, a set of dynamic region of interest image pairs is constructed; using the dynamic region of interest image pair set and the first preset convolutional neural network, the second preset convolutional neural network used for the third enhanced training is subjected to fourth enhanced training to complete the second enhanced training of the second preset convolutional neural network. Among them, the method of constructing a dynamic region of interest image pair set using the region of interest images corresponding to the multiple dynamic X-ray images includes: respectively extracting multiple name / ID information corresponding to the multiple dynamic X-ray images or their corresponding region of interest images; respectively pairing the multiple dynamic region of interest images with different name / ID information to construct a dynamic region of interest image pair set.

[0083] In the embodiments of the present disclosure and other possible embodiments, the method for performing a third enhanced training on the second preset convolutional neural network after the first enhanced training based on the multiple second dynamic X-ray image pairs and the first preset convolutional neural network includes: acquiring the multiple second dynamic X-ray image pairs and their corresponding multiple region of interest mask image pairs; determining the corresponding third deformation field based on the fixed image and the floating image of each pair of images in the multiple second dynamic X-ray image pairs and the second preset convolutional neural network of the first enhanced training; determining the corresponding sixth training deformation image based on the floating image in each pair of images and the corresponding third deformation field; calculating the seventh loss based on the sixth training deformation image corresponding to the floating image of each pair of images and the fixed image; dividing Determine the corresponding seventh training deformed image based on the mask image of the region of interest corresponding to the floating image in each pair of images and the corresponding third deformation field; calculate the eighth loss based on the seventh training deformed image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; respectively encode the seventh training deformed image and the mask image of the region of interest corresponding to the fixed image using the first preset convolutional neural network to obtain a fifth encoding image and a sixth encoding image; and calculate the ninth loss based on the fifth encoding image and the sixth encoding image; based on the seventh loss, the eighth loss and the ninth loss, adjust the parameters of the second preset convolutional neural network after the first enhanced training, and complete the third enhanced training of the second preset convolutional neural network after the first training.

[0084] In the embodiments of the present disclosure and other possible embodiments, the method of using the dynamic region-of-interest image pair set and the first preset convolutional neural network to perform fourth enhancement training on the second preset convolutional neural network used for the third enhancement training to complete the second enhancement training of the second preset convolutional neural network includes: obtaining the dynamic region-of-interest image pair set and its corresponding multiple region-of-interest mask image pairs; determining the corresponding fourth deformation field based on the fixed image and the floating image of each pair of images in the dynamic region-of-interest image pair set and the second preset convolutional neural network of the third enhancement training; determining the corresponding eighth training deformation image based on the floating image in each pair of images and the corresponding fourth deformation field; calculating the eighth training deformation image corresponding to the floating image of each pair of images and the fixed image respectively. The tenth loss; determining the corresponding ninth training deformed image based on the mask image of the region of interest corresponding to the floating image in each pair of images and the corresponding fourth deformation field; calculating the eleventh loss based on the ninth training deformed image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; respectively encoding the ninth training deformed image and the mask image of the region of interest corresponding to the fixed image using the first preset convolutional neural network to obtain a seventh encoding image and an eighth encoding image; and calculating the twelfth loss based on the seventh encoding image and the eighth encoding image; adjusting the parameters of the second preset convolutional neural network after the third enhanced training based on the tenth loss, the eleventh loss and the twelfth loss, and completing the second enhanced training of the second preset convolutional neural network after the first training.

[0085] In the embodiments of the present disclosure and other possible embodiments, when the multiple dynamic / static X-ray images used for training are configured as a first number of multiple dynamic X-ray images and a second number of multiple static X-ray images, the first number of multiple dynamic X-ray images, the second number of multiple static X-ray images and the first preset convolutional neural network are used to perform initial training on the second preset convolutional neural network used for registration; and the second preset convolutional neural network after the initial training is subjected to enhanced training using the first number of multiple dynamic X-ray images, the second number of multiple static X-ray images, multiple dynamic X-ray image pairs (a pair of X-ray images) corresponding to the region of interest images of the first number of multiple dynamic X-ray images and the second number of multiple static X-ray images and the first preset convolutional neural network.

[0086] First, in the embodiment of the present disclosure and other possible embodiments, the method for initially training the second preset convolutional neural network for registration using the first number of multiple dynamic X-ray images, the second number of multiple static X-ray images and the first preset convolutional neural network includes: constructing a dynamic / static X-ray image pair set using the first number of multiple dynamic X-ray images and the second number of multiple static X-ray images; initially training the second preset convolutional neural network for registration using the dynamic / static X-ray image pair set and the first preset convolutional neural network; wherein, the method for constructing a dynamic / static X-ray image pair set using the first number of multiple dynamic X-ray images and the second number of multiple static X-ray images includes: respectively extracting multiple name / ID information corresponding to the first number of multiple dynamic X-ray images and the second number of multiple static X-ray images; respectively pairing the multiple dynamic / static X-ray images with different name / ID information to construct a dynamic / static X-ray image pair set.

[0087] Secondly, in the embodiments of the present disclosure and other possible embodiments, the first number of multiple dynamic X-ray images and the second number of multiple static X-ray images are transformed respectively to obtain a first transformed X-ray image corresponding to each dynamic and static X-ray image; based on each dynamic and static X-ray image and its corresponding first transformed X-ray image, a plurality of first dynamic and static X-ray image pairs are constructed; based on the plurality of first dynamic and static X-ray image pairs and the first preset convolutional neural network, a first enhancement training is performed on the second preset convolutional neural network after the initial training; thereafter, a set of dynamic X-ray image pairs is constructed using the first number of multiple dynamic X-ray images; a second enhancement training is performed on the second preset convolutional neural network used for the first enhancement training using the first number of dynamic X-ray image pair sets and the first preset convolutional neural network, thereby completing the first enhancement training of the second preset convolutional neural network.

[0088] In the embodiments of the present disclosure and other possible embodiments, the method for performing a first enhanced training on the second preset convolutional neural network after the initial training based on the multiple first dynamic and static X-ray image pairs and the first preset convolutional neural network includes: acquiring the multiple first dynamic and static X-ray image pairs and their corresponding multiple region of interest mask image pairs; determining the corresponding first deformation field based on the fixed image and the floating image of each pair of images in the multiple first dynamic and static X-ray image pairs and the second preset convolutional neural network after the initial training; determining the corresponding first training deformation image based on the floating image in each pair of images and the corresponding first deformation field; calculating the first training loss based on the first training deformation image corresponding to the floating image of each pair of images and the fixed image; and calculating the first training loss based on the fixed image. Determine the corresponding second training deformed image based on the region of interest mask image corresponding to the floating image in each pair of images and the corresponding first training deformation field; calculate the second training loss based on the second training deformed image corresponding to the floating image of each pair of images and the region of interest mask image corresponding to the fixed image; encode the second training deformed image and the region of interest mask image corresponding to the fixed image using the first preset convolutional neural network to obtain a first encoding image and a second encoding image; and calculate the third training loss based on the first encoding image and the second encoding image; adjust the parameters of the second preset convolutional neural network after the initial training based on the first training loss, the second training loss and the third training loss to complete the first enhanced training of the second preset convolutional neural network after the initial training.

[0089] In the embodiments of the present disclosure and other possible embodiments, the method of performing second enhanced training on the second preset convolutional neural network used for the first enhanced training using the first number of dynamic X-ray image pair sets and the first preset convolutional neural network includes: obtaining the dynamic and static X-ray image pair sets and their corresponding multiple region of interest mask image pairs; determining the corresponding second deformation field based on the fixed image and floating image of each pair of images in the dynamic and static X-ray image pair sets and the second preset convolutional neural network after the first enhanced training; determining the corresponding third training deformed image based on the floating image in each pair of images and the corresponding second deformation field; calculating the fourth training loss based on the third training deformed image corresponding to the floating image of each pair of images and the fixed image; and calculating the fourth training loss based on the fixed image. Determine a corresponding fourth training deformation image based on the region of interest mask image corresponding to the floating image in each pair of images and the corresponding second training deformation field; calculate a fifth training loss based on the fourth training deformation image corresponding to the floating image of each pair of images and the region of interest mask image corresponding to the fixed image; encode the fourth training deformation image and the region of interest mask image corresponding to the fixed image using the first preset convolutional neural network to obtain a third encoding image and a fourth encoding image; and calculate a sixth training loss based on the third encoding image and the fourth encoding image; adjust the parameters of the second preset convolutional neural network after the first enhancement training based on the fourth training loss, the fifth training loss and the sixth training loss, and complete the second enhancement training of the second preset convolutional neural network after the first enhancement training.

[0090] Finally, in the embodiments of the present disclosure and other possible embodiments, it also includes: according to the method of transforming the first number of multiple dynamic X-ray images and the second number of multiple static X-ray images respectively, transforming the region of interest images corresponding to the first number of multiple dynamic X-ray images and the second number of multiple static X-ray images respectively, to obtain a second transformed X-ray image corresponding to each region of interest image; constructing a plurality of second dynamic and static X-ray image pairs based on each region of interest image and its corresponding second transformed X-ray image respectively; performing a third enhanced training on the second preset convolutional neural network after the first enhanced training based on the plurality of second dynamic and static X-ray image pairs and the first preset convolutional neural network; constructing a set of dynamic region of interest image pairs using the region of interest images corresponding to the first number of multiple dynamic X-ray images; performing a fourth enhanced training on the second preset convolutional neural network used for the third enhanced training using the dynamic region of interest image pair set and the first preset convolutional neural network, to complete the second enhanced training of the second preset convolutional neural network.

[0091] In the embodiments of the present disclosure and other possible embodiments, the method for performing a third enhanced training on the second preset convolutional neural network after the first enhanced training based on the multiple second dynamic and static X-ray image pairs and the first preset convolutional neural network includes: acquiring the multiple second dynamic and static X-ray image pairs and their corresponding multiple region of interest mask image pairs; determining the corresponding third deformation field based on the fixed image and the floating image of each pair of images in the multiple second dynamic and static X-ray image pairs and the second preset convolutional neural network of the first enhanced training; determining the corresponding sixth training deformation image based on the floating image in each pair of images and the corresponding third deformation field; calculating the seventh training deformation image based on the sixth training deformation image corresponding to the floating image of each pair of images and the fixed image, respectively. loss; determining the corresponding seventh training deformed image based on the mask image of the region of interest corresponding to the floating image in each pair of images and the corresponding third deformation field; calculating the eighth loss based on the seventh training deformed image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; respectively encoding the seventh training deformed image and the mask image of the region of interest corresponding to the fixed image using the first preset convolutional neural network to obtain a fifth encoding image and a sixth encoding image; and calculating the ninth loss based on the fifth encoding image and the sixth encoding image; based on the seventh loss, the eighth loss and the ninth loss, adjusting the parameters of the second preset convolutional neural network after the first enhanced training, and completing the third enhanced training of the second preset convolutional neural network after the first training.

[0092] In the embodiments of the present disclosure and other possible embodiments, the method of using the dynamic region-of-interest image pair set and the first preset convolutional neural network to perform fourth enhancement training on the second preset convolutional neural network used for the third enhancement training to complete the second enhancement training of the second preset convolutional neural network includes: obtaining the dynamic region-of-interest image pair set and its corresponding multiple region-of-interest mask image pairs; determining the corresponding fourth deformation field based on the fixed image and the floating image of each pair of images in the dynamic region-of-interest image pair set and the second preset convolutional neural network of the third enhancement training; determining the corresponding eighth training deformation image based on the floating image in each pair of images and the corresponding fourth deformation field; calculating the eighth training deformation image corresponding to the floating image of each pair of images and the fixed image respectively. The tenth loss; determining the corresponding ninth training deformed image based on the mask image of the region of interest corresponding to the floating image in each pair of images and the corresponding fourth deformation field; calculating the eleventh loss based on the ninth training deformed image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; respectively encoding the ninth training deformed image and the mask image of the region of interest corresponding to the fixed image using the first preset convolutional neural network to obtain a seventh encoding image and an eighth encoding image; and calculating the twelfth loss based on the seventh encoding image and the eighth encoding image; adjusting the parameters of the second preset convolutional neural network after the third enhanced training based on the tenth loss, the eleventh loss and the twelfth loss, and completing the second enhanced training of the second preset convolutional neural network after the first training.

[0093] In the embodiments of the present disclosure and other possible embodiments, the multiple name / ID information corresponding to the multiple dynamic X-ray images and the multiple static X-ray images are different; that is, the image pairs in the dynamic X-ray image pair set, the static X-ray image pair set, the dynamic / static X-ray image pair set, the dynamic region of interest image pair set or other X-ray image pairs or sets (for example, the first dynamic X-ray image pair, the second dynamic X-ray image pair, the first static X-ray image pair and the second static X-ray image pair, etc.) cannot be configured as the same X-ray image.

[0094] For example, in the embodiments of the present disclosure and other possible embodiments, the multiple dynamic / static X-ray images used for training are configured as multiple dynamic X-ray images corresponding to the chest, and the corresponding regions of interest can be configured as one or more of the left lung field, the right lung field, the heart, the diaphragm, etc. Among them, the number corresponding to the multiple dynamic X-ray images is 789. First, 635 dynamic X-ray images are selected from the Chinese collection of the public data set D1 (China Shenzhen Dataset-Chest X-ray (X-ray) Database). Specifically, the 635 dynamic X-ray images include 324 normal cases and 311 cases with tuberculosis manifestations. Since radiologists cannot make judgments on the lung fields, 24 cases with tuberculosis manifestations are excluded from this study. Secondly, 54 dynamic X-ray images are randomly selected from the Montgomery X-ray set in the public data set D2. Specifically, the 54 dynamic X-ray images include 47 normal cases and 7 cases with tuberculosis manifestations. Meanwhile, 72 dynamic X-ray images were selected from public datasets D3 (NIAID-TB portal program dataset [online]) and D4 (kaggle.RSNA pneumonia detection challenge [online]). Specifically, the 72 dynamic X-ray images included 7 cases of tuberculosis manifestations, 60 cases of pneumonia manifestations, and 5 normal cases. Finally, 15 dynamic X-ray images were collected from the Google website (dataset D5). The remaining 13 dynamic X-ray images (dataset D6) from chest X-ray videos (chest radiography) were collected during free breathing. Specifically, the chest X-ray video was collected from a 53-year-old female participant using a digital X-ray imaging system (manufacturer: Lanmage, acquisition mode: sequential point slices, exposure parameters: 78KV, 200mA, 50ms, flat panel detector: IRAY) for chest radiography. Obviously, the multiple dynamic / static X-ray images for training are configured as 789 static X-ray images; the dynamic X-ray image sequence to be registered is configured as 13 dynamic X-ray image sequences.

[0095] In an embodiment of the present disclosure, the method of using the multiple dynamic / static X-ray images and the first preset convolutional neural network to perform initial training on the second preset convolutional neural network used for registration includes: using the 789 static X-ray images to construct a static X-ray image pair set (789*789-789 pairs of images / image pairs); using the static X-ray image pair set and the first preset convolutional neural network to perform initial training on the second preset convolutional neural network used for registration.

[0096] In the embodiments of the present disclosure and other possible embodiments, the method of using the 789 static X-ray images and the multiple static X-ray image pairs corresponding to the region of interest images thereof (789*789-789 pairs of static images and the region of interest images corresponding to the 789*789-789 pairs of static images - lung field images / heart images) and the first preset convolutional neural network to perform enhanced training on the second preset convolutional neural network after initial training, comprises: performing affine transformation on the 789 static X-ray images respectively to obtain a first transformed X-ray image corresponding to each static X-ray image; constructing multiple first static X-ray image pairs based on each static X-ray image and its corresponding first transformed X-ray image respectively; and training the second preset convolutional neural network after initial training based on the multiple first static X-ray image pairs and the first preset convolutional neural network. The network performs the first enhancement training; according to the method of performing affine transformation on the multiple dynamic / static X-ray images respectively, the region of interest images corresponding to the 789 static X-ray images are transformed respectively to obtain the second transformed X-ray image corresponding to each region of interest image; based on each region of interest image and its corresponding second transformed X-ray image, a plurality of second static X-ray image pairs (789*789-789 pairs of region of interest images corresponding to static images - lung field image / heart image) are constructed respectively; based on the plurality of second static X-ray image pairs and the first preset convolutional neural network, the second preset convolutional neural network after the first enhancement training is subjected to the second enhancement training; further, based on the second preset convolutional neural network after the second enhancement training, any two region of interest images in the dynamic X-ray image sequence to be registered are registered.

[0097] Step S104: Based on the second preset convolutional neural network after the enhanced training, any two region of interest images in the dynamic X-ray image sequence to be registered are registered.

[0098] Based on the above, in the embodiments of the present disclosure and other possible embodiments, based on the second preset convolutional neural network after the enhanced training, any two images of the region of interest (lung field images / heart images corresponding to the dynamic X-ray images to be registered) in the dynamic X-ray image sequence to be registered (13 dynamic X-ray images to be registered) are registered.

[0099] In an embodiment of the present disclosure, the method for registering any two region of interest images in the dynamic X-ray image sequence to be registered based on the second preset convolutional neural network after the enhanced training includes: using the third preset convolutional neural network or preset segmentation model for the region of interest segmentation to perform the region of interest segmentation on the dynamic X-ray image sequence to be registered, respectively, to obtain a plurality of region of interest mask image sequences corresponding to the dynamic X-ray image sequence to be registered; and configuring the mask value corresponding to the region of interest mask image sequence as a set value; based on the configured region of interest mask image sequence and its corresponding dynamic X-ray image sequence to be registered, determining the region of interest image sequence corresponding to the dynamic X-ray image sequence to be registered; based on the second preset convolutional neural network after the enhanced training, registering any two region of interest images in the region of interest image sequence. Wherein, the set value can be configured as 1 or other values.

[0100] In an embodiment of the present disclosure, the second preset convolutional neural network based on the enhanced training is used to align any two region of interest images in the dynamic X-ray image sequence to be aligned, and the corresponding two region of interest images are respectively configured as image pairs corresponding to the maximum region of interest area and the minimum region of interest area in the region of interest images.

[0101] In the embodiments of the present disclosure and other possible embodiments, the third preset convolutional neural network used for segmenting the region of interest can be configured as one or more of FCN, Unet, Unet++, SegNet, RefineNet, nn Unet or their improved convolutional neural networks.

[0102] For example, in the embodiments of the present disclosure and other possible embodiments, the region of interest may be configured as the left lung field, the right lung field, or the entire lung field including the left lung field and the right lung field. Furthermore, the third preset convolutional neural network or preset segmentation model used for segmenting the region of interest may adopt the technical solutions corresponding to the lung area segmentation, lung disease assessment method and device, electronic device and storage medium (2023108100929), lung field image segmentation method, device and storage medium (2023106887446) previously disclosed by our company, or other existing lung field / lung area segmentation models.

[0103] The execution subject of the dynamic X-ray image registration method may be an image processing device, for example, the dynamic X-ray image registration method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the dynamic X-ray image registration method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0104] Those skilled in the art will understand that, in the above-mentioned dynamic X-ray image registration method of the specific implementation mode, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0105] The disclosed embodiment also proposes a dynamic X-ray image registration device, which includes: an acquisition unit, used to acquire a dynamic X-ray image sequence to be registered, multiple dynamic / static X-ray images for training and their corresponding first preset convolutional neural network for limiting the region of interest; an initial training unit, used to use the multiple dynamic / static X-ray images and the first preset convolutional neural network to perform initial training on a second preset convolutional neural network for registration; an enhanced training unit, used to use the multiple dynamic / static X-ray images and the multiple dynamic / static X-ray image pairs corresponding to the multiple dynamic / static X-ray images and the region of interest images and the first preset convolutional neural network to perform enhanced training on the second preset convolutional neural network after the initial training; and a registration unit, used to register any two region of interest images in the dynamic X-ray image sequence to be registered based on the second preset convolutional neural network after the enhanced training.

[0106] In an embodiment of the present disclosure, the registration device further includes: a segmentation unit; wherein the segmentation unit is used to segment the region of interest in the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images to obtain a corresponding region of interest mask image.

[0107] In the embodiments of the present disclosure and other possible embodiments, the segmentation unit is configured as a third preset convolutional neural network or a preset segmentation model that can be used for the segmentation of the region of interest. Among them, the third preset convolutional neural network that can be used for the segmentation of the region of interest can be configured as one or more of FCN, Unet, Unet++, SegNet, RefineNet, nnUnet or its improved convolutional neural network. For example, in the embodiments of the present disclosure and other possible embodiments, the region of interest can be configured as the left lung field, the right lung field, or the entire lung field including the lung field and the right lung field. Furthermore, the third preset convolutional neural network or preset segmentation model used for the segmentation of the region of interest can adopt the technical solutions corresponding to the lung area segmentation, lung disease assessment method and device, electronic device and storage medium (2023108100929), lung field image segmentation method, device and storage medium (2023106887446) previously disclosed by our company, or other existing lung field / lung area segmentation models.

[0108] In the embodiment of the present disclosure, the registration device further includes: a region of interest extraction unit; wherein the region of interest extraction unit is used to extract the region of interest image sequence and / or region of interest image in the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images based on the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images and their corresponding region of interest mask image sequence and / or region of interest mask image; wherein, based on the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images and their corresponding region of interest mask image sequence and / or region of interest mask image A region of interest mask image sequence and / or a region of interest mask image, extracting the region of interest image sequence and / or the region of interest image in the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images, comprising: performing pixel-level multiplication on the positions corresponding to the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images and their corresponding region of interest mask image sequence and / or the region of interest mask image, extracting the region of interest image sequence and / or the region of interest image in the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images.

[0109] In the disclosed embodiment, the registration device further includes: an area determination unit; wherein the area determination unit is used to determine the region of interest area sequence corresponding to all region of interest images in the dynamic X-ray image sequence to be registered, determine the image pair corresponding to the maximum region of interest area and the minimum region of interest area based on the region of interest area sequence, and register the image pair using the registration unit. Specifically, the second preset convolutional neural network after the enhanced training is used to register any two region of interest images in the dynamic X-ray image sequence to be registered, and the corresponding two region of interest images are respectively configured as the image pair corresponding to the maximum region of interest area and the minimum region of interest area in the region of interest images.

[0110] The embodiment of the present disclosure also proposes a dynamic X-ray image registration device, comprising: a processor; a memory for storing processor executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above-mentioned dynamic X-ray image registration method.

[0111] The embodiment of the present disclosure also proposes a dynamic X-ray image registration device, including: a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the dynamic X-ray image registration method described above is implemented.

[0112] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the dynamic X-ray image registration method described in the above method embodiment. Its specific implementation can refer to the description of the above dynamic X-ray image registration method embodiment. For the sake of brevity, it will not be repeated here.

[0113] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned dynamic X-ray image registration method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0114] The embodiment of the present disclosure also provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above-mentioned dynamic X-ray image registration method. The electronic device can be provided as a terminal, a server or other forms of equipment.

[0115] The disclosed embodiment also provides a computer program product, including a computer program / instruction, which implements the above-mentioned dynamic X-ray image registration method when executed by a processor.

[0116] The disclosed embodiments further provide a medical device that applies the above-mentioned registration method and / or includes the above-mentioned registration device and / or includes the above-mentioned computer program product.

[0117] Figure 2 8 is a block diagram of an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0118] Reference Figure 2 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0119] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0120] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as dynamic random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0121] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0122] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0123] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0124] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: home button, volume button, start button, and lock button.

[0125] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0126] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0127] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0128] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0129] Figure 3 1 is a block diagram of an electronic device 1900 according to an exemplary embodiment. For example, the electronic device 1900 may be provided as a server. Figure 3 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0130] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0131] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0132] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0133] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a dynamic random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0134] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0135] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0136] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0137] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0138] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0139] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0140] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A dynamic X-ray image registration method, characterized in that: include: Acquire a dynamic X-ray image sequence to be registered, a plurality of dynamic / static X-ray images for training, and a first preset convolutional neural network corresponding thereto for limiting a region of interest; Using the plurality of dynamic / static X-ray images and the first preset convolutional neural network, initially training a second preset convolutional neural network for registration; Performing enhanced training on the second preset convolutional neural network after initial training using the plurality of dynamic / static X-ray image pairs corresponding to the plurality of dynamic / static X-ray images and the region of interest images thereof and the first preset convolutional neural network; Based on the second preset convolutional neural network after the enhanced training, any two images of the region of interest in the dynamic X-ray image sequence to be registered are registered.

2. The registration method according to claim 1, characterized in that: The method of performing enhanced training on the second preset convolutional neural network after initial training using the multiple dynamic / static X-ray images and the multiple dynamic / static X-ray image pairs corresponding to the multiple dynamic / static X-ray images and the images of the region of interest and the first preset convolutional neural network comprises: transforming the multiple dynamic / static X-ray images respectively to obtain a first transformed X-ray image corresponding to each dynamic / static X-ray image; constructing multiple first dynamic / static X-ray image pairs based on each dynamic / static X-ray image and its corresponding first transformed X-ray image respectively; and performing a first enhanced training on the second preset convolutional neural network after initial training based on the multiple first dynamic / static X-ray image pairs and the first preset convolutional neural network; According to the method of transforming the multiple dynamic / static X-ray images respectively, the region of interest images corresponding to the multiple dynamic / static X-ray images are transformed respectively to obtain a second transformed X-ray image corresponding to each region of interest image; based on each region of interest image and its corresponding second transformed X-ray image, a plurality of second dynamic / static X-ray image pairs are constructed respectively; based on the plurality of second dynamic / static X-ray image pairs and the first preset convolutional neural network, a second enhanced training is performed on the second preset convolutional neural network after the first enhanced training; Furthermore, based on the second preset convolutional neural network after the second enhanced training, any two images of the region of interest in the dynamic X-ray image sequence to be registered are registered.

3. The registration method according to claim 2, characterized in that: The method for performing a first enhanced training on the second preset convolutional neural network after the initial training based on the multiple first dynamic / static X-ray image pairs and the first preset convolutional neural network comprises: obtaining the multiple first dynamic / static X-ray image pairs and the multiple corresponding region of interest mask image pairs; determining the corresponding first training deformation field based on the fixed image and the floating image of each pair of images in the multiple first dynamic / static X-ray image pairs and the second preset convolutional neural network after the initial training; determining the corresponding first deformed image based on the floating image in each pair of images and the corresponding first training deformation field; calculating the first loss based on the first deformed image corresponding to the floating image of each pair of images and the fixed image; determining the corresponding first deformation field based on the floating image in each pair of images; calculating the first loss based on the floating image in each pair of images; calculating the first deformation field based on the floating image in each pair of images. The invention relates to a method for obtaining a first image of a floating image of a fixed image and a second image of a floating image corresponding to the image and a corresponding first training deformation field, wherein the second deformation image is determined based on the mask image of the region of interest corresponding to the image and the corresponding first training deformation field; the second loss is calculated based on the second deformation image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; the second deformation image and the mask image of the region of interest corresponding to the fixed image are encoded using the first preset convolutional neural network to obtain a first encoding feature map and a second encoding feature map; and a third loss is calculated based on the first encoding feature map and the second encoding feature map; based on the first loss, the second loss and the third loss, the parameters of the second preset convolutional neural network after the initial training are adjusted to complete the first enhancement training of the second preset convolutional neural network after the initial training; and / or, Before acquiring the multiple region of interest mask image pairs corresponding to the multiple first dynamic / static X-ray image pairs, the region of interest segmentation is performed on the multiple first dynamic / static X-ray image pairs respectively using a third preset convolutional neural network or a preset segmentation model for the region of interest segmentation to obtain the multiple region of interest mask image pairs corresponding to the multiple dynamic / static X-ray images.

4. The registration method according to any one of claims 2 to 3, characterized in that: Before transforming the region of interest images corresponding to the multiple dynamic / static X-ray images to obtain the second transformed X-ray image corresponding to each region of interest image, the method of respectively determining the region of interest images corresponding to the multiple dynamic / static X-ray images comprises: respectively acquiring the region of interest mask images corresponding to the multiple dynamic / static X-ray images, and configuring the mask value corresponding to the region of interest mask image as a set value; determining the region of interest images corresponding to the multiple dynamic / static X-ray images based on the configured region of interest mask image and the corresponding multiple dynamic / static X-ray images; and / or, The method for performing a second enhanced training on the second preset convolutional neural network after the first enhanced training based on the multiple second dynamic / static X-ray image pairs and the first preset convolutional neural network comprises: obtaining the multiple second dynamic / static X-ray image pairs and their corresponding multiple region of interest mask image pairs; determining the corresponding second training deformation field based on the fixed image and the floating image of each pair of images in the multiple second dynamic / static X-ray image pairs and the second preset convolutional neural network of the first enhanced training; determining the corresponding third deformed image based on the floating image in each pair of images and the corresponding second training deformation field; calculating the fourth loss based on the third deformed image corresponding to the floating image of each pair of images and the fixed image; and calculating the fourth loss based on the third deformed image corresponding to the floating image of each pair of images and the fixed image. The invention discloses a method for obtaining a first enhanced convolutional neural network (CNN) comprising the steps of: determining a corresponding fourth deformed image based on a mask image of a region of interest corresponding to a floating image and a corresponding second training deformation field; calculating a fifth loss based on the fourth deformed image corresponding to the floating image of each pair of images and the mask image of the region of interest corresponding to the fixed image; encoding the fourth deformed image and the mask image of the region of interest corresponding to the fixed image using the first preset convolutional neural network, respectively, to obtain a third encoded feature map and a fourth encoded feature map; and calculating a sixth loss based on the third encoded feature map and the fourth encoded feature map; adjusting the parameters of the second preset convolutional neural network after the first enhanced training based on the fourth loss, the fifth loss and the sixth loss, and completing a second enhanced training of the second preset convolutional neural network after the first training.

5. The registration method according to any one of claims 1 to 4, characterized in that: Before acquiring a first preset convolutional neural network for limiting the region of interest corresponding to a plurality of dynamic / static X-ray images for training, using a third preset convolutional neural network or a preset segmentation model for segmenting the region of interest, respectively performing the region of interest segmentation on the plurality of dynamic / static X-ray images to obtain a plurality of region of interest mask images corresponding to the plurality of dynamic / static X-ray images; Using the plurality of region of interest mask images, training a first preset convolutional neural network for limiting the region of interest; and / or, The method of using the multiple dynamic / static X-ray images and the first preset convolutional neural network to initially train the second preset convolutional neural network for registration comprises: using the multiple dynamic / static X-ray images to construct a dynamic / static X-ray image pair set; using the dynamic / static X-ray image pair set and the first preset convolutional neural network to initially train the second preset convolutional neural network for registration; and / or, The method of initially training the second preset convolutional neural network for registration using the dynamic / static X-ray image pair set and the first preset convolutional neural network comprises: acquiring a plurality of region of interest mask images corresponding to the dynamic / static X-ray image pair set; Based on the fixed image and the floating image of each pair of images in the dynamic / static X-ray image pair set and the second preset convolutional neural network to be initially trained, the corresponding third training deformation field is determined; based on the floating image in each pair of images in the dynamic / static X-ray image pair set and the corresponding third training deformation field, the corresponding fifth deformation image is determined; based on the fifth deformation image corresponding to the floating image of each pair of images and the fixed image, the seventh loss is calculated; based on the mask image of the region of interest corresponding to the floating image in each pair of images and the corresponding third training deformation field, the corresponding sixth deformation image is determined; based on the mask image of the region of interest corresponding to the floating image in each pair of images and the corresponding third training deformation field, the eighth loss is calculated; the sixth deformation image and the mask image of the region of interest corresponding to the floating image of each pair of images are respectively encoded by the first preset convolutional neural network to obtain a fifth encoding feature map and a sixth encoding feature map; and the eighth loss is calculated based on the fifth encoding feature map and the sixth encoding feature map; based on the sixth loss, the seventh loss and the eighth loss, the parameters of the second preset convolutional neural network for initial training are adjusted to complete the initial training of the second preset convolutional neural network.

6. The registration method according to any one of claims 1 to 5, characterized in that: The method for registering any two region of interest images in the dynamic X-ray image sequence to be registered based on the second preset convolutional neural network after the enhanced training comprises: using a third preset convolutional neural network or a preset segmentation model for the region of interest segmentation to perform the region of interest segmentation on the dynamic X-ray image sequence to be registered, respectively, to obtain a plurality of region of interest mask image sequences corresponding to the dynamic X-ray image sequence to be registered; and configuring the mask value corresponding to the region of interest mask image sequence as a set value; based on the configured region of interest mask image sequence and its corresponding dynamic X-ray image sequence to be registered, determining The dynamic X-ray image sequence to be registered corresponds to a sequence of region of interest images; based on the second preset convolutional neural network after the enhanced training, any two region of interest images in the region of interest image sequence are registered; and / or the region of interest is configured as one or more of the left lung field, right lung field, heart, diaphragm, gastrointestinal tract, and breast; and / or, based on the second preset convolutional neural network after the enhanced training, any two region of interest images in the dynamic X-ray image sequence to be registered are registered, and the corresponding two region of interest images are respectively configured as image pairs corresponding to the maximum region of interest area and the minimum region of interest area in the region of interest images.

7. A dynamic X-ray image registration device, characterized in that: include: An acquisition unit, used for acquiring a dynamic X-ray image sequence to be registered, a plurality of dynamic / static X-ray images for training, and a first preset convolutional neural network corresponding thereto for limiting a region of interest; An initial training unit, configured to perform initial training on a second preset convolutional neural network for registration using the plurality of dynamic / static X-ray images and the first preset convolutional neural network; An enhanced training unit, configured to perform enhanced training on the second preset convolutional neural network after initial training by using the plurality of dynamic / static X-ray image pairs corresponding to the plurality of dynamic / static X-ray images and the images of the region of interest thereof and the first preset convolutional neural network; A registration unit, configured to register any two region-of-interest images in the dynamic X-ray image sequence to be registered based on the second preset convolutional neural network after the enhanced training; or, The invention comprises: a processor; a memory for storing instructions executable by the processor; The processor is configured to call the instructions stored in the memory to execute the registration method according to any one of claims 1 to 6; or The invention comprises: a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the registration method according to any one of claims 1 to 6.

8. The registration device according to claim 7, characterized in that: Also includes: A segmentation unit; wherein the segmentation unit is used to segment the region of interest in the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images to obtain a corresponding region of interest mask image; and / or, It also includes: a region of interest extraction unit; wherein the region of interest extraction unit is used to extract the region of interest image sequence and / or region of interest image in the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images based on the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images and their corresponding region of interest mask image sequence and / or region of interest mask image; Wherein, based on the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images and their corresponding region of interest mask image sequence and / or region of interest mask image, extracting the region of interest image sequence and / or region of interest image in the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images, including: performing pixel-level multiplication on the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images and the corresponding positions of the region of interest mask image sequence and / or the region of interest mask image, extracting the region of interest image sequence and / or region of interest image in the dynamic X-ray image sequence to be registered and / or the multiple dynamic / static X-ray images; and / or, It also includes: an area determination unit; wherein the area determination unit is used to determine a region of interest area sequence corresponding to all region of interest images in the dynamic X-ray image sequence to be registered, determine an image pair corresponding to a maximum region of interest area and a minimum region of interest area based on the region of interest area sequence, and use the registration unit to register the image pair.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the registration method according to any one of claims 1 to 6 is implemented.

10. A medical device, characterized in that: Apply the registration method as described in any one of claims 1 to 6 and / or include the registration device as described in any one of claims 7-8 and / or include the computer program product as described in claim 9.

Images