Dynamic image acquisition and stitching

Abstract

The invention relates to a computer-implemented method of acquiring and combining medical images of a subject. A first medical image of a subject acquired at a first field of view by a medical imaging device is analyzed with respect to inclusion of a region of interest of the subject. If the region of interest of the subject is not completely contained in the first medical image, at least one second field of view of the medical imaging device is determined, and data is generated for instructing the medical imaging device to acquire at least one second medical image of the subject with the at least one second field of view. The at least one second medical image contains at least a portion of the region of interest of the subject that is not contained in the first medical image. Further, the generated data is sent to a medical imaging device and at least one second medical image of the subject acquired in at least one second field of view is received and combined with the first medical image into a combined medical image. The combined medical image contains a region of interest of the subject and is provided to the user.

Description

TECHNICAL FIELD

[0001] The present invention relates to a computer-implemented method of acquiring and combining medical images of a subject, a medical imaging system, a data processing apparatus, a computer program and a computer readable storage medium. BACKGROUND

[0002] In conventional X-ray imaging and fluoroscopy applications, acquisitions are typically acquired in a static setup, i.e. with a fixed system geometry and collimator configuration. For conventional chest postero-anterior X-ray imaging, guidelines exist which describe the optimal positioning of the patient and collimator settings in order to ensure optimal acquisitions. However, due to the variability of patient anatomy or incidental findings, even a carefully planned acquisition can result in a suboptimal imaging zone. The implications range from unnecessary dose exposure, the need for a re-take, to a false diagnosis. While fluoroscopy applications suffer from similar limitations, they include an additional temporal component. For example, in swallowing examinations, contrast examinations, catheterization, etc., the region of interest can dynamically change during the examination, e.g. as a contrast agent passes through a passage of a particular organ. To this end, either a fixed image setup is employed, or manual interaction can be required.

[0003] Due to the static system configuration, X-ray imaging and fluoroscopy often result in suboptimal imaging zones and unnecessary dose exposure. At the same time, modern collimators allow for the acquisition of even very small structures by limiting the X-ray exposure to a specific region of interest.

[0004] Therefore, the inventors of the present invention have found that it would be advantageous to have a method which solves these problems of suboptimal imaging zones and unnecessary dose exposure while providing high quality images which allow for valuable diagnoses.

[0005] It should be noted that the patent application publication WO 2018 / 108923 Al discloses a radiographic system which can evaluate, based on a first image, whether a predetermined region of interest is fully captured in the first image, and if this is not the case, proposes to take a second image of the region of interest which is not captured in the first image. SUMMARY

[0006] It is an object of the present invention to provide an improved method of acquiring and combining medical images of a subject which limits the imaged zone to the necessary body or organ region and reduces the irradiation of tissue outside the region of interest.

[0007] The object of the present invention is solved by the subject matter of the independent claims, wherein further embodiments are incorporated in the dependent claims.

[0008] The described embodiments similarly relate to a computer-implemented method of acquiring and combining medical images of a subject, a medical imaging system, a data processing apparatus, a computer program, and a computer readable storage medium. The further described embodiments can be combined in any possible way. Synergistic effects can result from different combinations of embodiments, although they can not have been described in detail.

[0009] Furthermore, it should be noted that all embodiments of the present invention with respect to a method can be performed in the order of the described steps, however this need not be the only and necessary order of the steps of the method. The methods presented herein can be performed in another order of the disclosed steps without departing from the respective method embodiments, unless explicitly mentioned to the contrary hereinafter.

[0010] According to a first aspect of the present invention, a computer-implemented method of acquiring and combining medical images of a subject is provided. The method comprises the steps of receiving a first medical image of the subject acquired by a medical imaging device in a first field of view; analyzing the first medical image with respect to an encompassment of a region of interest of the subject in the first medical image; and in case the region of interest of the subject is not completely encompassed in the first medical image, determining at least one second field of view of the medical imaging device. The method further comprises the steps of generating data to instruct the medical imaging device to acquire at least one second medical image of the subject in the at least one second field of view, the at least one second medical image encompassing at least a portion of the region of interest of the subject not encompassed in the first medical image; and sending the generated data to the medical imaging device. The method further comprises the steps of receiving the at least one second medical image of the subject acquired by the medical imaging device in the at least one second field of view; combining the first medical image and the at least one second medical image into a combined medical image, the combined medical image encompassing the region of interest of the subject; and providing the combined medical image.

[0011] Hence, a dynamic acquisition scheme is proposed, preferably using an AI-based proposal system, to determine the need for acquiring at least one additional image and the corresponding imaging region in combination with an automated X-ray or fluoroscopy system capable of automatically adjusting and correcting the field of view to be imaged. The ability to change the field of view during acquisition based on already acquired image data provides a promising option to limit the imaged zone to the strictly necessary body or organ region, to dynamically extend the zone in case of imaging artifacts or incidental findings, and to reduce or even avoid irradiation of tissue outside the region of interest.

[0012] Preferably, the first and second images can comprise a plurality of images, respectively. In particular, when using, for example, a time series of images as input, the first medical image can comprise a plurality of input images. A time series of images can be understood as a sequence of images taken at successive equidistant time points. Each of these images can be analyzed with respect to the imaged region contained and for each of the plurality of images one or more second images can be acquired to correct the desired imaging region or field of view. In case of a time series, each of the plurality of input images can be corrected by the method of the present invention before the subsequent images of the time series are acquired. However, in case of correcting and extending previous images of a time series based on the proposed method, the proposed field of view can also be considered when acquiring subsequent images of the time series.

[0013] In particular, for tracking applications, the first acquired image can not contain the region of interest and multiple iterations of the method of the present invention can be necessary in order to cover the region of interest. In case of a fluoroscopy application, the temporal information can provide insight into the speed of distribution of a contrast agent, for example, in the body and thus can provide a better prediction of the diagnosis. For example, the subsequent images of the time series of images can be encoded as individual channels in the input image and the images can be analyzed using the method according to the present invention as described before.

[0014] In an embodiment of the present invention, the medical imaging device is an X-ray imaging device or a fluoroscopy device.

[0015] In an embodiment of the present invention, the generated data for instructing the medical imaging device to acquire at least one second medical image comprises instructions for changing the field of view of the medical imaging device by changing the position and / or orientation of a radiation source and / or a radiation detector of the medical imaging device.

[0016] In an embodiment of the present invention, the generated data for instructing the medical imaging device to acquire at least one second medical image comprises instructions for changing the field of view of the medical imaging device by changing the configuration of a collimator of the medical imaging device.

[0017] In an embodiment of the present invention, the medical imaging system comprises a multi-leaf collimator.

[0018] In an embodiment of the present invention, the generated data comprises the position, size and / or shape of the second field of view.

[0019] In an embodiment of the present invention, the steps of analyzing the first medical image and / or determining at least one second field of view of the medical imaging device are performed by a trained artificial intelligence.

[0020] In an embodiment of the present invention, the artificial intelligence comprises a regressor network or a region proposal network.

[0021] In embodiments of the application, the region of interest comprises an anatomical structure, a diffusion of contrast agent, a clinical finding or an inserted device of the subject.

[0022] In embodiments of the application, the first field of view of the first image is derived from a camera image or an X-ray pre-shot of the subject.

[0023] In embodiments of the application, the first medical images of the subject comprise a plurality of first medical images of the subject taken at successive points in time, and the step of determining at least one second field of view of the medical imaging device comprises analyzing a temporal change of the region of interest in the plurality of first medical images of the subject.

[0024] In embodiments of the application, the method comprises the step of proposing to a user to acquire at least one second medical image with the at least one second field of view.

[0025] According to another aspect of the application, there is provided a data processing apparatus comprising a processor configured to perform the steps of the method according to any of the preceding embodiments.

[0026] According to another aspect of the application, there is provided a medical imaging system comprising a data processing apparatus according to the preceding embodiment.

[0027] According to another aspect of the application, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method according to any of the preceding embodiments. The computer program can be executed on one or more processing units instructed to carry out the method according to any of the preceding embodiments.

[0028] According to another aspect of the application, there is provided a computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to any of the preceding embodiments.

[0029] Hence, the benefits provided by any of the previously cited aspects apply equally to all of the other aspects, and vice versa.

[0030] In summary, the present application relates to a computer-implemented method of acquiring and combining medical images of a subject. A first medical image of a subject acquired by a medical imaging device in a first field of view is analyzed with respect to a containment of a region of interest of the subject. In case the region of interest of the subject is not completely contained in the first medical image, at least one second field of view of the medical imaging device is determined and data for instructing the medical imaging device to acquire at least one second medical image of the subject in the at least one second field of view is generated. The at least one second medical image contains at least a portion of the region of interest of the subject not contained in the first medical image. Furthermore, the generated data is transmitted to the medical imaging device and the at least one second medical image of the subject acquired in the at least one second field of view is received and combined with the first medical image into a combined medical image. The combined medical image contains the region of interest of the subject and is provided to a user.

[0031] One of the advantages of embodiments of the present application to change the field of view during acquisition based on already acquired image data can be to limit the imaged zone to the strictly necessary body or organ region. Another advantage can be that the imaged zone can be dynamically extended in case of imaging artifacts or incidental findings. Another advantage can be that the irradiation of tissue outside the region of interest can be reduced or even avoided.

[0032] These advantages are non-limiting and other advantages can be envisaged in the context of the present application.

[0033] These aspects and embodiments will become apparent from and will be elucidated with respect to the example embodiments described hereinafter, and with respect to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A block diagram of a computer-implemented method of acquiring and combining medical images of a subject according to an embodiment of the present application is shown.

[0035] Figure 2 A schematic setup of a processing unit and a medical imaging system performing the steps of a method according to an embodiment of the present application is shown. LIST OF REFERENCE SIGNS 100 medical imaging device 110 subject 120 first medical image 121 first field of view 130 second medical image 131 second field of view 140 region of interest 150 combined medical image 160 generated data 170 collimator 200 data processing device DETAILED DESCRIPTION

[0036] Reference Figure 1 which shows a block diagram of a computer-implemented method of acquiring and combining medical images of a subject 110 according to an embodiment of the present application. The method comprises a step S110 of receiving a first medical image 120 of a subject 110 acquired by a medical imaging device 100 with a first field of view 121, a step S120 of analyzing the first medical image 120 with respect to a containment of a region of interest 140 of the subject 110 in the first medical image 120, and a step S130 of determining at least one second field of view 131 of the medical imaging device 100 in case the region of interest 140 of the subject 110 is not entirely contained in the first medical image 120. The method further comprises a step S140 of generating data 160 to instruct the medical imaging device 100 to acquire at least one second medical image 130 of the subject 110 with the at least one second field of view 131, the at least one second medical image 130 containing at least a portion of the region of interest 140 of the subject 110 not contained in the first medical image 120, and a step S150 of transmitting the generated data 160 to the medical imaging device 100. The method further comprises a step S160 of receiving the at least one second medical image 130 of the subject 110 acquired by the medical imaging device 110 with the at least one second field of view 131, a step S170 of combining the first medical image 120 and the at least one second medical image 130 into a combined medical image 150, the combined medical image 150 containing the region of interest 140 of the subject 110, and a step S180 of providing the combined medical image 150.

[0037] Accordingly, embodiments of the present application propose a method using a dynamic acquisition scheme, preferably with an AI-based proposal system, to determine the need for acquiring at least one additional image and corresponding imaging region in conjunction with an automated X-ray or fluoroscopy system capable of automatically adjusting and correcting the field of view to be imaged. The ability to change the field of view during acquisition based on already acquired image data provides promising options to limit the imaged zone to strictly necessary body or organ regions, dynamically expand the zone in case of imaging artifacts or incidental findings, and / or reduce or even avoid irradiation of tissue outside the region of interest.

[0038] The method according to the present application can use two main components: a processing unit 200 which analyzes the already acquired image data of the first medical image 120 and predicts the need for additional acquisitions, their location and / or expected size, and a medical imaging device 100 which can allow for dynamic acquisition of additional scans with adjusted collimation and / or positioning and / or orientation of parts of the medical imaging device 100. In embodiments, the medical imaging device 100 is an X-ray imaging device or a fluoroscopy device. Especially in X-ray applications, it can be desirable to reduce the radiation exposure to the patient. Therefore, the field of view of the imaging device 100 can advantageously be limited to a very small imaging region to avoid overexposure. To adjust the field of view, a collimator 170 can be used to spatially limit and confine the radiation beam.

[0039] For X-ray or fluoroscopy systems, different options can be utilized, like systems designed to collimate symmetrically around a central X-ray beam, or systems comprising a flexible collimator configuration.

[0040] Therefore, in embodiments, the generated data 160 for instructing the medical imaging device 100 to acquire at least one second medical image 130 comprises instructions for changing the field of view of the medical imaging device 100 by changing the position and / or orientation of the radiation source and / or radiation detector of the medical imaging device 100. Since many X-ray systems are designed to collimate symmetrically around a central X-ray beam, the configuration of the imaging system 100, e.g. tube and detector position, can be automatically adjusted based on the need for acquiring one or more additional images. Therefore, it can be proposed to use an automated imaging system which automatically adjusts the configuration of the system with respect to tube and detector position based on the results of the analysis of the already needed image data, like the first medical image 120.

[0041] Additionally or alternatively, in embodiments, the generated data 160 for instructing the medical imaging device 100 to acquire at least one second medical image 120 comprises instructions for changing the field of view of the medical imaging device 100 by changing the configuration of the collimator 170 of the medical imaging device 100. Therefore, X-ray systems with a flexible collimator configuration can be utilized, wherein the system geometry of the X-ray tube and detector can optionally remain unchanged, while the flexible collimator 170 allows for acquiring specific regions of interest with the medical imaging device 100. As in practical applications, the utilized field of view of the imaging device 100 can be the result of the position of the source, detector and collimator 170 in combination with the patient 110, it is to be understood that none of the previously defined parameters defines the field of view alone.

[0042] In embodiments, the medical imaging system 100 comprises a multi-leaf collimator as collimator 170. A multi-leaf collimator can be understood as a beam limiting device comprising a plurality of individual leaves of radiation absorbing material, which can be independently moved into and out of the path of the radiation beam in order to shape and / or change the intensity of the radiation beam.

[0043] In embodiments, the generated data 160 comprises a position, size and / or shape of the second field of view 131. Especially in case of a multi-leaf collimator, the field of view can be very complex and not limited to e.g. a rectangular shape. Moreover, depending on the determined at least one second field of view, it can be necessary to adjust the source, the detector and / or the collimator in order to acquire a second image.

[0044] In embodiments, the step of analyzing the first medical image 120 and / or determining at least one second field of view 131 of the medical imaging device 100 is performed by a trained artificial intelligence. Thus, a recommender engine can be used, which predicts the need and parameters of a subsequent image acquisition. For this purpose, different artificial intelligence (AI) algorithms can be considered. It is assumed that all of them generate at least two different types of output. A first output indicates whether an additional acquisition needs to be acquired in order to image the complete region of interest of the patient. A second output defines the system configuration for the additional acquisition. This can include the configuration of the collimator 170 as well as other information about the system geometry, e.g. tube and detector position. For the training and evaluation of the artificial intelligence algorithm, the process of sequential acquisition can be simulated by extracting parts from existing acquisitions to simulate incomplete images of the region of interest.

[0045] In embodiments, the artificial intelligence comprises a regressor network or a region proposal network. For training and evaluation, potential embodiments can include a regressor network, a region proposal network or conventional computer vision and machine learning techniques. A regressor network can be understood as an AI system with a deep learning backbone comprising a convolutional neural network, which can be extended by fully connected layers to predict different outputs given an input image. A region proposal network can be used in computer vision applications to detect objects. Thus, different image regions are evaluated in a similar sliding window fashion, while a proposal layer predicts candidate positions, which can later be integrated into individual predictions. Similar to a regressor network, this concept can be extended to predict multiple candidate configurations for a subsequent acquisition, which are then integrated. The use of a combination of conventional computer vision and machine learning techniques can result in a solution similar to the regressor network approach. Specifically, a combination of image processing features can be evaluated on sub-regions, after which a machine learning technique like support vector regression or decision trees is applied.

[0046] In embodiments, the region of interest comprises an anatomical structure, a spread of contrast agent, a clinical finding and / or an inserted device within the subject. Based on the initial image, the proposed method and system can be used to track the anatomical structure or device with minimal dose exposure to surrounding tissue.

[0047] In embodiments, the first field of view 121 of the first image 120 is derived from a camera image or an X-ray pre-shot of the subject 110. Thus, the initial acquisition of the first image 120 can be guided by a (RGBD) camera system or similar sensor or an X-ray pre-shot.

[0048] In embodiments, the first medical image of the subject comprises a plurality of first medical images of the subject taken at successive points in time, and the step of determining at least one second field of view of the medical imaging device comprises analyzing a temporal change of the region of interest in the plurality of first medical images of the subject.

[0049] Thus, in a dynamic environment such as fluoroscopy, and in order to predict good control parameters for the medical imaging system to acquire at least one second medical image, additional images from previous points in time, e.g. a time series of images, can be provided and used in order to facilitate, e.g., a speed estimation of a contrast agent.

[0050] In embodiments, the method comprises a step of proposing to a user an acquisition of at least one second medical image 130 with at least one second field of view 131. Additionally or alternatively, in case of a detected incidental finding, the proposed method and system can automatically propose to adjust the acquired field of view. In case of a fluoroscopy acquisition, the system can propose an additional scan region, or it can gradually increase the region, allowing a cinematic view over the entire region of interest.

[0051] Reference is made to Figure 2 , and to Figure 1 , Figure 2The processing unit 200 and the schematic setup of the medical imaging system 100 showing the steps of performing a method according to an embodiment of the present application are shown in Fig. 1. The medical imaging device 100 with a first setup of the collimator 170 is used to image at least a portion of the object 110 resulting in a first field of view 121 of the medical imaging device 100. The obtained first medical image 120 covers at least a portion of the region of interest 140 of the object 110. The processing unit 200 configured to perform the steps of the method according to an embodiment of the present application analyzes the first medical image 120 and decides whether the region of interest 140 is completely depicted in the first medical image 120. In case the region of interest 140 is not completely depicted in the first medical image 120, data 150 is generated to instruct the medical imaging device 100 to acquire at least a second medical image 130 with a second field of view 131 of the object 110 having the region of interest 140. The second field of view 131 can depend on the setup of the collimator 170. The second medical image 130 is combined with the first medical image 120, for example by stitching, to generate a combined medical image 150 completely containing the region of interest 140.

[0052] After acquiring one or more additional images, the second medical image 130, with the automated medical imaging system 100 according to the field of view determined by analyzing the first medical image 120, the one or more second medical images 130 are combined with the first medical image 120 into a single combined medical image 150. For example, by stitching at least one second image with the first image, an image advantageously containing the complete region of interest of the object 110 can be generated.

[0053] Thus, given an initial x-ray or fluoroscopy image, the method of the present application can be used to predict the location of an additional image acquisition of a second medical image 130 in addition to the existing image data of the first medical image 120. This additional image acquisition can be outside the existing image data and / or detector limits and can be acquired, for example, by changing the collimator configuration and / or by changing the setup of the x-ray tube and the x-ray detector.

[0054] Preferably, the first image 120 and the second image 130 can each comprise a plurality of images. In particular, when using a time series as input, the first medical image 120 can comprise a plurality of input images. Each of these images can be analyzed with respect to the contained imaging area and for each of the plurality of images one or more second images 120 can be acquired to correct the desired imaging area or field of view.

[0055] In particular, for a tracking application, the first acquired image can not contain the region of interest and multiple iterations of the method of the present application can be necessary in order to cover the region of interest.

[0056] According to another aspect of the application, there is provided a data processing apparatus 200 comprising a processor configured to perform the steps of the method according to any of the preceding embodiments.

[0057] According to another aspect of the application, there is provided a medical imaging system 100 comprising a data processing apparatus 200 according to the preceding embodiment.

[0058] According to another aspect of the application, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the steps of the method according to any of the preceding embodiments. The computer program can be executed on one or more processing units instructed to carry out the method according to any of the preceding embodiments.

[0059] According to another aspect of the application, there is provided a computer readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the steps of the method according to any of the preceding embodiments.

[0060] While the application has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. The application is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practising the claimed application, from a study of the drawings, the disclosure, and the dependent claims.

[0061] In the claims, the word "comprising" does not exclude other elements or steps, and the word "a" or "an" does not exclude a plurality. In the claims the term "comprising" does not exclude other elements or steps than those listed in the claim. In the claims the term "a" or "an" preceding the commencement of a clause with a plurality of elements does not exclude a larger number of such elements. In the claims the term "one" before the commencement of a clause with a plurality of elements does not exclude a larger number of such elements. In the claims the term "first", "second", "third", "fourth", "fifth" or "sixth" preceding the commencement of a clause with a plurality of elements does not exclude a larger number of such elements. Neither in the application nor in the any of the claims are used the terms first, second and third, one, two, three and the like solely to define the number of steps or elements in a claim; the use of these terms is solely for defining the nature of the first, second and third steps or elements with regard to each other. Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. A computer-implemented method for acquiring and combining medical images of an object (110), the method comprising the following steps: Receive a first medical image (120) of an object (110) acquired by a medical imaging device (100) in a first field of view (121), wherein the first medical image (120) of the object (110) includes multiple first medical images of the object (110) taken at successive time points; The first medical image (120) is analyzed regarding the inclusion of the region of interest (140) of the object (110) in the first medical image (120) and the temporal variation of the region of interest in the multiple first medical images of the object (110). Determine at least one second field of view (131) of the medical imaging device (100). Data (160) is generated to instruct the medical imaging device (100) to acquire at least one second medical image (130) of the object (110) with the at least one second field of view (131), the at least one second medical image (130) containing at least a portion of the region of interest (140) of the object (110) that was not included in the first medical image (120); The generated data (160) is sent to the medical imaging device (100). Receive the at least one second medical image (130) of the object (110) acquired by the medical imaging device (110) with the at least one second field of view (131). The first medical image (120) and the at least one second medical image (130) are combined into a combined medical image (150), the combined medical image (150) containing the region of interest (140) of the object (110); and Provide medical images (150) of the combination.

2. The method according to claim 1, wherein, The medical imaging device (100) is an X-ray imaging device or a fluorescence fluoroscopy device.

3. The method according to any one of claims 1 or 2, wherein, The generated data (160) for instructing the medical imaging device (100) to acquire at least one second medical image (130) when the first field of view (121) is different from the second field of view (131) includes: instructions for changing the field of view of the medical imaging device (100) by changing the position and / or orientation of the radiation source and / or radiation detector of the medical imaging device (100).

4. The method according to any one of the preceding claims, wherein, The generated data (160) for instructing the medical imaging device (100) to acquire at least one second medical image (130) when the first field of view (121) is different from the second field of view (131) includes: instructions for changing the field of view of the medical imaging device (100) by changing the configuration of the collimator (170) of the medical imaging device.

5. The method according to claim 4, wherein, The medical imaging system (100) includes a multi-leaf collimator (170).

6. The method according to any one of the preceding claims, wherein, The generated data (160) includes the position, size and / or shape of the at least one second field of view (131).

7. The method according to any one of the preceding claims, wherein, The steps of analyzing the first medical image (120) and / or determining at least one second field of view (131) of the medical imaging device (100) are performed by trained artificial intelligence.

8. The method according to claim 7, wherein, The artificial intelligence includes regressor networks or region proposal networks.

9. The method according to any one of the preceding claims, wherein, The region of interest (140) includes: anatomical structures, diffusion of contrast agents, clinical findings, or devices inserted into the object (110).

10. The method according to any one of the preceding claims, wherein, The method includes the following steps: proposing to the user to acquire the at least one second medical image (130) with the at least one second field of view (131).

11. A data processing apparatus (200) comprising a module for performing the steps of the method according to any one of claims 1 to 10.

12. A medical imaging system comprising the data processing device (200) according to claim 11.

13. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 10.

14. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 10.

Images