Scoring of X-ray projection images
Patent Information
- Application Number
- JP2024525037
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-10-27
- Filing Date
- 2022-10-21
- Publication Date
- 2024-11-13
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing methods for determining the optimal viewpoint of X-ray projection images, such as those used in medical procedures like aneurysm coil embolization and spinal fusion, rely on a trial-and-error approach, which can increase X-ray dose and result in suboptimal images, and manual analysis of multiple images is time-consuming.
A computer-implemented method using neural networks to predict an image viewpoint score value for X-ray projection images, combining subjective and objective assessments to evaluate the quality of the image perspective, allowing for efficient selection and archiving of high-quality images.
The method provides a systematic and efficient way to determine the optimal viewpoint for X-ray projection images, reducing X-ray dose and minimizing manual analysis time while ensuring high-quality image capture and archiving.
Smart Images

Figure 2024541941000001_ABST
Abstract
Description
[Technical field]
[0001] SUMMARY A computer-implemented method, system, and computer program product are disclosed that relate to determining an image viewpoint score value for an x-ray projection image. [Background technology]
[0002] Medical examinations often use X-ray projection images to provide information about a region of interest. Compared to 3D X-ray images, projection images, i.e., 2D images, can be generated with a smaller X-ray dose. However, projection images are subject to the integration of X-ray attenuation along the X-ray path, and it can be difficult to find a viewpoint of the X-ray imaging system relative to the region of interest, i.e., a viewing angle that provides the desired information in the projection image.
[0003] As an example, an aneurysm coil embolization procedure typically involves inserting a wire coil into the aneurysm's sac to reduce blood flow and thereby allow blood within the aneurysm to clot. The goal of aneurysm coil embolization is to fill the aneurysm's sac with enough coil to effectively treat the aneurysm while avoiding overfilling, which may cause the aneurysm to rupture or allow the coil material to flow into the parent vessel that is connected to the aneurysm's sac. An aneurysm coil embolization procedure is typically performed using x-ray projection images. To detect the flow of coil material into the parent vessel, the radiologist typically attempts to find a perspective of the x-ray imaging system that minimizes overlap of the sac with the parent vessel in the projection images. The radiologist also attempts to find a perspective of the x-ray imaging system that minimizes overlap of the parent vessel with other anatomical structures and also minimizes foreshortening of the parent vessel.
[0004] The need to find the optimal viewpoint of an X-ray imaging system to generate projection images also exists beyond vascular procedures. As another example, spinal fusion procedures performed to restore stability in severe arthritis or after removal of a herniated disc often involve the placement of pedicle screws in the vertebrae. To ensure that the screws are correctly placed, a viewpoint of the X-ray imaging system is desired that not only minimizes foreshortening of the screw, but also minimizes overlap of the screw with the spinal cord to ensure that spinal cord damage does not occur. Yet another example can be found in the field of orthopedics, where a viewpoint of the X-ray imaging system is desired that minimizes foreshortening of the target bone structure while minimizing its overlap with other anatomical structures. Therefore, it may be useful to evaluate the viewpoint of the X-ray projection image to provide an optimal viewpoint of the X-ray imaging system.
[0005] Because there are many factors that influence the selection of a viewpoint of an x-ray imaging system for producing an x-ray projection image, a trial-and-error approach is often used, where an initial promising viewpoint of the x-ray imaging system is adjusted until an image with an acceptable viewpoint is obtained. However, this can increase the x-ray dose to the subject and ultimately provide an image with only a marginally improved viewpoint.
[0006] An improvement over the trial-and-error approach to finding the optimal viewpoint for an x-ray imaging system is the analytical approach described in DL Wilson et al., "Determining X-ray projections for coil treatments of intracranial aneurysms," IEEE Transactions on Medical Imaging, Vol. 18, No. 10, pp. 973-980, October 1999, doi:10.1109 / 42.811309, which presents a method to automatically find angles that produce the desired x-ray projections. Summary of the Invention [Problem to be solved by the invention]
[0007] Besides finding the optimal viewpoint for an X-ray imaging system, it may also be useful to evaluate the viewpoint of X-ray projection images in other situations. As an example, medical examinations often require the collection of multiple X-ray projection images of an area of interest. After the medical examination, many of the collected images are of little use because the information they provide is duplicated in other images. To limit data storage requirements, it is beneficial to archive only images with viewpoints that meet predefined criteria. However, manual analysis of these images is time-consuming.
[0008] Therefore, there is room for improvement in the assessment of viewpoints in X-ray projection images. [Means for solving the problem]
[0009] According to one aspect of the present disclosure, there is provided a computer-implemented method for determining an image viewpoint score value for an x-ray projection image representing a region of interest within a subject, the method comprising: receiving a plurality of x-ray projection images, the x-ray projection images representing a region of interest from a plurality of different viewpoints of an x-ray imaging system relative to the region of interest; inputting the X-ray projection images into a neural network; generating a predicted image viewpoint score value for each of the x-ray projection images in response to the inputting step; wherein the neural network is trained to generate predicted image viewpoint score values for the x-ray projection images.
[0010] The predicted image viewpoint score value provides an indication of the quality of the viewpoint of the X-Ray image. The predicted image viewpoint score value can be used for various purposes, such as to determine which input X-Ray projection images provide acceptable viewpoints and therefore which viewpoints of the X-Ray imaging system to use to acquire further X-Ray images. The predicted image viewpoint score value can also be used to determine which input X-Ray images to archive.
[0011] Further aspects, features, and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments which proceeds with reference to the accompanying drawings. [Brief description of the drawings]
[0012] [Figure 1] FIG. 1 is a flow chart illustrating an example of a method for determining an image viewpoint score value for an X-ray projection image according to some aspects of the present disclosure. [Diagram 2] FIG. 2 is a schematic diagram illustrating an example of a system 100 for determining an image viewpoint score value of an X-ray projection image in accordance with some aspects of the present disclosure. [Diagram 3] FIG. 3 is a schematic diagram illustrating an example of a method for determining an image viewpoint score value s1 of an X-ray projection image using a neural network NN1 according to some aspects of the present disclosure. [Figure 4] FIG. 4 is a schematic diagram illustrating an example of a method for determining a combined image viewpoint score value s′ for an X-ray projection image according to some aspects of the present disclosure. [Figure 5A] FIG. 5A is a schematic diagram showing the perspective of a projection X-ray imaging system 130 relative to a region of interest 120 within a subject 220 (including the rotation angle a of the central ray of the projection X-ray imaging system 130 around the subject's longitudinal axis) in accordance with some aspects of the present disclosure. [Figure 5B]FIG. 5B is a schematic diagram showing the viewpoint of the projection X-ray imaging system 130 relative to the region of interest 120 within the subject 220 (including the inclination angle b of the central ray of the projection X-ray imaging system 130 relative to the skull-base axis of the subject 220) in accordance with some aspects of the present disclosure. [Figure 6] FIG. 6 is a flow chart illustrating an example of a method for training a second neural network NN2 to determine analytical image viewpoint score values for X-ray projection images according to some aspects of the present disclosure. [Figure 7] FIG. 7 is a schematic diagram illustrating an example of a method for training a second neural network NN2 to predict values of weights l1...n of metrics C1...n used to calculate an analytical image viewpoint score value s2 for an X-ray projection image, according to some aspects of the present disclosure. [Figure 8] FIG. 8 is a flow chart illustrating a first example of a method for training a neural network NN1 to determine an image viewpoint score value of an X-ray projection image according to some aspects of the present disclosure. [Figure 9] FIG. 9 is a schematic diagram illustrating a first example of a method for training a neural network NN1 to determine an image viewpoint score value of an X-ray projection image according to some aspects of the present disclosure. [Figure 10] FIG. 10 is a flow chart illustrating a second example of a method for training the neural network NN1 to determine image viewpoint score values of X-ray projection images in accordance with some aspects of the present disclosure. [Figure 11] FIG. 11 is a schematic diagram illustrating a second example of a method for training a neural network NN1 to determine an image viewpoint score value of an X-ray projection image according to some aspects of the present disclosure. [Figure 12] FIG. 12 is a schematic diagram illustrating an example of output image viewpoint score values according to some aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0013] With reference to the following description and figures, examples of the present disclosure are provided. In the following description, for purposes of explanation, many specific details of some examples are set forth. References herein to "examples," "implementations," or similar terms mean that features, structures, or characteristics described in connection with an example are included in at least one of the examples. It is further understood that features described in connection with one example may also be used in other examples, and that not all features are necessarily duplicated in each example for the sake of brevity. For example, features described in connection with a computer-implemented method may be implemented in a corresponding manner in computer program products and systems.
[0014] In the following description, reference is made to an example of a computer-implemented method that involves determining an image viewpoint score value for an X-ray image that represents a region of interest in a subject. Reference is made to an example of an X-ray projection image in the form of an angiogram image, where the region of interest is an aneurysm in the brain. The angiogram image may be generated using an angiography technique such as digital subtraction angiography (DSA). Angiogram images, i.e. images generated using a contrast agent, are typically used to visualize regions of interest in the vasculature, such as aneurysms. However, it is understood that these angiogram images are merely examples, and that the computer-implemented method disclosed herein may also be used with other types of X-ray projection images, such as fluoroscopic images, and images that represent regions of interest other than the vasculature. Thus, it is understood that the computer-implemented method may be used to determine an image viewpoint score value for X-ray projection images in general, and that the use of the method is not limited to X-ray images containing aneurysms or images of the brain.
[0015] It should be noted that the computer-implemented methods disclosed herein may be provided as a non-transitory computer-readable storage medium having computer-readable instructions stored thereon. The computer-readable instructions, when executed by at least one processor, cause the at least one processor to perform the method. That is, the computer-implemented methods may be implemented in a computer program product. The computer program product may be provided by dedicated hardware or dedicated hardware capable of executing software in association with appropriate software. When provided by a processor, the functionality of the method features may be provided by a single dedicated processor, a single shared processor, or multiple individual processors (some of which may be shared). For example, one or more functions of the method features may be provided by a processor shared in a networked processing architecture, such as a client / server architecture, a peer-to-peer architecture, the Internet, or a cloud.
[0016] Explicit use of the terms "processor" or "controller" should not be construed as referring solely to hardware capable of executing software, but implicitly includes, but is not limited to, digital signal processor (DSP) hardware, read only memory (ROMA) for storing software, random access memory (RAM), non-volatile storage devices, and the like. Furthermore, embodiments of the present disclosure may take the form of a computer program product accessible from a computer usable or computer readable storage medium, the computer program product providing program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer usable or computer readable storage medium may be any apparatus that includes the storage, communication, propagation, or transport of a program for use by or in connection with an instruction execution system, apparatus, or device. Media include electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or propagation media. Examples of computer readable media include semiconductor or solid state memory, magnetic tape, removable computer disks, random access memory (RAM), read only memory (ROM), rigid magnetic disks, and optical disks. Current examples of optical disks include compact disk-read only memory (CD-ROM), compact disk-read / write (CD-R / W), Blue-Ray, and DVD.
[0017] As mentioned previously, it is useful to evaluate the perspective of x-ray projection images in various situations.
[0018] One example of such a situation involves an aneurysm coil embolization procedure. The goal of aneurysm coil embolization is to fill the aneurysm pouch with enough coil to effectively treat the aneurysm while avoiding overfilling. To detect the inflow of coil material into the parent vessel, the radiologist typically attempts to find a viewpoint of the x-ray imaging system that minimizes overlap of the pouch with the parent vessel in the projection image. The radiologist also attempts to find a viewpoint of the x-ray imaging system that minimizes overlap of the parent vessel with other anatomical structures and also minimizes foreshortening of the parent vessel. It is desirable to minimize overlap of the parent vessel with other anatomical structures in order to ensure a clear view of the parent vessel. It is desirable to minimize foreshortening because if the longitudinal axis of the coil wire were aligned with the viewing angle, it would be possible to miss a large portion of the wire in the vessel.
[0019] Another situation in which it is useful to evaluate the perspective of X-ray projection images relates to image archiving. Medical examinations often require the collection of multiple X-ray projection images of an area of interest. After the medical examination, many of the collected images are of little use because the information they provide is duplicated in other images. To limit data storage requirements, it is beneficial to archive only those images that provide the desired perspective of the area of interest. However, manual analysis of these images is time-consuming.
[0020] There are other situations in which it is useful to evaluate the perspective of x-ray projection images.
[0021] Fig. 1 is a flow chart illustrating an example of a method for determining an image viewpoint score value of an X-ray projection image according to some aspects of the present disclosure. Fig. 2 is a schematic diagram illustrating an example of a system 100 for determining an image viewpoint score value of an X-ray projection image according to some aspects of the present disclosure. Operations described in connection with the method illustrated in Fig. 1 are also performed by the system 100 illustrated in Fig. 2. Similarly, operations described in connection with the system 100 are also performed in the method described with reference to Fig. 1.
[0022] 1, a computer-implemented method for determining an image viewpoint score value s1 for an X-ray projection image 110 representing a region of interest 120 within a subject is provided. The method comprises: Receiving (S110) a plurality of X-ray projection images 110, the X-ray projection images representing a region of interest 120 from a plurality of different viewpoints of an X-ray imaging system 130 relative to the region of interest; Inputting the X-ray projection image into a neural network NN1 (S120); generating (S130) a predicted image viewpoint score value s1 for each of the x-ray projection images in response to the input; where the neural network NN1 is trained to generate predicted image viewpoint score values s1 for the X-ray projection images.
[0023] The above method is also described with reference to Fig. 3. Fig. 3 is a schematic diagram illustrating an example of a method for determining an image viewpoint score value s1 of an X-ray image using a neural network NN1 according to some aspects of the present disclosure.
[0024] 1-3, in operation S110, a plurality of X-ray projection images 110 are received. The X-ray projection images 110 are generated by an X-ray imaging system. The X-ray projection images 110 may be generated by a projection X-ray imaging system that generates 2D X-ray images. A projection X-ray imaging system typically includes a support arm, such as a so-called "C-arm" or "O-arm", that supports an X-ray source-detector arrangement. Alternatively, a projection X-ray imaging system may include a support arm of a different shape than these examples. A projection X-ray imaging system typically generates the projection X-ray images using a support arm that is held in a static position relative to an imaging region during collection of image data. The X-ray projection images may be generated, for example, by a projection X-ray imaging system 130 shown in FIG. 2. As an example, an angiographic image may be generated by a Philips Azurion7 X-ray imaging system commercially available from Philips Healthcare Best (Netherlands).
[0025] The X-ray projection image 110 received in operation S110 may be an angiographic image or, for example, a fluoroscopic image, i.e., a live image. The angiographic projection image may be generated using a digital subtraction angiography (DSA) technique, in which each image is generated by subtracting from the image the corresponding pixel intensity of a background image. Alternatively, the X-ray projection image 110 received in operation S110 may be a synthetic projection image generated by projecting a 3D X-ray image (such as a 3D angiogram) generated by a volumetric X-ray imaging system at different viewpoints. In contrast to a projection X-ray imaging system, a volumetric X-ray imaging system typically generates image data while rotating or stepping an X-ray source-detector arrangement around an imaging region, and then reconstructs the image data acquired from multiple rotation angles into a 3D image, i.e., a volumetric image. Examples of volumetric X-ray imaging systems include a computed tomography (CT) imaging system, a cone-beam CT (CBCT) imaging system, and a spectral CT imaging system. 3D angiograms can be generated from volumetric imaging systems by performing imaging operations using a projection agent. Synthetic projection images can be generated from 3D x-ray images by projecting the 3D x-ray image onto a virtual detector based on the perspective of the x-ray source-detector arrangement relative to the 3D x-ray image.
[0026] The X-ray projection image 110 received in operation S110 is received from an X-ray imaging system, such as the projection X-ray imaging system 130 shown in FIG. 2, a computer readable storage medium, the Internet, the cloud, or the like. The X-ray projection image is received by one or more processors 210 shown in FIG. 2. The X-ray projection image is received via any form of data communication, such as wired communication, optical communication, and wireless communication. As some examples, when using wired or optical communication, the communication is via signals transmitted through electrical or optical cables, and when using wireless communication, the communication is via, for example, RF signals or optical signals.
[0027] The x-ray projection images 110 received in operation S110 represent the region of interest 120 from a number of different viewpoints, or orientations, of an x-ray imaging system relative to the region of interest. In this regard, the x-ray projection images are generated by adjusting the viewpoint, or orientation, of a projection x-ray imaging system 130, shown in Figure 2, to different viewpoints, or orientations, and generating an x-ray projection image at each viewpoint, or orientation. The adjustments are performed by stepping the viewpoint of the x-ray imaging system about the subject's cranio-distal axis and / or about the subject's longitudinal axis, for example.
[0028] In operation S120, the X-ray projection image is input to a neural network NN1. The neural network NN1 is trained to generate a predicted image viewpoint score value s1 for the X-ray projection image. The predicted image viewpoint score value s1 provides a subjective assessment of the quality of the viewpoint of the X-ray projection image. The subjective assessment depends on the ground truth training data used to train the neural network NN1. A high quality viewpoint is associated with an X-ray projection image, for example, in which the region of interest is separated from other features in the image. A high quality viewpoint is also associated with an X-ray projection image in which there is a small amount of foreshortening of a portion of the region of interest. A high quality viewpoint is also associated with an X-ray projection image in which there is little artifact from confusing features. The extent to which these factors affect the predicted image viewpoint score value s1 depends on the ground truth training data used to train the neural network NN1, and thus the predicted image viewpoint score value s1 provides a subjective assessment of the quality of the viewpoint of the X-ray projection image. Further details regarding the training of the neural network NN1 are provided below.
[0029] In operation S130, in response to inputting the X-ray projection images to the neural network NN1, a predicted image viewpoint score value s1 for each X-ray projection image is generated. The predicted image viewpoint score value s1 is output in various formats. In one example, the image viewpoint score value s1 is output by a graph for each X-ray projection image. An example of such a graph output is shown in the right part of FIG. 3. Another example of the output image viewpoint score value is shown in FIG. 12. FIG. 12 is a schematic diagram showing an example of the output image viewpoint score value according to some aspects of the present disclosure. In the example shown in FIG. 12, the image viewpoint score is output together with the corresponding X-ray projection image. A user can immediately know which image has the highest image viewpoint score value from the graphs shown in FIG. 3 and FIG. 12. Alternatively, the predicted image viewpoint score value s1 may be output in other formats, such as providing a numerical or graphic indicator within each image.
[0030] Thus, the predicted image viewpoint score value s1 provides a subjective assessment of the quality of the viewpoint of the X-ray projection image. The predicted image viewpoint score value s1 can be used for various purposes. For example, it can be used to determine which input images have acceptable viewpoints and therefore which viewpoints of the X-ray imaging system to use to generate further X-ray projection images. The predicted image viewpoint score value s1 can also be used to select which input X-ray projection images to archive. In one example, the method described with reference to FIG. 1 includes identifying the X-ray projection image and / or corresponding viewpoint of the X-ray imaging system for the region of interest that has the highest predicted image viewpoint score value.
[0031] The neural network NN1 may also generate a confidence value u1 for each of the predicted image viewpoint score values s1. In one example, the method described with reference to FIG. 1 also includes outputting the confidence value u1. The confidence value is output similarly to the image viewpoint score values s1. The confidence value represents the reliability of the prediction made by the neural network NN1 and allows a decision to be made based on the viewpoint score values s1. For example, if the reliability is low, it may be determined not to depend on the predicted image viewpoint score values s1. In one example, the confidence value of the predicted image viewpoint score values s1 may be used to weight the predicted image viewpoint score values s1.
[0032] An example of a technique for generating a confidence value associated with a prediction of a neural network is disclosed in a document entitled "Density eatimation in representation space to predict model uncertainty" by Ramalho, T et al. (https: / / arxiv.org / pdf / 1908.07235.pdf). A neural network NN1 is trained according to this technique to generate a confidence value. This allows the neural network NN1 to recognize and output a low confidence value when it is presented with an image that is very different from the training data set. The technique described in this document generates a confidence value by estimating the training data density in the representation space and determining whether the trained network is expected to make a correct prediction for the input by measuring the distance in the representation space between the input and its nearest neighbor in the training set. Another technique can also be used to generate a confidence value associated with the prediction of the neural network NN1. For example, a dropout technique can be used. The dropout technique involves repeatedly inputting the same data to the neural network and randomly eliminating a portion of the neurons from the neural network at each iteration while determining the output of the neural network. The output of the neural network is then analyzed to provide a mean and a variance, where the mean represents the final output and the magnitude of the variance indicates whether the neural network is consistent (in which case the variance is small) or inconsistent (in which case the variance is large) in its predictions.
[0033] In one embodiment, an analytical image viewpoint score value s2 is generated for each of the received X-ray projection images. The analytical image viewpoint score value s2 provides an objective assessment of the quality of the viewpoint of each X-ray projection image. In this embodiment, the analytical image viewpoint score value s2 is combined with the predicted image viewpoint score value s1 to provide a combined image viewpoint score value for each X-ray projection image. This embodiment is described with reference to FIG. 4. FIG. 4 is a schematic diagram illustrating an embodiment of a method for determining a combined image viewpoint score value s′ for X-ray projection images according to some aspects of the present disclosure.
[0034] Referring to FIG. 4, in this embodiment, the X-ray projection images represent the region of interest within the subject from corresponding viewpoints a, b of the X-ray imaging system relative to the region of interest, and the method described above with reference to FIG. receiving a 3D X-ray image 140 representative of the region of interest 120; For each received X-ray projection image 110 aligning a region of interest in the 3D X-ray image 140 with a region of interest in the X-ray projection images to provide viewpoints a, b of the X-ray imaging system 130 relative to the region of interest in the 3D X-ray image 140; Calculating an analytical image viewpoint score value s2 from the 3D X-ray image 140 based on viewpoints a, b of the X-ray imaging system 130 relative to the region of interest in the 3D X-ray image 140, the analytical image viewpoint score value s2 being calculated based on the following metric C 1…n calculating, from the 3D X-ray image 140, based on one or more of: a degree of overlap between a plurality of features within the region of interest, a foreshortening of one or more features within the region of interest, and a presence of one or more artifacts within the region of interest; combining the analytical image viewpoint score value s2 with the predicted image viewpoint score value s1 to provide a combined image viewpoint score value s' for the X-ray projection image.
[0035] In contrast to the predicted image viewpoint score value s1, which provides a subjective assessment of the viewpoint quality of the X-ray projection image, the analytical image viewpoint score value s2 provides an objective assessment of the viewpoint quality of the X-ray projection image. By providing a combined image viewpoint score value s' that depends on both the analytical image viewpoint score value s2 and the predicted image viewpoint score value s1, the combined image viewpoint score value s' provides an assessment in which the subjective bias of the predicted image viewpoint score value s1 introduced via the training of the neural network NN1 is compensated for by the objective analytical image viewpoint score value s2.
[0036] In this example, the X-ray projection images represent the region of interest from corresponding viewpoints a, b of the X-ray imaging system relative to the region of interest in the subject. The viewpoints are defined, for example, by a rotation angle a of the central ray of the X-ray imaging system about the longitudinal axis of the subject and an inclination angle b of the central ray of the X-ray imaging system relative to the cranio-cranial axis of the subject 220. An example of such viewpoints is shown in FIG. 5A and FIG. 5B. FIG. 5A is a schematic diagram illustrating a viewpoint of the projection X-ray imaging system 130 relative to the region of interest 120 in the subject 220 (including a rotation angle a of the central ray of the projection X-ray imaging system 130 about the longitudinal axis of the subject) in accordance with some aspects of the present disclosure. FIG. 5B is a schematic diagram illustrating a viewpoint of the projection X-ray imaging system 130 relative to the region of interest 120 in the subject 220 (including an inclination angle b of the central ray of the projection X-ray imaging system 130 relative to the cranio-cranial axis of the subject 220) in accordance with some aspects of the present disclosure.
[0037] In this example, a 3D X-ray image 140 is received that represents the region of interest 120. The 3D X-ray image may be generated using a volumetric imaging system. As described above for the X-ray projection images, the 3D X-ray image may be a 3D angiographic image. The 3D angiographic image may be generated using digital subtraction angiography techniques, for example.
[0038] In this example, the 3D X-ray image is received by one or more processors 210 shown in Figure 1. The 3D X-ray image is typically received from a volumetric X-ray imaging system, a computer readable storage medium, the internet, the cloud, etc. The 3D X-ray image 140 is received via any form of data communication, as described above for the X-ray projection image 110.
[0039] In this embodiment, the region of interest in the 3D X-ray image 140 is aligned with the region of interest in each X-ray projection image to provide viewpoints a, b of the X-ray imaging system relative to the region of interest in the 3D X-ray image 140. This alignment is performed based on known viewpoints relative to the subject of each of the volumetric imaging system generating the 3D X-ray image 140 and the projection X-ray imaging system generating the X-ray projection image 110. Alternatively, this alignment may be performed using an image matching technique, as known from the field of image alignment, in which synthetic viewpoints of the 3D X-ray image 140 at different viewpoints of the X-ray imaging system relative to the 3D X-ray image are generated and compared with the X-ray projection images until a viewpoint providing a matching image is found. In the former case, the viewpoint of the volumetric imaging system generating the 3D X-ray image 140 relative to the subject is usually known, since when the 3D X-ray image 140 is reconstructed, it is reconstructed relative to the orientation of the volumetric imaging system. During the generation of the 3D X-ray images, the subject lies on the patient table, and the orientation of the patient bed relative to the volumetric imaging system is known, so the orientation of the subject relative to the volumetric imaging system is also known. Similarly, when the X-ray projection images are generated, the viewpoint of the projection X-ray imaging system generating the X-ray projection images 110 relative to the subject is known, because again the patient lies on the patient bed. Also, with respect to the above viewpoint parameters a and b, the viewpoint of the X-ray imaging system relative to the patient bed is usually recorded for each X-ray projection image. Therefore, the registration performed in this operation can be performed by matching the orientation of the subject in the 3D X-ray images with the orientation of the subject in each X-ray projection image.
[0040] In this example, an analytical image viewpoint score value s2 for each X-ray projection image is calculated from the 3D X-ray image 140 and based on the X-ray imaging system viewpoints a, b relative to the region of interest in the 3D X-ray image 140. The X-ray imaging system viewpoints a, b relative to the region of interest in the 3D X-ray image 140 are determined using the above registration. This is illustrated in FIG. 4 by inputting the X-ray imaging system viewpoints a, b for the X-ray projection images into an analytical scoring module (ASM). The ASM uses this input viewpoint to calculate an analytical image viewpoint score value s2 from the 3D X-ray image 140 to provide the X-ray imaging system viewpoints a, b relative to the region of interest in the 3D X-ray image 140 via the registration. The functionality of the ASM may be provided by one or more processors, such as the processor illustrated in FIG. 2.
[0041] The analytical image viewpoint score value s2 is calculated using the following metric C 1…n : Calculated from the 3D X-ray image based on one or more of the following values: degree of overlap between multiple features within the region of interest, foreshortening of one or more features within the region of interest, and presence of one or more artifacts within the region of interest.
[0042] Techniques for computing such analytical image viewpoint scores are described in DL Wilson et al., "Determining X-ray projections for coil treatment of intracranial aneurysms," IEEE Transactions on Medical Imaging, Vol. 18, No. 10, pp. 973-980, October 1999, doi:10.1109 / 42.811309. In this document, a method is presented to automatically find angles that produce the desired X-ray projections. An algorithm is presented to compute factors such as overlap and foreshortening mentioned above. The algorithm uses as inputs i) a discretely defined 3D segmentation of the region of interest and surrounding anatomical structures, and ii) a model of the geometric and imaging parameters of the X-ray imaging system.
[0043] An example of the calculation of an analytical image viewpoint score value s2 for a 3D x-ray image including a region of interest 120 that is a cerebral aneurysm is shown below. First, a segmentation of the 3D x-ray image of the aneurysm is performed to assign voxels to the aneurysm sac and the parent vessels that feed the aneurysm. Then, simulated 2D projections of the 3D x-ray image are generated at multiple different viewpoints of the x-ray imaging system onto the 3D x-ray image. At each viewpoint, a value of an image viewpoint metric is calculated. An example of an image viewpoint metric is an overlap metric that is determined based on several aneurysm sac voxels that overlap with the parent vessels. The overlap metric value is calculated by projecting a virtual ray from the x-ray source of the x-ray imaging system, through the 3D x-ray image, onto the x-ray detector of the x-ray imaging system, and counting the number of aneurysm sac voxels that overlap with parent vessel voxels. Since it is usually desirable to provide a view without overlap between the aneurysm sac and the parent vessels, the overlap criterion should minimize the value of the overlap metric.
[0044] Similarly, a foreshortening metric for a portion of a blood vessel can be calculated by projecting a virtual ray from an X-ray imaging system's X-ray source, through a 3D X-ray image, onto an X-ray detector, and calculating the value of the average intensity of the portion of the blood vessel projected onto the X-ray detector. When the axis of the blood vessel is parallel to the X-ray detector plane, the apparent cross-sectional area of the blood vessel at the X-ray detector is maximum, and therefore the value of the foreshortening metric is minimum. As the axis of the blood vessel tilts away from the X-ray detector plane, the value of the foreshortening metric increases due to the decrease in the apparent cross-sectional area of the blood vessel at the X-ray detector and the increase in the integrated amount of contrast agent along the path of the virtual X-ray, until the axis of the portion of the blood vessel is aligned with the path of the virtual ray (at this viewpoint, the value of the foreshortening metric is maximum). Since it is usually desired to view the parent vessel as if it were parallel to the detector plane, i.e., to minimize foreshortening, the foreshortening criterion should minimize the value of the foreshortening metric of the parent vessel of the aneurysm.
[0045] Similarly, the presence of one or more artifacts in the region of interest can be detected to calculate a value of an artifact metric. As an example, the artifact metric represents the overlap of the region of interest with confusing features in the projection image. In an example where the region of interest is an aneurysm, examples of confusing features that may be present in the x-ray projection image include the shadow of the skull and metal artifacts from dental implants or fillings. Since it is usually desirable to optimize the clarity of the region of interest, the artifact criterion should minimize the value of the artifact metric.
[0046] Metrics such as overlap metric, foreshortening metric, and artifact metric provide an objective assessment of the quality of the viewpoint of an X-ray projection image and can be used alone or in combination to provide an analytical image viewpoint score value s2. 1…n If the analytical image viewpoint score value s2 is calculated using 1…n may also include
[0047] For example, if the region of interest is a sidewall aneurysm, and for simplicity, assume that the analytical image viewpoint score value s2 is based on only two metrics C1 and C2 (the metrics include the overlap C1 of the aneurysm sac with the parent vessel and the foreshortening C2 of the parent vessel), the analytical image viewpoint score value s2 is determined from the metrics C1 and C2 using the following formula: s2 = λ1C1 + λ2C2 (Equation 1) where C1 and C2 represent overlap and foreshortening metrics. The metrics can be calculated by the analytical scoring module ASM shown in Figure 4 using this formula. In this example, the weight l1 is selected to be larger than l2, because a large overlap is expected to be more detrimental to the quality of the viewpoint than foreshortening.
[0048] Criteria such as overlap, foreshortening, and artifact criteria may also be used, alone or in combination, to calculate an optimized analytical image viewpoint score value s2 to provide an optimal view of the region of interest. For example, in Equation 1, it may be desirable to minimize metric C1 (which provides optimal detection of coil migration into the parent vessel) and also minimize metric C2 (which provides an optimal view of the coil insertion into the aneurysm pouch from the connecting artery).
[0049] In this embodiment, the analytical image viewpoint score value s2 is combined with the predicted image viewpoint score value s1 generated by the neural network NN1 to provide a combined image viewpoint score value s' for the X-ray projection image, such that the subjective bias of the predicted image viewpoint score value s1 introduced through the training of the neural network NN1 is compensated for by the objective analytical image viewpoint score value s2.
[0050] The analytical image viewpoint score value s2 and the predicted image viewpoint score value s1 can be combined in various ways. For example, the image viewpoint score values s1 and s2 can be weighted. The image viewpoint score values can be weighted with weights that depend on the X-ray projection images, or the weights can be fixed. For example, the weights can be fixed such that the combined image viewpoint score value s' is the average of the image viewpoint score values s1 and s2.
[0051] In one embodiment, a confidence value u1 of the predicted image viewpoint score value s1 is used to weight both the predicted image viewpoint score value s1 and the analytical image viewpoint score value s2. In this embodiment, the combined image viewpoint score value s' can be calculated using the following formula: s'=u1·s1+(1-u1)·s2 (Formula 2) where u1 is in the range of 0 to 1. Thus, if the confidence values u1 of the predicted image viewpoint scores s1 are high, then these are used to give a higher weight to the predicted image viewpoint scores s1 than to the analytical image viewpoint scores s2.
[0052] As some examples, in some situations, the X-ray projection image 110 may have a very similar appearance or characteristics to the training data used to train the neural network NN1. In this situation, the neural network NN1 is expected to predict the image viewpoint score value s1 of the X-ray projection image 110 with a high confidence value u1. For example, u1=0.9 indicates that the image viewpoint score value s1 can be relied upon. In this situation, the combined image viewpoint score value is calculated as s'=0.9·s1+0.1·s2. In another situation, the X-ray projection image 110 may have few features in common with the training data used to train the neural network NN1. In this situation, the confidence value of the neural network NN1 is, for example, u1=0.1, and therefore the combined image viewpoint score value is calculated as s'=0.1·s1+0.9·s2. In another situation, the X-ray projection image 110 may have some similarity to the training data used to train the neural network NN1. In this situation, the confidence value of the neural network NN1 is, for example, u1=0.5 and the combined image viewpoint score value is calculated as s'=0.5·s1+0.5·s2, i.e., as the average of the predicted image viewpoint score value s1 and the analytical image viewpoint score value s2. Using the confidence value u1 of the predicted image viewpoint score value s1 in this way to weight both the predicted image viewpoint score value s1 and the analytical image viewpoint score value s2 avoids over-reliance on the less reliable image viewpoint score value s1.
[0053] In one embodiment, the predicted image viewpoint score value s1 may be omitted from the calculation of the combined image viewpoint score value s'. For example, if the confidence value of the predicted image viewpoint score value s1 does not exceed a predefined threshold, the predicted image viewpoint score value s1 is omitted from the calculation of the combined image viewpoint score value. This can be achieved, for example, by setting the value of s1 in Equation 2 to zero. In this way, over-reliance on the unreliable predicted image viewpoint score value s1 is also avoided.
[0054] As mentioned above, the analytical image viewpoint score value s2 is calculated by a number of metrics C 1…n For example, the analytical image viewpoint score value s2 may be calculated based on an overlap metric, a foreshortening metric, and also based on other metrics. In such cases, the metrics may have corresponding weights l, which define the importance of the metrics to the analytical image viewpoint score value s2. 1…n These weights l 1…n It is contemplated that various techniques may be used to set the value of the weight l. For example, 1…n The value of can be set based on user input or based on a reference value obtained from a lookup table. However, the weight l 1…n The task of setting the values of is not trivial, since in a real situation, they depend on factors such as the clinical procedure, the specific regions of interest in the X-ray image (e.g., type of aneurysm), but also on subjective factors such as the user's preferences. Therefore, the weights l for each X-ray projection image are 1…n It is advantageous to set the value of , so as to be efficient and reliable.
[0055] In one embodiment, the weight l 1…n The value of is set for each X-ray projection image using a neural network. In this embodiment, the analytical image viewpoint score value s2 is calculated based on a number of metrics, each of which has a corresponding weight l that defines the importance of the metric to the analytical image viewpoint score value s2. 1…n and the method described with reference to FIG. inputting the X-ray projection images into a second neural network NN2; In response to the input, the weights of the X-ray projection images l 1…n generating a prediction of Weight l 1…n The weights l used to calculate the analytical image viewpoint score value s2 based on the predicted values of 1…n Setting the value of The second neural network NN2 includes weights l of the X-ray projection images. 1…nIt is trained to generate predictions for
[0056] In this way, the weight l 1…n Setting the value of s2 is believed to efficiently provide a reliable analytical image viewpoint score value s2. Moreover, it overcomes the burden of manually setting the weights for each image and the limitations that come from setting the weights to fixed values.
[0057] This embodiment operates in a similar manner to that described in relation to FIG. 4, where each X-ray projection image 110 is input to a second neural network NN2, which assigns weights l 1…n The method further includes the additional operation of using the output of the second neural network NN2 to set the value of s2. By using a neural network to set these weights, a reliable value of the analytical image viewpoint score value s2 can be obtained. The training of the second neural network NN2 is described in more detail below.
[0058] In one embodiment, the second neural network NN2 includes a plurality of metrics C that are used to calculate an analytical image viewpoint score value s2. 1…n The neural network NN2 includes an additional input for receiving input data representing one or more physician preferences among the metric C. This is illustrated in FIG. 4 by an input to the second neural network NN2 labeled "Preferences." The user input is 1…n For example, a physician may prefer an overlap metric over a foreshortening metric. User input may be used to bias the output of the second neural network NN2 towards the user's preferred metric. The user input may come from a user input device such as a keyboard, mouse, touch screen, etc. This may be used to adjust the combined image viewpoint score value s' to reflect the user's preference for a particular view.
[0059] In another embodiment, the user preferences are incorporated into the combined image viewpoint score value s′ by performing inference using a second neural network NN2 that is selected based on the type of training data used to train the second neural network NN2. In this embodiment, the weights l used to calculate the analytical image viewpoint score value s′ are 1…n The value of the weight l 1…n The method described with reference to FIG. selecting a second neural network NN2 from a database of neural networks classified based on the type of training data used to train the second neural network NN2; , and the types of training data include training data for a particular interventional procedure, training data for a particular physician.
[0060] In this way, the resulting combined image viewpoint score value s' is adjusted to reflect the user's preferences for particular views of the region of interest.
[0061] If the same training data is used to train the neural network NN1 and the neural network NN2, the confidence value u1 generated by the neural network NN1 is used to calculate the metric C using a second neural network NN2. 1…n Weight of l 1…n or, alternatively, we can use these weights l from a lookup table. 1…nIn a situation where the neural network NN1 is presented with images different from its training data, the confidence value u1 of the neural network NN1 is expected to be low, and it can be decided to use a lookup table to provide the values of the weights rather than using NN2. Since it is likely that the second neural network NN2 will also result in an unreliable output when the neural network NN1 is presented with images different from its training data, it is more preferable to rely on the values of the lookup table for the weights rather than the potentially unreliable values provided by the second neural network NN2.
[0062] Thus, in one embodiment, the method described with reference to FIG. 1 comprises: i) determining the weights l used to calculate the analytical image viewpoint score value s2; 1…n ii) setting the value of the weight l based on a reference value obtained from a look-up table; or 1…n The weights l used to calculate the analytical image viewpoint score value s2 based on the predicted values of 1…n based on a confidence value u1 of the predicted image viewpoint score value s1 relative to a predefined threshold.
[0063] In one embodiment, the second neural network NN2 has weights l 1…n In one embodiment, the weights l 1…n A confidence value u2 of the predicted value of l is used to weight both the analytical image viewpoint score value s2 and the predicted image viewpoint score value s1. 1…n If the confidence value u2 of the prediction of is not above a predefined threshold, then the analytical image viewpoint score value s2 is omitted from providing the combined image viewpoint score value s' to prevent the less reliable analytical image viewpoint score value s2 from influencing the combined image viewpoint score value s' provided for the image.
[0064] In another embodiment, an analytical image viewpoint score value s2 is set for each X-ray projection image based on the values of similar X-ray projection images. This embodiment can be used in the absence of the second neural network NN2 and analytical scoring module shown in Figure 4, or in combination with both of these features. In this embodiment, the multiple X-ray projection images represent a region of interest from corresponding viewpoints a, b of the X-ray imaging system relative to the region of interest within the subject, and the method described with reference to Figure 1 further comprises: Obtaining a reference value of an analytical image viewpoint score s2 of the X-ray projection image from the database, the analytical image viewpoint score s2 being calculated based on the following metric C 1…n obtaining a representation of one or more of: a degree of overlap between a plurality of features within the region of interest, a foreshortening of one or more features within the region of interest, and a presence of one or more artifacts within the region of interest; outputting a reference analytical image viewpoint score value s2 of the X-ray projection image; Including, The reference value of the analytical image viewpoint score s2 is obtained from the database as follows: comparing the x-ray projection image to a database of reference x-ray projection images and corresponding reference analytical image viewpoint score values s2; The reference analytical image viewpoint score value s2 is obtained by selecting a reference analytical image viewpoint score value s2 from the database based on a calculated value of a similarity metric representing the similarity between the X-ray projection image and a reference X-ray projection image in the database.
[0065] In this embodiment, the reference analytical image viewpoint score value s2 stored in the database is set by an expert, and therefore the reference analytical image viewpoint score value s2 is adjusted to a certain extent for the image.
[0066] When used in combination with the neural network NN2 and the analytical scoring module ASM, the reference analytical image viewpoint score value s2 is calculated by the weights l 1…n For example, the weight l 1…nIf the confidence value u2 of the prediction of u2 does not exceed a predefined threshold, the reference analytical image viewpoint score value s2 is provided instead. In this way, it is avoided to generate the combined image viewpoint score value s′ using the unreliable weight values generated by the second neural network NN2.
[0067] In any of the above embodiments, a subsequent viewpoint of the x-ray imaging system is also determined. The subsequent viewpoint is output to inform an operator of the optimal viewpoint to use for generating further x-ray projection images. In this embodiment, the method described with reference to FIG. determining a subsequent viewpoint of an x-ray imaging system for generating an x-ray projection image of a region of interest within the subject, the subsequent viewpoint comprising: a predicted image viewpoint score value s1 generated by the neural network NN1 for the X-ray projection image and a corresponding viewpoint of the X-ray imaging system for the region of interest; and / or A combined image viewpoint score value is determined based on the provided combined image viewpoint score value for the received X-ray projection images and the corresponding viewpoint of the X-ray imaging system relative to the region of interest.
[0068] In this embodiment, the viewpoint of the X-ray imaging system corresponding to the highest predicted image viewpoint score value s1 is determined and used as the subsequent viewpoint. Alternatively, the viewpoint of the X-ray imaging system corresponding to the highest combined image viewpoint score value s1 may be used. This embodiment is used to determine the optimal viewing angle for imaging the region of interest. For example, a user can generate X-ray projection images 110 from different viewpoints of the X-ray imaging system for the region of interest by stepping the viewpoint of the X-ray imaging system by the above angles a and / or b to generate various different image viewpoint score values of the X-ray projection images 110. This procedure can be used to identify promising viewpoints for viewing the region of interest. By repeating this procedure with different step sizes, more optimal positions can be determined. In this way, the method provides an efficient technique for finding the optimal viewing angle for low X-ray dose.
[0069] The training of the neural network NN1 and the second neural network NN2 is detailed below.
[0070] Generally, training a neural network involves inputting a training dataset into the neural network and iteratively adjusting the parameters of the neural network until the trained neural network provides accurate output. Training is often performed using a graphics processing unit (GPU) or a dedicated neural processor such as a neural processing unit (NPU) or a tensor processing unit (TPU). Training often employs a centralized approach, using cloud- or mainframe-based neural processors to train the neural network. After training with the training dataset, the trained neural network is deployed to a device to analyze new input data during inference. The processing requirements during inference are significantly less than those during training, which allows neural networks to be deployed on a variety of systems, such as laptop computers, tablets, and mobile phones. Inference is performed by a central processing unit (CPU), GPU, NPU, or TPU, for example, on a server or in the cloud.
[0071] Thus, the process of training the above neural networks NN1 and NN2 includes adjusting their parameters. The parameters, more specifically, weights and biases, control the behavior of the activation function of the neural network. In supervised learning, the training process automatically adjusts the weights and biases so that when input data is presented, the neural network accurately provides the corresponding expected output data. To do this, the value of the loss function, i.e., the error, is calculated based on the difference between the predicted output data and the expected output data. The value of the loss function is calculated using functions such as negative log-likelihood loss, mean squared error, Huber loss, or cross-entropy loss. During training, the value of the loss function is typically minimized, and training is terminated when the value of the loss function meets a stopping criterion. Training may be terminated when the value of the loss function meets one or more of several criteria.
[0072] There are many known methods for solving loss minimization problems, including steepest descent and quasi-Newton methods. Many algorithms have been developed to implement these methods and their variants. These include, but are not limited to, stochastic gradient descent (SGD), batch steepest descent, mini-batch steepest descent, Gauss-Newton, Levenberg-Marquardt, momentum, Adam, Nadam, Adagrad, Adedelta, RMSProp, and Adamax "optimizers". These algorithms use the chain rule to calculate the derivatives of the loss function with respect to the model parameters. This process is called backpropagation. This is because the derivatives are calculated starting from the last layer or output layer and moving towards the first layer or input layer. These derivatives tell the algorithm how the model parameters need to be adjusted to minimize the error function. That is, the adjustments of the model parameters are made starting from the output layer and working backwards through the network until the input layer is reached. In the first training iteration, the initial weights and biases are often randomized. The neural network then predicts the output data, which is also random. Then, the weights and biases are adjusted using backpropagation. The training process is iterative by adjusting the weights and biases at each iteration. Training ends when the error, i.e., the difference between the predicted output data and the expected output data, is within an acceptable range of the training data or some validation data. The neural network is then deployed, and the trained neural network makes predictions on new input data using the trained values of its parameters. If the training process is successful, the trained neural network accurately predicts the expected output data from new input data.
[0073] In one embodiment, the second neural network NN2 comprises: Receiving volumetric training data (S210) including one or more 3D X-ray images 140' representing a region of interest; generating virtual projection image training data (S220) by projecting one or more of the 3D X-ray images 140' onto a virtual detector plane of the X-ray imaging system at a plurality of different viewpoints of the X-ray imaging system for each 3D X-ray image 140' to provide a plurality of composite projection images; Calculating (S230) for each composite projection image an analytical image viewpoint score value s2 of each corresponding viewpoint of the X-ray imaging system for the 3D X-ray image, the analytical image viewpoint score value s2 being calculated based on the following metric C 1…n Weights l are calculated from the 3D X-ray image 140′ based on one or more of the degree of overlap between multiple features in the region of interest, the foreshortening of one or more features in the region of interest, and the presence of one or more artifacts in the region of interest, and are used to calculate an analytical image viewpoint score value s2. 1…n The value of is set to an initial value, and calculating (S230); Selecting (S240) a subset 150 of synthetic projection images having analytical image viewpoint scores s2 that satisfy a predetermined selection criterion for use in training a second neural network NN2; Receiving ground truth image viewpoint score values for a selected subset 150 of synthetic projection images (S250); The weights of each synthetic projection image, l 1…n inputting (S260) the subset 150 of synthetic projection images into a second neural network NN2 to generate updated values of weights l 1…n and adjusting (S270) the parameters of the second neural network NN2 until the difference between the analytical image viewpoint score value s2 calculated for the synthetic projection image with the updated value of s2 and the corresponding ground truth image viewpoint score value satisfies a stopping criterion. 1…n It is trained to generate predictions for
[0074] This training method is described with reference to Fig. 6. Fig. 6 is a flow chart illustrating an example of a method for training a second neural network NN2 to determine an analytical image viewpoint score value s2 of an X-ray projection image, according to some aspects of the present disclosure. This training method is also described with reference to Fig. 7. Fig. 7 illustrates a metric C used to calculate an analytical image viewpoint score value s2 of an X-ray projection image, according to some aspects of the present disclosure. 1…n Weight of l 1…n FIG. 2 is a schematic diagram illustrating an example of a method for training a second neural network NN2 to predict values of
[0075] 6 and 7, in operation S210, volumetric training data is received. As shown in FIG. 7, the volumetric training data includes one or more 3D X-ray images 140' representing the region of interest. The volumetric training data can generally be received from a computer-readable storage medium, such as a database DB shown in FIG. 7, or from the Internet, cloud, etc. The 3D X-ray images are generally generated for the region of interest from multiple subjects. The 3D X-ray images are generated from subjects of different ages, genders, body mass index values, etc.
[0076] In operation S220, virtual projection image training data is generated by projecting one or more 3D X-ray images 140' onto a virtual detector plane of the X-ray imaging system at multiple different viewpoints of the X-ray imaging system for each 3D X-ray image 140' to provide multiple synthetic projection images. The viewpoints are randomly selected and cover a wide range of all possible viewpoints of the X-ray imaging system. Generating synthetic projection images and using them to train the second neural network addresses the challenge of obtaining various X-ray projection images of a region of interest from different viewpoints to train the second neural network NN2.
[0077] In operation S230, an analytical image viewpoint score value s2 is calculated for each composite projection image with respect to a corresponding viewpoint of the X-ray imaging system with respect to the 3D X-ray image. The analytical image viewpoint score value s2 is calculated from the 3D X-ray image using the corresponding viewpoint and is calculated using the techniques described above. Thus, the analytical image viewpoint score value s2 is calculated based on the following metric C 1…n In the first iteration, the weights l used to calculate the analytical image viewpoint score value s2 are based on one or more of the following: the degree of overlap between multiple features in the region of interest, the foreshortening of one or more features in the region of interest, and the presence of one or more artifacts in the region of interest. 1…n The value of is set to the initial value. For example, the n weights (i.e., l1 to l n ), then the initial value is set to 1 / n for each.
[0078] In operation S240, a subset 150 of synthetic projection images having analytical image viewpoint score values s2 that meet a predetermined selection criterion is selected for use in training the second neural network NN2. The selection criterion is, for example, that the analytical image viewpoint score value s2 is above a predetermined threshold. In this way, only the best views are selected for use in training the second neural network NN2. Reducing the amount of training data in this way improves the efficiency of training the second neural network NN2 to predict the analytical image viewpoint score value s2 with high confidence. This also increases the feasibility of accurately labeling the training data without significantly increasing the workload of experts providing subjective ground truth scoring.
[0079] Another selection criterion may be used for the synthetic projection images. For example, the selection criterion is set to include some synthetic projection images with low analytical image viewpoint score values s2 as well as some synthetic projection images with high analytical image viewpoint score values s2. Such a selection allows the second neural network NN2 to generate low analytical image viewpoint score values s2 with high confidence and high analytical image viewpoint score values s2 with high confidence. This is useful in identifying which predicted analytical image viewpoint score values s2 should or should not be used to calculate the combined image viewpoint score value s'.
[0080] In operation S250, ground truth image viewpoint score values are received for a selected subset of synthetic projection images 150. The ground truth image viewpoint score values s2 may be provided by an expert.
[0081] In operation S260, the subset 150 of synthetic projection images is input to a second neural network NN2, which calculates the weights l of each synthetic projection image. 1…n Next, in operation S270, the weight l is updated. 1…n The parameters of the second neural network NN2 are adjusted until a difference between the analytical image viewpoint score value s2 calculated for the synthetic projection image with the updated value of s2 and the corresponding ground truth image viewpoint score value satisfies a stopping criterion.
[0082] In this way, the second neural network NN2 adjusts the weights l to obtain the objective analytical image viewpoint score value s2 provided by the expert. 1…n Training the second neural network NN2 in this manner, and in particular training it using synthetic projection images, overcomes the challenge of obtaining real projection images from multiple viewpoints to train the second neural network NN2.
[0083] Next, an example of training the neural network NN1 described above with reference to Figures 3 and 4 will be described. This example will be described with reference to Figures 8 and 9. Figure 8 is a flow chart illustrating a first example of a method for training the neural network NN1 to determine an image viewpoint score value of an X-ray projection image according to some aspects of the present disclosure. Figure 9 is a schematic diagram illustrating a first example of a method for training the neural network NN1 to determine an image viewpoint score value of an X-ray projection image according to some aspects of the present disclosure.
[0084] In this embodiment, the neural network NN1 is Receiving (S310) x-ray projection image training data including a plurality of training projection images 110' representing a region of interest from different viewpoints of an x-ray imaging system relative to the region of interest, the training projection images including corresponding ground truth image viewpoint score values; The neural network NN1 is trained to generate a predicted image viewpoint score value s1 of the X-ray projection image by inputting the training projection image and the corresponding ground truth image viewpoint score value (S320) into the neural network NN1, and adjusting the parameters of the neural network NN1 until the difference between the image viewpoint score value s1 predicted by the neural network NN1 and the corresponding input ground truth image viewpoint score value meets a stopping criterion (S330). The operations S320 and S330 may be performed iteratively.
[0085] In operation S310, X-ray projection image training data is received. The X-ray projection image training data is received as described above in connection with the training data for the second neural network. The X-ray projection image training data may be received from a database DB, as shown in FIG. 9. The X-ray projection image training data includes training projection images 110' representing a region of interest from different viewpoints of an X-ray imaging system relative to the region of interest. The X-ray projection image training data is generated using a projection imaging system, such as a projection X-ray imaging system 130, to generate images of the region of interest from a plurality of different viewpoints a, b of the projection X-ray imaging system 130 relative to the region of interest. Corresponding ground truth image viewpoint score values of the images are generated by presenting the images to experts and receiving expert evaluations.
[0086] As shown in FIG. 9, an additional step of calculating an analytical image viewpoint score value s2 may be included for presenting the images to the expert. This additional step is used to provide a subset 150 of training projection images for presentation to the expert. The subset 150 represents training projection images having analytical image viewpoint score values that satisfy one or more predetermined conditions. This additional step is used, for example, to include in the training data only images having high analytical image viewpoint score values s2. Thus, the subset 150 represents training projection images having ground truth image viewpoint score values above a first threshold. Alternatively, the subset 150 may include images having low analytical image viewpoint score values s2 as well as images having high analytical image viewpoint score values s2. Thus, the training projection images may include at least some projection images having ground truth image viewpoint score values above a first threshold and at least some ground truth image viewpoint score values below a second threshold.
[0087] In operation S320, the training projection images and the corresponding ground truth image viewpoint score values are input to the neural network NN1. In operation S330, the parameters are adjusted until the difference between the image viewpoint score value s1 predicted by the neural network NN1 and the corresponding input ground truth image viewpoint score value satisfies a stopping criterion. The parameters are adjusted by calculating the value of an error function that represents this difference and using it to perform error backpropagation of the neural network NN1.
[0088] In this way, the neural network NN1 is trained to mimic the image viewpoint score values provided by the expert, and thus the image viewpoint score values predicted by the trained neural network NN1 incorporate the expert's bias.
[0089] A possible limitation of the method described with reference to FIG. 8 is its dependency on the availability of a training data set with images of the region of interest from multiple different viewpoints. In practice, collecting such a training data set is complex and time-consuming. In a further embodiment, a method for training a neural network NN1 is provided. In this method, synthetic projection images are generated from 3D X-ray images. This embodiment addresses the challenges in obtaining the desired training data. This embodiment is described with reference to FIGS. 10 and 11. FIG. 10 is a flow chart illustrating a second embodiment of a method for training a neural network NN1 to determine an image viewpoint score value for an X-ray projection image according to some aspects of the present disclosure. FIG. 11 is a schematic diagram illustrating a second embodiment of a method for training a neural network NN1 to determine an image viewpoint score value for an X-ray projection image according to some aspects of the present disclosure.
[0090] In this embodiment, the neural network NN1 is Receiving volumetric training data including one or more 3D X-ray images representing a region of interest (S410); generating virtual projection image training data (S420) to project one or more 3D X-ray images onto a virtual detector plane of the X-ray imaging system at a plurality of different viewpoints of the X-ray imaging system for each 3D X-ray image to provide a plurality of composite projection images; Calculating (S430) for each composite projection image an analytical image viewpoint score value s2 of each corresponding viewpoint of the X-ray imaging system for the 3D X-ray image, the analytical image viewpoint score value s2 being calculated based on the following metric C 1…n Calculating (S230) from the 3D X-ray image based on one or more of: a degree of overlap between a plurality of features within the region of interest, foreshortening of one or more features within the region of interest, and the presence of one or more artifacts within the region of interest; Selecting (S440) a subset of synthetic projection images having analytical image viewpoint scores s2 that satisfy a predetermined selection criterion for use in training the neural network NN1; receiving ground truth image viewpoint score values for a selected subset of synthetic projection images (S450); The neural network NN1 is trained to generate predicted image viewpoint score values s1 for X-ray projection images by inputting a subset of synthetic projection images and corresponding ground truth image viewpoint score values into the neural network NN1 (S460) and adjusting parameters of the neural network NN1 until the difference between the image viewpoint score value s1 predicted by the neural network NN1 and the corresponding input ground truth image viewpoint score value meets a stopping criterion (S470).
[0091] 10 and 11, in operation S410, volumetric training data is received. As shown in FIG. 7, the volumetric training data includes one or more 3D X-ray images 140′ representing the region of interest. The volumetric training data can generally be received from a computer-readable storage medium, such as a database DB shown in FIG. 11, or from the Internet, cloud, etc. The 3D X-ray images are generally generated for the region of interest from different subjects. The X-ray images are generated from subjects of different ages, genders, body mass index values, etc.
[0092] In operation S420, virtual projection image training data is generated, the virtual projection image training data including a plurality of composite projection images, the composite projection images being generated from one or more 3D x-ray images 140' in the manner described for operation S220 above.
[0093] In operation S430, an analytical image viewpoint score value s2 is calculated for each composite projection image relative to a corresponding viewpoint of the X-ray imaging system relative to the 3D X-ray image, as described above for operation S230.
[0094] In operation S440, a subset 150 of synthetic projection images is selected. This selection is performed as described above for operation S240, with the difference that in operation S440, the subset is now used to train the neural network NN1. The selection is performed to include at least some projection images having ground truth image viewpoint score values above a first threshold. Alternatively, the selection may be performed to include at least some projection images having ground truth image viewpoint score values above a first threshold and at least some ground truth image viewpoint score values below a second threshold.
[0095] In operation S450, ground truth image viewpoint score values for a selected subset of the synthetic projection images are received. The ground truth image viewpoint score values may be provided by an expert.
[0096] In operation S460, a subset of the synthetic projection images and the corresponding ground truth image viewpoint score values are input to the neural network NN1. In operation S470, parameters of the neural network NN1 are adjusted until the difference between the image viewpoint score value s1 predicted by the neural network NN1 and the corresponding input ground truth image viewpoint score value satisfies a stopping criterion. The parameters are adjusted by calculating a value of an error function representing this difference and using this to perform error backpropagation of the neural network NN1. Operations S460 and S470 may be performed iteratively.
[0097] In this way, the neural network NN1 is trained to mimic the subjective image viewpoint score values provided by the expert, and thus the image viewpoint score values predicted by the trained neural network NN1 incorporate the expert's bias. Moreover, this is achieved without requiring an extensive dataset of real projection training images from multiple different viewpoints, since synthetic projection images are used.
[0098] In this example, the weight l used to calculate the analytical image viewpoint score value s 1…n The values of l are generated by the second neural network NN2. Therefore, the second neural network NN2 is trained before the neural network NN1 and is used to train the neural network NN1. Alternatively, the trained neural network NN1 may use the weights l generated by the neural network NN2. 1…n Alternatively, the weights l used to calculate the analytical image viewpoint score value s2 may be retrained or fine-tuned. 1…n The value of may be set to a reference value.
[0099] According to another embodiment, a computer program product is provided that includes instructions that, when executed by one or more processors, cause the one or more processors to perform a method for determining an image viewpoint score value s1 of an X-ray projection image 110 representing a region of interest 120 of a subject, the method comprising: Receiving (S110) a plurality of X-ray projection images 110, the X-ray projection images representing a region of interest 120 from a plurality of different viewpoints of an X-ray imaging system 130 relative to the region of interest; Inputting the X-ray projection image into a neural network NN1 (S120); generating (S130) a predicted image viewpoint score value s1 for each of the x-ray projection images in response to the input; where the neural network NN1 is trained to generate predicted image viewpoint score values s1 for the X-ray projection images.
[0100] According to another embodiment, a system is provided for determining an image viewpoint score value s1 of an X-ray projection image 110 representing a region of interest 120 within a subject. The system 100 includes one or more processors 210, which include: Receiving (S110) a plurality of X-ray projection images 110, the X-ray projection images representing a region of interest 120 from a plurality of different viewpoints of an X-ray imaging system 130 relative to the region of interest; Inputting the X-ray projection image into a neural network NN1 (S120); In response to the input, a predicted image viewpoint score value s1 for each of the X-ray projection images is generated (S130), and the neural network NN1 is trained to generate the predicted image viewpoint score value s1 for the X-ray projection images.
[0101] An exemplary system 100 is shown in Figure 1. In some embodiments, the system 100 may also include two or more of a projection X-ray imaging system 130 for providing X-ray projection images 110, a monitor 240 for displaying the X-ray projection images 110, the predicted image viewpoint score value s1 for each X-ray projection image, and / or other output generated by the system, a patient bed 250, and a user input device (such as a keyboard, mouse, touch screen, etc.) for receiving user input, not shown in Figure 1.
[0102] It is to be understood that the above embodiments are illustrative of the present disclosure, but not limiting. Other embodiments are contemplated. For example, an embodiment described in connection with a computer-implemented method may also be provided in a corresponding manner by a computer program product, a computer-readable storage medium, or the system 100. It is to be understood that a feature described in connection with any one embodiment may be used alone or in combination with other described features, or in combination with one or more features of another embodiment, or in combination with other embodiments. Moreover, equivalents and modifications not described above may be used without departing from the scope of the invention as defined in the appended claims. In the claims, the word "comprises" does not exclude other elements or operations, and singular elements do not exclude a plurality. The mere fact that certain features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be advantageously used. Any reference signs in the claims should not be construed as limiting the scope.
Claims
1. 1. A computer-implemented method for determining an image viewpoint score value for an x-ray projection image representing a region of interest within a subject, the computer-implemented method comprising: receiving a plurality of x-ray projection images, the x-ray projection images representing a region of interest from a plurality of different viewpoints of an x-ray imaging system relative to the region of interest; inputting the X-ray projection images into a neural network; generating a predicted image viewpoint score value for each of the x-ray projection images in response to said inputting step; Including, A computer-implemented method, wherein the neural network is trained to generate the predicted image viewpoint score values for the x-ray projection images.
2. The computer-implemented method of claim 1 , wherein the neural network further generates a confidence value for each of the predicted image viewpoint score values, the computer-implemented method further comprising outputting the confidence values.
3. The X-ray projection images represent the region of interest from a corresponding viewpoint of the X-ray imaging system relative to the region of interest within the subject, and the computer-implemented method further comprises: receiving a 3D X-ray image representative of the region of interest; For each of the received x-ray projection images, registering the region of interest in the 3D X-ray image with the region of interest in the X-ray image to provide a perspective of the X-ray imaging system relative to the region of interest in the 3D X-ray image; calculating an analytical image viewpoint score value from the 3D X-ray image based on the viewpoint of the X-ray imaging system relative to the region of interest in the 3D X-ray image, the analytical image viewpoint score value being calculated from the 3D X-ray image based on one or more of the following metrics: a degree of overlap between features in the region of interest, foreshortening of one or more features in the region of interest, and the presence of one or more artifacts in the region of interest; combining the analytical image viewpoint score values and the predicted image viewpoint score values to provide a combined image viewpoint score value for the x-ray projection images; 3. The computer-implemented method of claim 1, comprising:
4. The analytical image viewpoint score value is calculated based on a plurality of metrics, the metrics including corresponding weights defining the importance of the metrics to the analytical image viewpoint score value, the computer-implemented method further comprising: setting the weighting values used to calculate the analytical image viewpoint score value based on user input or based on a criterion value obtained from a look-up table; and / or inputting the X-ray projection images into a second neural network; generating predictions of the weights for the x-ray projection images in response to the inputting step; setting values of the weights used to calculate the analytical image viewpoint score value based on the predicted values of the weights; Including, The computer-implemented method of claim 3 , wherein the second neural network is trained to generate the predicted values of the weights of the x-ray projection images.
5. The values of the weights used to calculate the analytical image viewpoint score value are set based on the predicted values of the weights, and the computer-implemented method further comprises: selecting the second neural network from a database of neural networks that are categorized based on the type of training data used to train the second neural network; The computer-implemented method of claim 4 , wherein the types of training data include training data for a specific interventional procedure, training data for a specific physician.
6. The neural network and / or the second neural network further generate a confidence value for each of the predicted image viewpoint score values and the predicted values of the weights, respectively, and the computer-implemented method further comprises: using the confidence value of the predicted image viewpoint score value to weight the predicted image viewpoint score value; using the confidence value of the predicted image viewpoint score value to weight both the predicted image viewpoint score value and the analytical image viewpoint score value; omitting the predicted image viewpoint score value from providing the combined image viewpoint score value if the confidence value of the predicted image viewpoint score value does not exceed a predetermined threshold; i) triggering a step of setting the values of the weights used to calculate the analytical image viewpoint score value based on reference values obtained from a look-up table, or ii) triggering a step of setting the values of the weights used to calculate the analytical image viewpoint score value based on the predicted values of the weights based on a confidence value of the predicted image viewpoint score value in relation to a predefined threshold value; using the confidence value of the prediction of the weight to weight both the analytical image viewpoint score value and the analytical image viewpoint score value; omitting the analytical image viewpoint score value from providing the combined image viewpoint score value if the confidence value of the predicted value of the weight does not exceed a predetermined threshold; 6. The computer-implemented method of claim 4 or 5, comprising one or more of the following:
7. The plurality of X-ray projection images represent the region of interest within the subject from corresponding viewpoints of the X-ray imaging system relative to the region of interest, and the computer-implemented method further comprises: obtaining, from a database, a reference value of an analytical image viewpoint score for the X-ray projection image, the analytical image viewpoint score representing one or more of the following metrics: a degree of overlap between a plurality of features within the region of interest, a foreshortening of one or more features within the region of interest, and a presence of one or more artifacts within the region of interest; outputting the reference analytical image viewpoint score values for the x-ray projection images; Including, The reference value of the analytical image viewpoint score is obtained from the database by: comparing said x-ray projection images to a database of reference x-ray projection images and corresponding reference analytical image viewpoint score values; selecting the reference analytical image viewpoint score value from the database based on a calculated similarity metric representative of a similarity between the X-ray projection image and the reference X-ray projection images in the database; The computer-implemented method of claim 1 , wherein the first and second inputs are obtained from the first and second inputs.
8. determining a subsequent viewpoint of the x-ray imaging system for generating an x-ray projection image of the region of interest within the subject, the subsequent viewpoint comprising: the predicted image viewpoint score values generated by the neural network for the X-ray projection images and the corresponding viewpoints of the X-ray imaging system for the region of interest; and / or the combined image viewpoint score value provided for the received x-ray projection image and the corresponding viewpoint of the x-ray imaging system for the region of interest; The computer-implemented method of claim 1 , wherein the step of determining the
9. The second neural network comprises: receiving volumetric training data including one or more 3D X-ray images representative of the region of interest; generating virtual projection image training data by projecting the one or more 3D X-ray images onto a virtual detector plane of the X-ray imaging system at a plurality of different viewpoints of the X-ray imaging system for each 3D X-ray image to provide a plurality of composite projection images; calculating, for each composite projection image, an analytical image viewpoint score value for each corresponding viewpoint of the X-ray imaging system relative to the 3D X-ray image, the analytical image viewpoint score value being calculated from the 3D X-ray image based on one or more of the following metrics: a degree of overlap between a plurality of features within the region of interest, a foreshortening of one or more features within the region of interest, and a presence of one or more artifacts within the region of interest, the values of the weights used to calculate the analytical image viewpoint score value being set to an initial value; selecting a subset of the synthetic projection images having analytical image viewpoint scores satisfying a predetermined selection criterion for use in training the second neural network; receiving ground truth image viewpoint score values for the selected subset of the synthetic projection images; inputting the subset of synthetic projection images into the second neural network to generate updated values of the weights for each synthetic projection image; and adjusting parameters of the second neural network until a difference between the analytical image viewpoint score value calculated for the synthetic projection images with the updated values of the weights and the corresponding ground truth image viewpoint score value satisfies a stopping criterion.
7. The computer-implemented method of claim 4, wherein the computer is trained to generate the predicted values of the weights of the X-ray projection images by
10. The neural network comprises: receiving x-ray projection image training data including a plurality of training projection images representing the region of interest from different viewpoints of an x-ray imaging system relative to the region of interest, the training projection images including corresponding ground truth image viewpoint score values; inputting the training projection images and the corresponding ground truth image viewpoint score values into the neural network; adjusting parameters of the neural network until a difference between the image viewpoint score values predicted by the neural network and the input corresponding ground truth image viewpoint score values satisfies a stopping criterion; The computer-implemented method of claim 1 , wherein the computer is trained to generate the predicted image viewpoint score values for the X-ray projection images by
11. The neural network comprises: receiving volumetric training data including one or more 3D X-ray images representative of the region of interest; generating virtual projection image training data by projecting the one or more 3D X-ray images onto a virtual detector plane of the X-ray imaging system at a plurality of different viewpoints of the X-ray imaging system for each 3D X-ray image to provide a plurality of composite projection images; calculating, for each composite projection image, an analytical image viewpoint score value for each corresponding viewpoint of the X-ray imaging system relative to the 3D X-ray image, the analytical image viewpoint score value being calculated from the 3D X-ray image based on one or more of the following metrics: a degree of overlap between features within the region of interest, foreshortening of one or more features within the region of interest, and the presence of one or more artifacts within the region of interest; selecting a subset of the synthetic projection images having analytical image viewpoint scores that meet predetermined selection criteria for use in training the neural network; receiving ground truth image viewpoint score values for the selected subset of the synthetic projection images; inputting the subset of the synthetic projection images and the corresponding ground truth image viewpoint score values into the neural network; adjusting parameters of the neural network until a difference between the image viewpoint score values predicted by the neural network and the input corresponding ground truth image viewpoint score values satisfies a stopping criterion; The computer-implemented method of claim 1 , wherein the computer is trained to generate the predicted image viewpoint score values for the X-ray projection images by
12. The computer-implemented method of claim 11 , wherein the weight values used to calculate the analytical image viewpoint score value are generated by the second neural network.
13. 12. The computer-implemented method of claim 10 or 11, wherein the selected subset of the training projection images or the synthetic projection images includes at least some projection images having ground truth image viewpoint score values above a first threshold and at least some ground truth image viewpoint score values below a second threshold.
14. A computer program comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the computer-implemented method of any one of claims 1 to 13.
15. 1. A system for determining an image viewpoint score value for an x-ray projection image representing a region of interest within a subject, the system comprising: one or more processors, the one or more processors comprising: receiving a plurality of x-ray projection images, the x-ray projection images representing a region of interest from a plurality of different viewpoints of an x-ray imaging system relative to the region of interest; inputting the X-ray projection images into a neural network; generating a predicted image viewpoint score value for each of the x-ray projection images in response to said inputting; Do the following: The neural network is trained to generate the predicted image viewpoint score values for the x-ray projection images.
