Providing a second key element of an examination region in an x-ray image

Abstract

The invention relates to providing a second key element of an examination region in an x-ray image. A computer-implemented method for providing a second key element of an examination region in an x-ray image, comprising: receiving first input data, wherein the first input data is an optical image of the examination region; applying a first trained function to the first input data, wherein a first output data is generated, wherein the first output data comprises a detected first key element in the optical image, determining a first collimation region based on the first key element; receiving second input data, wherein the second input data is an x-ray image of the examination region acquired using the first collimation region; applying a second trained function to the second input data, wherein a second output data is generated, wherein the second output data comprises a detected second key element in the x-ray image; providing the second output data, wherein the second output data comprises the second key element of the examination region.

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for a second key element of an examination area in an X-ray image, a computer-implemented method for providing a first trained function, a providing system, a computer program product, a computer-readable medium, a training system, and an X-ray system, wherein automatic collimation based on optical images is improved. Background Technology

[0002] X-ray systems, such as radiographic systems, mammographic systems, or fluoroscopy systems, consist of an X-ray source and an X-ray detector. The subject, particularly the patient, is positioned between the X-ray source and the X-ray detector so that an X-ray image of the area to be examined can be acquired. The X-ray beam from the X-ray source is confined by a collimator. The collimator defines a collimation zone. The typical shape of the collimation zone is a rectangle defined by collimator blades on its four sides. Within the collimation zone, X-rays penetrate the subject, and the X-rays that penetrate the subject are detected by the X-ray detector.

[0003] One known technique for defining collimator boundaries or collimation regions is to use an RGB camera in conjunction with a depth camera or a 3D camera. The camera in the acquisition room will first capture images from the patient or subject being examined. Several key features can be identified on the images, for example, using an AI model. These key features can then be used to define the collimator boundaries or collimation regions.

[0004] Automatic collimation is an essential part of acquiring X-ray images. A collimator is a device used to confine and reduce the beam of radiation, defining the boundaries of the X-ray image. Carefully selected collimator settings, or carefully chosen collimation areas, are one of the key aspects of improving radiographic imaging techniques. A carefully selected collimation area prevents unwanted exposure outside the region of interest. Furthermore, a carefully selected collimation area improves image quality by producing, for example, less scattered radiation generated outside the region of interest. The region of interest (e.g., the lung) is located within the examination area.

[0005] However, due to incorrect collimation settings, some essential parts of the organ to be imaged may not be fully shown in the X-ray image. For example, the collimation area may not be large enough to image the entire organ. Summary of the Invention

[0006] The object of the present invention is to provide a computer-implemented method for providing a second key element of an examination area in an X-ray image, a computer-implemented method for providing a first trained function, a providing system, a computer program product, a computer-readable medium, a training system, and an X-ray system, which improve the accuracy of automatic collimation of the examination area.

[0007] The object of the present invention is achieved by the following: a computer-implemented method for providing a second key element of an examination area in an X-ray image, a computer-implemented method for providing a first trained function, a providing system, a computer program product, a computer-readable medium, a training system, a computer program product, a computer-readable medium, and an X-ray system.

[0008] The solution according to the invention is described below with respect to the claimed providing system and the claimed method. Features, advantages, or alternative embodiments described herein can be assigned to other claimed objects, and vice versa. In other words, the claims to the providing system can be improved using features described or claimed in the context of the method. In this case, the functional features of the method are implemented by the target unit of the providing system.

[0009] Furthermore, the solutions according to the invention are described below with respect to methods and systems for providing a second key element of an examination region in an X-ray image and methods and systems for training a first trained function. Features, advantages, or alternative embodiments described herein can be assigned to other claimed objects, and vice versa. In other words, the claims to the methods and systems for providing the first trained function can be improved using features described or claimed in the context of methods and systems for providing a second key element of an examination region in an X-ray image, and vice versa.

[0010] Specifically, the trained function of the method and system for providing a second key element of the examination area in an X-ray image can be adjusted by the method and system for training a first trained function. Furthermore, the input data may include advantageous features and implementations for training the input data, and vice versa. Additionally, the output data may include advantageous features and implementations for outputting the training data, and vice versa.

[0011] The present invention relates to a computer-implemented method for providing a second key element of an examination region in an X-ray image, comprising:

[0012] Specifically, first input data is received via a first interface, wherein the first input data is an optical image of the inspection area.

[0013] Specifically, a first trained function is applied to the first input data via a computational unit, wherein first output data is generated, wherein the first output data includes a first key feature detected in the optical image, and a first collimation region is determined based on the first key feature.

[0014] Specifically, second input data is received via a first interface, wherein the second input data is an X-ray image of the examination area acquired using a first collimation region.

[0015] Specifically, a second trained function is applied to the second input data via a computational unit, wherein second output data is generated, wherein the second output data includes the second key element detected in the X-ray image.

[0016] - Specifically, second output data is provided through a second interface, wherein the second output data includes a second key element of the inspection area.

[0017] This invention can be applied to various X-ray systems, especially radiographic or fluoroscopic systems. X-ray systems include cameras, particularly 3D cameras, X-ray sources, and X-ray detectors. In this invention, an example of a chest X-ray image is considered to illustrate the invention. The invention can be applied to other anatomical regions of the body. Key elements, particularly the number and location, can vary depending on the body part. The first interface, the second interface, and other interfaces can be implemented as a single interface or separate interfaces.

[0018] Key elements can describe the location or coordinates of relevant anatomical features (e.g., lung boundaries, bones, joints, etc.). Key elements can be points, lines, areas, or volumes. In a preferred embodiment, a key element can be a point, also referred to as a key point. In another embodiment, a key element can be, for example, a line describing the boundary of a bone. Key elements can include shapes, such as points, lines, areas, or volumes, and locations. Location can be defined, for example, by the center of the shape. Key elements, and preferably sets of key elements, can describe anatomical features and / or their locations.

[0019] The first input data is an optical image of the area to be examined. The term "optical" describes the optical device used to acquire the optical image. A 2D or 3D camera can be used to acquire the optical image. The optical device can use wavelengths from 380 nm to 780 nm. The optical image may include color information of the area to be examined, particularly RGB information. The optical image may also include depth information of the area to be examined. In a preferred embodiment, the optical image includes both RGB and depth information. In another embodiment, the optical image includes either RGB information only or depth information only. In use cases in the field of mammography, the optical image may include only depth information. A depth camera can be used.

[0020] A first trained function (particularly a deep learning algorithm) is applied to the first input data, generating first output data. The first output data includes a first key feature detected in the optical image. A first collimation region is determined based on the first key feature. The first collimation region can be inferred or calculated based on the first key feature. The first key feature is detected in the optical image. A key feature can be defined as a characteristic feature within the examined region. Image recognition algorithms, pattern recognition algorithms, or other image processing algorithms can be used to detect the key feature. In particular, the first trained function can be based on a machine learning algorithm or a deep learning algorithm.

[0021] The second input data is an X-ray image of the examination area acquired using the first collimation region. A second trained function is applied to the second input data, generating second output data. The second trained function can be based on a machine learning algorithm or a deep learning algorithm. The second output data includes second key features detected in the X-ray image. The set of key features can include all detected second key features.

[0022] According to an aspect of the present invention, the method further includes the following steps:

[0023] - Check the completeness of the second critical element, and perform the following actions if the second critical element is incomplete:

[0024] Specifically, third input data is received through the first interface, wherein the third input data is an X-ray image of the examination area acquired using the collimation region, and also includes a second key element.

[0025] Specifically, a third trained function is applied to the third input data via a computational unit, wherein third output data is generated, wherein the third output data includes at least one estimated third key element to complete the second key element.

[0026] In one implementation, the completeness of the set of second key elements is checked. A certain number of key elements are required to set the collimation region. For example, four key elements can be used to set a rectangular collimation region.

[0027] In cases where the set of second key features is incomplete, third input data is received. The third input data includes an X-ray image of the examination area obtained using a first collimation region and second key features. The X-ray image of the examination area may correspond to the first collimation region. A third trained function is applied to the third input data, generating third output data. The third trained function may be based on a machine learning algorithm or a deep learning algorithm. The third output data includes at least one estimated third key feature to complete the set of second key features. Third output data is provided, comprising the complete set of second key features. The complete set of second key features may include at least one estimated third key feature.

[0028] According to an aspect of the invention, a complete set of second key elements is transmitted to an optical image. The second key elements can be transmitted to the optical image via image registration techniques. The key elements are selected such that they can be detected in both the optical and X-ray images. Mapping of the second key elements in the X-ray image and mapping of the first key elements in the optical image can be performed.

[0029] According to an aspect of the invention, a second collimation region is determined based on the complete set of transmitted second key elements. The second collimation region can be used to re-encode X-ray images.

[0030] According to an aspect of the invention, a second X-ray image of the examination area is acquired using a second collimation region.

[0031] In a preferred embodiment, a feedback loop can be used to check whether the collimation region is correct. The first collimation region and its boundary are defined by an optical image (particularly an RGB image combined with a depth image). The first collimation region is used to acquire, for example, an X-ray image of the chest.

[0032] The X-ray image will be processed by a second trained function, specifically embodied as an AI model, to detect key features related to key features detected in the optical image by a first trained function, which may be referred to as an RGB / depth key feature model. In the case of at least one key feature missing in the X-ray image, a third trained function can be used to estimate the location of at least one missing key feature. Clearly, at least one missing key feature should be located somewhere outside the image. The third trained function can estimate or predict the location of at least one missing key feature (also referred to as at least one third key feature).

[0033] After estimating at least one third key feature, the at least one third key feature and the detected second key feature can form a complete set of second key features. This complete set of second key features in the X-ray image coordinates can be mapped to the optical image, for example, using image registration techniques. In this way, the location of at least one third key feature can be determined in the optical image. In subsequent steps, a second collimation region can be calculated or extrapolated based on the complete set of second key features now available in the coordinates of the optical image.

[0034] A second X-ray image can be obtained using a second collimation region. This second X-ray image can be used again as second input data to determine second key features in the second X-ray image by applying a second trained function. If all second key features are found in the second X-ray image, it can be used for further steps, such as diagnosis by a radiologist. If at least one second key feature is missing again, the process of estimating at least one third key feature is repeated, and a third collimation region can be calculated. This method can be used iteratively to obtain an X-ray image with a complete set of second key features. An X-ray image with a complete set of second key features is considered a complete X-ray image of the area to be examined or the region of interest. An X-ray image with an incomplete set of second key features is considered an incomplete X-ray image of the area to be examined or the region of interest. An incomplete X-ray image can be considered a cropped X-ray image lacking details of the region of interest.

[0035] According to aspects of the invention, the selection of the key elements themselves should consider that the key elements are definable in both optical and X-ray images. Key elements can be defined at the location of body parts such as shoulders or structural features of the body such as bones. In a preferred embodiment, the key elements can be user-defined. In another embodiment, a set of key elements can be defined for a region of interest or area of ​​examination. For example, for a chest X-ray image, four key elements can be defined, which can be located at the four corners of a rectangle. The collimation region can be determined based on a first key element. The first key element can be located within the collimation region, for example, at a distance from the boundary of the collimation region.

[0036] A preferred embodiment of the invention allows the use of X-ray images to correct for potential collimation errors, which may be caused by determining the collimation region based on the optical image, particularly using a first trained function. Furthermore, the X-ray images can be repeatedly examined to ensure that all necessary key elements are available. In the event of missing key elements, the process is repeated to define a new collimation region.

[0037] The present invention also relates to a method for providing a computer implementation of a first trained function, comprising:

[0038] - Preferably, the first input training data is received through a first training interface, wherein the first input training data includes an optical image of the inspection area.

[0039] - Preferably, the first output training data is received through a second training interface, wherein the first output training data is related to the first input training data, and wherein the first output training data includes a second key element determined based on the X-ray image of the examination area.

[0040] - Preferably, a training computation unit is used to train the first function based on the first input training data and the first output training data.

[0041] - Preferably, the first trained function is provided through a third training interface.

[0042] In a preferred embodiment, the first trained function, the second trained function, and / or the third trained function may be based on deep learning methods. Typically, the trained functions mimic cognitive functions associated with other human brains. Specifically, by training based on training data, the trained functions can adapt to new environments and can detect and extrapolate patterns.

[0043] Typically, the parameters of a trained function can be tuned through training. Specifically, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representation learning (an alternative term is "feature learning") can be used. In particular, the parameters of a trained function can be iteratively tuned through a series of training steps.

[0044] Specifically, the trained function may include neural networks, support vector machines, decision trees, and / or Bayesian networks, and / or the trained function may be based on k-means clustering, Q-learning, genetic algorithms, and / or association rules. Specifically, the neural network may be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, the neural network may be an adversarial network, a deep adversarial network, and / or a generative adversarial network.

[0045] The basic idea of ​​this invention is the concept of retraining, which allows the entire system to become smarter by seeing more data. A pipeline for an automatic collimation method for X-ray images can include three AI models: a first trained function as a camera RGB / depth model for detecting a first key feature on the RGB image; a second trained function as an X-ray image model for detecting a second key feature on the X-ray image; and a third trained function as a model for estimating at least one third key feature missing on the X-ray image. The accuracy and stability of such a pipeline depend heavily on the performance of the AI ​​models used. To improve its performance, a retraining scheme for two models (the first trained function and the third trained function) can be used in the pipeline. This training can be performed on-site using data from one or more X-ray devices at the facility. Optionally, the user can examine the identified first or second key feature to influence the retraining, particularly when the data should be used for retraining based on the evaluation quality of the identified first or second key feature. For retraining, three cases based on the output of the X-ray image model can be considered. The second key element can be transmitted to the optical image for retraining, for example, as ground truth about the corresponding first key element.

[0046] The first scenario can occur when the second trained function identifies all the necessary key elements in the X-ray image, and after mapping the X-ray image to the optical image, the collimation region based on the second key elements is the same as the first collimation region based on the optical image. However, it is possible that some key elements identified by the two different models may be slightly off-center from each other. For example, three second key elements may have substantially the same position as their corresponding first key elements, and the first collimation region can be appropriately calculated even though another first key element is some distance from its corresponding second key element. The first trained function can be retrained based on the output of the second trained function.

[0047] In the second scenario, after mapping the second key element to the optical image, the first and second key elements will result in different collimation regions. The collimation region based on the second key element can be smaller than the first collimation region used for X-ray acquisition. In this case, it is not necessary to re-take the X-ray image because all necessary organs are visible in the image; however, the first collimation region used is wider than required, leading to overexposure of the patient. Therefore, it is more necessary to retrain the first trained function to prevent this from happening. This can be achieved by making the weighting of the training data from the second scenario higher than the weighting of the training data from the first scenario. For both the first and second scenarios, the second key element of the (first) X-ray image is used. For the third scenario, the second key element of the second X-ray image is used.

[0048] According to an aspect of the invention, the method further includes:

[0049] - Receive second input training data, wherein the second input training data includes the first X-ray image.

[0050] - Receive second output training data, wherein the second output training data is related to the second input training data, wherein the second output training data includes at least one estimated third key element.

[0051] - Train the third function based on the second input training data and the second output training data.

[0052] - Provides a third trained function.

[0053] The third and final scenario may require retraining both models: the first trained function and the third trained function. This scenario can occur when the second trained function fails to identify all the necessary second key features. The missing key features are estimated by the third trained function. A second collimation region can be defined by mapping at least one estimated third key feature onto the optical image. Since a portion of the organ is missing from the X-ray image, a new X-ray image should be taken. With the second X-ray image, the second key features will be identified again on the image by the second trained function, and the second key features can be compared with the output of the third trained function, which estimated the missing key features from the previous X-ray image. Therefore, both the first and third trained functions can be retrained.

[0054] To improve the pipeline, the first and third trained functions can be retrained. In a preferred embodiment, the second trained function should not be retrained based on the performance of other models during this process; therefore, the second trained function should have high accuracy. A retraining scheme based on user input for the second trained function can be considered. In this case, the user can check the output of the second trained function and, in case of errors, manually perform correct key feature detection. Later, these new second key features can be used together with the input image, such as an optical image or an X-ray image, for retraining.

[0055] Different weights can be considered for new trainable data based on user preferences. In this case, two factors can be considered: the category of the collimator output (good, over-collimated, under-collimated); and the amount of error between the actual key feature and the detected or estimated key feature. For example, the user can indicate that over-collimation cases that cause errors greater than a certain range should have a large impact during retraining because these cases are unacceptable in future events.

[0056] According to an aspect of the invention, the first collimation region determined based on the first key element is substantially the same as the second collimation region determined based on the second key element, and at least one of the first key elements is different from the corresponding second key element in the set of second key elements.

[0057] According to an aspect of the invention, the first collimation region determined based on the first key element is larger than the second collimation region determined based on the second key element.

[0058] According to an aspect of the invention, weights are applied to enhance the optimization of the size of the first collimation region relative to the inspection region.

[0059] According to an aspect of the invention, a method for providing a second key element of an examination region in an X-ray image can be used, wherein the trained function is provided by a method for providing a first trained function.

[0060] The present invention also relates to providing a system comprising:

[0061] - A first interface configured to receive input data, wherein the first input data is an optical image of the inspection area.

[0062] - A first computing unit configured to apply a first trained function to first input data, wherein first output data is generated, wherein the first output data includes a detected first key feature, and a first collimation region is determined based on the first key feature.

[0063] - A second interface configured to receive second input data, wherein the second input data is an X-ray image of the examination area acquired using the first collimation region.

[0064] - A second computing unit configured to apply a second trained function to the second input data, wherein second output data is generated, wherein the second output data includes the detected second key element.

[0065] - A third interface, configured to provide output data, wherein the second output data includes second key element data of the inspection area.

[0066] The second input data can be an X-ray image of the examination area corresponding to the first collimation region.

[0067] According to an aspect of the invention, the providing system further includes:

[0068] - An inspection unit configured to check the completeness of the set of second key elements, and to perform the following operations if the set of second key elements is incomplete:

[0069] - A fourth interface, configured to receive third input data, wherein the third input data includes an X-ray image of the examination area acquired using a first collimation region and a second key element.

[0070] - A third computational unit, configured to apply a third trained function to the third input data, wherein third output data is generated, wherein the third output data includes at least one estimated third key feature to complete the set of second key features.

[0071] - The fifth interface is configured to provide final output data, which includes a complete set of the second key elements.

[0072] The present invention also relates to a computer program product comprising instructions that, when executed by a providing system, cause the providing system to perform a method for providing a second key element of an examination area in an X-ray image.

[0073] The present invention also relates to a computer-readable medium comprising instructions which, when executed by a providing system, cause the providing system to perform a method for providing a second key element of an examination region in an X-ray image.

[0074] The present invention also relates to a training system, comprising:

[0075] - A first training interface configured to receive first input training data, wherein the first input training data includes an optical image of the inspection region.

[0076] - A second training interface configured to receive first output training data, wherein the first input training data is related to the first output training data, and wherein the first output training data includes a second key element determined based on the X-ray image of the examination area.

[0077] - A training computation unit configured to train a first function based on first input training data and first output training data.

[0078] - A third training interface, which is configured to provide the first trained function.

[0079] The present invention also relates to a computer program product comprising instructions that, when executed by a training system, cause the training system to perform a method for providing a first trained function.

[0080] The present invention also relates to a computer-readable medium comprising instructions which, when executed by a training system, cause the training system to perform a method for providing a first trained function.

[0081] The present invention also relates to an X-ray system including a system for providing X-rays.

[0082] Examples of embodiments of the present invention will now be described in more detail with the aid of the accompanying drawings. Attached Figure Description

[0083] Figure 1 This is a schematic representation of the method according to the invention for providing a second key element of an examination area in an X-ray image, as described in the first embodiment;

[0084] Figure 2 This is a schematic representation of the method according to the invention for providing a second key element of an examination area in an X-ray image, as described in the second embodiment;

[0085] Figure 3 This is a schematic representation of a method according to the present invention for mapping a second key element in an X-ray image to an optical image;

[0086] Figure 4 This is a schematic representation of a method for providing a trained function according to the present invention;

[0087] Figure 5 These are schematic representations of example X-ray images 50 and 51, which have complete collimation regions and erroneous collimation regions according to the present invention.

[0088] Figure 6 This is an illustrative representation of an example of a first scenario according to the present invention;

[0089] Figure 7 This is an illustrative representation of an example of a second scenario according to the present invention;

[0090] Figure 8 This is an illustrative representation of an example of a third scenario according to the present invention;

[0091] Figure 9 This is a schematic representation of an example of a second X-ray image according to the present invention;

[0092] Figure 10 This is a schematic representation of a neural network according to the present invention; and

[0093] Figure 11 This is a schematic representation of a convolutional neural network according to the present invention. Detailed Implementation

[0094] Figure 1 An embodiment of a computer-implemented method 10 for providing a second key element of an examination area in an X-ray image is shown. Method 10 includes:

[0095] - Receive 11 first input data, wherein the first input data is an optical image of the inspection area.

[0096] - Applying a first trained function to the first input data, wherein first output data is generated, wherein the first output data includes a first key element detected in the optical image, and a first collimation region is determined based on the first key element.

[0097] - Receive 13 second input data, wherein the second input data is an X-ray image of the examination area acquired using the first collimation region.

[0098] - Applying a second trained function to the second input data, wherein second output data is generated, wherein the second output data includes the second key element detected in the X-ray image.

[0099] - Provides 15 second output data, wherein the second output data includes the second key elements of the inspection area.

[0100] In a preferred embodiment, method 10 further includes the following steps:

[0101] - Check the completeness of the second key element (16). If the second key element is incomplete, perform the following actions:

[0102] - Receive 17 third input data, wherein the third input data is an X-ray image of the examination area acquired using the collimation region, and also includes a second key element.

[0103] - Apply 18 third trained functions to the third input data, wherein third output data is generated, wherein the third output data includes at least one estimated third key element to complete the second key element.

[0104] In a preferred embodiment, a complete set of second key elements is transmitted to an optical image. A second collimation region is determined based on the transmitted complete set of second key elements. A second X-ray image of the examined area is acquired using the second collimation region.

[0105] Figure 2 An embodiment of the method 10 according to the invention for providing a second key element of an examination region in an X-ray image is shown in a second embodiment. In step 21, an optical image 22 of the examination region of the patient 281 is acquired by a 3D camera 283. A first trained function 23 determines a first key element 24 in the optical image. Based on the first key element 24, a collimation region 26 is calculated in step 25.

[0106] In step 27, collimation region 26 is used to acquire X-ray image 28. The X-ray system includes X-ray source 280 and X-ray detector 282. Patient 281 is located between X-ray source 280 and X-ray detector 282. A second trained function 29 is applied to the X-ray image to determine a second key element 30.

[0107] In step 31, it is checked whether the second key element 30 forms a complete set of second key elements. If at least one second key element is missing, a third trained function 32 is applied to the X-ray image 28 to estimate at least one third key element 33 to complete the set of second key elements. The second collimation region 34 is determined based on the complete set of second key elements including at least one third key element 33. A second X-ray image is acquired, and the process begins again.

[0108] If a complete set of the second key elements 30 is found in X-ray image 28, then X-ray image 28 is complete.

[0109] Figure 2 An embodiment of the X-ray system 250 according to the present invention is also shown. The X-ray system 250 includes a providing system 260. The providing system 260 includes:

[0110] - A first interface 261, configured to receive input data, wherein the first input data is an optical image of the inspection area.

[0111] - A first computing unit 262 is configured to apply a first trained function to first input data, wherein first output data is generated, wherein the first output data includes a detected first key feature, and a first collimation region is determined based on the first key feature.

[0112] - A second interface 263, configured to receive second input data, wherein the second input data is an X-ray image of the examination area acquired using the first collimation region.

[0113] - A second computing unit 264 is configured to apply a second trained function to the second input data, wherein second output data is generated, wherein the second output data includes the detected second key element.

[0114] - A third interface 265 is configured to provide output data, wherein the second output data includes second key element data of the inspection area.

[0115] The X-ray system may include a training system 270. The training system includes:

[0116] - A first training interface 271 is configured to receive first input training data, wherein the first input training data includes an optical image of the inspection area.

[0117] - A second training interface 272, configured to receive first output training data, wherein the first input training data is related to the first output training data, wherein the first output training data includes a second key element determined based on the X-ray image of the examination area.

[0118] - Training computation unit 273, configured to train a first function based on first input training data and first output training data.

[0119] - A third training interface 274, which is configured to provide a first trained function.

[0120] Figure 3An embodiment of a method according to the present invention for mapping a second key element in an X-ray image to an optical image is shown. An X-ray image 28 having a second key element 30 and at least one third key element 33, and an optical image 22 having a first key element 24 are used as input. In step 35, the X-ray image 28 is mapped to the optical image 22, taking into account the first key element 24, the second key element 30, and at least one third key element 33. At least one third key element 33 is mapped to the optical image 36. In step 37, a second collimation region 38 is calculated.

[0121] Figure 4 An embodiment of a computer-implemented method 40 for providing a first trained function is shown, comprising:

[0122] - Receive 41 first input training data, wherein the first input training data includes an optical image of the inspection area.

[0123] - Receive 42 first output training data, wherein the first output training data is related to the first input training data, wherein the first output training data includes a second key element determined based on the X-ray image of the examination area.

[0124] - Train the first function 43 based on the first input training data and the first output training data.

[0125] - Provides 44 first trained functions.

[0126] Figure 5 Example X-ray images 50 and 51, showing both complete and incorrectly collimated regions according to the invention, are shown. The invention is illustrated using an example chest X-ray image. The invention can also be applied to other body parts. X-ray image 50 is an example of an incorrectly collimated region used to obtain a chest X-ray image; as can be seen, the lower part of the left lung is missing in the cropped X-ray image 50 due to incorrect collimator parameters. Complete X-ray image 51 shows the entire lung.

[0127] Figure 6 An example of a first scenario according to the invention is shown. Most of the first key element 62 and the second key element 63 are similarly located in the optical image 60, but one of the first key elements 62 is positioned at a distance from the second key element 63. The first collimation region 61 and the second collimation region 64 are substantially identical. The X-ray image 66 is shown as an overlay.

[0128] Figure 7An example of a second scenario according to the invention is shown. In the optical image, the second key element 73 on the right side is closer to the spine than the first key element 72. The first collimation region 74 is larger than the second collimation region 71. The first collimation region 74 does not necessarily need to be large. X-ray image 66 is shown as an overlay.

[0129] Figure 8 An example of a third scenario according to the invention is shown. In optical image 80, the first collimation region 81 is smaller than the desired second collimation region 82. In optical image 80, a first key element 84 is detected. In X-ray image 86, a second key element 83 is detected. The set of second key elements 83 is incomplete. Therefore, two third key elements 82 are estimated. The third key elements 82 are transmitted to optical image 80, and the third key elements 82 can be used to calculate the second collimation region.

[0130] Figure 9 An example of a second X-ray image according to the invention is shown. The third key element 82 can be compared with the newly detected second key element 87 in the second X-ray image 88. The first collimation region 89 is too small to obtain a complete X-ray image of the examined area.

[0131] Figure 10 An embodiment of the artificial neural network 100 is shown. Alternative terms for "artificial neural network" include "neural network," "artificial neural network," or "neural network."

[0132] The artificial neural network 100 includes nodes 120, ..., 132 and edges 140, ..., 142, wherein each edge 140, ..., 142 is a directed connection from a first node 120, ..., 132 to a second node 120, ..., 132. Typically, the first nodes 120, ..., 132 and the second nodes 120, ..., 132 are different nodes 120, ..., 132, but they may also be the same. For example, in... Figure 1 In the diagram, edge 140 is a directed connection from node 120 to node 123, while edge 142 is a directed connection from node 130 to node 132. Edges 140, ..., 142 from the first node 120, ..., 132 to the second node 120, ..., 132 are also represented as "entry edges" of the second node 120, ..., 132 and "exit edges" of the first node 120, ..., 132.

[0133] In this embodiment, nodes 120, ..., 132 of the artificial neural network 100 can be arranged in layers 110, ..., 113, wherein these layers may include an inherent order introduced by edges 140, ..., 142 between nodes 120, ..., 132. Specifically, edges 140, ..., 142 can only exist between adjacent node layers. In the illustrated embodiment, there is an input layer 110 that includes only nodes 120, ..., 122 without any entering edges, an output layer 113 that includes only nodes 131, 132 without any leaving edges, and hidden layers 111, 112 located between the input layer 110 and the output layer 113. Typically, the number of hidden layers 111, 112 can be arbitrarily chosen. The number of nodes 120, ..., 122 in the input layer 110 is typically related to the number of input values ​​of the neural network, while the number of nodes 131, 132 in the output layer 113 is typically related to the number of output values ​​of the neural network.

[0134] Specifically, (real) numbers can be assigned as values ​​to each node 120, ..., 132 of the neural network 100. Here, x (n) i This represents the value of the i-th node 120, ..., 132 in the nth layer 110, ..., 113. The values ​​of nodes 120, ..., 122 in the input layer 110 are equivalent to the input values ​​of the neural network 100, and the values ​​of nodes 131, 132 in the output layer 113 are equivalent to the output values ​​of the neural network 100. Furthermore, each edge 140, ..., 142 may include a weight as a real number, specifically a real number within the interval [-1, 1] or the interval [0, 1]. Here, w... (m,n) i,j Let represent the weight of the edge between the i-th node (120, ..., 132) of layer m (110, ..., 113) and the j-th node (120, ..., 132) of layer n (110, ..., 113). Furthermore, let w be the weight. (n,n+1) i,j Define the abbreviation w (n) i,j .

[0135] Specifically, in order to calculate the output value of neural network 100, the input value is propagated through the neural network. Specifically, the values ​​of nodes 120, ..., 132 in the (n+1)th layer 110, ..., 113 can be calculated based on the values ​​of nodes 120, ..., 132 in the nth layer 110, ..., 113 using the following formula:

[0136]

[0137] Here, the function f is the transfer function (another term is the "activation function"). Known transfer functions are step functions, sigmoid functions (e.g., logic functions, generalized logic functions, hyperbolic tangents, arctangents, error functions, smoothing step functions), or rectifier functions. Transfer functions are primarily used for normalization purposes.

[0138] Specifically, these values ​​are propagated layer by layer through the neural network, wherein the value of the input layer 110 is given by the input of the neural network 100, wherein the value of the first hidden layer 111 can be calculated based on the value of the input layer 110 of the neural network, wherein the value of the second hidden layer 112 can be calculated based on the value of the first hidden layer 111, and so on.

[0139] To set the value w of the edge (m,n) i,j Training data must be used to train the neural network 100. Specifically, the training data includes training input data and training output data (denoted as t). i For the training step, the neural network 100 is applied to the training input data to generate computational output data. Specifically, the training data and the computational output data include a number of values ​​equal to the number of nodes in the output layer.

[0140] Specifically, the comparison between the calculated output data and the training data is used to recursively adjust the weights within the neural network 100 (backpropagation algorithm). Specifically, the weights are varied according to the following formula:

[0141]

[0142] Where γ is the learning rate, and δ(n) is the numerical value when the (n+1)th layer is not the output layer. j It can be based on δ (n+1) j It is recursively calculated as follows:

[0143]

[0144] And when the (n+1)th layer is the output layer 113:

[0145]

[0146] Where f′ is the first derivative of the activation function, and y (n+1) j It is the comparison training value of the j-th node of the output layer 113.

[0147] Figure 11An embodiment of a convolutional neural network 200 is shown. In the illustrated embodiment, the convolutional neural network 200 includes an input layer 210, a convolutional layer 211, a pooling layer 212, a fully connected layer 213, and an output layer 214. Alternatively, the convolutional neural network 200 may include several convolutional layers 211, several pooling layers 212, and several fully connected layers 213, as well as other types of layers. The order of the layers can be arbitrarily chosen; typically, the fully connected layer 213 is used as the last layer before the output layer 214.

[0148] Specifically, within the convolutional neural network 200, the nodes 220, ..., 224 of a layer 210, ..., 214 can be considered as arranged as a d-dimensional matrix or a d-dimensional image. In particular, in the two-dimensional case, the values ​​of nodes 220, ..., 224 in the nth layer 210, ..., 214, indexed by i and j, can be represented as x. (n) [i, j]. However, the arrangement of nodes 220, ..., 224 in a layer 210, ..., 214 itself has no effect on the computation performed within the convolutional neural network 200, because these are given only by the structure and weights of the edges.

[0149] Specifically, the convolutional layer 211 is characterized in that the structure and weights of the incoming edges form a convolution operation based on a certain number of kernels. Specifically, the structure and weights of the incoming edges are chosen such that the value x of the node 220 based on the previous layer 210 is... (n-1) x is the value of node 221 of convolutional layer 211. (n) k Calculated as convolution x (n) k =K k *x (n-1) In the two-dimensional case, convolution is defined as:

[0150]

[0151] Here, the k-th kernel K k It is a d-dimensional matrix (a two-dimensional matrix in this embodiment), which is typically small compared to the number of nodes 220, ..., 224 (e.g., a 3×3 matrix or a 5×5 matrix). Specifically, this means that the weights of the incoming edges are not independent, but are chosen such that they produce the convolution equation. Specifically, for a kernel that is a 3×3 matrix, regardless of the number of nodes 220, ..., 224 in the corresponding layers 210, ..., 214, there are only 9 independent weights (each entry in the kernel matrix corresponds to one independent weight). Specifically, for convolutional layer 211, the number of nodes 221 in the convolutional layer is equal to the number of nodes 220 in the previous layer 210 multiplied by the number of kernels.

[0152] If the nodes 220 of the preceding layer 210 are arranged as a d-dimensional matrix, then using multiple kernels can be interpreted as adding other dimensions (represented as the "depth" dimension) so that the nodes 221 of the convolutional layer 221 are arranged as a (d+1)-dimensional matrix. If the nodes 220 of the preceding layer 210 have already been arranged as a (d+1)-dimensional matrix including the depth dimension, then using multiple kernels can be interpreted as extending along the depth dimension so that the nodes 221 of the convolutional layer 221 are also arranged as a (d+1)-dimensional matrix, where the size of this (d+1)-dimensional matrix relative to the depth dimension is a factor larger than the number of kernels in the previous layer 210.

[0153] The advantage of using convolutional layers 211 is that the spatial local correlation of the input data can be utilized by implementing local connection patterns between nodes of adjacent layers (in particular by connecting each node to a small region of the previous layer only).

[0154] In the illustrated embodiment, the input layer 210 includes 36 nodes 220 arranged as a two-dimensional 6×6 matrix. The convolutional layer 211 includes 72 nodes 221 arranged as two two-dimensional 6×6 matrices, each of which is the result of the convolution of the input layer values ​​with the kernel. Similarly, the nodes 221 of the convolutional layer 211 can be interpreted as arranged as a three-dimensional 6×6×2 matrix, where the last dimension is the depth dimension.

[0155] Pooling layer 212 can be characterized as the pooling operation formed by the structure and weights of the incoming edges and the activation functions of its nodes 222 based on a nonlinear pooling function f. For example, in the two-dimensional case, the pooling operation can be based on the value x of the nodes 221 of the previous layer 211. (n -1) x the value of node 222 in pooling layer 212 (n) The calculation is as follows:

[0156] x (n) [i, j] = f(x) (n-1) [id1, jd2], ..., x (n-1) [id1+d1-1,jd2+d2-1]).

[0157] In other words, by using pooling layer 212, the number of nodes 221 and 222 can be reduced by using a single node 222 calculated based on the number of neighboring nodes in the pooling layer, instead of the number d1·d2 of neighboring nodes 221 in the previous layer 211. Specifically, the pooling function f can be a maximum function, an average function, or an L2 norm function. Specifically, for pooling layer 212, the weights of the incoming edges are fixed and are not modified through training.

[0158] The advantage of using pooling layer 212 is that it reduces the number of nodes 221 and 222 and the number of parameters. This results in a reduction in the amount of computation in the network and controls overfitting.

[0159] In the shown implementation, pooling layer 212 replaces max pooling with only one node instead of the maximum of the four neighboring nodes. Max pooling is applied to each d-dimensional matrix of the previous layer; in this implementation, max pooling is applied to each of the two two-dimensional matrices, thereby reducing the number of nodes from 72 to 18.

[0160] The fully connected layer 213 can be characterized by the fact that there are most, in particular all, edges between nodes 222 of the previous layer 212 and nodes 223 of the fully connected layer 213, and that the weight of each of the edges can be adjusted individually.

[0161] In this embodiment, the nodes 222 of the preceding layer 212 of the fully connected layer 213 are displayed both as a two-dimensional matrix and as unrelated nodes (represented as lines of nodes, where the number of nodes is reduced for better presentation). In this embodiment, the number of nodes 223 in the fully connected layer 213 is equal to the number of nodes 222 in the preceding layer 212. Alternatively, the number of nodes 222 and 223 may be different.

[0162] Furthermore, in this embodiment, the value of node 224 in the output layer 214 is determined by applying the Softmax function to the value of node 223 in the previous layer 213. By applying the Softmax function, the sum of the values ​​of all nodes 224 in the output layer is 1, and all values ​​of all nodes 224 in the output layer are real numbers between 0 and 1. Specifically, if the input data is classified using the convolutional neural network 200, the values ​​of the output layer can be interpreted as the probability that the input data falls into one of the different categories.

[0163] Convolutional neural networks 200 can also include ReLU (an abbreviation for "rectified linear unit") layers. Specifically, the number and structure of nodes contained in a ReLU layer are equal to the number and structure of nodes contained in the previous layer. In particular, the value of each node in a ReLU layer is computed by applying a rectified function to the values ​​of the corresponding nodes in the previous layer. Examples of rectified functions are f(x) = max(0, x), the tangent hyperbolic function, or the sigmoid function.

[0164] Specifically, a convolutional neural network 200 can be trained based on the backpropagation algorithm. To prevent overfitting, regularization methods can be used, such as the loss of nodes 220, ..., 224, random pooling, the use of artificial data, weight decay based on L1 or L2 norm, or maximum norm constraints.

[0165] Although the invention has been described in further detail with reference to preferred embodiments, the invention is not limited to the disclosed examples, and those skilled in the art can derive other variations therefrom without departing from the scope of protection of the invention.

Claims

1. A computer-implemented method (10) for providing a second key element of an examination region in an X-ray image, comprising: Receive (11) first input data, wherein the first input data is an optical image of the inspection area. The first trained function (12) is applied to the first input data, wherein first output data is generated, wherein the first output data includes a first key element detected in the optical image, and a first collimation region is determined based on the first key element. Receive (13) second input data, wherein the second input data is an X-ray image of the examination area obtained using the first collimation region. The second trained function (14) is applied to the second input data, wherein second output data is generated, wherein the second output data includes the second key element detected in the X-ray image. Provide (15) the second output data, wherein the second output data includes a second key element of the inspection area.

2. The method according to claim 1, further comprising the following steps: Check (16) the completeness of the second key element. If the second key element is incomplete: Receive (17) third input data, wherein the third input data is an X-ray image of the examination area obtained using the collimation region, and also includes the second key element. The third trained function is applied (18) to the third input data, wherein a third output data is generated, wherein the third output data includes at least one estimated third key element to complete the second key element.

3. A method (40) for providing a computer implementation of a first trained function, comprising: Receive (41) first input training data, wherein the first input training data includes an optical image of the inspection area. Receive (42) first output training data, wherein the first output training data is related to the first input training data, wherein the first output training data includes a second key element determined based on the X-ray image of the examination area. The first function (43) is trained based on the first input training data and the first output training data. Provide (44) the first trained function.

4. The method of claim 3, further comprising: Receive second input training data, wherein the second input training data includes a first X-ray image. Receive second output training data, wherein the second output training data is related to the second input training data, and wherein the second output training data includes at least one estimated third key element. The third function is trained based on the second input training data and the second output training data. Provides a third trained function.

5. The method according to claim 3 or 4, wherein, The first collimation region determined based on the first key element and the second collimation region determined based on the second key element are substantially the same, and at least one of the first key elements is different from the corresponding second key element in the set of second key elements.

6. The method according to claim 3 or 4, wherein, The first collimation region determined based on the first key element is larger than the second collimation region determined based on the second key element.

7. The method according to claim 6, wherein, Weights are applied to enhance the optimization of the size of the first collimation region relative to the inspection region.

8. The method according to any one of claims 1 to 2, wherein, The trained function is provided by the method according to any one of claims 3 to 7.

9. A system providing (260) includes: A first interface (261) is configured to receive first input data, wherein the first input data is an optical image of the inspection area. A first computing unit (262) is configured to apply a first trained function to the first input data, wherein first output data is generated, wherein the first output data includes a detected first key feature, and a first collimation region is determined based on the first key feature. A second interface (263) is configured to receive second input data, wherein the second input data is an X-ray image of the examination area acquired using the first collimation region. A second computing unit (264) is configured to apply a second trained function to the second input data, wherein second output data is generated, wherein the second output data includes the detected second key element. A third interface (265) is configured to provide output data, wherein the second output data includes second key element data of the inspection area.

10. A computer program product comprising instructions that, when executed by a providing system, cause the providing system to perform the method according to any one of claims 1 to 2.

11. A computer-readable medium comprising instructions that, when executed by a providing system, cause the providing system to perform the method according to any one of claims 1 to 2.

12. A training system (270), comprising: A first training interface (271) is configured to receive first input training data, wherein the first input training data includes an optical image of the inspection region. A second training interface (272) is configured to receive first output training data, wherein the first input training data is related to the first output training data, and wherein the first output training data includes a second key element determined based on the X-ray image of the examination area. The training computation unit (273) is configured to train a first function based on the first input training data and the first output training data. The third training interface (274) is configured to provide the first trained function.

13. A computer program product comprising instructions that, when executed by a training system, cause the training system to perform the method according to any one of claims 3 to 7.

14. A computer-readable medium comprising instructions that, when executed by a training system, cause the training system to perform the method according to any one of claims 3 to 7.

15. An X-ray system (250) comprising the providing system according to claim 9.

Images