Method for determining environmental information based on an X-ray image, processing device, endoscopy device, computer program and data carrier

Abstract

A computer-implemented method for determining environmental information (22) based on at least one two-dimensional X-ray image (23) depicting at least a section of a third object (24-26) located within a patient (42), and at least one sensor information (27) based on at least one measurement value from at least one sensor (28-31) of the third object (24-26), wherein the environmental information (22) relates to material (32-34) in the vicinity of the third object (24-26) and / or an interaction of the third object (24-26) with this material (32-34), comprising the steps: - Receiving the X-ray image (23) and the sensor information (27), - Determining model parameters (35) or restricting the possible parameter values ​​of the model parameters (35) of a three-dimensional model (36) of the third object (24-26) depending on the X-ray image (23) to specify an X-ray-dependent model (37), wherein the three-dimensional model (36) describes a three-dimensional shape and / or pose of the third object (24-26) depending on the model parameters (35), - Determining the environmental information (22) as a function of the X-ray image-dependent model (37), wherein on the one hand the environmental information (22) and / or on the other hand the X-ray image-dependent model (37) additionally depend on the sensor information (27), and - Providing the environment information (22).

Description

[0001] The invention relates to a computer-implemented method for determining environmental information based on at least one two-dimensional X-ray image depicting at least a section of a third-party object located inside a patient. The invention also relates to a processing device, an endoscopy device, a computer program, and a data storage medium.

[0002] During medical treatment, it can often be relevant to obtain information about the environment surrounding a third-party object located within a patient, such as a medical implant or instrument. For example, it can be highly relevant to detect the presence of specific biological material, such as a calcification or a blood clot, within an implant, or the presence of an object to be removed, such as a gallstone, within an endoscopy basket. When positioning or verifying the location of an implant, it can be relevant to know what biological material is present in the surrounding area, for example, to check whether there is sufficient contact with a vessel wall.

[0003] In the field of endovascular aneurysm treatment, it is known to consider the interaction of a catheter with the surrounding vascular system to estimate the shape of an inserted catheter. Such an approach is discussed, for example, in the publication Jäckle S et al. Instrument localisation for endovascular aneurysm repair: comparison of two methods based on tracking systems or using imaging. Int J Med Robot. 2021; 17(6):e2327. https: / / doi.org / 10.1002 / rcs.2327. However, since this shape estimation assumes that the catheter adapts to the shape of vessels whose shape is determined based on a preoperative CT scan, actual interactions between the catheter and the vascular system cannot be detected or taken into account. Therefore, the identified shape does not allow any conclusions to be drawn about the properties of the biological material surrounding the catheter.

[0004] The invention is therefore based on the objective of enabling or improving the acquisition of information about the environment of a third-party object located within a patient, for example an implant or a medical tool.

[0005] The object is achieved according to the invention by a computer-implemented method for determining environmental information based on at least one two-dimensional X-ray image depicting at least a section of a third object located inside a patient, and at least one sensor information based on at least one measured value from at least one sensor of the third object, wherein the environmental information relates to material in the vicinity of the third object and / or an interaction of the third object with this material, wherein the method comprises the following steps: - Receiving the X-ray image and sensor information, - Determining model parameters or restricting the possible parameter values ​​of the model parameters of a three-dimensional model of the third object depending on the X-ray image to specify an X-ray-dependent model, wherein the three-dimensional model describes a three-dimensional shape and / or pose of the third object depending on the model parameters, - Determining environmental information as a function of the X-ray-dependent model, wherein on the one hand the environmental information and / or on the other hand the X-ray-dependent model additionally depend on the sensor information, and - Providing environmental information.

[0006] It was recognized that by combining sensor data from a third-party sensor with a three-dimensional model parameterized based on the X-ray image, the properties of materials in the third-party object's environment, or their interaction with the third-party object, can be determined with good accuracy even without acquiring three-dimensional image data. In particular, the environmental information can relate to biological or endogenous materials, or the interaction of the third-party object with biological or endogenous materials. Alternatively or additionally, the environmental information can also relate to artificial materials, such as bone cement or embolization materials used to fill an aneurysm during an embolic procedure, such as embolic spirals or coils, or embolization materials that harden upon contact with blood.

[0007] It is essential that the X-ray image be considered as the first information channel and the sensor information as the second, since using only one of these channels can lead to significant uncertainties. For example, if the third object's design is known, its three-dimensional shape and the forces acting on it (as determined by that shape) could be used to infer the consistency or elasticity of the surrounding biological material.However, estimating the three-dimensional shape of the third object solely on the basis of a single two-dimensional X-ray image, or even on the basis of a series of several successively taken two-dimensional X-ray images with the same imaging geometry, as is the case, for example, with fluoroscopy, is typically too inaccurate and can be particularly ambiguous, so that without taking sensor information into account, typically only a rough and error-prone estimation of the surrounding information would be possible.

[0008] Even the sensor information alone often does not allow for precise conclusions about local environmental properties, since, particularly due to the elastic deformability of third-party medical objects, the position of the sensor and sometimes also the relative position of different sensor components to each other is not known or at best very imprecisely known without using the x-ray-dependent model.

[0009] As will be explained in more detail later, the sensor information can provide additional information regarding the current shape of the third-party object and / or the forces acting on the third-party object due to the surrounding material. This can compensate for underdetermination or inaccuracy of the model parameters by requiring consistency between the X-ray-dependent model and the sensor information, thus making the X-ray-dependent model additionally dependent on the sensor information. Alternatively or additionally, the sensor information can also be used directly in determining environmental information, for example, to differentiate between various materials in the vicinity of the third-party object based on measured force, impedance, conductivity, or similar parameters.

[0010] Ideally, all model parameters of the three-dimensional model can be determined from the X-ray image and, optionally, the sensor information. However, it is also possible that only a restriction of the possible parameter values ​​for the model parameters can be achieved. For example, a system of equations can describe the relationship between image information, which is determined based on the X-ray image and describes, for instance, the positions of several predefined features in the X-ray image, and the model parameters, where the system of equations is underdetermined. In this case, the model parameters can, for example, be specified as a function of one or more free parameters, where the number of free parameters is reduced compared to the number of model parameters.If, initially, only the X-ray image is considered to limit the possible parameter values ​​of the model parameters, the values ​​of the free parameters, or at least limited value ranges for the free parameters, can be determined, for example, based on the sensor information.

[0011] The model parameters and / or the free parameters can have discrete possible values. For example, in the case of underdetermination, particularly when considering prior knowledge, only a few parameterization options for the three-dimensional model, or at least for a subset of its model parameters, may remain that are consistent with the X-ray image. In this case, for example, each of the parameterization options can be assigned a discrete value of a free parameter.

[0012] The three-dimensional model can, for example, describe an interface, particularly one defined by triangles, that follows the contour of the third object. The coordinates of the vertices of the triangles or other parts of the interface can thus describe the interface in the form of a coordinate grid.

[0013] As mentioned above, multiple X-ray images, especially those acquired with the same imaging geometry, can be considered to determine the model parameters or to narrow down the possible parameter values. This can be useful, for example, to exploit the temporal coherence of the movement of the third object through observation at multiple time points. This approach can at least partially eliminate ambiguities in the X-ray-dependent model that would remain if only a single X-ray image were considered.

[0014] The third-party object can be deformable, in particular, with at least one of the model parameters describing the deformation. In particular, the model parameter, or at least one of the model parameters, can describe an elastic deformation of the third-party object. Based on the deformation, especially the elastic deformation, forces with which the surrounding material acts on the third-party object can be identified.

[0015] For example, such deformation can reveal whether, and if so, which sections of the third-party object are deformed by the walls of a cavity in the patient containing the third-party object, such as a bile duct or a blood vessel, or whether solid organic material, such as a gallstone to be removed or a blood clot, is contained within a cavity of the third-party object. Suitable sensors, capable of measuring deformations or forces, can eliminate ambiguities in determining model parameters based on the X-ray image.

[0016] Detecting deformation of a third-party object can also be essential for estimating the precise position of the sensor and, in particular, the relative positions of different sensor components. For example, sensors for measuring the conductivity of surrounding material can have widely spaced electrodes as sensor components. Similarly, sensors that measure a relative electrical potential at different positions can have multiple widely spaced electrodes as sensor components.

[0017] Based on the X-ray image-dependent model describing the deformation, the significance of sensor information, for example regarding the resistance of the material and / or regarding an inductance influenced by the material, can thus be significantly increased.

[0018] Environmental information can be determined to identify whether and / or how much material of at least one specified material type is present in a cavity of the third-party object, and / or whether and / or in which area of ​​an outer surface of the third-party object it contacts an inner surface of a cavity in the patient that houses the third-party object. Additionally or alternatively, the environmental information can relate to the surface structure of the inner surface of the patient's cavity.

[0019] For example, environmental information can be used to determine whether an object of a specified material type, such as a calcification, a gallstone, or a fragment thereof, is contained within the cavity of the third-party object, and if so, its size. Detecting material to be removed from the cavity of the third-party object can be relevant, for instance, when using an endoscopy basket.

[0020] Additionally or alternatively, the environmental information can relate to at least one of the following material types: blood, blood clot, bone cement, an embolic material used to fill an aneurysm, and / or digestive fluid. The embolic material to be considered can be, in particular, metal wire (i.e., a coil) or an initially liquid embolic material that hardens upon contact with blood, for example, an ethylene-vinyl alcohol copolymer (EVOH), such as that marketed under the name Onyx®.

[0021] To detect contact between the outer surface of a third-party object and the inner surface of the patient's cavity, it is possible, for example, to differentiate between various potentially adjacent materials, such as a wall material and blood or digestive fluid, and / or to detect and evaluate forces acting on the outer surface. Based on the surface structure of the cavity wall, such as a vessel wall, a tumor and / or calcification in the cavity wall area can be detected. The surface structure can be determined, for example, from the shape of an elastically deformable third-party object as determined by the X-ray-dependent model, or from the forces acting on the third-party object.However, it is also possible, as a supplement or alternative, to consider electrical information determined by sensors, such as measured potentials, inductance and / or resistance, for the classification of the surface structure.

[0022] Environmental information, or processing information determined based on environmental information, can include the presence of a solid in a cavity of the third-party object and / or a control signal for an actuator to move the third-party object and / or a degree of filling of an aneurysm with an embolization material.

[0023] For example, it can be determined whether a gallstone or a fragment thereof is present in an endoscopy basket forming the third body. The presence of a solid can be recognized, in particular, by the determined shape of the third object or by the forces acting on it, the parameters of which can be determined as explained above.

[0024] The control signal can, for example, specify a motor current for retrieving a third-party object, such as an endoscopy basket. Environmental information can be used to determine, in particular, the force exerted by the third-party object on a solid object held in the endoscopy basket. This allows the motor current to be appropriately controlled, thus preventing unwanted breakage of the solid object or, for example, deliberately causing it to break.

[0025] Such a force can be determined, for example, based on a measured deformation of the third object and / or force sensors can directly measure at least one force on the third object and, based on the geometry of the force effect, which is specified by the model of the third object, the force acting locally on the solid body can be deduced.

[0026] The sensor, or at least one of the sensors on whose measured values ​​the sensor information is based, can be a force sensor and / or a pressure sensor and / or a deformation sensor and / or an electrical sensor for detecting an electrical resistance and / or an electrical impedance and / or an electrical potential.

[0027] Using a force sensor, a pressure sensor, and / or a deformation sensor, additional information regarding the shape of a deformable third-party object and / or the forces generating it can be obtained. This can, for example, overcome a potential underdetermination of the three-dimensional model resulting from solely evaluating the X-ray image, thus enabling a significantly more accurate determination of the actual shape of the third-party object. Based on this shape and the forces, pressures, etc., acting on the third-party object, the mechanical interaction of the third-party object with the material in its vicinity—especially with material located in a cavity of the third-party object and / or with material adjacent to its outer wall—can be quantified.This allows, for example, inferences to be made about the shape of a cavity in the body that accommodates the third-party object, thus enabling the detection of calcifications in a vessel. As explained above, the shape and / or forces can also be used to determine the presence of a solid body in a cavity of the third-party object, the degree to which the third-party object is in contact with a wall of the patient's cavity, and / or other relevant environmental information.

[0028] Based on known mechanical properties of the third-party object, it is also possible to infer properties of the materials surrounding the third-party object. This allows, for example, differentiation between liquid and solid materials in the vicinity of the third-party object—such as between vessel walls and blood contacting the outer wall of the third-party object—or even between different solid materials. Furthermore, in the case of solid materials, such as vessel walls, calcifications, and blood clots, their shape can be deduced, at least in the area where they contact the third-party object. This allows, for example, the detection of vessel wall deformation caused by a tumor or calcification, as well as the estimation of the size of an object captured in an endoscopy basket, which can then be provided as environmental information.

[0029] By detecting the electrical resistance, potential, or impedance of the material surrounding the third-party object using the sensor, different materials can be distinguished, for example, to differentiate whether an endoscopy basket contains only liquid, such as blood, digestive fluid, or bile, or a solid object, such as a gallstone or calcification. Based on the three-dimensional model parameterized using the X-ray image, the actual position of the respective sensor or sensor components is known with good accuracy. This is important because electrical potentials can depend strongly on the specific measurement position, and because, in resistance measurements, the relative position of the points between which the resistance measurement is taken, especially their distance, is highly relevant.For example, in induction measurements it is useful to take into account the influence of the third object itself and thus its shape on the inductance, which is also possible using the x-ray image-dependent model.

[0030] Preferably, at least one fiber optic shape sensor, for example based on a fiber Bragg grating, or alternatively or additionally at least one strain gauge and / or at least one piezo bending sensor, can be used as a deformation sensor.

[0031] In the X-ray image, a cavity of the patient containing the third object can be segmented and / or, based on the X-ray image, a three-dimensional environmental model of the environment of the third object can be parameterized depending on the X-ray image, whereby on the one hand the determination of the model parameters of the three-dimensional model of the third object or the restriction of the possible parameter values ​​of these model parameters and / or on the other hand the determination of the environmental information additionally depending on the segmentation of the cavity of the patient and / or the environmental model.

[0032] The patient's segmented cavity could be, for example, a blood vessel, a bile duct, or a section of the stomach or intestine. When determining or restricting the model parameters, the geometry of the patient's segmented cavity can be taken into account. For instance, when parameterizing the model of the third-party object, the boundary condition can be used that a forward projection of the third-party object, using the known imaging geometry, lies within the patient's segmented cavity. This prevents unrealistic parameterizations of the three-dimensional model where the modeled third-party object would protrude beyond the patient's cavity.

[0033] However, it is also possible, for example, to evaluate image content adjacent to the patient's segmented cavity. This can provide additional information for classifying material adjacent to the third-party object, enabling, for instance, the robust detection of calcifications and / or tumors adjacent to the patient's cavity.

[0034] Since extensive prior knowledge about the anatomy of patients generally exists, and further patient characteristics, such as size, weight and gender, as well as three-dimensional preoperative image data such as CT or MR image datasets, are typically known, a three-dimensional environmental model with relevant information can be generated based on a single X-ray image, for example by elastic deformation of an anatomical atlas, whereby certain features, such as tumors, can be recognized and localized at least in the directions parallel to the image plane of the X-ray image.

[0035] Several features of the third-party object can be specified, wherein the model positions of the features in the three-dimensional model depend on the model parameters, wherein the respective model position is assigned an image position in the X-ray image by a known imaging geometry of the respective X-ray image, wherein at least a subgroup of the specified features can be recognized in the X-ray image by means of feature recognition and their actual position in the X-ray image can be determined, wherein the model parameters can be determined by optimizing a cost function that depends on a respective distance of the image position of the respective feature from the actual position of this feature.

[0036] The described procedure ensures that the three-dimensional model replicates the information obtained from the X-ray image regarding the positions of the features, or at least does not deviate too significantly from it. Alternatively or additionally, the cost function can depend on a measure of the similarity between the representation of the respective detected feature in the X-ray image and its expected representation based on the three-dimensional model. This allows, for example, the shape of a feature in the three-dimensional model to be taken into account.

[0037] Depending on the model parameters, the three-dimensional model can define the positions of multiple model points in three-dimensional space. The three-dimensional model, or a predefined calculation rule, defines at least one piece of model information depending on the relative positions of at least a subset of the model points. The model parameters can be determined by optimizing a cost function that depends on a measure of the distance of the respective model information from the sensor information or a respective part of the sensor information. In particular, at least parts of the model points can be arranged at the model positions discussed above.

[0038] To take sensor information into account when parameterizing the x-ray image-dependent model, the values ​​of the model parameters can be varied in particular in order to minimize, by optimizing a common cost function, the deviation of the model from the x-ray image, at least with respect to the positions of detected features, and the deviation of the model information, which would have to be maintained for the sensor information according to the resulting model, from the sensor information.

[0039] Establishing a relationship between the model and sensor information is particularly straightforward when the sensor information, or a portion thereof, relates to a deformation of the third-party object, which can be detected, for example, by a bending sensor, or to forces acting on or within the third-party object, which can be detected, for example, by a force sensor. In general, the relationship between the relative position of various model points and the acting forces can be determined, for example, using the finite element method. However, many third-party objects relevant to medical imaging can also be described with sufficient accuracy by simpler models, such as a system of mass points coupled by springs and / or flexible beams.Such a description may, for example, be well suited to describe a third-party object that essentially consists of a wire mesh.

[0040] An example of a model for a complex object, namely a human body, is known from the publication Shetty, Karthik, et al. “BOSS: Bones, organs and skin shape model.” Computers in Biology and Medicine 165 (2023). The approaches described there can also be used for the parameterized description of complex third-party objects using model parameters.

[0041] The specification of the X-ray-dependent model and / or the model parameters and / or the three-dimensional environment model and / or the determination of environmental information can be performed by a respective machine learning-trained function. In general, a machine learning-trained function mimics the cognitive functions that humans associate with the thought processes of others. Through training based on training data, the trained function is particularly able to adapt to new circumstances and to recognize and extrapolate patterns. Another possible term for a "trained function" is a "machine learning-trained model."

[0042] Although trained functions can learn complex relationships, the application of even complex trained functions is often possible at least approximately in real time. At least one trained function can thus learn, during training, to provide results that essentially correspond to the results of a more computationally and / or memory-intensive investigation method used during training, or to the results of an evaluation performed or supported by an expert. Therefore, using a trained model can achieve a result that is just as good as a manual analysis of the X-ray image and sensor information by an expert, or a more complex computational method than the application of the trained function.

[0043] For example, the optimization described above, using a finite element method (which, depending on the desired model resolution, can be very computationally intensive and therefore unsuitable for real-time applications in some use cases), can be used during training to provide X-ray-dependent models for training datasets. This allows the trained function to learn to achieve similar results. This can make it possible to determine environmental information with minimal delay, for example, less than one second, so that environmental information can be used, for instance, during fluoroscopy or as part of surgical procedures.

[0044] In general, the parameters of a machine learning model can be adjusted through training to provide the trained model. The training can, in particular, be performed upstream of the method according to the invention and thus not be part of the method itself. Thus, a method for training a trained function is disclosed, which serves to implement, within the method according to the invention, the specification of the X-ray-image-dependent model and / or the model parameters and / or the three-dimensional environment model and / or the determination of the environment information. Alternatively, however, it would also be possible to perform the training as an additional upstream process step within the method according to the invention.

[0045] In particular, supervised training, semi-supervised training, unsupervised training, reinforcement learning, and / or active learning can be used. Furthermore, representational learning, also known as feature learning, can be employed. Specifically, the parameters of the machine learning models can be iteratively adjusted through multiple training steps. In particular, a specific cost function can be minimized during training. Specifically, the backpropagation algorithm can be used when training, for example, a neural network.

[0046] A machine learning model can, in particular, include a neural network, a support vector machine, a decision tree, a Bayesian network, and / or a transformer, and / or the machine learning model can be based on k-means clustering, Q-learning, genetic algorithms, and / or association rules. A neural network can, in particular, be a deep neural network, a convolutional neural network, or a convolutional deep neural network. Furthermore, a neural network can be an adversarial network, a deep adversarial network, and / or a generative adversarial network.

[0047] The third-party object can be an implant or a medical device. The environmental information can thus serve, in particular, to assess the correct positioning of an implant after a procedure or the condition of an implant that has been in the patient for some time, and / or to provide medical personnel with information regarding the environment of a medical device, which can be used, for example, during a medical procedure, especially one carried out independently of the method according to the invention.

[0048] As explained in more detail above, the environmental information may relate to material such as calcification and / or gallstone and / or blood and / or blood clot and / or bone cement and / or embolization material used to fill an aneurysm and / or digestive fluid and / or bile.

[0049] In addition to the method according to the invention, the invention relates to a processing device configured for carrying out the computer-implemented method according to the invention. The processing device can, for example, be designed as suitably programmed data processing equipment, or alternatively, the aforementioned functionality can be implemented, at least partially, by hardwiring. The processing device can be integrated into a medical imaging device, in particular an X-ray device, or be designed separately from it. It can, for example, be implemented as a workstation, server, or cloud solution.

[0050] Furthermore, the invention relates to an endoscopy device comprising a third-party object designed for insertion into a patient, an actuator for moving the third-party object, and a processing device according to the invention, wherein the processing device is additionally configured to control the actuator depending on environmental information. The endoscopy device may, in particular, include an X-ray device by which the X-ray image can be provided.

[0051] For example, the actuator could be a motor used to retrieve an endoscopy basket, which constitutes the third object. As explained in detail above, the motor current can be regulated, for instance, in such a way that the force exerted on a solid object, such as a gallstone, within the endoscopy basket is determined as environmental information and regulated to a setpoint by adjusting the motor current.

[0052] The invention also relates to a computer program with instructions designed to carry out the computer-implemented method according to the invention when executed on a data processing device.

[0053] Furthermore, the invention relates to a data carrier that includes the computer program according to the invention.

[0054] Furthermore, the features and details described in the invention, along with the aforementioned advantages, can also be applied to the other subject matter of the invention and vice versa.

[0055] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included.

[0056] Further advantages and details of the invention will become apparent from the following exemplary embodiments and the accompanying drawings. These schematically illustrate: Fig. 1 an embodiment of the endoscopy device according to the invention, comprising an embodiment of the processing device according to the invention, Fig. 2 a flowchart of an embodiment of the method according to the invention, Fig. 3-5 X-ray images in exemplary usage situations of embodiments of the method according to the invention, and Fig. 6. An exemplary structure of a function trained by machine learning.

[0057] Fig. Figure 1 schematically shows an endoscopy device 53 with a third-party object 24 designed for insertion into a patient 42, an actuator 45 for moving the third-party object 24, and a processing device 52. The endoscopy device 53 includes an X-ray device 55 to capture at least one X-ray image 23 by means of an X-ray source 56 and an X-ray detector 57, for example, in the context of fluoroscopy, which in the usage situation shown depicts the third-party object 24 located inside a patient 42.

[0058] X-ray image 23, as will be shown later in the section on Fig. 2, an exemplary embodiment of such processing will be explained in more detail, by the processing device 52 together with sensor information 27, which is based on at least one respective measured value of the sensors 28, 29 of the third object 24, in order to provide environmental information 22 that concerns material 32 in the environment of the third object 24 or an interaction of the third object 24 with this material.

[0059] As will be shown later based on the individual sections in the Fig. As will be explained in the exemplary usage situations shown in Figures 3-5, the environmental information 22 can be used, for example, to output a superimposed display 62 of the X-ray image 23 with the environmental information 22 or information derived therefrom to a user, for example via the display unit 63. For example, a specific material or a detected object made of that material can be highlighted in color. Additionally or alternatively, the processing unit 52 can control the actuator 45 depending on the environmental information 22. This can be used, for example, to optimally position a third-party object 24-26, such as a stent, or to control the retrieval of a third-party object 24-26, such as an endoscopy basket.

[0060] In this example, the method is implemented by a computer program 59, which is stored in a memory 58 of a data processing device 54. The method is executed by the processor 60 of the data processing device 54 through the execution of the instructions of the computer program 59; in other words, the processing device 52 is implemented through this programming of the data processing device 54.

[0061] An exemplary design of a procedure for determining the environmental information 22 is described below with reference to the one in Fig. Flowchart 2 is shown and explained. In this context, reference is also made to the information in the Fig. Figures 3-5, each represented as a schematic X-ray image, illustrate exemplary application situations of this method. The three application situations shown are purely exemplary, and the method can, in principle, be used for a multitude of other applications, as already discussed in the general section of the description.

[0062] In the example, at least one X-ray image 23 is initially acquired in step S1, which depicts at least a section of a third object 24-26 located inside a patient 42. In this example, according to... Fig. 3 an endoscopy basket as a third item 24, in the example according to Fig. 4 a stent as a third-party item 25 and in the example according to Fig. Figure 5 shows an endoscope 26 for introducing an embolizing material 47, in this example a platinum coil, as a third item 26. The further steps are initially described with a focus on the example according to Fig. 3 explains that there are some deviations and additions for the further examples according to Fig. 4 and Fig. 5 will be addressed.

[0063] In the Fig. In the example shown, the system is designed to detect, firstly, whether a solid object 43 to be removed from the body, in this example a gallstone 51, has been successfully picked up by the endoscopy basket 24. Secondly, if this is the case, the retraction of the third object 24 by the actuator 45 is to be controlled by a suitable rotor current setting.

[0064] In step S2, sensor information 27 is determined for this purpose, which, in the Fig. Case 3 shown is based on at least one respective measured value from sensors 28, 29 of the third object 24. In this example, sensor 29 is an inductance sensor and sensor 28 is a force sensor integrated into a control wire.

[0065] In step S3, a three-dimensional environment model 48 of the respective environment of the respective third object 24-26 is parameterized based on the X-ray image 23, describing at least the cavity 41 of the patient 42 that contains the third object 24-26. This can be achieved, for example, by elastically registering the X-ray image to an anatomical atlas or to previously acquired three-dimensional image data. Known properties of the depicted biological material, such as the known elasticity of tissue, can also be taken into account. Similar to the determination of the X-ray-image-dependent model of the third object described below, the model parameters can also be determined by a trained function, which is trained, for example, using training datasets based on three-dimensional imaging processes.

[0066] To reduce the complexity of the procedure, it would also be possible, for example, to segment only the cavity 41 of the patient 42 that receives the third object 24-26 in step S3. In principle, it would also be possible to completely disregard the surrounding area shown in the X-ray image 23.

[0067] In step S4, model parameters 35 of a three-dimensional model 36 describing the third object 24 are then determined based on the X-ray image 23 and, in the example, additionally based on the sensor information 27 and the environment model 48, in order to provide an X-ray-image-dependent model 37. Although a trained function 49 is used for this purpose in the example shown, a possible implementation of which will be discussed later with reference to Fig. In section 6, to better understand the relationships used or identified here, a different, typically more computationally intensive approach to determining these model parameters 35 will first be explained.

[0068] To determine the model parameters, a predefined list of features of the third-party object 24, potentially recognizable in the X-ray image, can be provided. Each of these features is assigned a model position in the three-dimensional model, which depends on the model parameters. Suitable features include, for example, the ends of the wires 64 of the endoscope basket, points of maximum and minimum curvature, and points that are maximally affected by a center line of the guide element 65. Each model position is assigned a corresponding image position in the X-ray image 23 by means of a known imaging geometry of the respective X-ray image 23.

[0069] By feature recognition, at least a subset of the predefined features in the X-ray image 23 can be identified, for example, by edge detection, by using scale-invariant features, or by a trained algorithm. This allows at least some of these features to be recognized in the X-ray image 23 and their actual positions within the image to be determined. Subsequently, the model parameters can be determined by minimizing a cost function that depends on the respective distance between the image position of the feature and its actual position.

[0070] As already explained in the general section, an additional measure for the similarity of the shapes of parts of the third object 24 found in the X-ray image, for example the wires 64, to the shape that results from a forward projection of the corresponding part of the model into the image plane of the X-ray image can be taken into account in the cost function.

[0071] For example, by using the finite element method as a calculation procedure, a relationship between the force acting on the endoscope basket, detected by sensor 28, and the resulting shape of the endoscope basket can be determined. Therefore, depending on the model parameters 35, model information can be determined that corresponds to a value of sensor information 27, or of the partial information of sensor information 27 relating to sensor 28, that would be expected for given values ​​of the model parameters 35. By using a cost function that additionally depends on a measure of the distance of the respective model information from sensor information 27 or the partial information, a more precise parameterization of the three-dimensional model can be achieved, and ambiguities that could arise from considering only the X-ray image 23 can be eliminated.

[0072] Additionally, the inductance determined by sensor 29 can also be taken into account when determining the model parameters 35. Based on the inductance determined as part of the sensor information 27, it can be determined whether a solid body 43, in particular a gallstone 51, is located within the cavity 38 of the endoscope basket, the presence of which would exert additional forces on the third object 24.

[0073] Based on the three-dimensional environment model 48, an additional boundary condition is specified in the example for the X-ray-dependent model 37 of the third object, namely that the third object 24-26 is located within the cavity 41 of the patient 42. A similar boundary condition could also be obtained if a two-dimensional segmentation of the cavity 41 was performed in step S3. In this case, for example, it could be required that a forward projection of the X-ray-dependent model 37 of the third object, according to the known imaging geometry of the X-ray image 23, is located within the segmented cavity 41.

[0074] Since the described procedure can be computationally intensive, particularly due to the use of the finite element method in optimization, it is often expedient to use a simpler model of the third subject 24 instead, as already explained in the general section, or as schematically shown in Fig. As shown in Figure 2, instead of using the method in the field, a trained function 49 is used to determine the model parameters 35. A neural network is used as an example of the trained function 49, the underlying principle of which will be explained later with reference to... Fig. Section 6 will be explained in more detail. For example, the X-ray image can first be processed by folding layers, whereby the result of this preprocessing can be fed together with the sensor information to one or more fully bonded layers.

[0075] In this example, the trained function 49 is based on supervised learning with error feedback, also known as backpropagation, using the gradient descent method. The training datasets used can include the respective X-ray image and sensor information as input data, and the model parameters as target output data. In principle, the model parameters can be specified by medical professionals who analyze the input data, taking into account additional information, such as information from three-dimensional imaging of the third object 24-26.However, it is also possible that at least parts of the target output data are determined from the input data using the robust but relatively computationally intensive approach explained above, so that the trained function 49 learns to provide similarly good results as the approach explained above with potentially significantly less computational effort.

[0076] In step S5, the environmental information 22 is determined based on the X-ray image-dependent model 37 and the sensor information 27. In the Fig. In the example shown, the determined environmental information can, in the simplest case, describe whether a specific solid 43, e.g., a gallstone 51, is present in the cavity 38 of the third object 24, i.e., within the endoscopy basket. Since the current shape of the endoscopy basket is already known from the X-ray-dependent model 37, it is sufficient to distinguish between a filling of the cavity with digestive fluid or bile and at least partial displacement of the digestive fluid or bile from the cavity 38 by the gallstone 51. Since these states lead to noticeably different inductances, the detection of a gallstone 51 in the endoscopy basket can, for example, depend on whether the inductance detected by the sensor 29 exceeds a limit value specified by the X-ray-dependent model 37.

[0077] Since the volume and shape of the cavity 38 of the third object 24 are predetermined by the X-ray-dependent model 37, the volume of the gallstone 51 can also be estimated based on the measured inductance, or more generally, how much material 32-34 of at least one predetermined material type, for example, gallstone material or digestive fluid, is present in the cavity 38 of the third object 24-26. Additionally, the surrounding model 48 or a segmentation of the gallstone 51 in the X-ray image 23 can optionally be used to estimate its dimensions in the image plane and its position.

[0078] Based on this information and the determined shape of the third object 24 and the force determined by means of the sensor 28, the force currently acting on the gallstone 51 can also be estimated.

[0079] As shown above, the force acting on the gallstone and its determined size, as determined in the example, depend on a multitude of measured variables and model parameters. In principle, the relationships between the various input variables and the respective output variables can be determined experimentally, for example, through preliminary tests conducted outside the body, in which three-dimensional X-ray data can also be used as a supplementary source of information. The results of these preliminary tests can be analyzed, for example, by regression analysis to provide an analytical relationship for calculating the respective output variable from the input variables. However, such relationships can often be determined with fewer input data points or preliminary tests if machine learning is used.Therefore, in the example, a trained function 50 is used to determine the environment information 22.

[0080] The trained function 50, like the trained function 49 described above, can be implemented, for example, by a neural network trained through supervised training using training datasets, such as a gradient descent method. The preliminary tests described above can be performed to provide training datasets.

[0081] The determined environmental information 22 is used in step S6 in the example according to Fig. 3 used to generate a control signal 44 for the schematically in Fig. The actuator 45, as shown in Figure 1, is used to move the third object 24, namely to retrieve the endoscope basket after picking up the gallstone 51. In this example, the control signal 44 specifies a motor current for the actuator 45. The motor current is adjusted such that the force acting on the gallstone 51, determined as environmental information 22, is regulated to a setpoint value. This ensures both the rapid retrieval of the endoscope basket and prevents it from breaking due to excessive force. If steps S1-S6 are repeated, for example, during fluoroscopy during surgery, the motor current can be regulated quasi-continuously.

[0082] In step S7, a superimposed representation 62 of the X-ray image 23 and the dimensions of the gallstone 51, determined as environmental information 22, is generated, which can be output to a user via the display device 63. It is also possible, of course, that in a variation of the procedure, only the control of the actuator 45 according to step S6 or the output of information to a user according to step S7 takes place, depending on the environmental situation 20.

[0083] The fundamental point, with regard to Fig. The procedure described in section 2 can also be used in a variety of other application situations to determine relevant environmental information 22 regarding the environment of a respective third-party object 24-26. X-ray images 23 for further application situations are shown schematically in the Fig. 4 and Fig. 5 shown.

[0084] In Fig. 4 is determined as environmental information 22 for a stent as a third-party object 25, specifying in which area of ​​an outer surface 39 of the stent it contacts an inner surface 40 of the cavity 41 of the patient 42, i.e., a blood vessel, which contains the third-party object 35. In the example shown, it is already apparent from the X-ray image 23 that a free space 33 remains between the stent and the inner surface 40 in a section of the stent. However, based on the X-ray image or a parameterization of the environmental model 48 or the X-ray-dependent model 37 of the third-party object 25 based solely on the X-ray image, it would not be clearly apparent to what extent this free space 33 extends into the periphery of the stent.

[0085] Therefore, additional sensor information from sensor 30 is evaluated. Sensor 30 may be an impedance sensor. Since blood and tissue in the vicinity of sensor 30 lead to significantly different detected impedances, the proposed combined use of the X-ray-dependent model 37 and the sensor information 27 also allows information about the extent of the free space 33 outside the image plane to be obtained.

[0086] Alternatively, it would also be possible, for example, to use sensor 30 in Fig. 4. To use a force sensor that detects forces acting on different segments in the circumferential direction of the stent, for example. This makes it possible to distinguish whether a respective circumferential segment of the stent is in contact with the vessel wall or borders the free space 39.

[0087] By detecting forces, for example, a surface structure of the inner surface 40 of the cavity 41 of the patient 42 can also be determined as environmental information 22. For example, the in Fig. 4 schematically represented tumor 61 have a different tissue elasticity than the surrounding tissue, so that in that section of the third object 25 which borders the tumor 61, different forces from other areas act on the outer surface 39 of the third object 25.

[0088] In the Fig. In the example shown in Figure 5, an endoscope for introducing an embolizing material 47, in this example a platinum coil, is depicted as a third object 26. Based on the X-ray image 23, or the environmental model 48 derived from it, and the X-ray-dependent model 37 of the third object 26, it can be seen that the endoscope is positioned in the region of the aneurysm 46 and that embolizing material 47 has already been introduced into it. However, due to the scattering and absorption behavior of the metal used as embolizing material 47, it is often not possible to determine from the X-ray image whether a sufficient degree of filling of the aneurysm 46 has already been achieved. In addition to the x-ray-dependent model 37 of the third object 26, at least one measured value of the sensor 31, which in the example is an impedance sensor, can thus be used as sensor information to determine the degree of filling of the aneurysm 46 as environmental information 22.

[0089] Since the inductance measured by the sensor 31 depends strongly on the distance of the sensor 31 to the aneurysm and the accumulation of embolizing material 47 contained therein, it is also necessary to take into account the position of the sensor 31 known from the x-ray-dependent model 37 in order to determine robust environmental information 22.

[0090] As explained above, the trained functions 49 and 50 can be used in the Fig. The example shown in point 2 can be implemented using neural networks. For the sake of simplicity, the properties of such a neural network will be briefly described below using a very simple example with reference to Fig. Section 6 explains that in real-world implementations, significantly larger numbers of input nodes, output nodes, and layers can be used. English terms for artificial neural network 1 are "artificial neural network", "neural network", "artificial neural net", or "neural net".

[0091] Artificial neural network 1 comprises nodes 6 to 18 and edges 19 to 21, where each edge 19 to 21 is a directed connection from a first node 6 to 18 to a second node 6 to 18. Generally, the first node 6 to 18 and the second node 6 to 18 are distinct nodes 6 to 18; however, it is also conceivable that the first node 6 to 18 and the second node 6 to 18 are identical. For example, in Fig.1. Edge 19 is a directed connection from node 6 to node 9, and edge 21 is a directed connection from node 16 to node 18. An edge 19 to 21 from a first node 6 to 18 to a second node 6 to 18 is called an incoming edge for the second node 6 to 18 and an outgoing edge for the first node 6 to 18.

[0092] In this embodiment, the nodes 6 to 18 of the artificial neural network 1 can be arranged in layers 2 to 5, wherein the layers can have an intrinsic order introduced by the edges 19 to 21 between the nodes 6 to 18. In particular, edges 19 to 21 can only be provided between adjacent layers of nodes 6 to 18. In the illustrated embodiment, there is an input layer 2 that contains only nodes 6, 7, and 8, each without an incoming edge. The output layer 5 comprises only nodes 17 and 18, each without outgoing edges, with hidden layers 3 and 4 further situated between the input layer 2 and the output layer 5. In the general case, the number of hidden layers 3 and 4 can be chosen arbitrarily.The number of nodes 6, 7, 8 in input layer 2 usually corresponds to the number of input values ​​into neural network 1, and the number of nodes 17, 18 in output layer 5 usually corresponds to the number of output values ​​of neural network 1.

[0093] In particular, a (real) number can be assigned to nodes 6 to 18 of neural network 1. Here, x denotes... (n) i The value of the i-th node 6 to 18 of the n-th layer 2 to 5. The values ​​of nodes 6, 7, 8 of input layer 2 are equivalent to the input values ​​of neural network 1, while the values ​​of nodes 17, 18 of output layer 5 are equivalent to the output values ​​of neural network 1. Furthermore, each edge 19, 20, 21 can be assigned a weight in the form of a real number. In particular, the weight is a real number in the interval [-1, 1] or in the interval [0, 1, ]. Here, w denotes (m,n) i,jthe weight of the edge between the i-th nodes 6 to 18 of the m-th layer 2 to 5 and the j-th nodes 6 to 18 of the n-th layer 2 to 5. Furthermore, the abbreviation wi,j(n) for the weight wi,j(n,n+1) defined.

[0094] To calculate the output values ​​of neural network 1, the input values ​​are propagated through neural network 1. In particular, the values ​​of nodes 6 to 18 of the (n+1)th layer 2 to 5 can be calculated based on the values ​​of nodes 6 to 18 of the nth layer 2 to 5 by xj(n+1)=f(∑ixi(n)⋅wi,j(n)).

[0095] Here, f is a transfer function, which can also be called an activation function. Well-known transfer functions include step functions, sigmoid functions (for example, the logistic function, the generalized logistic function, the hyperbolic tangent, the arctangent, the error function, the smoothstep function), and rectifier functions. The transfer function is primarily used for normalization purposes.

[0096] Specifically, the values ​​are propagated layer by layer through neural network 1, with values ​​of input layer 2 being given by the input data of neural network 1. Values ​​of the first hidden layer 3 can be calculated based on the values ​​of input layer 2 of neural network 1, values ​​of the second hidden layer 4 can be calculated based on the values ​​in the first hidden layer 3, and so on.

[0097] To determine the values wi,j(n) To be able to define the parameters for edges 19 to 21, neural network 1 must be trained using training data. Specifically, training data includes training input data and training output data, which are referred to below as t. i The training step involves applying neural network 1 to the training input data to determine the calculated output data. Specifically, the training output data and the calculated output data comprise a number of values, where this number is determined by the number of nodes 17 and 18 in output layer 5.

[0098] In particular, a comparison between the calculated output data and the training output data is used to recursively adjust the weights within neural network 1 (backpropagation algorithm). Specifically, the weights can be adjusted accordingly. w'i,j(n)=wi,j(n)−y⋅δj(n)⋅xi(n) to be changed, where γ is a learning rate and the numbers δj(n) can be calculated recursively as δj(n)=(∑kδk(n+1)⋅wj,k(n+1))⋅f'(∑ixi(n)⋅wi,j(n)) based on δj(n+1) if the (n+1)th layer is not the starting layer 5, and δj(n)=(xk(n+1)−tj(n+1))⋅f'(∑ixi(n)⋅wi,j(n)) if the (n+1)th layer is the output layer 5, where f' is the first derivative of the activation function and yj(n+1) the comparison training value for the j-th node 17, 18 of the initial layer 5 is. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Zitierte Nicht-Patentliteratur

[0000] Jäckle S et.al. Instrument localisation for endovascular aneurysm repair: comparison of two methods based on tracking systems or using imaging. Int J Med Robot. 2021; 17(6):e2327. https: / / doi.org / 10.1 002 / rcs.2327

[0003] Shetty, Karthik, et al. „BOSS: Bones, organs and skin shape model.“ Computers in Biology and Medicine 165 (2023

[0040]

Claims

[1] A computer-implemented method for determining environmental information (22) based on at least one two-dimensional X-ray image (23) depicting at least a section of a third object (24-26) located inside a patient (42) and at least one sensor information (27) based on at least one measurement value from at least one sensor (28-31) of the third object (24-26), wherein the environmental information (22) relates to material (32-34) in the vicinity of the third object (24-26) and / or an interaction of the third object (24-26) with this material (32-34), comprising the steps: - Receiving the X-ray image (23) and the sensor information (27), - Determining model parameters (35) or restricting the possible parameter values ​​of the model parameters (35) of a three-dimensional model (36) of the third object (24-26) depending on the X-ray image (23) to specify an X-ray-dependent model (37), wherein the three-dimensional model (36) describes a three-dimensional shape and / or pose of the third object (24-26) depending on the model parameters (35), - Determining the environmental information (22) as a function of the X-ray image-dependent model (37), wherein on the one hand the environmental information (22) and / or on the other hand the X-ray image-dependent model (37) additionally depend on the sensor information (27), and - Providing the environment information (22). [2] Computer-implemented method according to claim 1, characterized by , that the third object (24-26) is deformable, wherein at least one of the model parameters (35) describes the deformation. [3] Computer-implemented method according to claim 1 or 2, characterized by , that the environmental information (22) determines whether and / or how much material (32-34) of at least one specified material type is present in a cavity (38) of the third object (24-26), and / or whether and / or in which area part of an outer surface (39) of the third object (24-26) it contacts an inner surface (40) of a cavity (41) of the patient (42) accommodating the third object (24-26), and / or that the environmental information (22) concerns a surface structure of the inner surface (40) of the cavity (41) of the patient (42). [4] Computer-implemented method according to any one of the preceding claims, characterized by , that as environment information (22) or as processing information determined on the basis of the environment information (22) - the presence of a solid (43) in a cavity (38) of the third object (24-26) and / or - a control signal (44) for an actuator (45) to move the third object (24-26) and / or - the degree of filling of an aneurysm (46) is determined with an embolization material (47). [5] Computer-implemented method according to any one of the preceding claims, characterized by , that the sensor (28-31) or at least one of the sensors (28-31) on whose or whose measured values ​​the sensor information (27) is based is a force sensor and / or a pressure sensor and / or a deformation sensor and / or an electrical sensor for detecting an electrical resistance and / or an electrical impedance and / or an electrical potential. [6] Computer-implemented method according to any one of the preceding claims, characterized by, that in the X-ray image (23) a cavity (41) of the patient (42) receiving the third object (24-26) is segmented and / or based on the X-ray image (23) a three-dimensional environmental model (48) of the environment of the third object (24-26) is parameterized depending on the X-ray image (23), wherein on the one hand the determination of the model parameters (35) of the three-dimensional model (36) of the third object (24-26) or the restriction of the possible parameter values ​​of these model parameters (35) and / or on the other hand the determination of the environmental information (22) additionally depending on the segmentation of the cavity (41) of the patient (42) and / or the environmental model (48) is carried out. [7] Computer-implemented method according to any one of the preceding claims, characterized by, that several features of the third object (24-26) are specified, wherein the model positions of the features in the three-dimensional model (36) depend on the model parameters (35), wherein the respective model position is assigned an image position in the X-ray image (23) by a known imaging geometry of the respective X-ray image (23), wherein at least a subgroup of the specified features in the X-ray image (23) is recognized by feature recognition and their actual position in the X-ray image (23) is determined, wherein the model parameters (35) are determined by optimizing a cost function that depends on a respective distance of the image position of the respective feature from the actual position of this feature. [8] Computer-implemented method according to any one of the preceding claims, characterized by, that the three-dimensional model (36) specifies positions of several model points in three-dimensional space depending on the model parameters (35), wherein the three-dimensional model (36) or a specified calculation rule specifies at least one piece of model information depending on the relative positions of at least a subgroup of the model points, wherein the model parameters (35) are determined by optimizing the or a cost function which depends on a measure of the distance of the respective model information from the sensor information (27) or a respective part of the sensor information (27). [9] Computer-implemented method according to any one of the preceding claims, characterized by, that the specification of the x-ray image-dependent model (37) and / or the model parameters (35) and / or the three-dimensional environment model (48) and / or the determination of the environment information (22) is carried out by a respective machine learning-trained function (49, 50). [10] Computer-implemented method according to any one of the preceding claims, characterized by , that the third item (24-26) is an implant or a medical device. [11] Computer-implemented method according to any one of the preceding claims, characterized by , that the environmental information (22) relates as material a calcification and / or a gallstone (51) and / or blood and / or a blood clot and / or bone cement and / or an embolization material used to fill an aneurysm (47) and / or digestive fluid and / or bile. [12] Processing facility, characterized bythat it is equipped to carry out the computer-implemented method according to one of the preceding claims. [13] Endoscopy device comprising a third object (24-27) designed for insertion into a patient (42), an actuator (45) for moving the third object (24-27) and a processing device (52) according to claim 12, wherein the processing device (52) is additionally configured to control the actuator (45) depending on the environmental information (22). [14] Computer program with instructions configured to perform the computer-implemented method according to any one of claims 1 to 11 when executed on a data processing device (54). [15] Data storage devices, characterized by that it comprises the computer program (42) according to claim 14.

Images