Robotic system

Abstract

The invention is directed to a computing system (1) comprising an image input (117) configured to receive live images (IM) of a region of interest (51) of a patient from an imaging device (3) and a storage (14) configured to store data specific to this patient of at least a valve of the patient's heart,0 and different to the live images (IM). The computing system (1) further comprises an evaluation unit (111) configured to evaluate the patient-specific data, and configured to suggest a valve repair action (VRA) based on the evaluation, and a mapping unit (112) configured to map the suggested valve repair action (VRA) to one or more of the live images (IM) received via the imaging input (117) by way of visualizing the suggested valve repair action (VRA) in the one or more live images (IM). An image output (118) is configured to supply the one or more live images (IM-VRA) including the visualized suggested valve repair action.

Description

[0001] P192941PC00 1

[0002] ROBOTIC SYSTEM

[0003] TECHNICAL FIELD

[0004] The invention is directed to a robotic system, a computing system, and a computer- implemented method.

[0005] BACKGROUND

[0006] The human heart is a muscle with the function of a pump for pumping blood through the body. The human heart comprises four heart valves for controlling the flow of blood. The valves enable the blood flow to run through the heart in the right direction and at the right time. The valves are denoted as aortic valve, mitral valve, pulmonary or pulmonic valve and tricuspid valve.

[0007] The valves reside between the various chambers of the heart. The four chambers are denoted as the right atrium, the left atrium, the right ventricle and the left ventricle. Specifically, oxygen suffering blood is received from the venous blood system by the right atrium. It flows through the tricuspid valve into the right ventricle. The tricuspid valve, hence, is considered as inlet valve. Then, the blood is pumped by the right ventricle through the pulmonary valve into the lungs. The pulmonary valve, hence, is considered as outlet valve. In the lungs, the blood receives oxygen which oxygen rich blood then flows back to the heart and reaches the left atrium. From the left atrium, the blood flows through the mitral valve into the left ventricle. The mitral valve, hence, is considered as inlet valve. The blood is then pumped by the left ventricle through the aortic valve into the artery system of the body. The aortic valve, hence, is considered as outlet valve.

[0008] In general, the heart valves may suffer from certain defects. Such defects may include atresia, i.e., a heart valve is missing, stenosis, i.e., the valve gets stiff, or most common, regurgitation also referred to as insufficiency or leaking of the valve. In the case of insufficiency, the valve no longer fully closes such that portions of the blood stream back in form of a reverse or backward flow.

[0009] Insufficiency of the tricuspid valve is most often caused by other diseases of the body such as pulmonary hypertension. Hence, such tricuspid valve insufficiency is also referred to as secondary insufficiency given that it is caused by a primary disease instead of a P192941PC00 2 disease of the pulmonary valve itself. A standard approach therefore is to treat the primary disease, if possible. Insufficiency of the pulmonary valve again is most often caused by other diseases of the body such as infective endocarditis, pulmonary hypertension or rheumatic heart disease. Hence, such pulmonary valve insufficiency is also referred to as secondary insufficiency given that it is caused by a primary disease instead of a disease of the pulmonary valve itself. A standard approach therefor is to treat the primary disease, if possible.

[0010] Insufficiency of the mitral valve, however, often is not caused by any other primary disease but is caused by a defect of the mitral valve itself. Accordingly, the mitral valve may be repaired for which repair a set of repair actions are established, including replacement, repair, or other means. However, as of today it depends on the cardiologist or surgeon's skills to assess and propose the best treatment.

[0011] Hence, it is a desire to provide an improved basis for deciding on and / or executing heart valve reshaping actions, preferably for the mitral valve, but not limited to.

[0012] GENERAL DESCRIPTION OF THE INVENTION

[0013] According to a first aspect of the present invention, a computing system is provided. The computing system comprises an image input configured to receive live images of a region of interest of a patient. Such live images may be supplied from an imaging device. The image input of the computing system is preferably connected, e.g., wirebound or wireless, to the imaging device. The imaging device preferably is a 2D-camera or 3D-camera and provides 2D-images or 3D-images in the visual range or an echocardiogram. In an embodiment, the imaging device is an ultrasonic device providing ultrasonic images. In an embodiment, the imaging device is an endoscope. In an embodiment, the imaging device is a Transesophageal Echocardiography (TEE) transducer that is attached to a thin tube and passes through the mouth, down the throat and into the esophagus. In a different embodiment, the imaging device is a Transthoracic Echocardiography (TTE) in which the transducer is applied to the outside of the patient's body. In an embodiment, the imaging device is a computer tomograph, preferably low dose. In an embodiment, the imaging device includes a combination of any of the above imaging devices. In a very preferred embodiment, the imaging device includes an ultrasonic transducer and a computer tomograph, preferably low dose, providing images derived from both of these imaging components. P192941PC00 3

[0014] In an embodiment, the images received are transformed into 3D images. Hence, the live images preferably enable a surgeon to capture the third dimension of the field of interest which supports the surgeon performing surgical steps, in case of the live images being provided during surgery.

[0015] The image input of the computing system preferably implements an interface defining the image transfer from the imaging device to the computing system, such as an USB or HDMI interface, etc. Accordingly, it is desired that images are received by the computing system, preferably live images before surgery, also preferably during surgery. The imaging devices applied may be different between pre operation imaging and intra operation imaging. Live images may be images from a video stream, or may be still images extracted from a video stream, or may be still images taken by the imaging device. Any images of the region of interest received from the imaging device during the operation shall be considered as live images, even if these images are not real-time, and are e.g., buffered and only processed later on. Accordingly, the term "live" shall refer to any point in time of the session the camera / endoscope is in use in supplying images from the region of interest.

[0016] The region of interest to be imaged is preferably a portion of a patient’ s heart. Subject to the way the arrangement of the imaging device - e.g., in-situ / inside the body in case of an endoscope or TEE, or from the outside of the body in case of sternotomy or TTE - the section displayed may vary. In general, it is preferred that the imaging device supplies images at least including the region of interest, which region of interest may be identified upfront by medical personnel for a possible treatment or surgery. The region of interest comprises at least one of the heart valves, preferably the mitral valve, and preferably the adjacent heart chambers, in case of the mitral valve the left atrium and the left ventricle.

[0017] The computing system also comprises a storage configured to store data specific to the very patient the live images are transmitted from, which data refer to the region of interest of this patient, i.e., at least to a valve of this patient's heart. This patient-specific data preferably comprises patient-specific images of the region of interest, i.e., at least from the valve of the patient's heart under consideration. The patient-specific images are preferably images taken earlier in time, preferably pre-op, and preferably include echocardiogram images taken from the patient by an echocardiograph taken at an earlier session. However, patientspecific images may in addition or alternatively comprise cardiac computed tomography images provided by a computer tomograph. In addition to any of the previous kinds of images, or alternatively, the patient-specific images comprise cardiac computed magneto resonance images (MRI). The patient-specific images are not the live images received form the imaging P192941PC00 4 device, but are images different from the live-images, and preferably are taken at a different point in time, i.e., earlier in time than the live images, in other words not at the point in time of operating the imaging device. The patient-specific images may be taken by the same or a different imaging device, working according to the same or a different a imaging mechanism.

[0018] The patient-specific data, including any derivatives from, and in particular the patient-specific images, are evaluated by an evaluation unit. The evaluation unit finally suggests a valve repair action based on the evaluation of the patient-specific data. The evaluation is performed computer-implemented. The evaluation preferably is conducted prior to the patient undergoing surgery, and hence, prior to the imaging device transmitting the live images from the region of interest.

[0019] The computing system further comprises a mapping unit configured to map the suggested valve repair action to one or more of the live images received via the imaging input. The mapping unit may continuously map the valve repair action to the live images received, in case the live images represent a video stream or a sequence of still images. In a different variant, the mapping unit maps the valve repair action to a still image, e.g., a selected image out the images of the video live stream, or to a still image transmitted by the imaging device. The valve repair action is visualized in the one or more live images, preferably by projecting representations of the valve repair action such as one or more of symbols, identifiers, lines, etc. of the valve repair action onto the dedicated location in the live image / s received via the image input. An image output is provided and is configured to supply the images including the visualized suggested valve repair action, e.g., for display on a screen or display, which screen or display preferably is located in a hospital, preferably at a site close to the operating table, or a in a room from which the surgeon controls a robot assisted surgery of the patient, or for a display of an augmented reality device such as AR glasses that projects the field of surgery.

[0020] Accordingly, the present computing system supports deciding on the most appropriate valve repair action, and specifically supports a heart surgeon in her / his job by not only providing advice which valve repair action / s may have best prospects for a specific patient. It also projects the suggested and visualized valve repair action onto the live image / s transmitted from the imaging device, which imaging device is operated in a session preferably after the patient is sedated and after the imaging device is arranged in the patient's body or located relative to the patient's body to monitor the region of interest. In this regard, the computing system effects an improved execution of a valve repair action. P192941PC00 5

[0021] The computing system may be implemented as a distributed computing system, wherein at least two of the functional units of the computing system are located remote from each other. In one example, the evaluation unit may be implemented in the cloud, and / or may execute its task of identifying a preferred valve repair action ahead of the surgery. Instead, the mapping unit is preferably implemented on-site, and executes its task short before or at the time of surgery, subject to the availability of the live images. On-site preferably is defined on the site of a hospital given that the computing system in one embodiment is desired to interact with a robotic system, preferably a surgical robotic system.

[0022] When mapping the preferred valve repair action to the live image / s, the specific region in the region of interest, which region preferably at least includes the valve, may need to be identified in the live image / s first, e.g., by means of a feature identification unit allowing to detect the valve of interest and possibly neighbouring structures such as the neighbouring heart chambers in the live image / s. In case of different imaging mechanisms used between live imaging and taking the patient-specific images, e.g. the first being endoscope images, the second being ultrasonic images, the mapping preferably includes an mapping of features between the two images. In one embodiment, the patient-specific data comprises a patient-specific computing model of at least the concerned valve of the patient's heart, preferably also of the adjacent heart chambers, and in a preferred embodiment of the entire heart. The patient-specific computing model is a virtual model built from patient-specific data such as patient-specific images, preferably taken earlier in time, preferably pre-op, and which patient-specific images preferably include echocardiogram images taken from the patient and its region of interest respectively, e.g., by echocardiographs. In one embodiment, such echocardiographs may be converted into 3D images by means of a software system, e.g. Intelli Space™ by Philips™. Patient-specific images may comprise cardiac computed tomography images provided by a computer tomograph. In addition to any of the previous kinds of images, or alternatively, the patient-specific images comprise cardiac computed magneto resonance images (MRI).

[0023] The patient-specific computing model preferably is built upfront, i.e. prior to the surgery, and may be built elsewhere, e.g. on other servers of the hospital, or in the cloud. Subject to the modelling technique, a single model may provide both anatomic and functional information. In a different embodiment, two models are generated, preferably a patient-specific anatomical model describing the anatomy of at least the region of interest, and a patient-specific functional model preferably complementing the patient-specific anatomical model. The patient-specific functional model preferably models blood flow through the anatomy of the P192941PC00 6 anatomical model, i.e., specifically blood flow at least through the valve, and possibly through at least the adjacent chambers. The patient-specific computing model, hence, models specifically the patient's conditions in the region of interest. In particular, the patient-specific computing model is capable of determining a backflow of blood through the relevant valve in a closed state of the valve.

[0024] In one embodiment, the computational model is used for evaluating valve repair actions possibly to be applied to the patient, and is preferably used to suggest a preferred valve repair action. Hence, different valve repair actions and / or their corresponding impact on the anatomy and functionality can be simulated by the computing model. Accordingly, the evaluation unit includes a simulation function adapted to simulate anatomical modifications in the patient-specific computational model representing valve repair actions and their impacts on one or more measures that are considered as relevant indicator / s for a successful treatment of the patient. In particular such one or more measures includes at least a blood flow through the respective repaired valve of the heart in a backward direction through the valve in a closed state thereof. While a considerable blood flow in reverse direction through the closed valve is considered as indicator of an impaired valve, especially impaired in its closing function, a decrease or even an absence of such back flow of blood in the computing model simulating the valve after the corresponding valve repair action is a suitable indicator of a successful suggested valve repair action. Accordingly, the evaluation unit preferably is configured to, by way of the simulation function, determine the preferred valve repair action for receiving a minimum value of blood flow in a backward direction through the post-op valve in a closed state thereof.

[0025] Preferably, many different simulations are performed by the evaluation unit on the patient-specific computing model, wherein, per simulation, a value is determined for the at least one measure, e.g., the blood flow in reverse direction through the closed valve after surgery. Each different simulation may represent an anatomical modification of the patientspecific computing model, and as such represent a virtual valve repair action and, at its end, the state of the virtually repaired valve. Each different simulation results in a value of the at least one measure, such that the overall result of the different simulations is an array of values, also referred to as index. E.g., the index may indicate the various reverse blood flows in ml / ms per virtual valve repair action. In addition or alternatively, the measure may include one or more of pressure the relevant heart valve is exposed to, or deformation of the annulus, i.e. the ring around the heart valve. P192941PC00 7

[0026] The evaluation unit preferably is configured to determine the preferred valve repair action based on the index, and preferably is configured to determine the valve repair action that corresponds to the lowest value in the index.

[0027] The evaluation unit may include an optimization function adapted to optimize a measure in the patient-specific data, such as in the computational model, which computational model may also be understood and / or implemented as system of equations describing the anatomy and / or functionality of the patient's heart, or at least of the valve.

[0028] Annuloplasty rings are specially designed to help restore the mitral valve to its normal size and shape (the valve is often enlarged or distorted in a diseased state). This ring can be rigid, semi-flexible for flexible for preserving natural flexibility of the native normal mitral valve.

[0029] The value to be optimized may be the at least one measure such as the blood flow in reverse direction when the repaired valve is closed. The optimization function may comprise an algebraic optimization for optimizing the at least one measure under variation of parameters of the equations representing virtual valve repair actions. Specifically, the evaluation unit may be configured to, by way of the optimization function, determine the preferred valve repair action for receiving the minimum value of the blood flow in a backward direction through the post-op valve in a closed state thereof. In case of optimizing multiple measures, corresponding weights may be introduced in the optimization function.

[0030] Valve repair actions include surgical valve repair actions, but may also include non-surgical valve repair actions, such as a medicine treatment actions, or may even also include the advice not to apply any valve repair action and / or the recommendation to amend life style for improving parameters impacting heart valve diseases. A surgical repair action may include one or more of a resection of or incision into the impacted valve, a resection of or incision into parts of the valve, such as a leaflet, or a segment of a leaflet, closing an indentation or cleft in the valve, attaching an artificial valve, or parts of such as leaflet, or a portion of a leaflet, attachment of a previously incised portion of the valve, a folding and subsequent fixation of parts of the valve, such as a leaflet, or a portion thereof, applying annuloplasty, applying neochords, etc. The valve repair action preferably is selected from a list of available valve repair actions, such as the ones described before, preferably stored in the storage. Accordingly, the available valve repair actions may upfront be gathered in a data structure, and may represent a limited set of valve repair actions. However, valve repair actions from such set may also be combined and result in a valve repair action assembled from various entries of such set. In a preferred embodiment, an optimum valve repair action is suggested out P192941PC00 8 of the many possible valve repair actions defined in the set. Accordingly, it is envisaged that the evaluation unit preferably supplies a preferred valve repair action based on the model / s.

[0031] The evaluation unit may, in addition to the valve repair actions determined or selected from the set, preferably provide additional information, e,g, as to locations where to apply surgical steps, e.g. at which locations to start and stop resection cuts or incisions, e.g. at which locations or along which lines to stitch, e.g. at which locations or along which lines to add artificial replacements, etc. Accordingly, a valve repair action as suggested by the evaluation unit may include additional information either supporting a user of the computing system how to specifically perform the suggested valve repair action, or even enabling a robot to perform the valve repair action absent or only with limited manual intervention of the user. In the latter case, it is preferred that the valve repair action is represented or translated into a control language of a robot controller to be described below.

[0032] In one embodiment, the suggested valve repair action may also be visualized and / or displayed in the patient-specific computing model. In case the computing model may be visualized as 3D graphics, for example, cutting patterns, stitch patterns etc. may be visualized in the visualized 3D version, and be output to a display for illustrating this information to the user. Such 3D representation of the patient-specific computational model may also be used, by means of a corresponding user interface, to allow the user to simulate valve repair actions, e.g., by drafting, moving, amending structures in the computational model.

[0033] The mapping unit is configured to map the suggested valve repair action to the images received via the imaging input. Accordingly, the suggested valve repair action is visualized in the images received via the image input. Accordingly, the mapping unit projects the suggested valve repair action onto the live image / s from the region of interest from the patients’ heart. As indicated above in connection with the embodiment in which the suggested valve repair action is projected onto the 3D computational model of the patient, the suggested virtual repair action now is projected on the live image / s of the region of interest. Such visualization may include one or more of indicating a cutting pattern; indicating a stitching pattern; indicating a folding pattern; indicating an implant location, e.g. for an annuloplasty ring; indicating an attachment location, preferably for one or more of neo-chords and an artificial valve. Indicating may include one or more of drafting into the image / s at the correct location, projecting onto the image / s, flagging in and / or tagging to the image / s, adding comments, etc. For this reason, and as indicated above, the valve repair action as such preferably contains information in addition to the bare action, such as the location of the action, e.g. the tool to be used, etc. P192941PC00 9

[0034] In order to map the valve repair action onto the live images supplied from the imaging device, various pre-processing steps may be implemented: In one example, the mapping unit may comprise a feature identification or extraction unit capable to identify elements of the heart such as chambers, muscles, valves, etc. in the live images received from the imaging device. Such feature extraction unit may be required to continuously apply feature extraction given that the imaging device supplying the live images may be moved, turned, etc. Dependent on such a movement of the camera, the structures in the images may be viewed from different angles and / or perspectives and / or distances, and may be re-identified. Accordingly, it is preferred that the feature extraction unit may permanently be applied, and in particular in real-time with respect to the rate of new images arriving, in order to support a preferred real-time mapping of the visualized valve repair action onto the live images received.

[0035] In addition to the pre-processing step described above, another preprocessing step may be applied subject to the format of the suggested valve repair action. E.g. in case the valve repair action indicates locations where to apply cuts or stitches, for example, this location metrics preferably is mapped to the live image: Hence, if, for example, a partial resection in the anterior leaflet of the mitral valve is suggested at location x, the mapping unit is configured to identify the anterior leaflet by feature extraction, and the projects the suggested cut onto the leaflet taking mapping the scale of the live image and the scale of the computational model.

[0036] The image output of the computing system, again e.g. represented by an interface for transmitting images, provides the images received from the imaging device monitoring the region of interest in the patient’s heart in combination with the visualized valve repair action.

[0037] It is noted that the evaluation unit and the mapping unit preferably are functional units represented by software codes which may, however, run on a common processing unit, or on different processing units in case of an implementation of these two functional units at different, and in particular at remote locations. A processing unit may not only refer to a single processor but includes any kind of processing infrastructure such as processors, buffers, servers, etc.

[0038] In a preferred embodiment, the evaluation unit comprises a machine learning model, also referred to as first machine learning model, preferably a neuronal network. The machine learning model preferably is trained by training data sets. The training data sets may include one or more, preferably a combination of:

[0039] - first data sets, preferably first image data sets illustrating pre-op valves of patients; P192941PC00 10

[0040] - second data sets, preferably second image data sets illustrating post-op valves of these patients;

[0041] - associate valve repair actions applied for transforming the respective valves from the pre-op state into the post-op state;

[0042] - one or more associate measures, which may, e.g., include or consist of a reverse blood flow of the valve when the valve is closed, for the post-op valve, preferably measured or modelled at the time the second data sets are taken, and preferably also for the preop valve.

[0043] Preferably, a single training data set for a patient includes all of the above components, i.e., a first image illustrating the patient's pre-op valve, an associate valve repair action, and a second image illustrating the post-op valve of this patient. Such training data sets may be provided for hundreds, thousands or millions of different patients having undergone a valve repair action. In particular, it is desired to include training data sets with many different valve repair actions. As such the machine learning model is capable to learn which pre-op heart valve state was transformed by which kind of valve repair action into which associate post-op heart valve state. Preferably, the second data sets represent images of the post-op valves within a week after the applied valve repair action, preferably within a month after the applied valve repair action. Hence, the second images illustrate an early-stage post-op allowing to assess a success of the surgery. The measure or measures for the post-op state of the valve indicating a success of the treatment, e.g. the reverse blood flow of the valve when the valve is closed and, preferably the blood flow through the valve in an open state of the valve for ensuring good perfusion, may either be live monitored at the patient, and / or may be derived from the second image data sets, if possible, or may be derived from a computing model of the repaired valve, etc.. Preferably, the measure is also provided in the training data set for the pre-op valve, in order to allow the machine learning model to learn which valve repair action applied to which state of valve results in which improvement in the relevant measure. However, the relevant measure for a heart condition may not necessarily be limited to the reverse blood flow, but may include one or more additional measures and / or parameters as indicated above, e.g. one or more of pressure on the valve, deformation of the annulus or annuloplasty ring if any, which can be monitored, measured or otherwise determined in both pre-op and post-op state, and be added to the training data sets, both pre- and post-op. Post-op measures may also include one or more of absence of residual leaks, mobility of the valve, coaptation length, absence of stenosis (gradient), absence of systolic anterior motion (SAM). P192941PC00 11

[0044] In one embodiment, the multiple measures may be combined into an index allowing a quantification of the disease, and, hence, also a quantification of the improvement achieved by the valve repair action.

[0045] Generally, the image data sets provided for training purposes are images taken via imaging techniques for monitoring a heart, such one or more of echocardiograms, CT images, MRI images, visual images.

[0046] In another embodiment, the training data sets may include one or more, preferably a combination of

[0047] - first data sets, preferably first image data sets illustrating pre-op or pre-treatment valves of patients;

[0048] - additional data sets, preferably illustrating pre-op or pre-treatment valves of patients taken after the first data sets, e.g. between 1 month and 3 months after;

[0049] - third data sets, preferably third image data sets illustrating post-op or post-treatment valves of these patients;

[0050] - associate valve repair actions applied for transforming the respective valve from its preop or pre-treatment state into its post-op or post-treatment state;

[0051] - one or more associate measures, which may, e.g., include or consist of a reverse blood flow of the valve when the valve is closed, for the post-op valve, preferably measured or modelled at the time the third data sets are taken, and preferably also for the pre-op valve.

[0052] Preferably, a single training data set for a patient includes all of the above components, i.e. a first image illustrating the patient's pre-treatment pre-op valve, an associate valve repair action, and a third image illustrating the post-op valve of this patient, and the associate at least one measure at both times. Preferably, the third data sets represent images of the post-op valves after at least six months after the applied valve repair action, preferably after at least a year after the applied valve repair action, and, preferably, in addition also 2 years after surgery. All the other variants and embodiments explained above with respect to the first and second data sets and the corresponding valve repair actions applies in the same manner. As such, this machine learning model is capable to learn which pre-op heart valve state was transformed by which kind of valve repair action into which long-term post-op heart valve state. Such combination of training data sets preferably illustrates the long-term transition of the valve in response to the valve repair action.

[0053] In another embodiment, the training data sets may include: first data sets, see above; P192941PC00 12 second data sets, see above; third data sets, see above; and associate valve repair actions, see above; associate at least one measure, see above.

[0054] In such scenario, the machine learning model may indicate which valve repair action is preferred in view of both short-term and long-term improvement of the relevant measures.

[0055] In another embodiment, the above introduced simulation function may be extended by an aging function simulating the impact of time passing by on the repaired valve. Such aging function may also be integrated in the patient-specific computing model.

[0056] According to a further aspect of the present invention, a robotic system is provided for valve repair. The robotic system comprises a computing system according to any of the preceding embodiments, a user interface configured to receive a user's input, and a controller configured to control a robot according to the user's input and / or according to the valve repair action suggested by the computing system. Accordingly, the valve repair action suggested by the evaluation unit of the computing system either is fully or partly converted into instructions to control the robot. And / or, the images provided by the computing system including the visualized valve repair action supports a user or surgeon to at least partly manually control a robot via the user interface to execute the suggested valve repair action at the heart of the patient. In such a scenario, the user interface allows the user to manually control the robot via an input means of the user interface. The input means may include one or more of a pointing device such as a mouse, but more preferably a sensor based input device such as VR or AR glasses, or e.g. a console, e.g. a robot-specific console provided by the robot manufacturer for controlling the robot. In such embodiment, the live images including the visualized valve control action are displayed on the display of the robotic system. It is understood that also the displayed images are considered to belong to the user interface.

[0057] Prior to triggering or controlling the valve repair action supported by means of the robot, the input means may also be used or are configured to support identification of at least the region of interest in the live images received from the image input, i.e. to identify the valve as the region of interest.

[0058] Preferably, the robot comprises a valve reshaping tool configured to support reshaping of a valve of a heart of a human being according to the suggested valve reshaping action, and at least one arm for moving the valve reshaping tool in various directions. The robot P192941PC00 13 may comprise two arms or more and multiple valve reshaping tools if needed. Preferably, the valve reshaping tool is attached to the free end of the arm of the robot. The valve reshaping tool may be removably attached to the arm of the robot, e.g. in order to use the arm with different valve reshaping tools in a sequence. In a different embodiment, the valve reshaping tool is fixedly mounted to the arm of the robot, such that no interchange with other valve reshaping tools during operation is possible. In such scenario, different arms may be equipped with different valve reshaping tools. In a different scenario, the robot does not comprise any arm.

[0059] In one embodiment, it is sufficient to control the arm of the robot, and specifically to control a motion of the arm of the robot, by means of the controller. This is particularly the case if the valve reshaping tool is a passive tool such as a scalpel. Accordingly, the motion of the arm, be it a linear and / or rotational motion of the arm in any of the dimensions is sufficient to move the valve reshaping tool in any desired position by which movement the valve reshaping tool at the same time performs the assigned steps such as cutting tissue. In a different embodiment, however, the valve reshaping tool is an active tool in that it comprises an electrical input by which the valve reshaping tool may itself be controlled to the designated action, such as motion, applying pressure, suction etc. In such embodiment, the controller not only comprises a first output via which control signals are sent to the arm to control its motion, but also a second output configured to send control signals to control the valve reshaping tool.

[0060] The valve repair actions, hence, preferably are implemented by the controller sending control signals, via the first and possibly the second output to the arm and possibly to the valve reshaping tool. The arm preferably is positioned such that the valve reshaping tool is positioned relative to the target region, i.e. is the valve to be repaired. In a next step, the valve repair tool is controlled to perform a valve reshaping action, either by controlling the arm to move correspondingly, or by controlling the valve reshaping tool separate from the arm, or by controlling both the arm and the valve reshaping tools, via supplying corresponding signals to either one of the first output and the second output, or to both.

[0061] Preferably, the imaging device supplies images from the region of interest while the robot being controlled to perform valve repair actions. Preferably, the imaging device is an endoscope connected to the corresponding image input of the controller. The control of the robot can be embodied in different manners. In one embodiment, the live images supplied by the computing system and overlaid with visualized valve control actions may also illustrate at least the tip of the valve reshaping tool already located close to the P192941PC00 14 valve to be repaired. The input means of the user interface, such as a pointing device, may allow to position the valve reshaping tool, such that a movement of the input or pointing device on the screen is translated into a corresponding movement of the valve reshaping tool by the robot arm. In this scenario, the displayed images of the region of interest in combination with the visualized suggested valve repair action support the user in manually controlling the robot, e.g. by following a cutting line visualized in the live image by means of the pointing device. In a different scenario, the robot is fully automatically controlled to perform its actions according to the valve repair action. The valve repair action, for such purpose, is preferably translated into a control sequence for the robot, such that the robot is controlled by way of the corresponding control signals issued via the first and possibly the second output. Such fully automated valve repair action preferably is designed such that the user can interrupt at any time. It is preferred that even in the fully automated valve repair action the surgeon may intervene or modify any actions performed or to be performed by the robot via the user interface, which may be a console or AR glasses.

[0062] Preferably, the valve reshaping tool is embodied as one or more of

[0063] - a tool for adding support means to the valve for the valve to maintain a desired shape;

[0064] - a suction tool;

[0065] - a needle for stitching;

[0066] - a resection or incision tool for resecting incising part of or all of the valve, in particular a cutter;

[0067] - replacement tool for adding a valve replacement.

[0068] Although not desirable, valve replacement is sometimes compulsory because of excessive damage to the valve (endocarditis or calcification of the leaflets). In such cases, the software preferably detects the impossibility of repair and indicate whether replacement is desirable or compulsory. The surgeon or the robot will then replace the valve with a mechanical or biological prosthesis. In the highly unlikely event of the software malfunctioning, the system will switch to manual mode and the surgeon will have to decide for himself.

[0069] In terms of processing capability, it is preferred that the computing system comprises a processing unit implementing the mapping unit and the controller for the robot. This processing unit is preferably implemented at a location of the robot. The evaluation unit may also be implemented in the processing unit at the location of the robot. Or, it may be implemented by a different processing unit installed elsewhere remote from the location of the robot, e.g. in the cloud. P192941PC00 15

[0070] According to a further aspect of the present invention, a computer- implemented method is provided, comprising receiving live images from a region of interest of a patient, receiving patient-specific data of at least a valve of the patient's heart different from the live images, evaluating the patient-specific data, suggesting a valve repair action based on the evaluation, mapping the suggested valve repair action to one or more of the live images received by way of visualizing the suggested valve repair action in the one or more live images, and outputting the one or more images including the visualized suggested valve repair action.

[0071] Preferably, the method comprises one or more of the following steps or elements:

[0072] - the patient-specific data comprises a patient-specific computing model of at least the valve of the patient's heart;

[0073] - the patient-specific computing comprises a model of at least the valve of the patient's heart and the adjacent heart chambers;

[0074] - the valve is the mitral valve of the heart of the patient and the adjacent chambers are left atrium and left ventricle;

[0075] - the patient-specific data comprises patient-specific images including one or more of echocardiogram images, cardiac computed tomography images, cardiac MRI;

[0076] - generating the patient-specific computing model from the patient-specific images;

[0077] - generating a patient-specific anatomical computing model of at least the valve and the adjacent chambers from the patient-specific images;

[0078] - generating the patient specific anatomical model by means of a modelling tool;

[0079] - generating a patient-specific functional computing model of blood flow at least through the valve from the patient-specific images;

[0080] - generating the patient-specific functional computing model of blood flow at least through the valve from the patient-specific anatomical model;

[0081] - generating the patient-specific functional computing model by means of another modelling tool;

[0082] - simulating anatomical modifications in the patient-specific computational model representing valve repair actions, determining values of at least one measure per simulation, and determining the suggested valve repair based on the values of the at least one measure; P192941PC00 16

[0083] - optimizing at least one measure in the patient-specific computational model subject to configurations of the patient-specific computational model representing valve repair actions, and determining the suggested valve repair action as a result of the optimization process;

[0084] - the at least one measure includes blood flow in a backward direction through the valve in a closed state thereof;

[0085] - the at least one measure of blood flow in backward direction through the valve in a closed state thereof is extracted from the patient-specific computational model and its simulated states or configurations respectively;

[0086] - determining the suggested valve repair action for receiving the minimum value of the blood flow in a backward direction through the post-op valve in a closed state thereof;

[0087] - evaluating the patient-specific data by means of a machine learning model trained by image data sets illustrating at least the respective valve in one or more of echocardiogram images, cardiac computed tomography images, and cardiac MRI;

[0088] - the machine learning model being trained by first data sets, preferably by first image data sets illustrating pre-op valves of a multitude of patients, and by second data sets, preferably by second image data sets illustrating post-op valves of these patients, and by data representing associate valve repair actions applied for transforming the respective patient's valve from its pre-op state into its post-op state;

[0089] - the second data sets illustrate the post-op valves within a week after the applied valve repair action, preferably within a month after the applied valve repair action;

[0090] - the patient-specific data comprises pre-op patient-specific images of at least the valve of the patient, inputting the pre-op patient-specific images into the machine learning model, and in response to the input the machine learning model outputting the suggested valve repair action;

[0091] - the pre-op patient-specific images are taken prior to the live images;

[0092] - the pre-op patient-specific images including one or more of echocardiogram images, cardiac computed tomography images, and cardiac MRI;

[0093] - the post-op patient-specific images including one or more of echocardiogram images, cardiac computed tomography images, and cardiac MRI;

[0094] - the patient-specific data including one or more of the patient's age, sex, weight, height, vital parameters including one or more of blood pressure, lab parameters including blood composition, and / or diseases including one or more of diabetes, psychiatric diseases; P192941PC00 17

[0095] - providing a data set containing a limited number of valve repair actions from which the suggested valve repair action is determined;

[0096] - identifying features in the live images received from the image input, and mapping the visualized suggested valve repair action to the features identified in the one or more live images, preferably by one or more of: o projecting locations of surgical actions into the images; o indicating a cutting pattern; o indicating a stitching pattern; o indicating a folding pattern; o indicating an implant location, preferably an annuloplasty ring; o indicating an attachment location, preferably for one or more of neo-chords and an artificial valve.

[0097] According to a further aspect of the present invention, a computer program element, preferably computer program product, is provided comprising computer readable code means configured to, when executed by a processing unit, perform the steps of a method according to any one of the preceding claims embodiments.

[0098] BRIEF DESCRIPTION OF THE DRAWINGS

[0099] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings. The illustrations are for clarity in facilitating one skilled in the art in understanding the invention in conjunction with the detailed description. In the drawings:

[0100] FIG. 1 illustrates is a block diagram of a robotic system according to an embodiment of the present invention;

[0101] FIGs. 2 to 4 each illustrate a block diagram of a computing system according to different embodiments of the present invention;

[0102] Figure 5 illustrates an example of a live-image in combination with a visualized suggested valve repair action, as output by one of a computing system, a robot system and a computer- implemented method according to embodiments of the present invention. P192941PC00 18

[0103] DETAILED DESCRIPTION OF THE DRAWINGS

[0104] FIG. 1 illustrates a block diagram of a robotic system according to an embodiment of the present invention. The robotic system is envisaged to be used in heart surgery, and preferably in heart valve surgery. It may be applicable in both, minimally invasive heart surgery, or in invasive heat surgery requiring median sternotomy.

[0105] The robotic system comprises a robot 2 and a computing system 1. The computing system 1 comprises a controller 113 configured to control the robot 2. The robot 2 comprises an arm 21 rotatable around one or more joints. The arm 21 is controlled in its motion by way of a robot control signal RC supplied at a first output 114 of the controller 113, which first output 114 may also represent an interface suitable for controlling the robot 2. The interface may also be bi-directional, e.g. signalling back to the controller 113 e.g. status signals from one or more position sensors, status signals from a drive of the robot 2, etc.

[0106] The robot 2 preferably is configured for surgical operations, fully or semiautomated. While in some of conventional robots 2 a tool at the end of the arm of the robot (also referred to end-of-arm-tool) is a passive tool and solely is operated by way of moving the arm 21 of the robot 2, i.e. by motion control of the robot 2, it is presently envisaged that the end-of-arm-tool is an active tool, specifically a valve reshaping tool 4. Accordingly, the valve reshaping tool 4 preferably is controlled in its actions in addition to its motion via the robot 2. For this purpose, a second output 115 of the controller 113 is connected to the valve reshaping tool 4, for controlling the actions of the valve reshaping tool 4 by way of a valve reshaping signal RV. Again, the second output 115 of the controller 113 may be an interface, preferably bi-directional, in order to also receive signals from the valve reshaping tool 4, e.g. status signals or the like.

[0107] Hence, the controller 113 controls both, the robot 2 and the valve reshaping tool 4 by means of the first output 114 by which control signals RC are sent to the robot 2 to control its motion, and by means of the second output 115 by which control signals RV are sent to the valve reshaping tool 4 to control its functions.

[0108] In addition, the controller 113 preferably receives signals via an image input 111. The signals comprise images IM of a region of interest 51 from an imaging device 3. It is assumed that the imaging device 3 is arranged and configured to image the region of interest 51, which presently is the heart of a human being, or specific regions of such heart, such as the mitral valve as indicated by reference numeral 512. Such imaging device 3 may, in a preferred P192941PC00 19 embodiment, be an endoscope, indicated by a tube 31 which serves to transfer illumination in direction of the region of interest 51 and 2D images to the computing system 1. Such images IM are also denoted as live images IM

[0109] Given that the live images IM from the imaging device 3 are desired to support the work of a surgeon, it is preferred that the images IM supplied by the imaging device 3 actually show the target region 51 on which the robot 2 and the valve reshaping tool 4 shall finally act on. The positioning of such imaging device 3, e.g., in form of the endoscope, preferably is performed by an operator.

[0110] A user interface 12, such as a graphical user interface of the computing system 1 may comprise a user input 121. The user input 121 may allow to identify the region of interest 51 within the target volume 5. For this reason, the user interface 12 preferably comprises a display 122 on which the live images IM are displayed. The user input, e.g., a pointer, a mouse, etc. may serve to confine the region of interest in the target volume 5.

[0111] As is indicated in Fig. 1, the target volume 5 presently is the heart, while the region of interest 51 is the region around and including a valve of the heart. In a very preferred embodiment, the valve is the mitral valve 512 which is adjacent the left atrium 511 and the left ventricle 513. In the illustration of Fig. 1, the mitral valve 512 is in a closed position. The flow of blood is from the left atrium 511 through the open mitral valve 512 into the left ventricle 513 as is indicated by the arrow. Upon closure of the mitral valve 512, such flow of blood in the forward direction is temporarily stopped until in the next cycle the mitral valve 512 is opened again.

[0112] Patients with a dysfunction of the mitral valve, in particular with a mitral valve insufficiency, suffer from an insufficient closure of the mitral valve such that during the closure of the mitral valve, there still is a blood flow in reverse direction, as is indicated by the small arrow and the reference numeral BF, from the left ventricle 513 to the left atrium 511. Such dysfunction of the mitral valve may be repaired by surgery subject to the medical indication. Such repair may include one or more of adding support means to the mitral valve or parts of it, such as its leaflets, such as replacement portions, replacement fibres, components for fixing the leaflets of the mitral valve in a certain position, removing fibres, removing leaflets of the valve, or removing the entire mitral valve. For doing so, the valve reshaping tool comprises one or more tools performing such actions, such as a scalpel, a needle, a suction device, etc.

[0113] In one embodiment, the robot 2 and / or the valve reshaping tool 4 are controlled by the controller 113 which in turn operates in response to the input of a surgeon. P192941PC00 20

[0114] The surgeon may, e.g., via a console, manually trigger the actions to be executed by the robot 2. In such scenario, the surgeon may rely on the live images IM supplied by the endoscope and / or pre-op images.

[0115] In another embodiment, the motions of the robot 2 and / or the functions to be performed by the valve reshaping tool 4 are controlled fully automated by the computing system 1. In such a scenario, a suggested valve reshaping action preferably is translated into control code interpretable by the controller 113 for executing the desired steps and / or motions and / or actions by the robot 2.

[0116] The computing system 1 further comprises an evaluation unit 111 and a mapping unit 112. All the three entities - including the controller 113 - may be represented by a processing unit 11 and define functionality within the processing unit 11 in form of dedicated software. In other embodiments, the controller 113 may be implemented by a separate processing unit dedicated to controlling functions. In addition to the processing unit 11 and to the user interface 12, a storage 14 is provided in the computing system 1.

[0117] The storage 14 stores patient-specific images. Such images may include echocardiograms ECHO, that are taken earlier in time during a screening of the patient's heart, for example. These patient-specific images preferably are used for generating a patient-specific computational model PSCM that is also stored in the storage 14. The PSCM preferably models the present state at least of the region of interest of the patient's heart, e.g. the valve and the adjacent chambers, and in another embodiment may model the entire target volume 5, i.e. the patient's heart. A model in the present context means a computational model which simulates the anatomy and / or function of the region of interest, preferably by way of mathematical equations. The computational model may be enhanced by a 3D graphical interface illustrating, e.g., the anatomy of the region of interest in a 3D view, such as known from CAD tools, which 3D representation in turn may allow a user to modify the model according to needs.

[0118] In one embodiment, the patient-specific computational model PSCM includes anatomy and functionality, the latter indicating primarily blood flow and dynamics of blood flow, in a single model. In other variants the patient-specific computational model PSCM comprises two models, i.e. an anatomical model, and a functional model. The patient-specific computational model PSCM preferably is built from patient-specific data fed to the computing system, such as by the echocardiograms ECHO. Known modelling tools may include Intelli Space ®.

[0119] It is preferred, that the PSCM is capable of providing at least one measure that is considered as relevant measure or indicator for the present valve dysfunction, as well as P192941PC00 21 for an assessment of the success of the valve repair action that is intended to be performed, and / or of any valve repair action that may be considered before selecting the most promising one, also referred to as suggested valve repair action.

[0120] Accordingly, the reverse blood flow during closure is a preferred computed measure from the patient-specific computing model PSCM, but not measured with the patient. Given that it is desired to computer-support the decision taking of how to best repair the valve in order to reduce the blood flow in reverse direction during valve closure or even bring it to or close to zero, the patient-specific computing model PSCM preferably is evaluated by the evaluation unit 111. The evaluation unit 111 preferably modifies the PSCM, or, in different words, generates different configurations of the PSCM, which modifications or configurations represent different approaches in valve reshaping, i.e. the application of different valve reshaping actions. Such variants may, e.g., include the virtual insertion of a clove-covered ring around the valve to shape the leaflets of the valve into a desired fit, aka annuloplasty, or a removal of loose portions of the leaflets of the valve, aka quadrangular resection, or a resuspension of leaflets with artificial fibres such as Gore, or a complete removal of the valve and the subsequent attachment of an artificial valve.

[0121] A result of such evaluation is a suggested valve reshaping action, that seems most promising in view of the various simulated valve reshaping actions. Such evaluation may be conducted iteratively, in that, e.g., for a given valve reshaping action such as the resection of a leaflet of the valve, various resection lines, aka cutting lines are iteratively developed with fed back to the simulation.

[0122] In a different embodiment, a non-iterative approach is applied in the simulation such that the at least one measure is captured per valve repair action, i.e., from the simulated state of the modified PSCM, and is assembled in an array also referred to as index. The array of values of BFi may represent the measure values that in total represent the index in a sense that all blood flow values BFi are in a scale between low reverse blood flow and high reverse blood flow. These values BFi, collectively referred to as index, are then compared with each other. At least one of the corresponding valve repair actions is selected as suggested valve repair action in view of its promising at least one measure. Other measures may also be taken from the corresponding PSCM in its modified or configured state, and maybe evaluated standalone or in combination with the other measures. In one embodiment, a minimum blood flow BFXis detected in the index, representing the modified PSCM resulting in the lowest reverse blood flow BFXat closed, reshaped valve. P192941PC00 22

[0123] The suggested valve repair action VRA is then input to the mapping tool 112. The mapping tool 112 translates the suggested valve repair action VRA into graphics, preferably graphical elements such as lines, dots, symbols, etc. preferably in combination with a legend explaining the graphical elements if not self-explaining. Preferably, the suggested valve repair action not only consists of one or more specific actions, but also comprises positional information where to apply the one or more specific actions. Such positional information preferably is 3D positional information.

[0124] On the other hand, the mapping unit 112 receives the live images IM from the imaging device 3. The mapping unit 112 preferably contains a feature identification or extraction unit configured to identify features and / or structures in the live images IM. Specifically, the feature identification unit is capable of identifying anatomical structures in the live images IM, such as the leaflets of the valve, the chambers, muscles, etc.

[0125] In a next step, the visualized valve repair action is mapped to one or more of the live images IM. For doing so, it is preferred that the relevant structures are already identified in the image / s. The mapping unit 112 then projects the graphical elements to the identified structures, at the desired position. A live image IM such prepared is denoted as live image including the visualized suggested valve repair action IM- VRA, which is output via image output 118 to the display 122 of the user interface 12, and is displayed there, preferable to support the surgeon during surgery, or for monitoring the fully or semi-automated valve repair action executed by the robot 2, given that the valve reshaping tool 4 may also become visible in the live images IM during operation. In the latter case, a translation of the suggested valve repair action VRA into instructions INS executable by the controller 113 and upon execution controlling the robot 2 preferably is input to the controller 113 for fully or at least semiautomated execution of the valve repair action by the robot 2. The computing system 1 as illustrated in Figure 1 may, instead, also be operated absent the robot 2, and as such represent a computing system according to an embodiment of the present invention.

[0126] FIGs.. 2 to 4 each illustrates a block diagram of a computing system according to different embodiments of the present invention. These computing systems 1 vary in some aspects in comparison with the computing system shown in Figure 1. Hence, the description is limited to these variations. Same elements and signals are referred by the same reference numerals across the Figures 1 to 4.

[0127] In Figure 2, the computing system 1 additionally comprises a modelling tool 114 implemented by corresponding software executed on the processing unit 11. However, the modelling tool 114 may also be implemented remote from, in particular the mapping unit 112, P192941PC00 23 which is indicated by the dashed separation of the modelling tool 114 from the other components. Still, the modelling tool 114 is considered as part of the computing system 1.

[0128] The modelling tool 114 is used to generate the patient-specific computing model PSCM of the region of interest in the patient's heart. Accordingly, e.g., the computing system is fed by echocardiograms ECHO, which preferably are stored in the storage 14. These echocardiograms ECHO as such may represent patient-specific data, and in particular patientspecific images PS-IM, that may directly or after pre-processing be input to the modelling tool 114. Based on these patient-specific images PS-IM, the modelling tool 114 generates the patient-specific computing model PSCM, that preferably is stored for later usage in the storage 14. The PSCM may then be input to the evaluation unit 111 in order to determine the suggest valve repair action VRA. All these steps may be performed prior to any surgery, preferably offline from the live-imaging of the patient's region of interest 51. At the time of the live- imaging, the mapping unit 112 is preferably activated, and the suggested valve repair action visualized in one or more of the life images IM is output, and preferably is displayed on display 122.

[0129] In Figure 3, the evaluation unit 111 of the computing system 1 specifically comprises a (first) machine learning model ML1. Accordingly, the input of the evaluation unit 111 no longer is a patient-specific computing model PSCM representing the patient-specific data to be evaluated. Instead, the evaluation unit 111 comprises the machine learning model ML1 that is fed with the patient-specific data, and specifically with the patient specific image data PS-IM, that e.g. may be represented by echocardiograms, pre-processed or non-pre- processed, of the region of interest, in a pre-op state of the patient.

[0130] The machine learning model ML1 preferably is trained by first image data sets, preferably of the same kind like the patient-specific image data PS-IM that are available for consideration / evaluation, hence, in the present embodiment also echocardiograms of the same region of interest of multiple, preferably many thousands of different patients, pre-op state, with a dysfunction of the respective valve. The training data includes second image data sets, illustrating post-op images of the region of interest, post-op state, i.e. after a valve repair action, for the same patients the first image data sets were provided. The second image data sets preferably also include images of the same kind like the first image data sets and the patient-specific images PS-IM, presently echocardiograms. The training data sets preferably also include additional data which denotes the applied valve repair action applied to the respective patient, and, preferably the at least one measure post-op, i.e. after valve repair. Accordingly, the images of the first images set and of the second image set of a given patient P192941PC00 24 are preferably linked or tagged by the applied valve repair action, in order to enable the machine learning model to learn the effect of a given valve repair action on a given dysfunction of the valve.

[0131] When the machine learning model ML1 is fed by the patient-specific pre-op image data PS-IM, it provides a suggested valve repair action VRA based on its previous training.

[0132] It is noted that the second set of image data the machine learning model ML1 is trained with presently includes post-op images, such echocardiograms, taken at a rather shot time after the surgery, e.g. at most a week or a month after surgery. Accordingly, the machine learning model ML1 suggests a valve repair action VRA that is considered best when looking at the state of patient short after surgery.

[0133] In a different approach, instead of the second set of image data as defined above, a third image data set is used as training data, again illustrating post-op images of the patients, however, taken at a later point in time, e.g., at least half a year or at least a year after surgery. Accordingly, the corresponding (second) machine learning model ML2 preferably evaluates the longer-term effect of valve repair actions, given that the monitoring period is extended. Such embodiment of a computing system is illustrated in Figure 4. In a different embodiment, the (second) machine learning model ML2 is additionally trained with the second image data set, e.g. defined according to the embodiment of Figure 3. In such embodiment, both, short-term and long-term images, with respect to the point in time of surgery, are taken into account such that the suggested valve repair action preferably takes the short-term and long-term trend into account.

[0134] Figure 5 illustrates an example of a live-image in combination with a visualized suggested valve repair action, as output by one of a computing system, a robot system and a computer-implemented method according to embodiments of the present invention. In a top view on the valve, AL indicates the anterior leaflet, PL indicates the posterior leaflet, and the line between AL and PL indicates the valve in a closed state. EM refers to excessive material in the posterior leaflet, that impacts the functioning of the valve. An according valve repair action may include resection of the excessive material EM. The visualization, hence, may comprise dashed resection lines. P192941PC00 25

[0135] Reference List

[0136] 1 Computing System

[0137] 11 Processing Unit

[0138] 111 Evaluation Unit

[0139] 112 Mapping Unit

[0140] 113 Controller

[0141] 114 Modelling Tool

[0142] 115 First Output

[0143] 116 Second Output

[0144] 117 Image Input

[0145] 118 Image Output

[0146] 12 User Interface

[0147] 121 Input device

[0148] 122 Display

[0149] 14 Storage

[0150] 2 Robot

[0151] 21 Arm

[0152] 3 Imaging Device

[0153] 31 Endoscope

[0154] 4 Valve Reshaping Tool

[0155] 5 Target Volume

[0156] 51 Region of Interest

[0157] 511 Left atrium

[0158] 512 Mitral Valve

[0159] 513 Left Ventricle

Claims

P192941PC00 26CLAIMS1. Computing system (1), comprising an image input (117) configured to receive live images (IM) of a region of interest (51) of a patient from an imaging device (3); a storage (14) configured to store data specific to the patient o of at least a valve of the patient's heart; o different to the live images (IM); an evaluation unit (111) configured to evaluate the patient-specific data, and configured to suggest a valve repair action (VRA) based on the evaluation; a mapping unit (112) configured to map the suggested valve repair action (VRA) to one or more of the live images (IM) received via the imaging input (117) by way of visualizing the suggested valve repair action (VRA) in the one or more live images (IM); and an image output (118) configured to supply the one or more live images (IM- VRA) including the visualized suggested valve repair action.

2. Computing system (1) according to claim 1, wherein the storage (14) stores the patient-specific data, wherein the evaluation unit (111) is configured to receive the patient-specific data as input from the storage (14), wherein the mapping unit (112) is configured to receive the suggested valve repair action (VRA) from the evaluation unit (111) and the images (IM) from the image input (117) or from a memory buffering the images (IM) received from the image input (117), and comprising a display (122) connected to the image output (118) and configured to the receive the images (IM- VRA) including the visualized valve repair action from the mapping unit (112).

3. Computing system according to claim 1 or claim 2, wherein the patient-specific data comprises a patient-specific computing model (PSCM) of at least the valve of the patient's heart, preferably wherein the patient-specific computing model (PSCM) comprises a model of at least the valve of the patient's heart and the adjacent heart chambers, preferably wherein the patient-specific computing model (PSCM) comprises a patient-specific anatomical computing model and / or a virtual patient-specific functional computing model of blood flow at least through the valve,P192941PC00 27 preferably wherein the valve is the mitral valve (512) of the heart of the patient and the adjacent chambers are left atrium (511) and left ventricle (513).

4. Computing system according to claim 3, wherein the evaluation unit (111) includes an optimization function adapted to optimize a measure in the patient-specific computational model (PSCM) subject to a valve repair action (VRA) applied, preferably wherein the evaluation unit (111) is configured to, by way of the optimization function, determine a preferred valve repair action (VRA) that achieves an optimum value of the at least one measure, preferably wherein the at least one measure includes blood flow in a backward direction through the valve in a closed state thereof, preferably wherein the evaluation unit (111) is configured to, by way of the optimization function, determine the suggested valve repair action (VRA) for receiving the minimum value of the blood flow in a backward direction through the post-op valve in a closed state thereof.

5. Computing system according to claim 3 or claim 4, wherein the evaluation unit (111) includes a simulation function adapted to simulate anatomical modifications in the patient-specific computational model (PSCM) representing valve repair actions (VRA) and to determine corresponding values of the at least one measure, preferably wherein the at least one measure includes blood flow in a backward direction through the valve in a closed state thereof, preferably wherein the evaluation unit (111) is configured to, by way of the simulation function, determine the suggested valve repair action (VRA) for receiving the minimum value of the blood flow in a backward direction through the post-op valve in a closed state thereof.

6. Computing system according to claim 5, wherein the evaluation unit (111) is configured to determine an index comprising the values of the at least one measure for the various simulated anatomical modifications, wherein the evaluation unit (111) is configured to determine the suggested valve repair action (VRA) based on the index, preferably wherein the evaluation unit (111) is configured to determine the suggested valve repair action (VRA) corresponding to the lowest value in the index.P192941PC00 287. Computing system according to any of the preceding claims, wherein the evaluation unit (111) comprises a machine learning model (ML1) trained by first data sets, preferably by first image data sets illustrating pre-op valves of a multitude of patients, and by second data sets, preferably by second image data sets illustrating post-op valves of these patients, and by associate valve repair actions applied for transforming the respective valve from its pre-op state into its post-op state, preferably wherein the second data sets illustrate the post-op valves within a week after the applied valve repair action, preferably within a month after the applied valve repair action, preferably wherein the first and second image data sets include one or more of echocardiogram images (ECHO); cardiac computed tomography images; cardiac MRI.

8. Computing system according to claim 7, wherein the patient-specific data comprises pre-op patient-specific images (PS-IM) of at least the valve of the patient, configured to input the pre-op patient-specific images (PS-IM) to the machine learning model (ML1), and in response the machine learning model (ML1) being configured to output the suggested valve repair action (VRA), preferably wherein the pre-op patient-specific images (PS-IM) are taken prior to the live images (IM); preferably wherein the pre-op patient-specific images (PS-IM) include one or more of echocardiogram images (ECHO); cardiac computed tomography images; cardiac MRI.

9. Computing system according to any of the preceding claims, providing a data set containing a limited number of valve repair actions (VRA), preferably wherein the data set of valve repair actions (VRA) is stored in the storage (14).

10. Computing system according to any of the preceding claims,P192941PC00 29 wherein the mapping unit (112) comprises an identification unit configured to identify features in the live images (IM), and wherein the mapping unit (112) is configured to map the suggested virtual repair action (VRA) to the identified features.

11. Computing system according to any of the preceding claims, wherein the visualization of the valve repair action (VRA) in the live images (IM) includes one or more of: projecting locations of surgical actions into the live images (IM); indicating a cutting pattern in the live images (IM); indicating a stitching pattern in the live images (IM); indicating a folding pattern in the live images (IM); indicating an implant location in the live images (IM), preferably for annuloplasty ring; indicating an attachment location in the live images (IM), preferably for one or more of neo-chords and an artificial valve.

12. Robotic system for valve repair, comprising a robot (2) configured to support reshaping of a valve of a heart of a patient, a computing system (1) according to any of the preceding claims, a user interface (12) configured to receive a user's input, and a controller (113) configured to control the robot (2) according to the user's input and / or according to the suggested valve repair action (VRA).

13. Robotic system according to claim 12, comprising wherein the user interface (12) includes the displayed images (IM-VRA) comprising the visualized suggested valve repair action (VRA), and wherein the user interface (12) comprises input means (121) configured to trigger at least portions of the suggested valve repair action (VRA) according to the user’s input.

14. Robotic system according to claim 13, wherein the input means (121) are configured to support identification of at least one region of interest (51) in the live images (IM) received from the image input (117).P192941PC00 3015. Robotic system according to any of the preceding claims 12 to 14, wherein the computing system (1) comprises a processing unit (11), the processing unit (11) implementing the mapping unit (112) and the controller (113), preferably wherein the evaluation unit (111) is arranged remote from the mapping unit (112) and the controller (113), preferably wherein the evaluation unit (111) is implemented in a cloud computing system, preferably wherein the mapping unit (112) and the controller (111) are implemented at a location of the robot (2).

16. Robotic system according to any of the preceding claims 12 to 15, wherein the robot (2) comprises a valve reshaping tool (4) configured to support reshaping of a valve of a heart of a human being according to the suggested valve reshaping action (VRA), and at least one arm for moving the valve reshaping tool (4) in various directions, wherein the controller (113) comprises a first output (113) configured to send control signals (RC) to control a motion of the at least one arm of the robot (2).

17. Robotic system according to claim 16, wherein the controller (113) comprises a second output (114) configured to send control signals (RV) to control the valve reshaping tool (4).

18. Robotic system according to claim 16 or claim 17, wherein the controller (113) is configured to position, via controlling the at least one arm, the valve reshaping tool (4) relative to the target region (51),19. Robotic system according to any of the preceding claims 16 to 18, wherein the controller (113) is configured to implement the at least portions of the suggested valve repair action (VRA) by controlling the arm and / or the valve reshaping tool (4) via the control signals sent via the first and / or second output (115, 116).

20. Robotic system according to claim 16, wherein the valve reshaping tool (4) comprises one or more of:- a tool for adding support means to the valve for the valve to maintain a desired shape,- a needle for stitching,P192941PC00 31- a resection tool for resecting part of or all of the valve and / or- replacement tool for adding a valve replacement.

21. Robotic system of any of the preceding claims 12 to 20, wherein the imaging device (3) includes an endoscope connected to the image input (117).

22. A computer-implemented method, comprising: receiving live images (IM) from a region of interest (51) of a patient, receiving patient-specific data of at least a valve of the patient's heart different from the live images (IM), evaluating the patient-specific data, suggesting a valve repair action (VRA) based on the evaluation, mapping the suggested valve repair action (VRA) to one or more of the live images (IM) received by way of visualizing the suggested valve repair action (VRA) in the one or more live images (IM), and outputting the one or more images (IM- VRA) including the visualized suggested valve repair action.

23. The computer-implemented method according to claim 22, wherein the patient-specific data comprises a patient-specific computing model (PSCM) of at least the valve of the patient's heart, preferably wherein the patient-specific computing model (PSCM) comprises a model of at least the valve of the patient's heart and the adjacent heart chambers, preferably wherein the valve is the mitral valve (512) of the heart of the patient and the adjacent chambers are left atrium (511) and left ventricle (513).

24. The computer-implemented method according to claim 22 or claim 23, wherein the patient-specific data comprises patient-specific images including one or more of echocardiogram images (ECHO); cardiac computed tomography images; cardiac MRI.

25. The computer-implemented method according to claim 23 and claim 24, comprising generating the patient-specific computing model (PSCM) from the patient-specific images.P192941PC00 3226. The computer-implemented method according to claim 25, generating a patient-specific anatomical computing model of at least the valve and the adjacent chambers from the patient-specific images, preferably generating the patient specific anatomical model by means of a modelling tool.

27. The computer-implemented method according to claim 25, generating a patient-specific functional computing model of blood flow at least through the valve from the patient-specific images, preferably generating the patient-specific functional computing model of blood flow at least through the valve from the patient-specific anatomical model, preferably generating the patient-specific functional computing model by means of another modelling tool.

27. The computer-implemented method according to any of the preceding claims 23 to 25, comprising simulating anatomical modifications in the patient-specific computational model (PSCM) representing valve repair actions (VRA), determining values of at least one measure per simulation, and determining the suggested valve repair action (VRA) based on the values of the at least one measure.

28. The computer-implemented method according to any of the preceding claims 23 to 27, comprising optimizing at least one measure in the patient-specific computational model (PSCM) subject to configurations of the patient-specific computational model (PSCM) representing valve repair actions (VRA), determining the suggested valve repair action (VRA) as a result of the optimization process.

29. The computer-implemented method according to claim 26 or claim 27, wherein the at least one measure includes blood flow in a backward direction through the valve in a closed state thereof,P192941PC00 33 wherein the at least one measure of blood flow in backward direction through the valve in a closed state thereof is extracted from the patient-specific computational model (PSCM) and its simulated states or configurations respectively.

30. The computer-implemented method according to claim 29, comprising determining the suggested valve repair action (VRA) for receiving the minimum value of the blood flow in a backward direction through the post-op valve in a closed state thereof.

31. The computer-implemented method of any of the preceding claims 22 to 30, comprising evaluating the patient-specific data by means of a machine learning model (ML1) trained by image data sets illustrating at least the respective valve in echocardiogram images (ECHO); cardiac computed tomography images; cardiac MRI.

32. The computer-implemented method of claim 31, comprising wherein the machine learning model (ML1) is trained by first data sets, preferably by first image data sets illustrating pre-op valves of a multitude of patients, and by second data sets, preferably by second image data sets illustrating post-op valves of these patients, and by data representing associate valve repair actions applied for transforming the respective patient's valve from its pre-op state into its post-op state, preferably wherein the second data sets illustrate the post-op valves within a week after the applied valve repair action, preferably within a month after the applied valve repair action.

33. The computer-implemented method of claim 32, comprising wherein the patient-specific data comprises pre-op patient-specific images (PS-IM) of at least the valve of the patient, inputting the pre-op patient-specific images (PS-IM) into the machine learning model (ML1), and in response to the input the machine learning model (ML1) outputting the suggested valve repair action (VRA), preferably wherein the pre-op patient-specific images (PS-IM) are taken prior to the live images (IM), preferably wherein the pre-op patient-specific images (PS-IM) include one or more ofP192941PC00 34 echocardiogram images (ECHO); cardiac computed tomography images; cardiac MRI.

34. The computer-implemented method of any of the claims 22 to 33, comprising wherein the patient-specific data includes one or more of the patient's age, sex, weight, height, vital parameters including one or more of blood pressure, lab parameters including blood composition, and / or diseases including one or more of diabetes.

35. The computer-implemented method of any of the preceding claims 22 to 34, providing a data set containing a limited number of valve repair actions (VRA) from which the suggested valve repair action (VRA) is determined.

36. The computer-implemented method of any of the preceding claims 22 to 35, identifying features in the live images (IM) received from the image input (117), and mapping the visualized suggested valve repair action (VRA) to the features identified in the one or more live images (IM), preferably by one or more of: projecting locations of surgical actions into the images (IM); indicating a cutting pattern; indicating a stitching pattern; indicating a folding pattern; indicating an implant location, preferably an annuloplasty ring; indicating an attachment location, preferably for one or more of neo-chords and an artificial valve.

37. A computer program element, preferably computer program product, comprising computer readable code means configured to, when executed by a processing unit, perform the steps of the method according to any one of the preceding claims 22 to 36.

Images