Generating an interaction zone for an interventional medical device

Abstract

A concept for facilitating interaction with an interventional medical device depicted in an image. The location of one or more features of at least one interventional medical device is identified. For each feature, the size and shape of an interaction region are determined, wherein the interaction region (when provided on the display of the user interface) facilitates interaction with the corresponding feature and / or interventional medical device. The size of the interaction region is selected to be larger than the size of the feature. The location of the interaction region is selected based on the position / location of the corresponding feature in the image. Region information is then output to define the size, shape, and position of the interaction region relative to the image.

Description

Technical Field

[0001] This invention relates to the field of user interfaces, and more particularly to user interfaces that provide interactive images. Background Technology

[0002] Standard user interfaces typically require users to operate user input devices (such as a mouse) to interact with or click on devices / components in an image.

[0003] User interfaces are increasingly being used in clinical interventional procedures to guide, tune, and / or visualize interventional medical devices throughout the procedure. In such embodiments, users can interact with the user interface to modify parameters of the interventional medical device, visualize the device, and / or monitor its location / progress within the patient.

[0004] However, during clinical intervention procedures, operating user input devices to interact with user interface tools can be inconvenient or cumbersome for clinicians. There is a need to provide a user interface system that allows users to interact with the user interface more easily with greater accuracy and / or precision. Summary of the Invention

[0005] This invention is defined by the claims.

[0006] According to an example of one aspect of the invention, a computer-implemented method is provided for facilitating interaction with an interventional medical device for displaying an image at a user interface.

[0007] The computer-implemented method includes: obtaining the position of at least one feature of one or more interventional medical devices relative to an image (e.g., a two-dimensional (2D) image) for display at a user interface (e.g., a two-dimensional (2D) user interface), or within an image (e.g., a two-dimensional (2D) image) for display at a user interface (e.g., a two-dimensional (2D) user interface), the image containing a two-dimensional (2D) visual representation of each or more interventional medical devices; performing a region generation process, the region generation process including, for each of the one or more features, determining the size and shape of an interactive region for overlaying a portion of a display image onto the user interface and for which a user can interact via the user interface, wherein the interactive region has a larger size than the size of the feature associated with the interactive region; for each interactive region, determining the position of the interactive region relative to the image based on the position of the associated feature; and outputting region information including at least information about the size, shape, and position of each interactive region. Specifically, the 2D visual representation of the interventional device is a projection of the interventional device (which is inside the patient – ​​navigating in a 3D volume) onto a 2D surface of the image (or a screen-user interface).

[0008] This invention facilitates new user interfaces for providing improved user-machine interaction. The location of one or more interventional medical devices (such as catheters, stents, or guidewires) within an image is established. Interactive regions are generated for at least one feature of the interventional medical device(s), each defining a display area of ​​the user interface with which the user can interact (e.g., click or select). Information about the interactive regions is output as region information to allow the user interface to display the image and the interactive regions. This method thereby facilitates new user interfaces with interactive regions larger than the features of the interventional medical device the user might wish to interact with.

[0009] By defining the size of the interactive region as larger than the size of its corresponding feature, the proposed concept makes it easier to select and / or interact with one or more features of an interventional medical device within an image. In particular, this concept reduces the need for users to carefully select the precise location and position of the interventional medical device (features) when interacting with images, thus providing a simpler, faster, and easier-to-use user interface. The proposed concept can also be more intuitive in use.

[0010] The present invention is particularly advantageous when the interventional medical device is an elongated device. This is because interacting with elongated interventional medical devices (such as guidewires or catheters) in an image is particularly cumbersome and / or challenging due to the precision required for such interaction. By providing a larger interaction area (i.e., larger than the features of the elongated interventional medical device), the necessary precision is reduced, thereby increasing the ease with which the user can interact with the features of the elongated interventional medical device.

[0011] The step of determining the location of the interaction region for each interaction region may include determining the location of the interaction region for each interaction region such that the interaction region and at least a portion of the feature associated with the interaction region overlap.

[0012] In some embodiments, the at least one feature includes the entirety of the interventional medical device.

[0013] In some embodiments, the step of determining the shape of the interaction region includes setting the shape of the interaction region to be the same as the shape of the feature associated with the interaction region.

[0014] The step of obtaining the location of at least one feature may include processing the image to identify the location of at least one feature of the one or more interventional medical devices.

[0015] Other embodiments for obtaining the location of at least one feature will be apparent to those skilled in the art. For example, an external source can provide information about the location of at least one feature within an image. Suitable examples of external sources include: an image processor (which preprocesses an image to identify one or more features), or a shape sensing data provider (such as an optical sensor for interventional medical devices), or a combination thereof.

[0016] Methods for processing images to identify the location of at least one feature will be apparent to those skilled in the art, and may include, for example, segmenting the image using one or more deep learning or image analysis methods.

[0017] Optionally, the step of generating the interactive area includes determining at least one visual display property of the interactive area, wherein the area information further includes information about the determined at least one visual display property of each interactive area. Each interactive area preferably has at least one unique visual display property. This increases the ease with which users can distinguish different interactive areas and thus different interventional medical devices.

[0018] As an example, visual display properties may include the color, shadow, opacity, and / or pattern fill of the interactive area.

[0019] The embodiment may also include the step of obtaining identification information that identifies the type of each interventional medical device, wherein each interaction area and any other interaction area associated with the characteristics of the same type of interventional medical device share common display properties, wherein the display properties are different for any interaction area corresponding to different types of interventional medical devices.

[0020] This embodiment enables users to more easily identify different types of interventional medical devices and their associated zones, for example, allowing users to more quickly identify all interventional medical devices of a specific type by looking for common display properties.

[0021] The type of device can include, for example, the identity and / or category of interventional medical devices. As an example, devices can be categorized into two classes: shape-sensing devices or non-shape-sensing devices. In another example, devices can be categorized based on their size and / or thickness (e.g., devices with a thickness less than 10 mm and devices with a thickness greater than 10 mm). Of course, the type of device can include any combination of the foregoing (e.g., the type could be a stationary device with a thickness less than 10 mm).

[0022] Optionally, the size of each interactive region responds to the number of features (at least one) to which the region generation process is performed in the image.

[0023] The method further includes the step of obtaining identification information that identifies the type of each interventional medical device, wherein: the region generation process includes receiving user input that identifies one or more desired types of interventional medical devices, and generating an interactive region only for at least one feature of the one or more desired types of interventional medical devices.

[0024] Therefore, users can restrict the generated regions to include only regions of a specific type. This increases the ease with which users can select the desired interventional medical device (by restricting the generated regions to the desired type that matches the desired interventional medical device).

[0025] Preferably, all interactive areas are sized, shaped, and / or positioned to prevent overlap between different interactive areas. Therefore, the space occupied by the areas can depend on how many areas are generated. For example, if fewer interventional medical devices are present, this results in larger areas to reduce the likelihood of the user mistakenly selecting or clicking on an unwanted area.

[0026] The images include data obtained from medical images (such as X-ray images, CT images, ultrasound, etc.).

[0027] A computer program product is also proposed, comprising a computer-readable storage medium having computer-readable program instructions thereon, the computer-readable program instructions being configured to cause the processing system to perform any of the methods described herein when the computer-readable program instructions are executed by the processing system.

[0028] An interactive area processing unit is also proposed to facilitate interaction with interventional medical devices displayed in images at the user interface.

[0029] The interactive region processing unit is adapted to: obtain the position of at least one feature of one or more interventional medical devices relative to an image for display at a user interface, the image containing a two-dimensional visual representation of each of the one or more interventional medical devices; perform a region generation process, the region generation process including, for each of the one or more features, determining the size and shape of an interactive region, the interactive region being used to overlay a portion of the display image onto the user interface and allowing the user to interact with the interactive region via the user interface, wherein the interactive region has a larger size than the size of the feature associated with the interactive region; for each interactive region, determining the position of the interactive region relative to the image based on the position of the associated feature; and outputting region information including at least information about the size, shape, and position of each interactive region.

[0030] Some embodiments provide a user interface system including a user interface and a processor. The processor includes a user interface processing unit and an interactive area processing unit. The user interface processing unit is adapted to acquire image and area information, and control the user interface based on the acquired image and area information to provide interactive images.

[0031] The user interface processing unit can respond to user interactions with the interaction area via the user interface by executing one or more predefined actions.

[0032] These and other aspects of the invention will become apparent with reference to one or more embodiments described below, and will be illustrated with reference to one or more embodiments described below. Attached Figure Description

[0033] To better understand the invention and to more clearly illustrate how it can be implemented, reference will now be made to the accompanying drawings by way of example only, wherein:

[0034] Figure 1 The figure illustrates a user interface system according to an embodiment of the present invention;

[0035] Figure 2 The method according to an embodiment of the present invention is illustrated; and

[0036] Figures 3 to 7 The illustrations depict methods for generating interactive regions according to different embodiments of the present invention. Detailed Implementation

[0037] The present invention will be described with reference to the accompanying drawings.

[0038] It should be understood that the detailed descriptions and specific examples, while indicating exemplary embodiments of the apparatus, system, and method, are intended for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, system, and method of the invention will be better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are merely schematic and not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to denote the same or similar parts.

[0039] This invention provides a concept for facilitating interaction with an interventional medical device depicted in an image. The location of one or more features of at least one interventional medical device is obtained. For each feature, the size and shape of an interaction region are determined, wherein the interaction region (when provided on the display of the user interface) facilitates interaction with the corresponding feature and / or interventional medical device. The size of the interaction region is selected to be larger than the size of the feature. The location of the interaction region is selected based on the position / location of the corresponding feature in the image. Region information is then output to define the size, shape, and location of the interaction region relative to the image.

[0040] The embodiments are based on the understanding that precise interaction with the display of features of interventional medical devices is difficult to perform, especially during interventional clinical procedures, due to the small size of interventional medical devices and the difficulty of manipulating them when interacting with the user interface. It has been recognized that the required precision can be reduced by sizing the interaction area to be larger than the size of the features of the interventional medical device, thereby providing a better user interface to facilitate human-computer interaction.

[0041] This invention can be used in any clinical setting where interventional medical devices are used, such as in an operating room or intensive care unit.

[0042] In the context of this invention, features may include the entire interventional medical device or a part of the interventional medical device, such as a logo, tip, or component portion of the interventional medical device. Therefore, interaction with the interactive area can facilitate interaction with the entire interventional medical device and / or a part thereof.

[0043] Figure 1 The illustration shows a user interface system 100 according to an embodiment of the present invention. The user interface system includes a user interface 110, a processor 120, and (optionally) a database 130.

[0044] User interface systems can be part of a holistic medical system used in conjunction with interventional (medical / clinical) devices. Specifically, a user interface system can allow users to visualize interventional medical devices and / or their control parameters, their environment, and / or the visualization of the interventional medical device itself.

[0045] User interface 110 is adapted to display interactive images, for example, on a two-dimensional screen 115. Interaction with the interactive images can be via touch (e.g., if the two-dimensional screen is a touch-sensitive screen), via voice input, or any other input mechanism (e.g., mouse or keyboard input). Therefore, user interface 110 may include input mechanisms for enabling a user to interact with the interactive images.

[0046] In this invention, the interactive image is formed in two aspects: the image (i.e., providing visual information about the object, particularly providing a visual representation of an interventional medical device) and one or more interactive areas superimposed on the image (a portion thereof) with which the user can interact.

[0047] Images may include data obtained from medical images, such as X-ray images, CT images, MRI images, etc. In some embodiments, the interventional medical device depicts within the medical image itself.

[0048] In some embodiments, the visual representation of the medical device is obtained from a device positioning system (e.g., an optical fiber connected to the medical device), which can be used to identify the position of the medical device (relative to an image or display) and to generate a visual representation of the interventional medical device on the displayed image. This can be represented on a default or model image of a human anatomy.

[0049] In some embodiments, the displayed image is a combination of, for example, an underlying medical image of an object and a visual representation of an interventional medical device superimposed on the underlying medical image, the visual representation indicating its position relative to the object. This embodiment is not required, and the displayed image may alternatively consist of, for example, a single medical image.

[0050] Processor 120 is adapted to control interactive images displayed by a user interface and to respond to any user interaction with the interactive images. In particular, user interface processing unit or module 121 obtains images for display, for example, from database 130 or directly from image generating equipment (such as an X-ray system and / or device positioning system), and performs actions (e.g., controlling external devices and / or controlling the display of interactive images) in response to user interaction at user interface 110.

[0051] Interactive area processing unit 122 generates interactive areas for an image. An interactive area defines the location and / or region of an interactive image that a user can interact with to induce one or more actions to be performed. Specifically, the interactive area is overlaid on a portion of the image to provide the interactive area of ​​the image. The interactive area may have a visual representation (e.g., a specific shadow, color, outline opacity, and / or pattern fill) or may be transparent / invisible. Different types of interactions with the interactive area (e.g., tap or long press) can trigger different actions.

[0052] The displayed interactive images include one or more interventional medical devices, such as those used in invasive clinical procedures. Each interactive area represents a feature of one or more interventional medical devices. The interactive areas effectively enable users to interact with the corresponding features of one or more interventional medical devices and / or the corresponding interventional medical device itself.

[0053] For the purpose of improving understanding of the background of the present invention, a general process for providing interactive images is provided below. However, the embodiments are not limited to the precise steps performed in this process.

[0054] In this general process, the user interface processing unit 121 of processor 120 obtains, for example, a (non-interactive) image from database 130 for display at user interface 115. The obtained image is an image of one or more interventional medical devices. The interaction area processing unit 122 of processor 120 generates one or more interactive areas for at least one feature (such as the logo of the interventional medical device or the entire interventional medical device itself) of the obtained image. The interaction area processing unit 122 generates information about the generated interactive areas (e.g., at least their size, shape, and position relative to the image) and outputs this information as area information. User interface processing unit 121 obtains the area information from interaction processing unit 122 and controls user interface 110 to provide interactive images based on the obtained image and area information.

[0055] Subsequent interactions between the user and the interactive image via user interface 110 can trigger one or more actions. For example, a first interaction with the interactive area can cause the user interface to "zoom in" on features associated with the interactive area. A second, different interaction with the interactive area can cause features associated with the interactive area to be highlighted, for example, for easier viewing.

[0056] Different actions can be performed based on the selected interaction area and / or the type of interaction (such as interaction mode, e.g., long press, short press, or double tap).

[0057] Those skilled in the art will understand that these possible actions are merely examples, and that any possible response to interaction with the interactive area will be implemented. Further examples will be provided later in the description.

[0058] This invention relates to a process for generating interactive regions for an image, thereby facilitating interaction with the interactive image and / or features of an interventional medical device depicted within the image. Therefore, the interactive region processing unit 121 itself can be an embodiment of this invention.

[0059] The basic concept of this invention is that the size of the generated interactive regions is larger than the size of the features they represent. This facilitates easier interaction with interactive images and / or features of interventional medical devices depicted within the images.

[0060] The present invention is particularly advantageous when the interventional medical device is an elongated device. This is because interacting with elongated interventional medical devices (such as guidewires or catheters) in an image is particularly cumbersome and / or challenging due to the precision required for such interaction. By providing a larger interaction area (i.e., larger than the features of the elongated interventional medical device), the necessary precision is reduced, thereby increasing the ease with which the user can interact with the features of the elongated interventional medical device.

[0061] Figure 2 A method 200 according to an embodiment of the present invention is shown. Method 200 can be executed by an interaction processing unit 121.

[0062] Method 200 includes step 202 of obtaining the position of at least one feature of one or more interventional medical devices relative to an image for display at a user interface, the image containing a visual representation of each or more interventional medical devices. The position of the feature (within the image) can be derived in a variety of ways.

[0063] Suitable examples of images may include data about X-ray images, ultrasound images, CT images, etc. The images used for display depict one or more interventional medical devices. These can be represented as physical entities within the image.

[0064] As previously stated, features may include the entire interventional medical device or a part of the interventional medical device, such as the device's logo, top, or component portion.

[0065] In some examples, the location of features is determined by processing the image using machine learning algorithms or other image analysis methods, such as image segmentation techniques. These examples allow the location of features and, alternatively, their identities to be established directly from the image.

[0066] In other examples, the location of the features is provided by an external source (e.g., an external image analysis unit) or from shape sensing device 150 (e.g., an imaging device adapted to physically track the position of the interventional medical device relative to the image being captured). A suitable example of a shape sensing device is an optical fiber coupled to the interventional medical device, which provides optical shape sensing information that can be superimposed on the image.

[0067] In some embodiments, machine learning methods may be used, for example, to identify multiple possible locations of a feature. As described later, an interactive region may be generated for each possible location to allow the user to select the correct location of the feature.

[0068] Of course, combinations of the aforementioned methods (i.e., hybrid methods) can be used; for example, using external data to identify general locations and performing image analysis to identify precise locations. Other methods for determining the location of features of interventional medical devices within an image will be apparent to those skilled in the art.

[0069] In some embodiments, step 202 may be combined with step 201 such that recognizing an image includes obtaining information about the location of features within the image.

[0070] Method 200 also includes step 203 of performing a region generation process. This process includes, for each of one or more features, (at least) determining the size and shape of an interactive region. As previously discussed, the interactive region is used to overlay a portion of a displayed image onto a user interface and allows the user to interact with the image and / or features of an interventional medical device depicted within the image via the user interface.

[0071] The size of each interactive region (relative to the image) is configured to be larger than the feature associated with the interactive region (“associated feature”). The size of a region or feature can be defined by the area of ​​the image occupied by the region or feature.

[0072] For example, when an image undergoes an image analysis method, the size of the feature can be determined during step 202. Other methods for determining the size of the feature will be apparent to those skilled in the art, such as based on the known size of the feature (e.g., the known size of an interventional medical device).

[0073] The method also includes step 204, which determines the position of an interactive region relative to the image based on the position of features associated with the interactive region (i.e., “associated features”).

[0074] This location can be defined, for example, as the coordinates or relative position of the interactive region within the image (e.g., "top left"). In some examples, the location can be defined relative to the corresponding feature (e.g., 10 pixels to the right of the feature or at the location of the feature).

[0075] Preferably, the position of each interactive region is configured such that each interactive region overlaps with at least a portion of its associated feature. This embodiment allows interactive regions to be associated with feature graphics in an intuitive and clear manner, while maximizing the available space in the interactive image for providing the interactive regions.

[0076] Effectively, steps 203 and 204 together define the shape, size, and position of the interactive area within the interactive image (which may be displayed / provided by the user interface). Specifically, steps 203 and 204 define the shape, size, and position of the interactive area with respect to (i.e., relative to) the image. This can be done, for example, by determining the pixel coordinates of the area's position within the image or by determining any other information about where the interactive area should be positioned relative to the image (e.g., "top left quadrant" or "bottom half").

[0077] The shape of the interactive area can be selected or determined in a variety of ways.

[0078] For example, the shape of the interactive area can be controlled based on the shape of the feature corresponding to the interactive area (e.g., to match the shape of the feature). This helps users intuitively understand which feature the interactive area corresponds to.

[0079] In another example, the shape of the interactive area is predetermined (e.g., square, rectangle, circle, triangle, etc.). In other examples, the shape of the interactive area can be controlled by the user, for example, in response to user input. In still other examples, the shape of the interactive area depends on the number (and / or location) of the generated interactive areas, for example, to prevent overlapping of interactive areas.

[0080] Other embodiments will be apparent to those skilled in the art, such that the defined shapes are not limited to any specific examples described herein.

[0081] In a preferred example, the size and / or shape of each interactive area is configured such that the interactive area has a width / thickness greater than the feature of its corresponding interventional medical device. This increases the ease with which the user can interact with the feature (e.g., compared to embodiments where the width / thickness of the interactive area is the same as the feature of the interventional medical device).

[0082] Preferably, the size, shape, and / or position of each interactive area are configured such that the interactive areas do not overlap. This helps to avoid confusion about which interactive area is associated with which feature.

[0083] Preferably, the size, shape, and / or position of each interactive region are configured such that the area occupied by the interactive region(s) of the interactive image is greater than half of the entire image (e.g., the whole image), for example, greater than 3 / 4. Therefore, the space occupied by the interactive region can be maximized.

[0084] Preferably, the size (and optional shape and / or position) of each interactive area is configured to maximize the average size of the interactive areas while aiming to minimize the range of sizes of the interactive areas. This embodiment is particularly preferred when the size, shape, and / or position of each interactive area is configured such that the interactive areas do not overlap with each other.

[0085] The size, shape, and / or position of each interactive region can be controlled based on the number of at least one feature of its region generation process. For example, the more features, the smaller the size of the interactive region.

[0086] Preferably, the interactive regions can be sized, shaped, and / or positioned such that they are arranged relative to the image to maximize the size of each interactive region while avoiding overlap. In another preferred example, the interactive regions are also shaped to have the same shape as the corresponding features (but different sizes).

[0087] The foregoing embodiments for determining size, shape, and / or position effectively achieve background adaptive determination of one or more interactive areas.

[0088] Method 200 also includes step 205 of outputting region information that includes at least information about the size, shape, and location of each interactive region.

[0089] In some examples, the region information also identifies at least one visual display property of the interactive region (e.g., color, opacity, fill pattern, outline, etc.). For this purpose, the region generation process performed in step 203 may further include defining one or more visual display properties (and / or their values) for the corresponding interactive region for each located feature. In some embodiments, each interactive region may have at least one unique visual display property to help distinguish the interactive regions from each other.

[0090] In some embodiments, a user may be able to define one or more display properties for an interactive area. Thus, area information may be based on user input (e.g., received at the user interface) to identify at least one visual display property of the interactive area, for example, defining the area's color as pink instead of yellow.

[0091] In some examples, the region information also provides a (text) label (e.g., "1" or "2") for each region, which can be displayed along with the interactive region at the user interface. For this purpose, the region generation process performed in step 203 may further include generating at least one (text) label for the corresponding interactive region for each located feature. The label for each interactive region can be unique (for a specific interactive image). The label can be later used to enable the user to interact with the interactive region, for example, by using voice commands to interact with a specific interactive region that has a label.

[0092] Region information can be used to control or define the interactive areas of interactive images. Therefore, interactive displays can be controlled to provide interactive areas overlaid on the image based on region information.

[0093] Some methods may also include the step of displaying images and interactive areas at the user interface, thereby providing interactive images at the user interface. However, this is not necessary to realize the basic concept of facilitating the provision of interactive areas for images.

[0094] Some methods may also include responding to user interactions with the interactive area by performing one or more actions. Again, this embodiment is not essential to the basic concepts.

[0095] Optionally, the method may include step 201 of identifying a (medical) image for display at a user interface, the image containing one or more interventional medical devices. Step 201 may include obtaining the image itself or simply obtaining information about the image including one or more interventional medical devices.

[0096] Figures 3 to 6 This illustration is intended to depict an embodiment used to define the size, shape, and location (i.e., region information) of one or more interactive regions. It is illustrated graphically for conceptual understanding purposes, but those skilled in the art will understand that the process of generating region information does not need to be performed graphically.

[0097] Figure 3 Various embodiments for generating an interactive region for an image 300 of a single interventional medical device 310 are conceptually illustrated. This process can be performed by an interactive region processing unit. Here, the interventional medical device 310 includes a guidewire.

[0098] In particular, Figure 3 Different interactive images 350, 360, and 370 are conceptually illustrated, with dashed lines used to depict the size, shape, and position of interactive areas 355, 365, and 375. Each interactive image 350, 360, and 370 is a result of implementing different embodiments or examples of the invention.

[0099] In the illustrated example, each interactive region 355, 365, 375 corresponds to a feature of interventional medical device 310, which in this context refers to the entire interventional medical device. In other words, the feature of the interventional medical device is the shape and / or outline of interventional medical device 310. For clarity, where appropriate, references to "interventional medical device" in the following paragraphs may be replaced by the term "feature of interventional medical device".

[0100] In the first example, image 300 is processed to generate a first interactive image 350 having a first interactive region 355.

[0101] The size of the first interactive area is configured to be larger than the size of the interventional medical device 355. For example, the size of the interactive area may be configured to be larger than the size of the interventional medical device by a predetermined percentage or amount (e.g., the number of pixels), or configured such that the thickness of the interactive area is greater than the thickness of the interventional medical device by a predetermined amount (e.g., the number of pixels), and so on.

[0102] The position of the first interactive area 355 depends on the position of the interventional medical device 355. Here, the first interactive area is positioned to overlap with the interventional medical device 355, and specifically, the interventional medical device is positioned in the middle / center of the interactive area.

[0103] The first interaction region 355 is shaped to be similar to and / or identical to the shape of the interventional medical device 355. Therefore, the shape of the first interaction region depends on the shape of the interventional medical device 355.

[0104] In the second example, image 300 is processed to generate a second interactive image 360 ​​with a second interactive region 365. The second interactive region 365 differs from the first interactive region 355 in that the shape of the second interactive region 365 is predetermined (here: rectangular) and it is not directly based on the shape of the interventional medical device.

[0105] In the third example, image 300 is processed to generate a third interactive image 370 with a third interactive region 375. The third interactive region 375 is sized to occupy the entire image. This embodiment is possible only if the interactive image contains only a single interventional medical device (feature).

[0106] As shown in the figure, in order to help identify the interactive area, the interactive area can have at least one non-zero visual display property (e.g., associated with a specific color, opacity, fill pattern, outline, etc.).

[0107] Figure 4 Other examples or embodiments for generating interactive areas for image 400 are conceptually illustrated. Here, image 400 includes multiple interventional medical devices 411, 412, such as multiple guidewires.

[0108] In particular, Figure 4 Different interactive images 450, 460, and 470 are conceptually illustrated, wherein the dimensions, shapes, and positions of interactive areas 451, 452, 461, 462, 471, and 472 are illustrated using dashed lines. Each interactive image 450, 460, and 470 is a result of implementing different embodiments or examples of the present invention.

[0109] As mentioned earlier, each interactive region 451, 452, 461, 462, 471, and 472 corresponds to a feature of interventional medical devices 411 and 412, which in this context refers to the entire interventional medical device.

[0110] In the first example, image 400 is processed to generate a first interactive image 450. The first interactive image 450 includes a first interactive region 451 and a second interactive region 452, each interactive region corresponding to a corresponding interventional medical device. In the first example, the shape of the interactive region matches the shape of the corresponding interventional medical device.

[0111] In the second example, image 400 is processed to generate a second interactive image 460. The second interactive image includes a third interactive region 461 and a fourth interactive region 462, each corresponding to a corresponding interventional medical device. The third and fourth interactive regions (respectively) differ from the first and second interactive regions in that their shapes are independent of the shape of the corresponding interventional medical device.

[0112] In the third example, image 400 is processed to generate a third interactive image 470. The third interactive image includes a fifth interactive region 471 and a sixth interactive region 472, each corresponding to a corresponding interventional medical device. The fifth and sixth interactive regions (respectively) differ from the third and fourth interactive regions in that the regions are sized such that they together fill the entire interactive image.

[0113] like Figure 4 As shown, to aid in the identification of interactive areas, each interactive area may have at least one non-zero visual display property (e.g., associated with a specific color, opacity, fill pattern, outline, etc.). Specifically, each interactive area may have at least one unique visual display property to distinguish it from others. In the illustrated example, each interactive area has a different fill pattern.

[0114] Figure 5 Other example embodiments for generating interactive areas for image 500 are conceptually illustrated. Here, the image includes various types of interventional medical devices 511, 512, such as guidewire 511 and catheter 512.

[0115] In particular, Figure 5 Different interactive images 550 and 560 are conceptually illustrated, wherein the size, shape, and position of interactive areas 551, 552, 561, and 562 are shown using dashed lines. Each interactive image 550 and 560 is a result of implementing different embodiments or examples of the invention.

[0116] As mentioned earlier, each interactive region 551, 552, 561, and 562 corresponds to a feature of interventional medical devices 511 and 512, which in this context refers to the interventional medical devices themselves.

[0117] In the first example, image 500 is processed to generate a first interactive image 550. The first interactive image 550 includes a first interactive region 551 and a second interactive region 552. The first interactive region 551 corresponds to a guidewire 511, and the second interactive region 552 corresponds to a catheter 512. Each interactive region is shaped based on the shape of its associated interventional medical device.

[0118] In the second example, image 500 is processed to generate a second interactive image 560. The second interactive image 560 includes a third interactive region 561 and a fourth interactive region 562. The third interactive region 561 corresponds to a guidewire 511, and the fourth interactive region 562 corresponds to a catheter 512. The third and fourth interactive regions differ from the first and second interactive regions in that their shapes are independent of the shape of the associated interventional medical device, and they are sized to fill the interactive image.

[0119] like Figures 3 to 5 As shown, in a preferred example, the interaction areas are sized, shaped, and positioned such that they do not overlap.

[0120] However, this is not necessary, and the areas can be configured to overlap at least partially. In such embodiments, the user (when interacting with overlapping areas) may be asked to confirm, for example, through the use of a pop-up menu, whichever area they intend to interact with.

[0121] In the illustrated example, each interactive area is positioned to overlap with its corresponding feature (when displayed with the corresponding image). However, while preferred, this is not a necessary aspect of the invention.

[0122] However, preferably, the interactive region is configured to be graphically associated with its corresponding feature. As previously mentioned, this can be done by positioning the interactive region to overlap with the feature.

[0123] In other examples, this is done using graphical indicators or annotations (such as labels or arrows that identify features corresponding to the interactive area). In such examples, the area information may include information about the location, size, and shape of such graphical indicators. Therefore, the area generation process may also include generating information about one or more graphical indicators.

[0124] Therefore, region information can include information for graphically associating each interactive region with its corresponding feature. When an interactive region is configured to overlap with its corresponding feature, the location information of the region information provides this information.

[0125] In the previously described example, each feature of the interventional medical device is already the interventional medical device itself. In other words, for example, the interactive area has been generated only for the interventional medical device, rather than for a part or segment of the interventional medical device.

[0126] However, in some embodiments, the features of the interventional medical device include segments or portions of the interventional medical device, such as markings of the interventional medical device. Suitable examples of markings of interventional medical devices will be apparent to those skilled in the art.

[0127] By way of example only, the features of an interventional medical device may include the end or tip of the interventional medical device. In some examples, the feature of the interventional medical device is a predicted end or tip of the interventional medical device.

[0128] As another example, the features of interventional medical devices may include (radiosensitive) markings on the interventional medical device. A suitable example of the use of radiosensitive markings suitable for use as features exists in FEVAR (fenestrated intravascular aortic repair) stent grafts.

[0129] As yet another example, the features of interventional medical devices may include specific elements of the interventional medical device, such as gate markings on stent grafts.

[0130] Other suitable examples of features(s) of interventional medical devices will be apparent to those skilled in the art.

[0131] Generating interactive regions for each interventional medical device (feature) in the image is not necessary. Rather, it is sufficient to generate only a subset of the possible interactive regions.

[0132] As an example, in some embodiments, interactive areas may be generated only for certain desired types of interventional medical devices (features). Specifically, the type can be the desired type of a feature and / or the desired type of the interventional medical device. The desired type can be defined via user input or other signals (e.g., provided via a user interface).

[0133] exist Figure 6The diagram illustrates the process, in which an interactive region is generated for an image 600 displaying two different types of interventional medical devices (guidewire 611 and catheter 612). User input 620 indicates the desired type of interventional medical device (e.g., guidewire), and a single interactive region 655 is generated only for the guidewire. Therefore, the generated interactive image 650 includes a single interactive region 655 for the guidewire 611 (even if more than one interactive region could have been generated).

[0134] Return to reference Figure 2 This process can be implemented in step 203. In particular, the region generation process may include receiving user input identifying one or more desired types of interventional medical devices, and generating interactive regions only for at least one feature of each identified interventional medical device of the desired type.

[0135] The desired type can be a medical (sub)category or label for interventional medical devices, such as: “guidewire,” “catheter,” “stent graft,” “sheet,” etc. Exemplary subcategories may include, for example, “Davis catheter,” “pigtail catheter,” or “cobra catheter.” Other suitable categories and subcategories for interventional medical devices will be apparent to those skilled in the art.

[0136] However, the type is not limited to such labels, but can include any category or subcategory of the desired interventional medical device. For example, a device can be classified into two types: shape-sensing devices and non-shape-sensing devices. As another example, a device can be classified into two different types: mobile devices and non-mobile (i.e., stationary) devices. As yet another example, a device can be classified into different thicknesses (i.e., the type is a category of thickness), such as: <10mm thick; >10mm and <50mm thick; >50mm thick; and so on.

[0137] Therefore, "type" can be any category of interventional medical device, where the category can be defined using any suitable characteristics of the interventional medical device (e.g., shape, size, medical purpose / intention, age, thickness, diameter, etc.).

[0138] Continue to refer to Figure 2 The type of interventional medical device can be obtained in step 202, for example, by obtaining identification information that identifies the type of each / interventional medical device. The type acquisition process can be performed automatically, for example, using an image recognition process or based on external information (e.g., which identifies the type and / or identity of the interventional medical device contained in the image).

[0139] In embodiments that identify the type of interventional medical device, step 203 may further include defining one or more visual display properties (and / or their values) for a corresponding interactive area for each feature, wherein features corresponding to the same type(s) of interventional medical devices share the same value for the first display property (e.g., having the same color). Preferably, interactive areas for features of different interventional medical devices have different values ​​for the first display property (e.g., having different colors). This helps to distinguish between different types of features.

[0140] It will be obvious that when different display properties are set for different types of interventional medical devices, area generation is not limited to the desired type of device.

[0141] exist Figure 7 The diagram illustrates the process, in which interactive areas are generated for an image 700 displaying four interventional medical devices of two different types: a first guidewire 711; a second guidewire 712; a first catheter 713; and a second catheter 714. The first guidewire 711 and the first catheter 713 are associated with each other, as are the second guidewire 712 and the second catheter 714.

[0142] In interactive image 750, a first set of interactive regions 751, 752 (i.e., "first type") is generated for the guidewire, and a second set of interactive regions 753, 754 (i.e., "second type") is generated for the catheter. The interactive regions in the first set (i.e., interactive regions of the same type) share the same value for the display property (here, pattern fill), and so do the interactive regions in the second set. However, the interactive regions in different sets have different values ​​for that display property (i.e., different types of pattern fill).

[0143] Therefore, features corresponding to one or more interventional medical devices of the same type share the same value for the first display property, while the interaction areas for features of different interventional medical devices have different values ​​for the first display property. This helps to distinguish between different types of features.

[0144] As previously mentioned, region information can be used (e.g., by the user interface processing unit) to define one or more interactive regions of the interactive image. The interactive image is provided by a user interface that enables the user to interact with the interactive region(s).

[0145] Depending on the type of user interface, users can interact with the interactive area in a variety of ways.

[0146] Preferably, the user interface is adapted to receive touch and / or voice input for interaction, for example, via a touch screen and / or microphone.

[0147] To improve the convenience of voice input, it is preferable that the interactive areas are labeled when displayed. This allows users to interact with the interactive areas by recalling or audibly referencing the displayed labels. Therefore, embodiments that generate area information (i.e., the area generation process) may include generating a unique label for each interactive area so that the interactive areas can be distinguished from each other. The labels can be arbitrary (e.g., numbers) or can relate to the characteristics or identity or type of interventional medical device (e.g., “Catheter 1” or “Possible Mark 1” or “Cadre Mark”).

[0148] It has been previously described how interaction with an interactive area can trigger one or more actions when displayed by a user interface. The exact actions(s) performed when interacting with the interactive area depend on implementation details and are not considered essential to the concept of this invention.

[0149] Interactive areas can, for example, enable users to interact with the system or environment to guide, tune, or visualize the intervention device or its parameters.

[0150] Specifically, interaction with the interaction area can trigger actions related to features (associated with the interaction area) and / or interventional medical devices (associated with the interaction area). These actions may involve the environment of the corresponding feature and / or interventional medical device, or the physical properties of the feature or interventional medical device.

[0151] The triggered actions can involve the display of features or interventional medical devices, such as highlighting features or interventional medical devices, selecting features or interventional medical devices, removing interventional medical devices from the display, and so on. Selecting a feature enables further interaction with the feature, such as modifying the parameters of the associated interventional device.

[0152] Other examples may include highlighting / outlining selected features or devices to facilitate identification during operation, or zooming in on selected devices (the ends) to increase the ease of tracking the movement of medical devices.

[0153] The triggered actions can involve the interventional medical device itself, such as modifying the nature of the (physical) interventional medical device or automatically recording the process involving the selected interventional medical device that is taking place, such as to ensure that the necessary device has not been removed.

[0154] Another response to interaction with the interactive area can be to modify or alter the image based on features or interventional medical devices associated with the interactive area. For example, if the image is part of a video composed of multiple images, the “best” appearance of the selected device (i.e., the device associated with the interactive area the user interacts with) can be searched within the video and that image can be displayed. The “best” appearance could be the image with the highest contrast between the selected device and the background.

[0155] As another example, multiple interactive regions can correspond to possible or predicted locations of the same feature of the interventional device (e.g., the predicted end of the interventional device). Interacting with an interactive region can cause the corresponding location to be selected as the correct location of the feature, and other interactive regions can be discarded. Of course, the image can be reprocessed based on this additional information to generate new interactive regions.

[0156] As another example, interactive images can be used to annotate underlying images, for instance, based on user interactions with interactive areas, to generate base ground truth data for further training machine learning-based algorithms. In other words, a user can interact with one or more interactive areas to label one or more features of the interventional device associated with those areas, generating labeled images as base ground truth data. This base ground truth data can then be used to train machine learning methods, such as image segmentation algorithms or automatic labelers.

[0157] In other examples, interactive images can be used to select the location and properties of devices to be recorded for use in automated reporting. Therefore, interaction with the interactive area can initiate the recording of the location and / or properties of one or more features of the intervention device associated with the interactive area.

[0158] Therefore, those skilled in the art will understand that various different actions can be performed in response to user interaction with the interactive area, and further examples will be obvious to those skilled in the art.

[0159] Previously, in some embodiments, machine learning algorithms have been described to automatically determine the location and / or identification of features of interventional medical devices within an image. This may include, for example, the automatic identification of the location (or possible location) of the interventional medical device or the prediction of the location (or possible location) of markings (such as the top) of the interventional medical device.

[0160] A machine learning algorithm is any self-trained algorithm that processes input data to produce or predict output data. Here, the input data is an image or image data, and the output data includes one or more features of an interventional medical device.

[0161] Suitable machine learning algorithms used in this invention will be apparent to those skilled in the art. Examples of suitable machine learning algorithms include decision tree algorithms and artificial neural networks. Other machine learning algorithms, such as logistic regression, support vector machines, or Naive Bayes models, are suitable alternatives.

[0162] Artificial neural networks (or simply neural networks) are inspired by the human brain. A neural network consists of layers, each containing multiple neurons. Each neuron performs a mathematical operation. Specifically, each neuron can include different weighted combinations of a single type of transformation (e.g., the same type of transformation, a sigmoid, etc., but with different weights). In processing input data, the mathematical operation of each neuron is performed on the input data to produce a numerical output, and the outputs of each layer in the neural network are sequentially fed to the next layer. The final layer provides the output.

[0163] Methods for training machine learning algorithms are well-known. Typically, such methods involve obtaining a training dataset that includes training input data entries and corresponding training output data entries. An initialized machine learning algorithm is applied to each input data entry to generate a predicted output data entry. The error between the predicted output data entry and its corresponding training output data entry is used to modify the machine learning algorithm. This process can be repeated until the error converges and the predicted output data entry is sufficiently similar to the training output data entry (e.g., ±1%). This is often referred to as a supervised learning technique.

[0164] For example, in machine learning algorithms formed by neural networks, the mathematical operations (weights) of each neuron can be modified until the error converges. Known methods for modifying neural networks include gradient descent, backpropagation, and others.

[0165] The training input data entries correspond to example images. The training output data entries correspond to one or more features of the corresponding example from the interventional medical device.

[0166] Those skilled in the art, through studying the accompanying drawings, the disclosure, and the claims, will be able to understand and implement variations of the disclosed embodiments when practicing the claimed invention. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude a plurality.

[0167] Computer programs can be stored / distributed on suitable media, such as optical storage media or solid-state media supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.

[0168] Although certain measures are described in different dependent claims, this does not imply that combinations of these measures cannot be used advantageously. If the term "suitable" is used in the claims or description, it should be noted that the term "suitable" is intended to be equivalent to the term "configured as." No reference numerals in the claims should be construed as limiting the scope.

Claims

1. A computer-implemented method (200) for facilitating interaction with an interventional medical device for displaying an image at a user interface (110), the computer-implemented method comprising: Obtain (202) the location of at least one feature of one or more interventional medical devices within a two-dimensional image (300, 400, 500, 600, 700) for 2D display at a user interface, the image containing a two-dimensional (2D) visual representation of each or more interventional medical devices (310, 411, 412, 511, 512, 611, 612, 711, 712, 713, 714); Obtain identification information that identifies the type of each interventional medical device; Perform a region generation process (203) comprising determining, for each of one or more features, the size and shape of an interactive region (355, 365, 375, 451, 452, 461, 462, 471, 472, 551, 552, 561, 562, 655, 751, 752, 753, 754), the interactive region being used to overlay a portion of a display image onto the user interface and to allow the user to interact with the interactive region via the user interface, wherein the interactive region has a larger size than the size of the feature associated with the interactive region, and wherein the region generation process comprises receiving user input (620) identifying one or more desired types of interventional medical devices (612), and generating an interactive region (655) only for at least one feature of the one or more desired types of interventional medical devices; For each interactive region, the position of the interactive region relative to the image is determined (204) based on the location of the associated features; and The output (205) includes at least the region information about the size, shape and location of each interactive region.

2. The computer-implemented method according to claim 1, wherein, The step (204) of determining the position of the interaction region for each interaction region includes: determining the position of the interaction region for each interaction region such that the interaction region and at least a portion of the feature associated with the interaction region overlap.

3. The computer-implemented method according to claim 1 or 2, wherein, The at least one feature includes the entirety of the interventional medical device, or a part of the interventional medical device.

4. The computer-implemented method according to claim 3, wherein, The portion of the interventional medical device includes the device's logo, top, or component portion.

5. The computer-implemented method according to claim 1 or 2, wherein, The step of determining (203) the shape of the interaction area includes setting the shape of the interaction area to be the same as the shape of the feature associated with the interaction area.

6. The computer-implemented method according to claim 1 or 2, wherein, The step (202) of obtaining the location of at least one feature includes processing the image to identify the location of at least one feature of the one or more interventional medical devices.

7. The computer-implemented method according to claim 1 or 2, wherein, The step (203) of generating the interactive area includes determining at least one visual display property of the interactive area, wherein the area information further includes information about the determined at least one visual display property of each interactive area.

8. The computer-implemented method according to claim 7, wherein, Each interactive area has at least one unique visual display property.

9. The computer-implemented method according to claim 7, wherein, Visual display properties include the color, shadow, opacity, and / or pattern fill of the interactive area.

10. The computer-implemented method of claim 7, further comprising the step of obtaining identification information that identifies the type of each interventional medical device, wherein, Each interactive area and any other interactive area associated with the characteristics of the same type of interventional medical device share common display properties, wherein the display properties are different for any interactive area corresponding to a different type of interventional medical device.

11. The computer-implemented method according to claim 1 or 2, wherein, The size of each interactive region responds to the number of features in the image to which the region generation process is performed.

12. The computer-implemented method according to claim 1 or 2, wherein, All interactive areas are sized, shaped, and / or positioned to prevent overlap between different interactive areas.

13. The computer-implemented method according to claim 1 or 2, wherein, The images include data obtained from medical images.

14. A computer program product comprising a computer-readable storage medium having computer-readable program instructions thereon, the computer-readable program instructions being configured to cause the processing system to perform the method according to any one of claims 1 to 13 when executed by the processing system.

15. An interaction area processing unit (122) for facilitating interaction with an interventional medical device for displaying images (300, 400, 500, 600, 700) at a user interface (110), the interaction area processing unit being adapted to: Obtain (202) the location of at least one feature of one or more interventional medical devices within a two-dimensional image (300, 400, 500, 600, 700) for two-dimensional display at a user interface, the two-dimensional image containing a two-dimensional visual representation of each or more interventional medical devices (310, 411, 412, 511, 512, 611, 612, 711, 712, 713, 714); Obtain identification information that identifies the type of each interventional medical device; Perform a region generation process (203), which includes determining the size and shape of an interactive region (355, 365, 375, 451, 452, 461, 462, 471, 472, 551, 552, 561, 562, 655, 751, 752, 753, 754) for each of one or more features. The interactive region is used to overlay a portion of a display image onto the user interface, and the user can interact with the interactive region via the user interface. The interactive area has a larger size than the size of the features associated with the interactive area, and wherein the area generation process includes receiving user input (620) identifying one or more desired types of interventional medical devices (612), and generating an interactive area (655) only for at least one feature of the one or more desired types of interventional medical devices; For each interactive region, the position of the interactive region relative to the image is determined (204) based on the location of the associated features; and The output (205) includes at least the region information about the size, shape and location of each interactive region.

Images