Apparatus and method for supporting a user based on images

Abstract

The invention relates to a device and a method for supporting a user based on an image. The device according to the invention is used for supporting a user of an apparatus based on an image. The apparatus is configured to apply a treatment medium at variable positions in a region of a tissue surface to be treated. Furthermore, the device comprises an image capturing device which is configured to capture the region to be treated. The device comprises an evaluation device which is configured to determine a treatment trajectory of the treatment medium relative to the region to be treated from the data of the image capturing device and to determine a spatially resolved dose of the region to be treated based on the treatment trajectory. The device further comprises a presentation device which is configured to optically display the treatment trajectory and / or the spatially resolved dose for the user relative to the tissue. With the device according to the invention, it is possible to optically indicate to the user which positions of the region of the tissue surface to be treated have already been treated and / or the dose with which the respective treatment position has been treated.

Description

Technical Field

[0001] The present invention relates to an apparatus and method for a user of an image-supported (medical) device configured to apply a processing medium to a tissue surface. Background Technology

[0002] Devices that allow users to apply a treatment medium, such as non-thermal plasma, to a small area on a tissue surface to be treated in a handheld manner are generally known. For example, a user treats tissue surfaces such as skin, mucous membranes, or wounds by applying a treatment medium such as cold plasma (non-thermal plasma) or argon plasma with low energy to the tissue surface.

[0003] The treatment of the tissue surface causes little or no thermal effect within the tissue, thus preventing coagulation or carbonization. The local temperature of the tissue surface remains below the denaturation temperature of proteins (approximately 60–70°C). The treated tissue surface does not undergo optical changes, making it impossible for the user to identify which area of ​​the tissue surface has been treated. The user also cannot identify the dosage and duration of effect received at individual sites within the treated area.

[0004] Therefore, users have to remember as accurately as possible which areas and during what time period the processing medium was applied.

[0005] This places higher demands on the user's concentration. Furthermore, the estimation of the duration for which the treatment medium has been applied to a specific location is limited by the user's individual perception. Additionally, due to the lack of a visible effect of the treatment on the tissue, users may tend to move over already treated areas repeatedly, potentially leading to overdosing.

[0006] However, similarly, it is possible that users may not treat individual areas at all or only treat them for a short period of time, and therefore, these areas may suffer from insufficient dosage.

[0007] This can lead to significant variations in the quality and uniformity of the treatment medium applied to the tissue surface from time to time. Consequently, the treatment results are difficult to reproduce and cannot be examined and / or recorded in an objective manner.

[0008] Methods exist to improve homogeneity, where the treatment medium is applied to the tissue surface using a newly developed large-size plasma source. For example, DE 10 2014 220 488 A1 proposes a large-size plasma source for cold plasma at atmospheric pressure, which allows for the treatment of relatively large areas (e.g., approximately 100 cm²). 2In particular, when the area to be processed does not have a simple shape (e.g., rectangle, square, ellipse, or circle) but is complex and / or irregular, it is possible that locations outside the area to be processed may be accidentally processed.

[0009] Other known methods propose automating the application of the treatment medium. For example, DE 10 2010 011 64 A1 proposes a plasma source that moves automatically relative to the tissue surface by means of a moving device controlled by a control unit. However, this directly affects the treatment process. Furthermore, complete automation of the application requires significant development work and is therefore very costly.

[0010] The paper "Concept for Improved Handling Ensures Effective Contactless Plasma Treatment of Patients with kINPen® MED", Applied Sciences, Bd. 10, Nr. 17, Art. Nr. 17, Jan. 2020, doi:10.3390 / app10176133, proposes a sensor system that determines the distance between the instrument and the treatment area and illuminates the area where plasma is to be applied. However, this only indicates the user at the current point in time. Therefore, the user must continuously remember which areas were treated and for how much time.

[0011] Additionally, WO 2011 / 044248 A2 discloses a system in which a surgical area is monitored by means of a video camera during an ablation procedure, thereby enabling a processor to assess the quality of damage generated in the tissue during ablation. Based on the determined quality, color markers are displayed in real-time images of the surgical area.

[0012] DE 10 2015 100 927 A1 describes an image-based assistive device for surgeons during surgical interventions. The device includes a camera and an image processing unit configured to identify used medical instruments without requiring labels attached to the instruments. Summary of the Invention

[0013] Therefore, the object of the present invention is to provide an apparatus and method that allows the user of the instrument to indicate which part of the tissue to be treated has been treated, by means of which the treatment medium can be applied to the tissue surface in a hand-held manner.

[0014] This objective is achieved by means of an image-based user support device according to an embodiment of the present invention.

[0015] The apparatus according to the invention includes an instrument, an image capturing device, an evaluation device, and a presentation device. The instrument is configured to apply a treatment medium to a variable location on a tissue surface within a region to be treated. The image capturing device is configured to detect the region to be treated. The evaluation device is configured to determine the treatment trajectory of the treatment medium relative to the region to be treated based on data from the image capturing device, and to determine a spatially resolved dose of the region to be treated based on the treatment trajectory. Furthermore, the presentation device is configured to visually display the treatment trajectory and / or spatially resolved dose of the tissue to a user.

[0016] In doing so, the user can be visually indicated which locations on the tissue surface have been treated. Specifically, the user can be shown which dose has been received at which location. Therefore, the user can decide where in the treatment area the treatment medium should still be applied to achieve the most uniform distribution of the treatment medium possible. This reduces the risk of overdose or underdose at individual locations in the treatment area.

[0017] This can lead to increased uniformity of the treatment medium applied to the tissue surface, thereby enabling the evaluation and reproduction of the application results.

[0018] The device according to the invention is characterized by an evaluation device, by means of which the processing trajectory of the processing medium relative to the area to be processed is determined. Thus, the spatially resolved dose of the area to be processed can be determined based on the processing trajectory. The processing trajectory and / or the spatially resolved dose are visually presented relative to the area to be processed by a presentation device.

[0019] Accordingly, the spatially resolved dosage can be a dose value assigned to a location within the treatment area. This dose value can have a binary state (e.g., treated or untreated) or a continuous state (e.g., a percentage value relative to a predetermined desired dose). The treatment trace can be a time-dependent progression of the location and / or size (e.g., diameter or radius) of the application area. The application area is preferably the region on the tissue surface where the treatment medium has been applied by means of the instrument at the current point in time. For example, the area can be elliptical or circular. The evaluation device can preferably be a computer or processor device.

[0020] The presentation device may include a speaker unit by which the spatially resolved dose can be acoustically presented. For example, it may output sound with varying characteristics (frequency (pitch), loudness, or sound sequence depending on the dose) for its purpose. These characteristics may depend on whether the treatment medium impacts tissue that has already been treated or is still untreated. Furthermore, changes in sound characteristics can be used to indicate that the treatment medium has been on the tissue for too long.

[0021] Preferably, the treatment medium is plasma, particularly low-temperature or non-thermal plasma. This plasma preferably does not produce a macroscopically visible thermal effect on the tissue surface. During treatment using such plasma, the local temperature of the tissue surface can be maintained below the denaturation temperature of the protein, i.e., below the temperature limit of 40 to 70°C, preferably below 60 to 70°C. Non-thermal plasma is plasma not in thermal equilibrium, therefore, the temperatures of the particles (ions, electrons, or neutral particles) contained therein vary significantly. In low-temperature plasma, the plasma is also not in thermal equilibrium, so the temperatures of the particles (ions, electrons, or neutral particles) contained in the low-temperature plasma also vary. However, the amount of temperature difference among the contained particles is lower compared to non-thermal plasma. Furthermore, the applied treatment medium can be low-energy argon plasma, for which no visible tissue changes are produced on the treated tissue surface. This is at least the case where the impact of the plasma at the same location in the tissue does not exceed a maximum duration of, for example, a few seconds.

[0022] Preferably, the image capture device is configured to capture the region to be processed at least partially in an image sequence comprising multiple individual images. The processing medium can be detected and tracked in each image of the image sequence. The image sequence can also be a continuous stream of individual images, whereby the current individual image is considered a real-time image of the detected region.

[0023] In a preferred embodiment, the evaluation device is configured to identify the head of the processing medium and / or instrument in an image sequence, and / or preferably track it across multiple individual images in the image sequence. In doing so, the position of the processing medium relative to the area to be treated on the tissue surface can be determined. If the processing medium is occluded by the head of the instrument in a current individual image, the position of the instrument can be used to estimate the position of the processing medium. To identify the head of the instrument, image recognition algorithms from the field of machine learning, such as neural networks, support vector machines, etc., can be used.

[0024] Preferably, the evaluation device is configured to determine the location and / or size of the applied treatment medium in the current single image of the image sequence. This allows for tracking the treatment medium in the image sequence in the simplest and most robust possible manner. For example, the location of the treatment medium can be the center point of the application area in which the treatment medium impacts the tissue surface presented in the single image. For example, the application area can be an ellipse or a circle. Thus, the size of the applied treatment area can be the radius, diameter, or area of ​​the region where the treatment medium impacts the tissue surface.

[0025] Specifically, the evaluation device can be configured to determine the position, orientation, and / or size of the head of the instrument in a current single image of an image sequence. Based on additional information, such as the position, orientation, and / or size of the head, the location of the treatment medium can be determined or estimated even if the instrument occludes the treatment medium.

[0026] In a simplified embodiment, the position of the head alone can be used as a measure of the position of the treatment medium, thereby enabling the detection and determination of the treatment medium's location. This allows for the identification of difficult-to-detect media, such as weak treatment media and / or treatment media with irregularly shaped impact areas on tissue surfaces. Furthermore, this reduces the computational power required for implementation.

[0027] In a preferred embodiment, the evaluation device is configured to identify the processing medium in the data of the image capture device based on color and / or shape features. Preferably, the evaluation device is configured to identify the processing medium based on at least one of the following features: color value range, brightness value range, and saturation value range. By using color and / or shape features to identify the processing medium, the processing medium can be detected in the simplest and most robust way possible.

[0028] Specifically, the evaluation device can be configured to identify the head of the instrument from the data of the image capture device based on color and / or shape features or based on markings attached to the head. In doing so, a very simple method that does not require high computational power can be used to identify the head of the instrument, thereby determining the position of the head relative to the area to be processed.

[0029] Preferably, the evaluation device is configured to identify the processing medium in a search area within a current single image by means of a search routine. This search area for the processing medium may vary based on the position, orientation, and / or size of the head. This allows the search area for the processing medium to be limited to a smaller region, such as the area at the end of the head (directly) adjacent to the instrument.

[0030] In a preferred embodiment, the evaluation device is configured to determine the predicted position of the processing medium in the current single image based on the position and / or velocity of the processing medium in the previous single image.

[0031] Preferably, the search region in the search routine for identifying the processing medium in the current single image varies based on the predicted location. For example, a state estimate of the location (predicted location) of the processing medium for the current single image can be determined using an alpha-beta filter or a Kalman filter. The search region of the search routine can be located around this predicted location. Thus, the size of the search region can be set to a predetermined value, for example. Alternatively, the size of the search region can vary based on the predicted location and / or the velocity and / or the covariance value of the predicted location. For example, the search region can be reduced in the case of a high covariance value, and conversely, expanded in the case of a low covariance value. Doing so improves the tracking of the processing medium.

[0032] Specifically, the search region defined within the current single image based on the predicted location of the processing medium can be used to eliminate wrong positive mismatches. Accordingly, the search routine can continuously search the processing medium throughout the entire current single image. However, detections located outside the search region can be eliminated as wrong positive mismatches. For example, eliminated wrong positive mismatches can be directly ignored. Doing so reduces the number of wrong positive mismatches.

[0033] Alternatively, the evaluation device can be configured to define the area to be treated before the user applies the processing medium. For example, the user can trace the area to be treated back within the field of view of an image capture device using an instrument, without applying the processing medium before application begins. The evaluation device can be configured to identify and track the head of the instrument to define the desired area to be treated in this way. The area to be treated defined in this way can have any shape of arbitrary complexity, as it can be manually entered by the user. Alternatively, the definition of the area to be treated can also be performed by the user, for example, on a touchscreen. If the user now applies the processing medium to the tissue surface, specific detections of the processing medium located outside the area to be treated can be eliminated as false positive mismatches. This allows for a reduction in false positive mismatches that can be coupled in, such as through light reflections from metal clamps outside the area to be treated.

[0034] Preferably, the evaluation device is configured to identify characteristic image features of the region to be processed or reference markers applied to the area on the tissue surface within the image capture device, and to precisely allocate the processing trajectory space to the region to be processed. To identify these characteristic image features, image recognition algorithms from the field of machine learning, such as neural networks, particularly convolutional neural networks, support vector machines, etc., can also be used. For example, the reference markers can be colored dots or dots arranged in a pattern that is particularly easy to detect in the region to be processed.

[0035] In a preferred embodiment, the presentation device includes (augmented reality, AR) glasses configured to spatially and precisely project a processing trajectory and / or spatially resolved dose of the area to be processed into the field of view of the glasses.

[0036] In another preferred embodiment, the evaluation device is communicatively connected to the instrument's supply unit, wherein the evaluation device is configured to control the application of the treatment medium through the instrument based on local dosage. In doing so, overdosing in individual ranges can be automatically avoided.

[0037] Preferably, the evaluation device is communicatively connected to the instrument's supply unit, thereby configuring the evaluation device to determine the spatially resolved dose based on measurement information from the supply unit. This measurement information can be transmitted from the supply unit to the evaluation device, thereby enabling more accurate determination of the spatially resolved dose.

[0038] In a particular embodiment, the instrument may be part of a processing apparatus that includes the instrument and a supply unit for supplying operating means to the instrument. For example, the operating means may be a fluid such as a gas or liquid, and / or voltage, particularly radio frequency AC voltage. For this purpose, the supply unit may include a controllable (radio frequency, RF) generator. This generator can provide the power required for the instrument to apply the processing medium. Furthermore, the generator may include a control unit, which is communicatively connected, for example, to an evaluation device. Communication between the control unit and the evaluation device may be bidirectional. For example, information and / or measurement information regarding the processing trajectory and / or spatially resolved dose, such as ignition identification, current, voltage, power, and / or resistance, may be transmitted from the control unit to the evaluation device. Similarly, other components of the supply unit, such as a gas supply device, may be provided with a control unit and communicatively connected to the evaluation device, for example, for exchanging and / or influencing data regarding gas flow rate or gas pressure. Thus, components of the supply unit may be devices capable of operating without the presence of corresponding other components.

[0039] Preferably, the evaluation unit is configured to divide the region to be processed into multiple resolution units, and for each of these resolution units, a dose is determined based on the processing trajectory. For example, a resolution unit may be a pixel.

[0040] Furthermore, the objective is also achieved by means of a method for image-based user support according to an embodiment of the present invention: The method according to the invention includes detecting a region to be treated, in which a treatment medium can be applied at a movable location by means of an instrument. Furthermore, the method includes determining a treatment trajectory of the treatment medium relative to the region. Based on the treatment trajectory, a spatially resolved dose for the region to be treated is determined. Additionally, the method includes presenting the treatment trajectory and / or the spatially resolved dose with respect to the region to be treated.

[0041] All the features and advantages described with reference to the apparatus according to the invention can also be used in the method according to the invention. Attached Figure Description

[0042] Additional details of the advantageous embodiments or the invention itself will be apparent from the accompanying drawings, description, and claims. The drawings show: Figure 1 : A schematic diagram of an embodiment of an image-based user support device according to the present invention. Figure 2 : A schematic diagram of another embodiment of the device for image-based user support according to the present invention. Figure 3 : A schematic diagram of another embodiment of the device for image-based user support according to the present invention. Figures 4a to 4c Examples of the region to be processed and examples of related real-time images of that region, illustrating the processing trajectory. Figure 5 Example of a real-time image of the area to be processed. Figure 6 A flowchart of an embodiment of the method for image-based user support according to the present invention. Figure 7 Another example of a real-time image of a region processed using a predicted location of the processing medium. Detailed Implementation

[0043] exist Figure 1 The image shows device 1, which is used to support a user of (medical) device 2 by means of imaging.

[0044] Device 1 includes an instrument 2, by which a user applies a treatment medium P to a patient's treatment area B. The instrument 2 can be freely moved by the user in space. The movement of the instrument can virtually follow any path in the x, y, z directions, and thus, it can also pivot in any direction. By moving and / or rotating the instrument 2, the user can change the position where the treatment medium P impacts the tissue surface in the treatment area B.

[0045] At the distal end 7 of the device 2, the device 2 includes a head 6 for generating a treatment medium, such as plasma, particularly non-equilibrium plasma (cold plasma). This plasma exits the distal end 7 of the device 2 and impacts the tissue surface. The distal end 7 of the device 2 is the end closer to the tissue surface when the treatment medium is applied. Conversely, the proximal end 8 of the device 2 is the end further away from the tissue surface when the treatment medium is applied. For example, the tissue surface can be skin, mucous membrane, or a wound in the skin or mucous membrane. The device 2 is supplied by a supply unit 9, which includes an RF generator having electricity and a suitable gas, such as argon.

[0046] In this embodiment, the region B to be processed is captured at least partially within the field of view (FoV) of the image capture device 3. For example, the image capture device may be a (video) camera. Preferably, the image capture device 3 may be configured to create an image sequence comprising a plurality of individual images, wherein the image capture device 3 may also generate a current single image (real-time image) of the region B to be processed. The image capture device 3 may be arranged stationary or movable (handheld) relative to the region B to be processed. If the image capture device 3 is movable relative to region B, the field of view (FoV) may move relative to region B during processing.

[0047] Image capture device 3 is communicatively connected to evaluation device 4. Evaluation device 4 is configured to receive and evaluate data from image capture device 3. In evaluation device 4, the processing trajectory T of processing medium P relative to region B to be processed can be determined from the data of image capture device 3. Then, based on the processing trajectory T, the spatial resolution dose D of region B to be processed can be determined in evaluation device 4. If image capture device 3 is movably arranged relative to region B, then, for example, characteristic features 11 of region B to be processed can be identified in each image of an image sequence in evaluation device 4, and the movement of image capture device 3 can be compensated.

[0048] The evaluation device 4 is also communicatively connected to the presentation device. In this embodiment, the presentation device 5 is configured as a display device 5a, for example, a screen. In this example, the presentation device 5 is configured to optically and virtually present the processing trajectory T to the user on an image of the tissue. The presentation device 5 can modify the transparency or color intensity of the displayed processing trajectory T according to the local dose D. Thus, the effect of the medium on tissue that itself has no trajectory is made visible in the form of a trajectory.

[0049] exist Figure 1 In the real-time image of the area to be processed, the processing trajectory is spatially precisely shown. Figure 1The real-time image also shows characteristic feature 11. The evaluation device 4 is further configured to identify characteristic feature 11 in each image of the image sequence, calculate the processing trajectory T of the position of characteristic feature relative to the current single image (real-time image) of the image sequence, and display it therein.

[0050] Alternatively, the reference marker 12 may be adhesively attached to the tissue surface in the area to be processed B. For example, the reference marker 12 may have an optical pattern that can be easily identified by the evaluation device 4. In a preferred embodiment, two or more reference markers are adhesively attached to the tissue surface to allow for precise determination of the spatial orientation of each individual image.

[0051] exist Figure 2 The illustration shows another embodiment of the device according to the invention. Instead of display device 5a, in this example, the processing trajectory T and the spatially resolved dose are projected directly onto the area B to be processed by means of projection device 5b. For example, projection device 5b may include a laser or light-emitting diode as a light source, and additional optical components, such as lenses and mirrors, to illustrate the processing trajectory T accordingly.

[0052] Alternatively or additionally, the presentation device 5 may also include augmented reality (AR) glasses 5c, which may also be connected to the evaluation device 4, for example, via a wired or wireless means. The AR glasses 5c are configured to project a spatially resolved dose of the area to be processed into the field of view of the glasses in a spatially precise manner.

[0053] Figure 3 The figure shows another embodiment of the device 1 according to the present invention. Figure 3 The examples basically correspond to Figure 1 The embodiment differs in that the evaluation device 4 is communicatively connected to the supply unit 9, which, among other things, includes an RF generator and a gas supply and supply device 2.

[0054] Because of this communication connection, the evaluation device 4 can access the measurement and control data of the supply unit 9 and use them for dosage calculation. For example, the measurement or control data may be the ignition detection, power, output power, current, voltage, or resistance of the RF generator, or, in the case of the gas supply unit, gas flow rate or gas pressure.

[0055] In possible embodiments, the values ​​of the ideal dose and / or the limits of the minimum and / or maximum dose to the tissue may also be stored in the evaluation device. These can be preset in a constant manner, or can be individually defined by the user using appropriate input devices, such as those suitable for the patient or the intervention that must be performed.

[0056] In this embodiment, the evaluation device 4 can be configured to adjust the application of the treatment medium P based on the local dose determined for the area where the head 6 of the instrument 2 is currently located. In this way, for example, if the area being treated is close to the stored, desired, or maximum dose—that is, if, for example, 70%, 80%, or 90% of the desired or maximum dose has been reached—the application of the treatment medium through the instrument can be reduced. If the area to be treated has already been treated with the desired or maximum dose, the application of the treatment medium can also be completely turned off. If the head of the instrument subsequently moves over a tissue area where the desired or maximum dose has not yet been reached, the application of the treatment medium is increased or permitted again. This can be achieved, for example, because the evaluation device 4 provides information about the treatment trajectory T or the local dose D to the control unit of the RF generator in the supply unit 9. It is also possible that the evaluation device 4 sends a signal to the control unit for reducing or increasing power, or also for turning the power supplied to the instrument 2 by the RF generator of the supply unit 9 on or off.

[0057] Spatial-resolved dose can be calculated based on the residence time of the processing medium P at various locations within the region B to be treated, and adjusted processing parameters, such as processing mode and / or processing power. In a simplified embodiment, spatial-resolved dose can also be calculated using only the residence time if it can be assumed that a predetermined processing mode and / or predetermined processing power is used for intervention.

[0058] exist Figures 4a to 4c In the image, an example of region B to be processed is illustrated together with the associated real-time image.

[0059] exist Figures 4a to 4c On the upper left side of the image, the area to be treated, B, is shown, where plasma P is applied to the tissue surface G using instrument 2. Figures 4a to 4c The upper right side of the image shows the associated real-time images.

[0060] exist Figure 4a In the real-time image, the current position of plasma P relative to the region to be processed B has been determined by the evaluation device 4. Furthermore, the processing trajectory T, i.e., the time-dependent progression of the position of plasma P within the region to be processed B, is illustrated in the real-time image.

[0061] Alternatively or additionally, instead of the processing trajectory T, the dose D may also be shown in the real-time image. This is achieved, for example, by having the color or transparency of the processing trajectory T shown in the real-time image vary according to the local dose D.

[0062] Figure 4b It shows Figure 4a An example of region B to be processed. (Compared to...) Figure 4aIn contrast, device 2 has been further moved, causing a change in the position where the treatment medium P is applied. Figure 4b In this process, the processing medium P is obscured by the head 6 of the instrument 2. For this reason, it is impossible for the evaluation device 4 to directly locate the processing medium P in the current single image of the image sequence. In this case, the evaluation device 4 determines the position and orientation of the head 6 of the instrument 2 relative to the region B to be processed.

[0063] For this purpose, the evaluation device 4, for example, determines the main axis A of the head 6 of the instrument 2, and determines the distal orientation of the head 6 based on the main axis A. Based on the position and orientation of the head 6, the evaluation device 4 determines the position of the obscured treatment medium P. In one embodiment, the evaluation device 4 may use the distal end 7 of the head 6 of the instrument 2 as the position of the treatment medium P on the tissue surface.

[0064] In another embodiment, the evaluation device can determine the position of the processing medium P by directionally pushing the head 6 of the instrument 2 along the main axis A of the instrument head 6. In doing so, accurate determination of the position of the processing medium can be achieved even if the processing medium P is occluded in the current single image.

[0065] Figure 4c It shows Figure 4a and 4b The area to be processed, B, has been moved by the user, thus changing the current position of the device applying the processing medium P.

[0066] exist Figure 5 The image schematically illustrates a magnified portion of a real-time image. The region B to be processed shown in this real-time image is divided into multiple resolution elements C11 to Cmn. For each resolution element, a dose D is determined, calculated based on the processing trajectory T. For example, in... Figure 5In this example, no processing medium P is applied in resolution cell C11 because the processing trace T does not extend through resolution cell C11. Therefore, the dose D11 of resolution cell C11 has a value of 0. In this example, the processing trace T extends partially through resolution cell C12. Therefore, for resolution cell C12, evaluation device 4 can determine the measure of dose D12 based on the duration of time the processing medium P is located within resolution cell C12. Alternatively, evaluation device 4 can also measure it using a portion of the region of the resolution cell on which the processing medium P has been applied. This can be advantageous, particularly for the accuracy of dose D12 in the edge regions of the processing medium P. For example, if the position of the processing medium P is determined to be the center point of the detected processing medium P in the current single image, and the boundary of the processing medium P is placed around the center point determined in the current single image, then the boundary of the processing medium P can extend between multiple resolution cells, such that, particularly in the edge regions of some resolution cells, a region is partially processed by the processing medium P. Preferably, a combination of the two methods is advantageous, taking into account the time-dependent residence time of the processing medium P in the decomposition unit and the processed regional portion. Figure 5 In this process, the evaluation device 4 can determine the current position 13 of the processing medium P as the center point of a region that is at least substantially circular, for example, where the processing medium P is currently applied. Furthermore, for example, the evaluation device 4 can also determine the diameter 14 of this region as the size of the currently applied processing medium.

[0067] Furthermore, the evaluation device 4 can also use measurements controlled by the instrument 2, such as ignition identification, current, voltage, power, or resistance, to improve the accuracy of the positional progression of the treatment medium P (treatment trajectory T) and the dose determined thereby. The relationship between these measurements and the treatment trajectory can be obtained with the aid of time information.

[0068] exist Figure 6 The diagram shows an example flowchart of a method for image-based user support according to the present invention, performed by evaluation device 4 and other components of device 2.

[0069] In method step V1, the region B to be processed is captured, within which the processing medium P is applied at a manually changeable location. For example, region B can be captured in an image sequence comprising multiple individual images. For further processing steps of the tracking loop, such as (method step V11), the current single image is now used.

[0070] In method step V2, the processing trajectory T of the processing medium P relative to region B can be determined. For this purpose, it can first be determined in method step V20 whether the field of view (FoV) of the image capturing device 3 has moved relative to region B in the current single image. For example, the movement of the field of view (FoV) can occur due to movement of the image capturing device 3 relative to the tissue surface, such as translation and / or rotation, or vice versa. This movement can be determined by comparing the position and location of the feature 11 in the current single image with the position and location of the feature 11 in a previous single image. In this way, the movement that has occurred can be quantitatively determined and can be compensated for. Subsequently, the processing medium P can be searched in the current single image, and the position 13 and / or size 14 of the currently applied processing medium P can be determined (method step V21). If the processing medium P cannot be found in the current single image (query in method step V22), the head 6 of the instrument 2 can be searched, and the position, orientation, and / or size of the head 6 can be determined (method step V23).

[0071] If the processing medium P has been found in the current single image, i.e., the current position 13 and / or size 14 of the processing medium P has been determined for the current single image, the processing trajectory T can be determined or updated in method step V25. For this purpose, for example, the determined current position 13 and the size of the processing medium P can be stored for each corresponding single image, such that the processing trajectory T generates an image sequence from the positions 13 and sizes 14 of the individual images as a whole. The spatially resolved dose D can now be determined by the spatial progression of the position and size of the processing medium (processing trajectory) and, preferably, by an assessment of the time-dependent progression (further in method step V25). If the position of the processing medium P is not directly found in the current single image, but the position of the head 6 of the device 2 has been determined (queried in method step V24), the position of the processing medium P can be determined (estimated) based on the position, orientation, and / or size of the head 6 of the device 2. If the position of the head 6 of the device 2 also cannot be found, the method continues in method step V11, where the new current single image is now used as the database for sub-steps V21 to V25 of method steps V2.

[0072] Steps V21 / V22 and V23 / V25 can also be interchanged in their order. First, the position, orientation, and / or size of the head 6 can be determined. Based on these parameters, the search routine can now be modified to search the search area of ​​the processing medium P within the current single image. Preferably, this search area is now limited to the region adjacent to the distal end 7 of the head 6. In doing so, it is no longer necessary to search the processing medium P throughout the entire current single image.

[0073] Furthermore, method steps V21 and V23 can be executed in parallel.

[0074] In a simplified embodiment, instead of the location of the processed medium P (sub-steps V21 and V22), the location of the head 6 (sub-step V23) can also be used as a measure of the location of the processed medium P. Therefore, in this example, only sub-steps V23 to V25 of method step V2 are executed.

[0075] Preferably, the processing trajectory T is updated in method step V25 based on the estimated position of the processing medium P.

[0076] Based on the processing trajectory T, the spatially resolved dose of the region B to be processed can be determined in method step V3.

[0077] In method step V4, the processing trajectory T and / or spatially resolved dose D can now be plotted relative to the region to be processed B, since the processing trajectory is shown in the real-time image.

[0078] After method step V4, method step V11 is followed again, where the new current single image is used as the database for tracking the method steps of the loop.

[0079] exist Figure 7 The diagram schematically illustrates a magnified portion of a real-time image. In this embodiment, the evaluation device is configured to determine a predicted position Pos*_t of the processing medium P for the current single image. This predicted position Pos*_t can be determined based on the previous position Pos*_t-1 and / or the previous velocity V_t-1 of the processing medium P. In this example, a search region S is defined around the predicted position Pos*_t, in which a search routine identifies the processing medium in the current single image. The position Pos_t of the processing medium P detected in the current single image lies within the search region S.

[0080] Alternatively, the defined search area S can also be used to eliminate false positive mismatches fp, where the processing medium P is detected outside the search area S, for example, due to light reflection.

[0081] Furthermore, there is a possibility that the size of the search region S can be adjusted according to the velocity V_t-1. If the velocity V_t-1 is high, the search region S will be larger. However, if the velocity V_t-1 is low, the search region S can also be smaller.

[0082] The device 1 for image-based user support described so far operates as follows: To perform the treatment, the user may use device 2, such as a plasma applicator, to apply plasma at a desired dose to the area to be treated on the tissue surface in a handheld manner. During the treatment, the user may, for example, view a display device 5a on which the area to be treated on the tissue surface is displayed, and / or wear AR glasses 5c. Furthermore, during the treatment, the treatment trajectory T and / or dose D may be projected onto the area to be treated on the tissue surface in a manner visible to the user.

[0083] While performing the treatment, the user is now informed which locations within the treatment area have been treated with plasma. Preferably, the dose received at each location within the treatment area is further indicated to the user. This allows the user to apply plasma to untreated locations or locations with excessively low doses.

[0084] For example, in the treatment of cervical intraepithelial neoplasia (CIN), a video colonoscopy is used to monitor the area of ​​the mucosal surface to be treated. In this case, the assessment device can be connected to a pre-existing video colonoscopy, allowing the assessment device to evaluate the (video) data from the video colonoscopy. Similarly, in other interventions, cameras from endoscopes or laparoscopes can be used, for example.

[0085] For example, CIN treatment can be performed by applying non-thermal plasma to the mucosal surface. Data from the video colonoscopy can now be used by the evaluation device to determine the time-dependent progression of the plasma's position (treatment trajectory) and thereby determine the spatially resolved dose. This treatment trajectory or spatially resolved dose can be displayed accordingly on the video colonoscopy screen and / or other screens.

[0086] The device 1 according to the invention is used to support a user of the instrument 2 based on an image. The instrument 2 is configured to apply a treatment medium at a variable location in a region to be treated on a tissue surface. Furthermore, the device includes an image capturing device configured to capture the region to be treated. The device 1 includes an evaluation device 4 configured to determine the treatment trajectory of the treatment medium relative to the region to be treated from data from the image capturing device, and to determine a spatially resolved dose of the region to be treated based on the treatment trajectory. The device 1 also includes a presentation device 5 configured to optically display the treatment trajectory and / or the spatially resolved dose relative to the tissue for the user. Using the device 1 according to the invention, the user can be optically indicated which locations in the region to be treated on the tissue surface have been treated and / or the doses applied to the corresponding treatment locations.

[0087] Figure label: 1. Device for Image-Based User Support 2. Instruments 3. Image capture device (camera) 4. Evaluation device 5 Presentation Device 5a screen 5b Projection device 5c (AR-) Glasses 6. The head of the instrument 7. The distal end of the instrument 8. Proximal end of the instrument 9. Supply Unit (RF Generator, Gas Supply) 10 Processing Unit 11. Characteristics of the region 12 Reference Markers 13. Location of the processing medium 14. Size (diameter) of the processing medium A. Main axis of the instrument B. Area to be processed C11, …, Cmn resolving units D dosage fp erroneous positive mismatch FoV field of view P is the processing medium (non-thermal plasma). Pos_t represents the position of the processing medium in the current single image. Pos_t-1 represents the position of the processing medium within the previous single image. V_t-1 is the speed at which the processing medium was processed in the previous single image. Pos*_t represents the predicted location of the processing medium in the current single image. T Processing Trajectory S Search Area V1 Method Steps for Capturing the Region to be Processed V2 Methods and Steps for Determining the Processing Trajectory V3 Methods and steps for determining spatially resolved dose V4 presents the method steps for processing trajectory and / or dose. V11 Method Steps for Using the Current Single Image V20 Method and steps for determining possible changes in the field of view (FoV) relative to region B V21 Search and process methods for media (determining the location and / or size of the media) V22 Request Steps: Has the processing medium been found? V23 Methods and procedures for searching the head of an instrument (determining the head's position, orientation, and / or size) V24 Request Steps: Did the headers find? V25 Methods and Steps for Determining or Updating Processing Trajectories x-axis of the instrument y-axis of the instrument z is the z-axis of the instrument.

Claims

1. An apparatus (1) for supporting a user based on an image, comprising: - Instrument (2), which is configured to apply a treatment medium (P) at a variable position in the area to be treated (B); - Image capture device (3), which is configured to capture the area to be processed (B); - An evaluation device (4) configured to determine, based on data from the image capture device (3), the processing trajectory (T) of the processing medium (P) relative to the region to be processed (B), and to determine, based on the processing trajectory (T), the spatial resolution dose (D) of the region to be processed (B); and - Presentation devices (5, 5a, 5b, 5c) are configured to optically display the processing trajectory (T) and / or the spatially resolved dose (D) to the user relative to the area to be processed (B). The evaluation device (4) is communicatively connected to the supply unit (9) of the instrument (2), thereby the evaluation device (4) is configured to determine the spatially resolved dose (D) based on the measurement parameters transmitted from the supply unit (9) to the evaluation device (4).

2. The apparatus (1) according to claim 1, characterized in that, The processing medium (P) that can be applied is plasma.

3. The apparatus (1) according to claim 2, characterized in that, The plasma is a low-temperature or non-thermal plasma.

4. The apparatus (1) according to claim 1 or 2, characterized in that, The image capture device (3) is configured to capture at least part of the region to be processed (B) through an image sequence comprising multiple individual images.

5. The apparatus (1) according to claim 4, characterized in that, The evaluation device (4) is configured to identify the processing medium (P) and / or the head (6) of the instrument (2) in the image sequence.

6. The apparatus (1) according to claim 5, characterized in that, The evaluation device (4) is configured to track the head (6) of the processing medium (P) and / or the instrument (2) on multiple individual images in the image sequence.

7. The apparatus (1) according to claim 4, characterized in that, The evaluation device (4) is configured to determine the position and / or size of the applied processing medium (P) in the current single image of the image sequence.

8. The apparatus (1) according to claim 5, characterized in that, The evaluation device (4) is configured to determine the position, orientation and / or size of the head (6) of the instrument (2) in the current single image of the image sequence.

9. The apparatus (1) according to claim 1 or 2, characterized in that, The evaluation device (4) is configured to identify the processing medium (P) in the data of the image capture device (3) based on color and / or shape features.

10. The apparatus (1) according to claim 9, characterized in that, The evaluation device (4) is configured to identify the processing medium (P) in the data of the image capture device (3) based on at least one of the following characteristics: color value range, brightness value range, and saturation value range.

11. The apparatus (1) according to claim 5, characterized in that, The evaluation device (4) is configured to identify the head (6) from the data of the image capture device (3) based on color and / or shape features or based on a mark attached to the head (6).

12. The apparatus (1) according to claim 5, characterized in that, The evaluation device (4) is configured to identify the processing medium (P) in a search region (S) in the current single image by means of a search routine, thereby modifying the search region for the processing medium (P) based on the position, orientation and / or size of the head (6).

13. The apparatus (1) according to claim 12, characterized in that, The evaluation device (4) is configured to determine the predicted position (Pos*_t) of the processing medium (P) in the current single image based on the position (Pos_t-1) and / or velocity (V_t-1) of the processing medium (P) from a previous single image.

14. The apparatus (1) according to claim 13, characterized in that, The search routine identifies the search region (S) of the processing medium (P) in the current single image and modifies it based on the predicted position (Pos*_t) and / or the velocity (V_t-1).

15. The apparatus (1) according to claim 1 or 2, characterized in that, The evaluation device (4) is configured such that the area to be processed (B) can be defined by the user before the processing medium (P) is applied.

16. The apparatus (1) according to claim 1 or 2, characterized in that, The evaluation device (4) is configured to identify characteristic image features of the region to be processed (B) in the data of the image capture device (3), or to identify reference marks applied in the region (B) on the tissue surface, and to assign the processing trajectory (T) to the region to be processed (B) in a spatially precise manner.

17. The apparatus (1) according to claim 1 or 2, characterized in that, The evaluation device (4) is configured to control the application of the treatment medium (P) through the instrument (2) based on the local dose.

18. A method for supporting a user based on an image, comprising: - Capture (V1) the area (B) to be processed in which the processing medium (P) can be applied at a variable position; - Determine (V2) the processing trajectory (T) of the processing medium (P) relative to the region (B); - Based on the processing trajectory (T) and the measurement parameters transmitted from the supply unit (9) to the evaluation device (4) via the communication connection, determine (V3) the spatially resolved dose (D) of the area to be treated (B). - To present (V4) the processing trajectory and / or the spatially resolved dose (D) regarding the region to be processed (B).

Images