Automatic positioning of structures

Abstract

According to a computer-implemented method for automatically positioning a structure (9), a single image (17) is displayed on a display surface (12) that maps an object having a structure (9). A gaze acquisition system (18) acquires a gaze movement of a user (19) and generates a gaze acquisition signal based on the gaze movement. A computing unit (13) is used to identify at least one image point (20) of the single image (17) based on the gaze acquisition signal, the at least one image point at least partially mapping the structure (9).

Description

Technical Field

[0001] The invention relates to a computer-implemented method for automatically positioning a structure, wherein a single image of an object with the structure is displayed on a display surface. The invention also relates to a device for automatically positioning a structure, a computer-implemented method for training a software module and a computer program. Background Art

[0002] In various applications in which objects are analyzed or manipulated by a person in an image-assisted manner, such as in the field of medical applications, the correct determination of the position of structures in the image data is of decisive importance for further method steps. In medicine, this is the case, for example, for analysis methods or diagnostic methods or for interventions supported by imaging methods (e.g. fluoroscopy-based methods). In particular, for many medical applications, the correct segmentation of anatomical objects is decisive for further treatment steps or analysis steps.

[0003] Existing segmentation methods and other methods for locating structures in image data require user input or selection of image points or image regions by the user in order to provide suitable results. For example, the user may manually place seed points, which may also be referred to as initial points for segmentation, within the anatomical structure of interest or draw an approximate contour around the structure.

[0004] Manual interaction of the user with the positioning system or segmentation algorithm is disadvantageous because it interrupts the user's workflow, distracts the user's attention and takes up corresponding time. In addition, the interaction requires physical manipulation of the input device by the user, which is particularly disadvantageous in a sterile environment. Summary of the invention

[0005] Therefore, the technical problem to be solved by the present invention is to provide an improved design for automatically locating a structure in an image, wherein the improved design can achieve a higher degree of automation and reduce manual steps for locating the structure.

[0006] According to the invention, the above-mentioned technical problem is solved by the corresponding subject matter of the invention. Advantageous refinements and further embodiments are the subject matter of the invention.

[0007] The improved design is based on the following idea: the gaze movement of the user is captured by means of a gaze capture system, and at least one image point related to the structure is identified, in particular automatically identified, based on the captured gaze movement.

[0008] According to an independent aspect of the improved design, a computer-implemented method for automatically positioning a structure is proposed. In this case, a single image that maps an object having the structure is displayed on a display surface, in particular a display surface of a display device, in particular by means of a computing unit. In particular, during the display of the single image on the display surface, a line of sight movement of a user, in particular a user of an imaging device or a device for automatically positioning the structure is captured by means of a line of sight capture system. Based on the captured line of sight movement, a line of sight capture signal is generated, in particular by means of a line of sight capture system. With the computing unit, at least one image point of the single image is identified, in particular automatically, based on the line of sight capture signal, wherein the at least one image point at least partially maps the structure.

[0009] Positioning a structure may be understood as determining the position of the structure or a part of a structure.

[0010] The object may be, for example, a part of a human or animal body, such as one or more organs or other anatomical objects, or a part of a plant. The object may also be an inanimate object.

[0011] In particular, a structure is a region of interest of an object, i.e., a region relevant to the analysis, treatment or manipulation of the object. For example, it can be a point or region of an object. In medicine, a structure can be a part of an anatomical structure or an anatomical object. A structure can be, for example, an organ, an anatomical vessel, a tumor or a corresponding part thereof. Here, the term tumor can be based on a broad understanding, so that in addition to the neoplasms of body tissue, other tissue solid tumors, cysts, etc. can also be included.

[0012] In particular, a single image is a representation of a corresponding image data set, which contains, for example, at least the grayscale values ​​of individual pixels of the image data set and the associated positions of the individual pixels.

[0013] In particular, the single image may be a two-dimensional image of the object or a two-dimensional part of a three-dimensional representation of the object.

[0014] In particular, the at least one pixel is a group of pixels, for example a coherent group of pixels.

[0015] The image data set may be present in a stored form, for example. To display a single image on a display surface of a display device, the computing unit may access the stored image data set. To display the image data set, the computing unit or the display device may also receive the image data set in real time from the imaging device.

[0016] At least one pixel at least partially maps a structure, for example, which can be understood as meaning that a complete mapping of the structure includes at least one pixel. In other words, the structure is represented in a single image by a large number of pixels including at least one pixel.

[0017] In particular, the recognition of the at least one pixel comprises the determination of the position of the at least one pixel, in particular of each of the at least one pixel, in the individual images, ie in particular the localization of the at least one pixel.

[0018] In other words, according to the improved design, it is determined which image points of the single image correspond to the user's line of sight movement or to the line of sight position on the display surface determined by the line of sight movement. In particular, by associating at least one image point, this goes beyond simply determining the line of sight position on the display surface and tracking the line of sight position if necessary.

[0019] By means of the improved design, use is made of the fact that the user of the device can usually distinguish relevant image areas from less relevant image areas with high accuracy and reliability based on corresponding experience or expertise. Accordingly, the user's gaze is preferably moved to the location on the display surface that belongs to the structure. Thus, in particular in the case of a physician or other medical personnel, the user's expert knowledge, which is present in the form of the user's gaze movement and is "stored" to a certain extent, is used to identify at least one image point without further manual interaction of the user with the system.

[0020] Thus, the positioning of the structure or a possible subsequent anatomical segmentation is fully or partially automated and the manual interaction of the user with the system is reduced, so that the user's workflow does not have to be interrupted in order to, for example, manually set or define seed points or approximate contours of the structure. In particular in a sterile environment, it is advantageous to thereby dispense with the operation of input devices such as a computer mouse or keyboard.

[0021] Therefore, according to the improved design, the user's experience can be effectively utilized without having to explicitly express the user's experience according to rules, work instructions, etc.

[0022] The at least one recognized image point can be used, for example, by a computing unit for segmentation of the structure, in particular anatomical segmentation.

[0023] In various embodiments, the at least one recognized image point can be used by means of a computing unit to localize a characteristic position or a relevant position in the object, ie, to localize a so-called segment marker on the image.

[0024] According to at least one embodiment of the computer-implemented method for automatically positioning a structure according to the improved design, the gaze position on the display surface is determined based on the gaze acquisition signal by means of a computing unit. In particular, at least one image point is determined based on the gaze position by means of the computing unit.

[0025] In other words, based on the user's line of sight movement, determine which point or area of ​​the display surface the user is looking at, or which area of ​​the display surface the user is focusing on. By associating the line of sight position with at least one image point, determine at least one image point based on the line of sight position.

[0026] Since the at least one image point corresponds to the viewing position of the user, it can be assumed that the at least one image point represents a region of increased interest to the user, so that this region of increased interest to the user can be used to locate the structure and, for example, to segment the structure.

[0027] For example, using an image region corresponding to a line-of-sight position as a seed point for segmentation may replace manual input of a seed point by a user.

[0028] According to at least one embodiment, an image sequence comprising a single image and at least one further single image is displayed on a display surface, in particular by means of a computing unit. In particular, a gaze acquisition system is used to acquire a gaze movement of a user, for example continuously, during the display of the image sequence, and in particular, a gaze acquisition signal is generated based on the gaze movement of the user acquired during the display of the image sequence, in particular by means of the gaze acquisition system.

[0029] In particular, to display the image sequence on the display surface, individual images, in particular an individual image and at least one further individual image, are displayed repeatedly in sequence and if appropriate periodically, wherein each individual image or further individual image is displayed for a predefined display duration.

[0030] In particular, capturing the gaze movement of the user during display of the image sequence may be understood as capturing the gaze movement during display of a single image and during display of each of the further single images.

[0031] In various embodiments, the individual image and the at least one further individual image form a plurality of at least two individual images, in particular at least five, preferably at least ten or at least 30 individual images.

[0032] The display of a sequence of images allows more information or a higher information density to be provided on the display surface by means of a dynamic representation of an object or structure, such as the movement of the object or structure. In medicine, for example, blood flow can also be visualized by means of a sequence of images.

[0033] Due to the higher information density provided to the user, the user can identify structures with higher reliability or accuracy. Accordingly, the analysis of the line of sight acquisition signal or line of sight movement is also more reliable or more accurate. Therefore, a more accurate positioning of the structure can be achieved and, if necessary, a more accurate segmentation of the structure can be achieved.

[0034] According to at least one embodiment of the method, at least one further image point is respectively identified for each of the at least one further individual image, in particular by means of a computing unit, depending on the line of sight detection signal, wherein the respective at least one further image point at least partially maps the structure.

[0035] In this case, the at least one further pixel can correspond, for example, to at least one pixel of a single image or can be located at a different location of the structure.

[0036] According to at least one embodiment, a corresponding additional viewing position on the display surface is determined for each additional individual image based on the viewing detection signal, in particular by means of a computing unit. Based on the corresponding additional viewing position, at least one corresponding additional image point is determined.

[0037] According to at least one embodiment, the position of the at least one detected pixel and the corresponding position of each of the at least one further pixel, if applicable, is stored on a storage medium, in particular by means of a computing unit and in particular as a function of the line of sight detection signal.

[0038] According to at least one embodiment, the display of the image sequence is repeated at least once, in particular multiple times, for example periodically. By means of the gaze acquisition system, gaze movement is acquired during multiple repetitions, i.e., during at least two repetitions or during all repetitions, and a gaze acquisition signal is generated based on the gaze movement acquired during the multiple repetitions.

[0039] By means of a number of repetitions, if necessary, the position of the image point or the user's gaze position can be verified and / or additional gaze positions can be determined for a single image or for further single images. Accordingly, at least one image point or respectively at least one further image point can be determined with greater accuracy and greater reliability.

[0040] According to at least one embodiment, the display of the image sequence is paused, for example, by means of a computing unit. During the pause, in particular, the single image or one of the further single images is displayed statically. During the pause, the gaze movement is captured by means of a gaze capture system, and a gaze capture signal is generated based on the gaze movement captured during the pause.

[0041] The pause may be initiated, for example, automatically or by user input.

[0042] In this case, for example, a pause can be effected for a predefined duration or the pause can be ended by a further user input.

[0043] By pausing and the correspondingly longer duration for which the user can gaze at the corresponding individual images, additional sight positions and corresponding image points of the mapped structures can be acquired or determined for each individual image. In particular, the number of required repetitions of the image sequence can be reduced and / or the localization of the structures can be performed more accurately.

[0044] According to at least one embodiment, the object is imaged by means of an imaging device in order to generate an individual image and, if appropriate, at least one further individual image.

[0045] In particular, the imaging device can be an X-ray device, a computer tomography device, a magnetic resonance tomography device, a device for radioactive nuclear radiation or an ultrasound device.

[0046] In particular, an imaging device is a device which allows time-resolved or time-dependent mapping of an object.

[0047] According to at least one embodiment, at least one gaze movement of at least one other user is captured by the gaze capture system, and based on the captured at least one other gaze movement, at least one other gaze capture signal is generated by the gaze capture system. At least one image point is identified from the gaze capture signal and the other gaze capture signal by means of a computing unit.

[0048] In this embodiment, not only the experience of the user who performs the analysis or manipulation of the object, but also the experience of other users, such as auxiliary users, can be used to locate the structure. Thus, further improved accuracy or reliability of positioning or segmentation can be achieved.

[0049] Here, the sight acquisition system may include, for example, a separate sight acquisition system for each of the user and the other users. For example, the separate sight acquisition systems may be designed as devices that can be worn on the head, and the corresponding users may wear the devices similar to glasses, for example.

[0050] According to at least one embodiment, the structure is segmented at least partially automatically by means of a computing unit based on at least one pixel and, if appropriate, at least one further pixel.

[0051] In particular, the segmentation of a structure can be understood as identifying all relevant image regions belonging to the structure in a single image or in at least one further single image. In particular, the structure as a region of interest is thereby separated from less relevant regions (eg, surrounding regions around the structure).

[0052] In particular, at least partially automatic segmentation can be understood to mean that, with the aid of the computing unit, regions of structures are automatically identified based on at least one pixel or at least one further pixel, which regions of the structure do not themselves correspond to the at least one pixel or at least one further pixel. Manual input by the user (e.g. for confirming, correcting or supplementing the segmentation) is possible, but not necessary.

[0053] For example, at least one image point and / or at least one further image point can be used as a corresponding seed point for segmentation. The segmentation can be performed by means of a computing unit, for example based on conventional image processing methods (e.g. methods for edge detection, for contrast analysis or for gray value analysis), or according to an algorithm that has been trained by machine learning.

[0054] According to at least one embodiment, the structures are displayed visually emphasized in individual images or in a sequence of images on the display surface, in particular by means of a computing unit.

[0055] According to another independent aspect of the improved design, a method for automatically segmenting an anatomical structure is provided. Here, a single image mapping an object having a structure is displayed on a display surface, a user's line of sight movement is captured by a line of sight capture system, and a line of sight capture signal is generated based on the line of sight movement. By means of a computing unit, at least one image point of the single image is identified based on the line of sight capture signal, wherein the at least one image point at least partially maps the structure. The structure is segmented at least partially automatically based on the at least one image point by means of the computing unit.

[0056] Further embodiments of the method for automatically segmenting an anatomical structure follow directly from different embodiments of the method for automatically localizing a structure.

[0057] According to a further independent aspect of the improved design, a computer-implemented method for training a software module for machine learning, in particular a software module comprising a computer program or software for machine learning, is proposed. In this case, for example, a training single image of a training object mapped with a training structure is displayed on a display surface, in particular a display device, by means of a training computing unit. The gaze movement of the user is acquired by means of a gaze acquisition system, and based on this, i.e. based on the acquired gaze movement, a training signal is generated, in particular by means of a training computing unit. The training structure is segmented at least partially automatically based on the training signal by means of the training computing unit. Based on the result of the segmentation, the software module is trained by means of the training computing unit.

[0058] In particular, the software module is a source code stored on a computer-readable storage medium, or a compilation of multiple source codes. The training computing unit can access the software module in order to implement the corresponding source code or the compilation of source codes. The training computing unit can also access the computer-readable storage medium by writing, in order to, for example, adjust the software module to train the software module, in particular to adjust or change the parameters of the software module.

[0059] In particular, software or a computer program for machine learning may be understood as software or a computer program that has been trained according to a method for machine learning or can be trained according to a method for machine learning.

[0060] The software module for machine learning can be designed, for example, as an automatic classifier, an artificial neural network (particularly a convolutional artificial neural network), a software module for reinforcement learning, a software module for cluster analysis, a support vector machine, etc.

[0061] In particular, the training of the software module by means of the training computing unit includes adjusting one or more parameters of the software module or source code stored on the software module according to the segmentation results. In the case of a neural network, the training includes in particular adjusting the corresponding weights of the neural network according to the segmentation results.

[0062] According to at least one embodiment, the training of the software module includes comparing the segmentation result with a reference result, ie, for example, a predefined reference segmentation of the structure. Alternatively or additionally, the training of the software module may also include verifying or falsifying the automatic segmentation by user input.

[0063] According to at least one embodiment, a computer-implemented method for training a software module is performed during or as part of a computer-implemented method for automatically positioning a structure according to an improved design.

[0064] Therefore, in the method for automatic positioning according to the improved design, during the operational use of the software module, the training or the method for training the software module can be used in real time. In particular, the computing unit can use the software module in operational use to segment the structure.

[0065] In particular, the training computing unit may be a computing unit or a further computing unit.

[0066] In various embodiments of the method for training a software module, a single image of the method for automatically locating a structure is used as a training single image. The training object then corresponds to the object, the reference structure corresponds to the structure, and the training signal corresponds to the line of sight acquisition signal.

[0067] According to at least one embodiment, at least one pixel of a training single image is identified based on the training signal by means of a training computing unit, wherein the at least one pixel of the training single image at least partially maps a training structure. The training structure is segmented at least partially automatically based on the at least one pixel by means of the training computing unit.

[0068] According to at least one embodiment of the computer-implemented method for automatically positioning a structure, the structure is segmented at least partially automatically with the aid of a software module for machine learning, in particular by means of a computing unit, wherein the software module is trained or has been trained according to a computer-implemented method for training a software module for machine learning according to an improved design.

[0069] The advantages of the software module for machine learning are high speed, robustness and flexible or universal usability and applicability. As described, the quality of the segmentation can be continuously improved during operation.

[0070] According to another independent aspect of the improved design, a device for automatically locating a structure is provided. The device has a display device with a display surface and a computing unit, and the device is configured to display a single image mapping an object having a structure on the display surface. The device has a sight line acquisition system, which is configured to acquire the sight line movement of a user of the device and generate a sight line acquisition signal based on the sight line movement. The computing unit is configured to identify at least one image point of the single image based on the sight line acquisition signal, wherein the at least one image point at least partially maps the structure.

[0071] Gaze acquisition systems can also be called eye trackers.

[0072] According to at least one embodiment of the device, the device has an imaging device which is configured to map the object in order to generate a single image.

[0073] For example, the imaging device is an X-ray device, a computer tomography device, a magnetic resonance tomography device or an ultrasound device.

[0074] According to at least one embodiment, the computing unit is configured to segment the structure at least partially automatically based on the at least one image point.

[0075] According to at least one embodiment, the device has a software module for machine learning, for example a computer-readable storage medium, on which the software module is stored, wherein the software module has been trained according to the method for training a software module for machine learning according to the improved design. The computing unit is configured to segment the structure at least partially automatically with the aid of the software module.

[0076] Further embodiments of the device are derived directly from different designs of the computer-implemented method for automatically positioning a structure according to the improved design and from different designs of the computer-implemented method for training a software module for machine learning according to the improved design, and vice versa. In particular, the device according to the improved design can be configured or programmed to perform the method according to the improved design, or the device performs such a method.

[0077] According to another independent aspect of the improved design, a training device for training a software module for machine learning is proposed. The training device has a display device with a display surface and a training calculation unit, and the training device is configured to display a training single image of a training object with a training structure mapped on the display surface. The training device has a sight line acquisition system, which is configured to acquire the sight line movement of the user of the training device and generate a training signal based on the sight line movement. The training calculation unit is configured to automatically segment the training structure at least partially based on the training signal. The training calculation unit is configured to train the software module based on the segmentation result.

[0078] Further embodiments of the training device result directly from different embodiments of the computer-implemented method for training a software module according to an improved embodiment.

[0079] According to a further independent aspect of the improved design, a computer program with commands is provided. When the computer program is executed by the device for automatically positioning a structure according to the improved design, in particular by a computing unit of the device, the commands cause the device to execute the method for automatically positioning a structure according to the improved design or the method for training a software module for machine learning according to the improved design.

[0080] According to a further independent aspect of the improved design, a computer-readable storage medium is provided, on which a computer program according to the improved design is stored.

[0081] Features and feature combinations mentioned previously in the description and subsequently mentioned in the description of the figures and / or shown individually in the figures may be used not only in the respectively stated combination but also in other combinations without departing from the scope of the invention. Embodiments and feature combinations which do not have all the features of the originally formulated independent claims and / or which exceed or deviate from the feature combinations stated in the subsequent references to the claims are also considered disclosed. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] The present invention will be described in more detail below based on specific embodiments and related schematic diagrams. In the accompanying drawings, identical or functionally identical elements may have identical reference numerals. The description of identical or functionally identical elements need not be repeated relative to different drawings when necessary.

[0083] In the attached figure:

[0084] Figure 1 A schematic diagram showing an exemplary embodiment of an apparatus for automatically positioning a structure according to an improved design;

[0085] Figure 2 A flow chart illustrating an exemplary embodiment of a method for automatically positioning a structure according to an improved design;

[0086] Figure 3 A single image with a structure is shown;

[0087] Figure 4 showing a further single image with the structure; and

[0088] Figure 5 A schematic diagram of the segmented structure is shown. DETAILED DESCRIPTION

[0089] Figure 1 Detailed description of the invention is given in FIG. 1 , which is a schematic diagram of an exemplary embodiment of a device 10 for automatically positioning a structure according to an improved design.

[0090] The device 10 has a computing unit 13 and a display device 11 (eg a monitor with a display surface 12 ).

[0091] The device 10 has, for example, an imaging device 22 which is arranged Figure 1 In the example of is designed by way of example but not limitation as an X-ray device, for example as a C-arm X-ray device.

[0092] In this example, the imaging device 22 has an X-ray source 23 and an X-ray detector 24 and a support surface 26 on which an object 27 (e.g., a person) can be arranged. The X-ray source 23 emits X-rays, which at least partially pass through the object 27 and are correspondingly collected by the X-ray detector 24.

[0093] The X-ray detector 24 is connected to the computing unit 13 , in particular to the X-ray image processor 14 of the computing unit 13 , in order to transmit detector signals based on the detected X-rays to the X-ray image processor 14 .

[0094] The X-ray image processor 14 can generate image data, in particular based on the detector signals, and can display the image data on the display surface 12 as a single image or as an image sequence of a plurality of successive single images.

[0095] In addition, the device 10 includes an eye tracker 18, which is connected to the analysis unit 15 of the computing unit 13. The eye tracker 18 can collect the visual movement of the user 19 who is looking at the display surface 12, generate a visual collection signal based on the collected visual movement, and transmit the visual collection signal to the analysis unit 15.

[0096] In this case, the eye tracker 18 can be of fixed design and, for example, can be arranged on the display device 11. However, the eye tracker 18 can also be designed as a mobile eye tracker as a head-wearable or so-called head-mounted eye tracker, which the user 19 wears on his head like glasses.

[0097] In various embodiments, the head-wearable output device (also referred to as video glasses or head-mounted display) may include a display device 11, in particular a display surface 12, and an eye tracker 18. For example, the head-wearable output device may be designed as so-called virtual reality glasses or augmented reality glasses.

[0098] The computing unit 13 has a storage medium with a software module 16, which contains a segmentation algorithm in the form of corresponding source code. In this case, the segmentation algorithm can contain, in particular, an artificial neural network or another software framework based on machine learning.

[0099] Reference below Figure 2 The working principle of the device 10 is explained in more detail. Figure 2 Detailed description of the invention A flow chart of an exemplary embodiment of a method for automatically positioning a structure according to an improved design is shown in FIG.

[0100] In step 1 of the method, the imaging device 22 generates a detector signal by means of the X-ray detector 24 and transmits the detector signal to the X-ray image processor 14 .

[0101] In step 2 , based on the detector signals, the X-ray image processor 14 generates one or more image data sets corresponding to one or more individual images, and displays at least one individual image 17 on the display surface 12 .

[0102] In step 3, while the individual images 17 are displayed on the display surface 12, the gaze movement of the user 19 is detected by means of the eye tracker 18. Based on this, a gaze detection signal is generated and transmitted to the evaluation unit 15, as described.

[0103] In step 4 of the method, the analysis unit 15 determines one or more gaze positions of the user 19 on the display surface 12 based on the gaze acquisition signal. In other words, the analysis unit 15 determines which position or positions the user 19 is observing or fixating on the display surface 12, that is, in particular on the corresponding single image 17. To this end, for example, based on the gaze acquisition signal, the analysis unit 15 can determine and process the corresponding dwelling duration of the user 19's gaze at the corresponding position of the display surface 12.

[0104] Likewise, in step 4 , the evaluation unit 15 receives the generated image data set from the X-ray image processor 14 and associates the determined viewing position with a corresponding region of the individual images 17 , ie in particular with a region of an image point of the individual images 17 .

[0105] This is Figure 3 , where a single image 17 is shown, for example, showing the vascular system of a human being, which has been mapped by means of an imaging device 22. Different gaze positions 20, 20', 20", 20'" are shown, at which the gaze of a user 19 has been fixed for a certain time and which can be correspondingly identified by the analysis unit 15 based on the gaze acquisition signal and can be associated with corresponding image points of the single image 17. A part of the vascular system, in particular a single blood vessel, is a structure 9 which is to be segmented, for example, by means of the device 10.

[0106] In optional step 5 of the method, for example, the steps explained with respect to steps 2, 3 and 4 of the method are repeated, wherein the single image 17 can be displayed statically during this time. In this way, the number of identified points on the structure 9 and the number of viewing positions 20, 20', 20", 20'" in the single image 17 can be correspondingly increased.

[0107] Alternatively or additionally, the steps explained with respect to steps 2 to 4 can be repeated for another single image 17' of the image sequence in step 5. For example, the image sequence can be played as a loop on the display surface 12. For example, the user 19 can pause the playback in order to observe one of the single images 17, 17' more accurately and, as described, further increase the number of identified gaze positions.

[0108] Figure 4 In step 5 of the method, as described with respect to the single image 17, in particular in the further single image 17' further viewing positions 21, 21', 21", 21'" on the vessel 9 or structure 9 are identified.

[0109] As from Figure 3 and Figure 4As can be seen from the comparison, the sight lines positions 20, 20', 20", 20'" of the first single image 17 are not necessarily the same as the other sight lines positions 21, 21', 21", 21'" of the other single image 17'. As a result, the amount of information that can be considered by the computing unit 13 for segmentation can be increased. In particular, the computing unit 13 can assume with corresponding probabilities that all sight lines positions 20, 20', 20", 20'", 21, 21', 21", 21'" are located on the structure 9 or approximately on the structure 9.

[0110] In step 6 of the method, the analysis unit 15 implements a segmentation algorithm of the software module 16. In particular, the analysis unit 15 uses regions or image points of the individual images 17, 17' that are associated with the viewing positions 20, 20', 20", 20'", 21, 21', 21", 21'" (e.g. the positions of these regions or image points) in order to identify a coherent region that defines the structure 9 as completely as possible.

[0111] Figure 5 The result of the segmentation of the structure 9 is schematically shown in . The result of the segmentation can be understood as coherent or incoherent regions of image points of one or more individual images 17 , 17 ′ which have been identified as being associated with the structure 9 .

[0112] like Figure 5 As shown in , this information can be displayed on the display area 12 , for example, by means of the evaluation unit 15 or the X-ray image processor 14 , by visually emphasizing the result of the segmentation.

[0113] In an optional step 7 of the method, the segmentation result can be verified or falsified, for example, by a user 19. Based on the result of the verification or falsification, the computing unit 13, in particular the analysis unit 15, can adjust the parameters of the segmentation algorithm (for example, the weights of the neural network used for this purpose) in step 8 in order to reduce errors and thereby further train and optimize the software module 16, in particular the already pre-trained one.

[0114] Alternatively or additionally, in step 7 , the analysis unit 15 can compare the segmentation result with predefined reference data and, based on the result of the comparison, carry out an adjustment of the parameters of the software module 16 .

[0115] In the further segmentation by means of the method or device 10 according to the improved design, the adapted software modules 16 can then be used.

[0116] In other embodiments, the software module 16 is trained in advance, ie, before operation of the device 10 or the software module 16 is started.

[0117] As described, according to the improved design, in particular in the medical segmentation of anatomical objects, it is possible to achieve an increased degree of automation for locating structures on the object and thus make manual interaction of the user at least partially superfluous.

[0118] Doctors and other users usually have experience in identifying structures in images from imaging methods, in particular in image sequences, as used, for example, in the diagnosis, analysis or treatment of patients with radiation support. According to an improved design, the experience of these personnel can be used directly to automatically locate the structure through their line of sight movements.

[0119] According to an improved design, a gaze acquisition system or an eye tracker is used, which continuously monitors the gaze of the user in different embodiments. As a result, the exact gaze position of the user can be tracked. The eye tracker is synchronized with the computing unit and, if necessary, with the imaging device, so that the gaze direction or gaze position of the user can also be tracked over a certain period of time, in particular over a large number of individual images in an image sequence.

[0120] In this way, information can be extracted from the gaze movement. This gaze information can be integrated into the localization algorithm or the segmentation algorithm to replace manual input via other input devices.

[0121] The line of sight information may also make landmark recognition easier. If, for example, there is stenosis in an angiographic image, the user's line of sight may contain information about the location of the stenosis. In some embodiments, the line of sight information may also support diagnosis, as the user's line of sight direction may follow different patterns for different diagnostic results. For example, the line of sight information for a coronary angiography in the presence of stenosis may be different from the line of sight information for a coronary angiography in the presence of a healthy vessel.

[0122] In various embodiments, the gaze acquisition signal or gaze information can be used as an input parameter for a machine learning-based algorithm to perform localization or segmentation. In this way, the algorithm can use additional information "stored" in the user's gaze movement.

[0123] With different embodiments of the improved design, for example, mouse clicks or other manual inputs for manually defining initialization points or regions for segmentation can be replaced and the degree of automation in the segmentation can thus be significantly increased.

[0124] A further advantage of the improved design is that since the user intuitively follows the contrasting blood vessels or organs with his gaze, the gaze information in the form of the gaze acquisition signal of the computing unit allows compensation for object movements caused, for example, by breathing or organ movements.

[0125] Compared to conventional methods for manually positioning structures, the improved design does not require interrupting the user's workflow. Since the user is usually already using the display surface, the eye movement can be captured by the user himself without further overhead.

[0126] Due to the increased degree of automation, workflows, in particular clinical workflows, can be accelerated.

Claims

1. A computer-implemented method for automatically positioning a structure (9), wherein: Displaying a single image (17) of an object having the structure (9) on a display surface (12); It is characterized in that Using a sight line acquisition system (18) to acquire the sight line movement of a user (19), and generating a sight line acquisition signal based on the sight line movement; and By means of a computing unit (13), at least one image point (20) of a single image (17) is identified based on the line of sight acquisition signal, wherein the at least one image point (20) at least partially reflects the structure (9), and Displaying an image sequence comprising the single image (17) and at least one further single image (17') on the display surface (12); During display of the image sequence, the gaze movement is acquired by means of the gaze acquisition system (18), and the gaze acquisition signal is generated based on the gaze movement; And wherein, based on the line of sight acquisition signal, at least one further image point (21) is respectively identified for each further single image (17'), wherein the at least one further image point (21) at least partially maps the structure (9).

2. The computer-implemented method of claim 1, It is characterized in that Determining the sight line position on the display surface (12) according to the sight line acquisition signal by means of the calculation unit (13); and The at least one image point (20) is determined based on the line-of-sight position.

3. The computer-implemented method of claim 1, It is characterized in that Repeating the display of the image sequence at least once; The gaze movement is detected by means of the gaze detection system (18) during a plurality of repetitions, and the gaze detection signal is generated based on the gaze movement.

4. The computer-implemented method according to claim 1 or 2, It is characterized in that Pause the display of the image sequence; During the pause, displaying one of the single image (17) or another single image (17'); During the pause, the sight line movement is captured by means of the sight line capture system (18), and the sight line capture signal is generated based on the sight line movement.

5. The computer-implemented method according to claim 1 or 2, It is characterized in that Using the sight line acquisition system (18) to acquire the sight line movement of another user, and generating another sight line acquisition signal based on the sight line movement; as well as The at least one image point (20) is identified by means of the computing unit (13) based on the further line of sight detection signal.

6. The computer-implemented method according to claim 1 or 2, It is characterized in that The object is mapped by means of an imaging device (22) in order to generate the single image (17).

7. The computer-implemented method according to claim 1 or 2, It is characterized in that The structure (9) is segmented at least partially automatically based on the at least one image point (17) by means of the computing unit (13).

8. A computer-implemented method for training a software module (16) for machine learning, wherein: Displaying a training single image of a training object having a training structure mapped thereon on a display surface (12); Using a sight line acquisition system (18) to acquire the sight line movement of a user (19), and generating a training signal based on the sight line movement; as well as segmenting the training structure at least partially automatically based on the training signal by means of a training computing unit; training the software module (16) with the aid of the training computing unit based on the results of the segmentation, By means of a computing unit, at least one pixel of a single training image is identified based on the training signal, wherein the at least one pixel at least partially reflects the training structure, and displaying an image sequence comprising the training single image and at least one further training single image on the display surface; capturing the gaze movement by means of the gaze capture system during display of the image sequence, and generating the training signal based on the gaze movement; And wherein, based on the training signal, at least one further image point is respectively identified for each further individual training image, wherein the at least one further image point at least partially reflects the training structure.

9. The computer-implemented method of claim 8, It is characterized in that The structure (9) is segmented at least partially automatically by means of a software module (16) for machine learning, wherein the software module (16) is trained or has been trained according to the method as claimed in claim 8.

10. An apparatus for automatically positioning a structure (9), the apparatus (10) comprising a display device (11) with a display surface (12) and a computing unit (13), the apparatus being configured to display a single image (17) of an object having the structure (9) on the display surface (12); It is characterized in that The device (10) has a sight line acquisition system (18), which is configured to acquire sight line movement of a user (19) of the device (10) and generate a sight line acquisition signal based on the sight line movement; as well as The computing unit (13) is configured to identify at least one image point (20) of the single image (17) based on the line of sight acquisition signal, wherein the at least one image point (20) at least partially maps the structure (9), Using the device, a sequence of images comprising the single image and at least one further single image is displayed on the display surface; and acquiring the sight line movement by means of the sight line acquisition system during display of the image sequence, and generating the sight line acquisition signal based on the sight line movement; And wherein, based on the line of sight acquisition signal, at least one further image point is respectively identified for each further individual image, wherein the at least one further image point at least partially maps the structure.

11. The device according to claim 10, It is characterized in that The computing unit (13) is configured to segment the structure (9) at least partially automatically based on the at least one image point (20).

12. The device according to claim 11, It is characterized in that The device (10) has a software module (16) for machine learning, which has been trained according to the method according to claim 8; and The computing unit (13) is configured to segment the structure (9) at least partially automatically with the aid of the software module (16).

13. A computer program product having instructions which, when the computer program product is executed by a device (10) according to any one of claims 10 to 12, cause the device (10) to carry out a method according to any one of claims 1 to 9.

Images