Method and device for image processing, and computer-readable storage medium

Abstract

According to various aspects, a method is provided which comprises: receiving a three-dimensional reference image of a region in which an object to be manipulated is arranged, the three-dimensional reference image having a plurality of volume pixels; carrying out a three-dimensional rendering using the plurality of volume pixels in order to generate a rendered reference image; receiving one or more volume pixel coordinates for forming one or more target points in the rendered reference image; arranging the volume pixels of the three-dimensional reference image in discrete image layers; identifying, for each image layer, the volume pixels that are part of the target points in that image layer; generating image-layer-specific instructions for an action on the basis of the identified volume pixels; generating an instruction for starting a process and generating control information, the control information containing instructions for performing, in the region, the action for manipulating the object according to the target points; sending the instruction for starting the process and the control information to a manipulation device.

Description

[0001] Be see e ibung

[0002] Method and apparatus for image processing and computer-readable storage medium

[0003] Several aspects of this disclosure relate to a method and apparatus for image processing and a computer-readable storage medium.

[0004] In many fields, such as biology and medicine, three-dimensional (3D) images of objects are used to facilitate the planning of object manipulations, such as sectioning a sample. Currently, user-planned manipulations are performed in a two-dimensional plane.

[0005] Conventional methods in this field include, for example, the reconstruction of the 3D positions of free-floating particles or bacteria from bright-field microscopy and the use of holographic tweezers to capture or move these objects in 3D, allowing the user to manipulate the object's position in real time. This results in a volumetric image whose isosurface contours represent the surface of the bacterium quite accurately, but not its entire volume (S. Ferretti, S. Bianchi, G. Frangipane, and R. Di Leonardo. A virtual reality interface for the immersive manipulation of live microscopic systems. 11(l): 7610. doi:10.1038 / s41598-021-87004-5).

[0006] Another conventional method involves a VR pipetting system to assist in in-vitro fertilization. This system required the use of a pen or glove for manual pipetting. In this method, three images are captured with different focal lengths, and the 3D position and size of a microscopic object are determined from these images. This is followed by a microscopic transformation of the objects into the coordinates of the immersive space (VR space). With this method, only predefined objects (such as oocytes) can be visualized in the VR space, with their size and position determined from three images taken at different focal lengths. (K. Yokoe, T. Aoyama, T. Fujishiro, M. Takeuchi, and Y. Hasegawa. An immersive micro-manipulation system using real-time 3D imaging microscope and 3D operation interface for high-speed and accurate micro-manipulation. 9(1):16. doi: 10.1186 / s40648-022-00228-6) .

[0007] US 20150032414 Al describes a method for 3D measurement using a laser scanning microscope in a 3D measurement room with a virtual reality (VR) system.

[0008] Several aspects of this invention constitute a device and a method for performing a three-dimensional rendering of a three-dimensional reference image of an object with a plurality of volume pixels. This enables a 3D visualization of the three-dimensional object, which can be done, for example, in virtual reality (VR) or augmented reality (AR). In virtual reality / augmented reality, the user can select volume pixels that form one or more target points (e.g., a plurality of target points). Furthermore, control information for a process, e.g., optical manipulation using an optical manipulation device, can be generated based on the formed target points. In the present description, aspects described with respect to "one or more target points" can also apply analogously to "a plurality of target points" and vice versa.

[0009] Exemplary embodiments of the invention are shown in the figures and are explained in more detail below with reference to the following figures, in which:

[0010] Figure 1 shows an example of a device for carrying out a method for performing a three-dimensional rendering of a three-dimensional reference image of an object with a plurality of volume pixels; and

[0011] Figure 2 shows a 3D visualization 208 from the user's point of view.

[0012] Figure 3A shows image layers 320, 322, 324, 326, 328, 330 of a reference image in three-dimensional space.

[0013] Figure 3B shows a variety of target points selected by a user using the 3D visualization device.

[0014] Figure 4A shows the image layers 320, 322, 324, 326, 328, 330 of the reference image in two-dimensional space in projection onto the xz-plane.

[0015] Figure 4B shows the multitude of target points selected by a user using the 3D visualization device.

[0016] Figure 5A and Figure 5B show an assignment of the target points to the individual image layers 320, 322, 324, 326, 328, 330 and illustrate one step of a procedure for generating image-specific instructions.

[0017] Figure 6A, Figure 6B and Figure 60 show the assignment of the target points 506, 508, 514 to the individual image layers 322, 324, 330 in a top view along the z-axis 300, which is orthogonal to the respective layers.

[0018] The following detailed description refers to the accompanying drawings, which form part thereof and in which specific embodiments of the invention are shown for illustrative purposes. In this respect, directional terminology such as "top," "bottom," "front," "back," "anterior," "rear," etc., is used with reference to the orientation of the described figure(s). Since components of embodiments can be positioned in a number of different orientations, the directional terminology serves for illustration and is in no way restrictive. It is understood that other embodiments may be used and structural or logical modifications may be made without deviating from the scope of protection of the present invention.It is understood that the features of the various exemplary embodiments described herein can be combined with one another, unless specifically stated otherwise. The following detailed description is therefore not to be interpreted in a restrictive sense, and the scope of protection of the present invention is defined by the attached claims.

[0019] Within the scope of this description, the terms "connected," "attached," and "coupled" are used to describe both direct and indirect connections, direct or indirect links, and direct or indirect couplings. In the figures, identical or similar elements are labeled with identical reference symbols where appropriate.

[0020] Fig. 1 shows an example of a device for carrying out a method for performing a three-dimensional rendering of a three-dimensional reference image of an object with a plurality of volume pixels. A user generates an input (e.g., tactile or by speech) for a 3D visualization device 100. In response to the received input, the 3D visualization device 100 generates a command 102 corresponding to the input and sends it to a process-executing device 104. The command 102 contains an instruction to start an imaging process. The process-executing device 104 receives the command 102 from the 3D visualization device 100 and accordingly generates another command 106 according to a stored computer program and sends the further command 106 to an imaging device 112 to start an imaging process.Additionally, the further command 106 can include an instruction regarding the parameters of the imaging process, such as image size, image resolution, etc.

[0021] The imaging device 112 performs the imaging process. In the imaging process, the imaging device 112, for example, scans a predefined area and captures one or more digital images of the predefined area. The predefined area may contain one or more objects, which are captured by the imaging device 112 in the one or more digital images. One or more digital images form one or more three-dimensional reference images 108, each with a plurality of volume pixels.

[0022] After completion of the imaging process (figuratively speaking, after all desired digital images have been acquired), the imaging device 112 sends the three-dimensional reference image 108 to the process-executing device 104. The process-executing device 104 performs a three-dimensional (3D) rendering of the reference image 108 using the multitude of volume pixels. The result is (at least) one rendered reference image 101. The process-executing device 104 sends the rendered reference image 101 to the 3D visualization device 100.

[0023] The 3D visualization device 100 receives the rendered reference image 101, stores it, and visually displays the rendered reference image 101 to the user in a 3D space. The user selects one or more volume pixel coordinates to create one or more target points 103 (e.g., to create a plurality of target points 103) in the rendered reference image 101. This is done, for example, by means of tactile input using, for example, VR input devices (as an example of a VR control). The 3D visualization device 100 stores the plurality of target points 103. Furthermore, the 3D visualization device 100 sends the plurality of target points 103 to the process-executing device 104.

[0024] The process-executing device 104 receives the plurality of target points 103. Based on the plurality of target points 103, the process-executing device 104 generates a start command 110 to start a process and control information.

[0025] 113 for the process, wherein the control information contains instructions to perform an action in the volume of the object according to the multitude of target points.

[0026] The process-executing device 104 sends the start command 110 to an actuator, figuratively speaking, to a manipulation device.

[0027] 114 (for example, an optical manipulation device). The manipulation device 114 implements the instructions and performs the instructed action, for example, an optical manipulation of the object (at the target points, e.g., along a trajectory defined by the target points or in an area defined by the target points).

[0028] The manipulation can involve any suitable manipulation of the object within the area. For example, the manipulation can involve a change in the object's shape, a displacement of the object (e.g., along one or more directions), a rotation of the object, the removal of a part of the object, etc.

[0029] The process can be repeated as often as needed.

[0030] Fig. 2 shows a 3D visualization 208 from the user's point of view, with the multitude of target points 103 in the rendered reference image.

[0031] Figure 101 is shown. Reference symbols 204 and 206 denote (e.g., VR) controls with which the user selects the volume points that comprise the multitude of target points 103. The 3D visualization 208 is shown / displayed to the user by means of the 3D visualization device 100. The multitude of target points 103 can, for example, form the following: a three-dimensional trajectory; a multitude of 2D trajectories; a surface; felt shapes.

[0032] The imaging device 112, with which the three-dimensional reference image is generated, can, for example, be configured as

[0033] Microscope, for example a fluorescence microscope or an electron microscope; medical imaging diagnostic device, for example a magnetic resonance tomography scanner or a computed tomography (CT) scanner.

[0034] The generation of the three-dimensional reference image can be carried out by a suitable method performed by the imaging device 112, such as a method based on fluorescence microscopy, a method based on electron microscopy, a method based on magnetic resonance tomography, a method based on computed tomography, etc.

[0035] It should be noted that the procedure can also be used to define a clinical target volume and / or other structures on computed tomography (CT) / magnetic resonance imaging (MRI) images for radiotherapy.

[0036] 3D rendering of volumes in 3D space can be achieved using the technology of so-called volume raycasting, in which a ray for each screen pixel, originating from the plane closest to the camera, is projected perspectively through the piece of volumetric data, and color and transparency information is thus determined. In a preferred configuration, the manipulation device 114 can be configured as an optical manipulation device. In essence, the manipulation device 114 can be set up for manipulating an object using optical means. For example, an optical manipulation device can be configured as one of the following types:

[0037] Optogenetics device; optical tweezers;

[0038] FRAP (fluorescence recovery after photobleaching) technology;

[0039] Laser ablation device.

[0040] The action performed by the manipulation device 114 can therefore, depending on the type of device, involve a corresponding (e.g. optical) manipulation of the object.

[0041] For example, in the case of an optogenetics device, the action may involve controlling cells using light (within the area / trajectory defined by the target points). Another example is that the action may involve capturing and / or moving objects using optical tweezers (within the area / trajectory defined by the target points). A further example is that the action may involve focusing a laser beam and / or moving a focused laser beam within the area / trajectory defined by the target points to ablate the object.

[0042] The 3D visualization device 100 can be configured as one of the following types:

[0043] Augmented Reality (AR) device;

[0044] Holographic display (also known as holographic display);

[0045] Virtual Reality (VR) device.

[0046] The AR device can have at least one control element. The holographic display can be connected to a control element. The VR device can include user glasses and at least one control element.

[0047] The VR control can be configured to generate the volume pixel coordinates of the volume pixels to form the plurality of target points 103 from manual inputs by a user of the VR control, with the plurality of target points 103 being integrated into the reference image 101.

[0048] The process-executing device 104 can be configured to perform the following operations, wherein these operations can be performed for each image layer of the three-dimensional reference image 101:

[0049] Arranging the volume pixels of the reference image 101 into discrete image layers;

[0050] Identifying the volume pixels that are part of the multitude of target points in a given image layer;

[0051] Generating image layer-specific instructions, for example for laser ablation or other manipulation, based on the identified volume pixels.

[0052] These identified volume pixels are the ones that are to be ablated with the laser or otherwise manipulated.

[0053] The devices involved in the process can be implemented as a single unit or as two (or more) separate units. For example, the process can be carried out by a single processing device that implements the functionality of both the process-executing device and the 3D visualization device. Alternatively, the process can be carried out by a process-executing device and a 3D visualization device that are separate from each other but communicatively coupled (e.g., via a network such as a wired or wireless network). Figure 3A shows the image layers 320, 322, 324, 326, 328, and 330 of a reference image in three-dimensional space. Reference numerals 304, 302, and 300 denote the x, y, and z axes, respectively. Reference numeral 308 denotes the plurality of image layers of the reference image.Each individual image layer 320, 322, 324, 326, 328, 330 from the multitude of image layers 308 is arranged orthogonally to the z-axis 300 and represents a plane defined by x and y coordinates.

[0054] Additionally, it may be provided that one or more synthetic intermediate layers are generated between adjacent image layers of the reference image in order to enable a finer image layer resolution.

[0055] Fig. 3B shows a variety of target points selected by a user using the 3D visualization device.

[0056] Reference symbols 304, 302, and 300 refer to the x, y, and z axes, respectively. Reference symbol 310 denotes the multiple target points selected by a user using the 3D visualization device 100.

[0057] Each target point from the multitude of target points 310 can be described by three coordinates (X, Y, Z).

[0058] Fig. 4A shows the image layers 320, 322, 324, 326, 328, 330 of the reference image in two-dimensional space in projection onto the xz-plane.

[0059] Fig. 4B shows the multitude of target points 310 selected by a user using the 3D visualization device, prior to a discretizing assignment of the target points to the individual image layers 320, 322, 324, 326, 328, 330.

[0060] Fig. 5A and Fig. 5B show an assignment of the target points to the individual image layers 320, 322, 324, 326, 328, 330 and illustrate a step of a procedure for generating image-specific instructions.

[0061] Fig. 6A, Fig. 6B and Fig. 6C show an assignment of the target points 506, 508, 514 to the individual image layers 322, 324, 330 in a top view.

[0062] In this step, each target point from the multitude of target points is assigned the image layer of the reference image that is closest to that target point on the z-axis. For example, first target points are assigned to a second image layer 322. The first target points are projected onto the second image layer 322 as projected first target points 506. In the same way, second target points are assigned to a third image layer 324 and are projected onto the third image layer 324 as projected second target points 508.

[0063] Similarly, a third target point is assigned to a fourth image layer 326 and projected onto the fourth image layer 326 as projected third target point 510. Furthermore, a fourth target point is assigned to a fifth image layer 328 and projected onto the fifth image layer 328 as projected fourth target point 512. Finally, in this example, a fifth target point is assigned to a sixth image layer 330 and projected onto the sixth image layer 330 as projected fifth target point 514.

[0064] Generally, further target points from the multitude of target points are assigned to these respective adjacent image layers. Based on this assignment, image layer-specific instructions for the manipulation device are generated.

[0065] The image layer-specific instructions contain information specifying which projected target points, each defined by x- and y-coordinates, are to be manipulated in which of the respective image layers 320, 322, 324, 326, 328, 330 of the reference image (and thus, figuratively, of the object to be manipulated). Subsequently, the instruction to start the process and to generate the control information is created, whereby this control information contains the image-specific instructions.

[0066] The instructions specify that an action to manipulate the object must be performed according to the projected target points. These projected target points are each defined by x and y coordinates and assigned to image layers 320, 322, 324, 326, 328, or 330 of the reference image—and thus, intuitively, to the corresponding layers of the object to be manipulated.

[0067] For example, if the manipulation device is a laser ablation device, the action consists of ablating specific areas of the object using a laser. The instructions in this case may specify that the laser—based on the coordinates of the projected target points—selectively ablates those points of the object that correspond to the projected target points with a predefined laser beam intensity.

[0068] Subsequently, the instruction to start the procedure and the control information containing the corresponding instructions are transmitted to the manipulation device 114.

[0069] For the example that the manipulation device 114 is the laser ablation device, a laser ablation procedure is started.

Claims

Patent claims 1. Procedure, comprising: Receiving a three-dimensional reference image of an area in which an object to be manipulated is arranged, wherein the three-dimensional reference image has a plurality of volume pixels; From driving a three-dimensional rendering using the multitude of volume pixels to generate a rendered reference image; Receiving one or more volume pixel coordinates to create one or more target points in the rendered reference image; Arranging the volume pixels of the three-dimensional reference image into discrete image layers; Identifying the volume pixels that are part of the target points in a given image layer; Generating image layer-specific instructions for an action based on the identified volume pixels; Generating an instruction to start a process and generating control information, wherein the control information contains instructions to perform the action to manipulate the object according to the target points; Sending the instruction to start the process and the control information to a manipulation device.

2. Method according to claim 1, further comprising: performing the instructed action by means of the manipulation device in the area for manipulating the object according to the target points.

3. Method according to claim 1 or 2, further comprising: generating the three-dimensional reference image by means of an imaging device.

4. Method according to claim 3, wherein the imaging device is a microscope.

5. Method according to claim 4, wherein the microscope is a fluorescence microscope, and wherein the generation of the three-dimensional reference image comprises an imaging method based on fluorescence microscopy.

6. Method according to claim 4, wherein the microscope is an electron microscope, and wherein the generation of the three-dimensional reference image comprises an imaging method based on electron microscopy.

7. Method according to claim 3, wherein the imaging device is a magnetic resonance tomography (MRI) scanner, and wherein the generation of the three-dimensional reference image comprises an imaging method based on magnetic resonance tomography.

8. Method according to claim 3, wherein the imaging device is a computed tomography (CT) scanner, and wherein the generation of the three-dimensional reference image comprises an imaging method based on computed tomography.

9. Method according to any one of claims 1 to 8, wherein the manipulation device is an optical manipulation device and the action comprises optical manipulation of the object in the area corresponding to the target points.

10. Method according to claim 9, wherein the optical manipulation device is an optogenetics device.

11. Method according to claim 9, wherein the optical manipulation device comprises optical tweezers.

12. Method according to claim 9, wherein the optical manipulation device comprises FRAP (fluorescence recovery after photobleaching) technology.

13. Method according to claim 9, wherein the optical manipulation device is a laser ablation device.

14. Method according to any one of claims 1 to 13, wherein the identified volume pixels are the volume pixels to be manipulated.

15. Method according to any one of claims 1 to 14, further comprising: Visualizing the rendered reference image using a 3D visualization device, and Receiving an input from the 3D visualization device to generate the volume pixel coordinates.

16. Method according to claim 15, wherein the 3D visualization device is an augmented reality (AR) device.

17. Method according to claim 15, wherein the 3D visualization device is a virtual reality (VR) device.

18. Method according to claim 17, wherein the virtual reality (VR) device comprises VR glasses and at least one VR control element.

19. Method according to claim 16, wherein the augmented reality (AR) device comprises at least one AR control element.

20. Method according to claim 18 or 19 , • wherein the AR / VR control generates the volume pixel coordinates of the volume pixels to form the one or more target points from inputs of a user of the AR / VR control; and • the method further includes the integration of one or more target points into the rendered reference image.

21. Device comprising: at least one processor configured to execute a method according to any one of claims 1 to 20.

22. Computer-readable storage medium in which instructions are stored which, when executed by at least one processor, implement a method according to any one of claims 1 to 20.

Images