Optical 3D Scanning for Collision Avoidance in Microscopic Systems

Abstract

The present invention relates to a collision avoidance system and method for an X-ray CT microscope, which processes image data of an object at different angles and generates a model of the object. The model is then used to configure the microscope for operation and can avoid collisions between the microscope and the object.

Description

Technical Field

[0001] X-ray computed tomography (CT) is a technique for non-destructively examining and analyzing the internal structure of an object. Generally, as X-rays penetrate the object, the X-rays are absorbed or scattered. The X-rays that have not been absorbed or scattered are then detected by a detector system. The image formed at the detector system is called an X-ray projection. Then, as the object is scanned at different angles, a tomographic volume dataset is reconstructed from a series of such projections by means of a standard CT reconstruction algorithm.

[0002] X-ray CT systems have many different configurations. In X-ray microscopy systems, the X-ray source and detector are large, while the object being scanned is usually small. Thus, the X-ray source and detector are mostly fixed, and the object rotates in the X-ray beam. This is in contrast to medical CT systems, where the patient is fixed and the radiation source and detector rotate around the patient. Background Art

[0003] In many cases, the object scanned in an X-ray microscopy system has an unknown inherent shape. For example, even when a CAD model is available or the object is from a core sample of known specifications, it is often still unknown whether the object is precisely aligned. Moreover, when different regions of interest are selected and the object is realigned in the beam path, the alignment may vary. The problem thus caused is that when the object is moved for scanning (mainly rotation), the object may collide with the scanning setup (the part of the X-ray source or detector closest to the object). The X-ray source and / or detector need to be moved closer to the object to obtain optimal system performance, so the challenge of avoiding collisions usually becomes more difficult.

[0004] Similar setups exist in other microscopy / tomography systems operating in other electromagnetic spectral regions, such as optical coherence tomographs and confocal microscopes (optical projection tomographs). Other examples include scanning electron microscopes (SEM) and focused ion beam (FIB) systems - i.e., charged particle imaging systems.

[0005] If a 3D model of the object (e.g., a 3D representation such as mesh or surface data) and a 3D model of the setup are available, collisions can be avoided. Another use of such a digital 3D model of the object is to predict the resulting X-ray images by ray tracing. Such simulated images can be used for imaging region selection, contrast enhancement, system configuration (e.g., power supply voltage or power), or fully automated scanning, as well as for improving X-ray reconstruction. Summary of the Invention

[0006] Generally speaking, according to one aspect, the present invention features an anti-collision system for a microscope. The system includes: one or more cameras for capturing images of an object loaded onto the microscope; and a computer that processes image data when the object is at different angles with respect to the cameras, generates a model of the object, and configures the microscope for operation using the model.

[0007] In certain embodiments, the computer uses the model to avoid collisions between the microscope and the object.

[0008] Moreover, the microscope can be, for example, an X-ray microscope, a scanning electron microscope, a focused ion beam system, or an optical microscope.

[0009] Preferably, the computer further has models for configuring the source subsystem and / or the detector subsystem of the microscope as well. In addition, the computer receives, for example, current position data of the source subsystem and / or the detector subsystem. Additionally, the computer can present a display including the model of the object.

[0010] In certain embodiments, a light source is provided for illuminating the object under the control of the computer. The light source can illuminate the object in different colors and / or against different object backgrounds.

[0011] Generally speaking, according to another aspect, the present invention features an anti-collision method for a microscope. The method includes capturing image data of the object at different angles, generating a model of the object, and using the model to configure the microscope for operation.

[0012] Generally speaking, according to yet another aspect, the present invention features a user interface presented on a display of a microscopy system. The interface includes: controls for moving the source stage and / or the object stage and / or the detector stage; and an image area where a model of the object to be imaged is presented.

[0013] The above and other features of the present invention, including various novel construction details and combinations of components and other advantages, will now be described more specifically with reference to the accompanying drawings and in the claims. It should be understood that the specific methods and devices embodying the present invention are shown by way of illustration and not as a limitation of the present invention. The principles and features of the present invention can be employed in several embodiments without departing from the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In the drawings, reference numerals in different views refer to the same parts. The drawings are not necessarily drawn to scale; rather, the emphasis is on illustrating the principles of the present invention. In the figures:

[0015] Figure 1 is a schematic diagram of an X-ray microscopy system applying the present invention in a certain embodiment;

[0016] Figure 2 is a flowchart showing the operations of performing a collision avoidance application on a computer system of a microsystem;

[0017] Figure 3 and Figure 4 shows loading an object into a microscope under different lighting conditions.

[0018] Figures 5 to 10 shows a user interface generated by the microscope. DETAILED DESCRIPTION

[0019] The present invention will be described more fully hereinafter with reference to the accompanying drawings, in which illustrative embodiments of the invention are shown. However, the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0020] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Additionally, the singular forms and the article "a", "an" and "the" are also intended to include the plural forms unless specifically stated otherwise. It can be further understood that the terms "comprises" and / or "comprising", when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should also be understood that when an element (including a component or subsystem) is referred to as being and / or shown as connected or coupled to another element, this element can be directly connected or coupled to the other element, or intervening elements may be present.

[0021] All terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined. It should also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning that is consistent with their context in the relevant art field and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0022] Figure 1 is a schematic diagram of a microsystem 200 to which the present invention can be applied.

[0023] The microscopic system 200 shown in the figure is an X-ray CT system and generally includes several subsystems. The X-ray source subsystem 102 generates a polychromatic or possibly monochromatic X-ray beam 103. The object carrier subsystem 110 with an object holder 112 holds the object 114 in the beam and positions and repositions it to enable scanning of the object 114 in the static beams 103, 105. After the beam 105 has been modulated by the object, the detector subsystem 118 detects the beam 105. A base 107, such as a platform or an optical bench, provides a stable foundation for the microscopic system 200 and its subsystems.

[0024] Generally speaking, the object carrier subsystem 110 has the ability to position and rotate the object 114 in the beam 103. Therefore, the object carrier subsystem 110 usually includes a linear stage and a rotary stage. The example shown in the figure has a precision three-axis stage 150, which is very precise and only translates and positions the object along the x-axis, y-axis, and z-axis within a relatively small travel range. This allows the region of interest of the object 114 to be located within the beams 103 / 105. The 3-axis stage 150 is mounted on a θ stage 152, which rotates the object 114 in the beam about the y-axis. The θ stage 152 is in turn mounted on the base 107.

[0025] In some embodiments, the source subsystem 102 is typically a synchrotron X-ray radiation source, or alternatively, it is a "laboratory X-ray source".

[0026] As used herein, a "laboratory X-ray source" is any suitable X-ray source other than a synchrotron X-ray radiation source. The laboratory X-ray source 102 can be an X-ray tube, where electrons are accelerated by an electric field in a vacuum and shot into a metal target, and X-rays are emitted as the electrons decelerate in the metal. Generally, depending on the type of metal target used, such sources produce a continuous background X-ray spectrum and a combination of intensity spikes at certain energies, which are derived from the characteristic curve of the selected target material. Additionally, the X-ray beam is divergent and lacks spatial and temporal coherence.

[0027] In one example, the source subsystem 102 is a rotating anode type source or a microfocus source with a tungsten target. Target materials containing molybdenum, gold, platinum, silver, or copper can also be used. Preferably, a transmission configuration is used, where the electron beam strikes a thin target from its back side. The X-rays emitted from the other side of the target material are used as the beam 103.

[0028] The X-ray beam generated by the source subsystem 102 is typically adjusted to suppress excess energy or radiation wavelengths. For example, by using an energy filter, such as one held in a filter wheel 160, which is designed to select a desired X-ray energy range (bandwidth), to eliminate or attenuate the unwanted wavelengths present in the beam. Adjustment is also often provided by a collimator or a condenser.

[0029] For other types of microscopy systems, different sources will be used. For example, if the microscopy system 200 is a scanning electron microscope (SEM), the source subsystem 102 generates an electron beam. If the microscopy system is a focused ion beam (FIB) system, the source subsystem 102 generates an ion beam. Finally, if the microscopy system operates in the optical state, the source subsystem 102 generates a light beam.

[0030] When the object 114 is exposed to X-rays or other beams 103, the X-ray photons or particles that propagate through the object form a modulated beam 105, which is received by the detector subsystem 118. In some other instances, an objective lens is used to form an image onto the detector subsystem 118 of the microscopy system 200.

[0031] Typically, an enlarged projected image of the object 114 is formed on the detector subsystem 118. The magnification factor is equal to the inverse ratio of the source-to-object distance 202 to the source-to-detector distance 204.

[0032] To obtain high resolution, the X-ray CT system 200 utilizes a detector subsystem 118 with extremely high resolution and may combine positioning the object 114 close to the X-ray source subsystem 102. In this case, the resolution of the X-ray image is limited by the resolution of the detector subsystem 118, the focal spot size of the X-ray source subsystem 102, the position of the object 114, and the geometric magnification of the object 114 on the detector subsystem 118. In other instances, the object may be positioned close to the detector subsystem, especially when the detector subsystem employs optical magnification.

[0033] For other types of microscopy systems, an appropriate detector (electron, particle, optical) is selected.

[0034] Typically, the source subsystem 102 and the detector subsystem 118 are mounted on respective z-axis stages. For example, in the illustrated instance, the source subsystem 102 is mounted to the base 107 via the source stage 154, while the detector subsystem 118 is mounted to the base 107 via the detector stage 156. In fact, the source stage 154 and the detector stage 156 are low-precision and long-travel stages that allow the source subsystem 102 and the detector subsystem 118 to be moved into position, typically extremely close to the object during scanning, and then retracted to allow the object to be removed from the object holder 112 of the object stage subsystem 110, a new object to be loaded onto it, and / or the object to be repositioned on it.

[0035] The operation of the microscopy system 200 and the scanning of the object 114 are controlled by the computer subsystem 124, which typically includes an image processor 120 and a controller 122.

[0036] The computer system 124 includes one or more processors 160 and their data storage resources such as disks or solid state drives and the memory MEM. The processor 160 executes the operating system 162, and various applications run on the operating system 162 to allow a user to control and operate the microscopy system 200. A display device 136 connected to the computer subsystem 124 displays information from the microscopy system 200, such as tomographic reconstructions, in a graphical user interface. User input devices 135 such as a touch screen, a computer mouse, and / or a keyboard enable interaction between an operator and the computer subsystem 124.

[0037] The controller 122 allows the computer subsystem 124 to control and manage components in the X-ray CT microscope 200 under software control. The controller can be a stand-alone computer system suitable for handling real-time operations or an application program executed on the processor 160. The source subsystem 102 includes a control interface 130 that allows it to be controlled and monitored by the controller 122. Similarly, the object stage subsystem 110 and the detector subsystem 118 have respective control interfaces 132, 134 to allow them to be controlled and monitored by the computer subsystem 124 via the controller 122.

[0038] To configure the microscopy system 200 to scan an object and adjust other parameters such as geometric magnification, the operator uses a user interface presented on the display device to adjust the source-to-object distance 202 and the source-to-detector distance 204 through the respective operations of the source stage 154 and the detector stage 156 to achieve a desired scan setup.

[0039] Specifically, the source stage 154 and the detector stage 156 include respective motor encoder systems or other actuator systems that allow the computer system 124 to position the respective X-ray source subsystem 102 and detector subsystem 118 to specified positions via the controller 122 through the control interfaces 130, 134. Additionally, the source stage 154 and the detector stage 156 signal their actual positions to the controller 122.

[0040] The operator of the system operates the object stage subsystem 110 under automatic control via the computer subsystem, the controller 122, and the control interfaces 130, 132, 134 to perform a CT scan. Typically, the object stage subsystem 110 will position the object by rotating the object around an axis orthogonal to the optical axes of the X-ray beams 103, 105 by controlling the θ stage 152, and / or by positioning the object in the x-axis, y-axis, and z-axis directions by using the stage 150.

[0041] The detector subsystem 118 creates an image representation of photons or particles from the attenuated beam 105. For example, in the case of an X-ray system, it typically includes a scintillator and a spatially resolving electron detector. The image formed at the detector subsystem 118 is also referred to as a projection or projection image.

[0042] Using a user interface presented on a display device, the operator defines / selects scan settings that include CT scan parameters. They include voltage settings that help determine the X-ray energy spectrum and exposure time on the X-ray source subsystem 102. The operator also typically selects other settings, such as the field of view of the X-ray beam 103 incident on the object 114, the number of X-ray projection images created with respect to the object 114. Generally, the scan settings include the angle by which the gantry subsystem 110 rotates and positions the object. Additionally, the source-to-object distance 202 and source-to-detector distance 204 are often specified and translated into the necessary positions or settings of the source gantry 154 and detector gantry 156 as part of the scan settings.

[0043] The computer subsystem 124, possibly assisted by its image processor 120, receives from the detector subsystem 118 a set of images associated with each rotation angle of the object 114 to build a scan. The image processor 120 uses a CT reconstruction algorithm to combine the projection images to create 3D tomographic volume information of the object. The reconstruction algorithm can be an analytical algorithm, where convolution or frequency domain filtering of the projection data is combined with backprojection onto a reconstruction grid. Alternatively, the reconstruction algorithm can be an iterative algorithm, where numerical linear algebra or optimization theory techniques are used to solve the discretized scheme of the projection process, which can include modeling the physical characteristics of the imaging system.

[0044] This microscopy system 200 further adds an optical camera 210, such as a video camera, which collects image data of the object 114 held in the object holder 112. The camera is typically mounted directly or indirectly to the system base 107 via a mounting system 215 such as a bracket. Generally, the optical camera 210 collects images in the visible spectral portion and / or adjacent spectral regions such as infrared. Generally, the optical camera 210 has a CCD or CMOS image sensor. Also included is a light source 212 that illuminates the object in the spectral region used by the optical camera. Preferably, the light source 212 illuminates the object 114 in multiple spectral regions (colors), and the optical camera 210 collects different color images in each of those spectral regions. Additionally, the light source preferably selectively illuminates the object in one operating mode and the background of the object in another operating mode.

[0045] Operation:

[0046] Figure 2 Shows the process performed by the collision avoidance application 230 executed on the computer system 124.

[0047] In step 410, the collision avoidance application 230 controls the object stage system 110 of the microscopic system to first rotate the object 114 360 degrees about the y-axis by controlling the θ stage 152. To save time, this rotation can be performed continuously. As the object 114 rotates, the optical camera 210 records videos and / or sequential images of the object and provides the image data to the computer subsystem via, for example, a USB cable. The image data is stored by the collision avoidance application 230 of the computer subsystem 124. By controlling the light source 212, the scanning is repeated several times (twice) using different lighting settings (see below).

[0048] The current implementation uses two independent lighting conditions (see Figure 3 ). The idea is that by controlling the light source 212 by the collision avoidance application, the object and the background are illuminated in the first image 310, while only the background is illuminated in the second image 312.

[0049] In Figure 2 step 412, the collision avoidance application 230 isolates the object in the image data. In one implementation, the application analyzes the image data according to the two lighting conditions. If the background appears the same in both images, the collision avoidance application 230 performs a simple image subtraction to isolate the object 114 in the image data.

[0050] Of course, it is difficult to achieve perfect lighting conditions in practice to make the background the same. Therefore, in one example, the collision avoidance application 230 executed on the computer subsystem 124 uses a calibration step to match the background intensities of the two lighting conditions. After adjusting for different background lighting intensities, the unmixed image calculated by the collision avoidance application 230 is the difference between the foreground and the background. The image calculated in this way isolates the object 114 and corresponds to the regular reflection image of the object (in front of the dark background at this time). The collision avoidance application 230 obtains a second unmixed image by taking the reciprocal of the transmission image contrast and subtracting the average background intensity (which can be estimated from the first unmixed image). This image corresponds to the shadow contour of the object (see Figure 4 ). The collision avoidance application 230 fuses these two images (averaging) for reconstruction and filters them using a tomographic high-pass filter (such as a "ram-lak" filter). Alternatively, any of these images can be used for object reconstruction. For example, only using the background image corresponds to shadow carving reconstruction.

[0051] In other examples, the user draws a bounding box around the object at several angles via a user input device 135 such as a mouse and keyboard, requiring at least 2 orthogonal angles to specify the extent of the object 114. This can be used to define the contour of the object or refine (trim or expand) the constructed 3D contour.

[0052] Figure 5 Shows a user interface 500 generated by a collision avoidance application 230 executing on an operating system 162 of a computer system 124, and the user interface is presented on a display device 136. It includes an image area 512 that displays current image data received from an optical camera 210 when a select webcam view button 520 is selected, and the webcam view button 520 operates as a radio button relative to a system model button 522.

[0053] The user interface 500 allows selection of different scan setting parameters. An objective selector 524 allows dropdown selection of different objectives in an inverted triangle of a detector subsystem 118. Also included are a voltage setting 526 and a power setting 528 for a source subsystem 102.

[0054] The user interface 500 includes a source position section 514. The source position section 514 includes position control buttons for operating a source stage 154 and a step size. It also includes a source position reading that displays the current position of the source stage and thus the current position of the source subsystem 102.

[0055] The user interface 500 includes a detector position section 516. The detector position section 516 includes position control buttons for operating a detector stage 156 and a step size. It also includes a detector position reading that displays the current position of the detector stage and thus the current position of the detector subsystem 118.

[0056] The user interface 500 also includes an object position section 518. It also displays the position of an object by showing the x-axis, y-axis, and z-axis positions of a 3-axis sample stage 150. It also displays the θ position of the object by showing the position of a θ stage 152. There are separate position control buttons for each axis of the 3-axis sample stage 150 and the θ stage 152 as well as separate step size settings.

[0057] Also shown in the figure is the user interface 500 generated by the computer system 124 and displayed on the display device 136, which generates a graphical crosshair overlay 534 in the image area 512 when a crosshair overlay button 520 is selected. When a Fit All overlay button 532 is selected, a graphical box overlay is generated around the object in the image area 512.

[0058] Figure 6 Shows a user interface generated by a computer system 124 and displayed on a display device for performing a second optical scan of an object under different lighting conditions. In this example, a light source 212 only illuminates the background.

[0059] Review Figure 2, in step 412, the collision avoidance application 230 preprocesses the video data from the camera 210. The collision avoidance application 230 unmixes the lighting conditions to separate the foreground (objects) and the background (the rest of the camera's field of view).

[0060] Then, in step 414, the collision avoidance application 230 uses the image data from the camera 210 to reconstruct a 3D volumetric model of the object 114. In one instance, the collision avoidance application performs a specially weighted filtered back-projection algorithm to reconstruct the object based on the image data. Generally, this 3D model is a shell or boundary. Such a model represents the surface, i.e., the boundary of the object, rather than its volume. That is, in other instances, the model is a solid model that defines the volume of the object, such as a solid model established using constructive solid geometry.

[0061] Meanwhile, the collision avoidance application also includes a 3D system model 235 of the microscopy system 200, especially the parts of the microscopy system 200 that may collide with the sample 144. Thus, generally the 3D system model 235 includes models of the X-ray source subsystem 102 and the detector subsystem 118. This model may be a shell or boundary generated based on the image data from the camera. In other instances, it can be a solid model created as part of the design or manufacture of the system 200.

[0062] In addition, the collision avoidance application receives the current position 338 of the source stage, the current position of the detector stage, the x-axis, y-axis, and z-axis positions of the 3-axis sample stage 150, and the position of the θ stage 152.

[0063] In addition, when the detector subsystem includes a turret 119 with multiple objective lenses, the collision avoidance application receives, via the interface 134, the currently selected objective lens of the detector subsystem. This information is very important when different objective lenses have different clearances relative to the object 114.

[0064] After the 3D reconstruction into a voxel volume, in step 416, a point cloud of the object is generated. Specifically, the object is segmented. This segmentation can be done by thresholding, hysteresis thresholding, principal curves, or machine learning. Then, for example, the volume is converted into a point cloud.

[0065] Then, the collision avoidance application generates a 3D model of the area around the object 114 by combining the point cloud of the object, the 3D system model 235, the current position data 338, and the currently selected object. Then, in step 418, when the model view button 522 is selected, the model is presented on the display device 136.

[0066] Figure 7Disclosed is a user interface 500 generated by a computer system 124 and displayed on a display device 136 after object reconstruction. The real-time video image of the object and the parts of the system 100 near the object are replaced with the generated 3D solid model. The model includes a point cloud of the 3D rendered object 114M, the rendered source subsystem 112M, and the rendered detector subsystem 118M. The relative positions of the rendered object 114M, the rendered source subsystem 112M, and the rendered detector subsystem 118M are located based on the current position data 338 characterizing the scanning settings.

[0067] At this time, in step 420, object reconstruction and the 3D solid model are used to configure the microscope for X-ray scanning of the object.

[0068] In one example, collisions are determined by using the triangular intersection between the model representing the system hardware and the object. In the case of using an STL (Standard Triangle Language) formatted model, the bounding box of each component of the model is provided, and if these bounding boxes intersect, each triangle of the model is analyzed. If the bounding boxes of these triangles intersect, an intersection test is performed on each triangle to infer whether a collision is likely to occur.

[0069] In the current implementation, the user is limited to moving one stage or axis at a time. If a collision is considered to occur before continuing or performing a movement, the movement is interrupted.

[0070] There are different times to test for collisions. One of them is before each movement commanded by the user. Also, if a collision occurs on the axis, the system moves the axis to the nearest non-collision position and notifies the user that this is the nearest position.

[0071] Collisions can also be tested before a compound movement, such as during the process of changing the beamline on the detector. For example, when the user commands the system to change from an objective lens mounted on a turret to a macro lens mounted beside the turret, which is a lens with a larger field of view, the system performs a collision test before allowing the switch to all motor startups.

[0072] In addition, the system can also test for collisions before acquisition, such as during a tomographic scan when moving the object along the X-axis, Y-axis, Z-axis, and θ-axis. The system tests all these movements before allowing acquisition.

[0073] Figure 8 and Figure 9 Disclosed is a user interface generated by a computer system 124 and displayed on a display device. As Figure 8 shown, object reconstruction is used to move the detector subsystem to a position close to the object without colliding with the object. Then, asFigure 9 As shown, object reconstruction is used to move the source subsystem closer to the object without colliding with the object. The system is ready to perform an X-ray scan as Figure 10 shown.

[0074] In addition, using the constructed 3D solid model, an optimized scan trajectory (total number of projections and / or total number of projections at each angle for any specific rotation angle) is created by the collision avoidance application to reduce subsequent X-ray tomography artifacts (e.g., optimizing HART or determining if 180+ fan angles allow for faster scans).

[0075] In addition, the positioning of the source subsystem and the detector subsystem can be optimized, which can not only avoid collisions but also enable scans with the fastest or best resolution or the largest field of view.

[0076] Finally, during the initial hardware setup process, the 3D solid model can be used to intuitively guide the positioning of the source and the detector.

[0077] As an alternative to meshing, volumes can be used to directly calculate collisions (using distance transforms to the object surface).

[0078] The computer system 124 needs to perform a full 3D scan of the object and thus requires a 360-degree rotation. This applies to other alternative techniques such as strip projection or laser line scanning that can be used as an addition to or an alternative to the above-mentioned camera and back-projection reconstruction. Continuous rotation significantly reduces the time required for such scans. The recorded images consist of a static background and a (rotating) foreground. For reconstruction, the computer system 124 should isolate the foreground part of the image to obtain the object. This method uses images from different lighting conditions to perform such isolation. There may also be other methods, such as generating a background model in the absence of the object or modeling the housing of the microscope.

[0079] In one instance, object reconstruction is (weighted) filtered back-projection. The computer system preferably uses a new form of weighting that is calculated based on the intensity constancy (color can also be used) assumption commonly used in optical flow calculations. Other forms of weighting can be derived from a stereo (multi)-camera system, where the weights come from the intensity constancy between simultaneous stereo (multi-camera) images. Another option is a fringe projection system.

[0080] After FBP reconstruction, the 3D volume contains a 3D representation of the object. At this time, the volume can be directly used to calculate collisions (using distance transforms) or to simulate X-ray images (by filling the object with a predetermined material). Alternatively, the volume is segmented (e.g., by thresholding), and the resulting point cloud is converted to a mesh representation (e.g., by triangulation, alpha shape, or similar algorithms).

[0081] Background subtraction:

[0082] Background subtraction is an unsolved problem and an active research area. There are many methods in the literature. The method closest to the method of the present invention is the green screen technique that separates the background using colors. Although their performance is well optimized, at least two colors are required to segment each type of object, and the problem of uneven background still occurs (the housing (door) of the X-ray microscope cannot be changed).

[0083] Background subtraction records the background image (without objects) and attempts to remove (subtract) it from the final image. This is highly restrictive and depends on other changes in the scene (such as lighting or shadows caused by moving parts). The two lighting procedures proposed by the present invention have the following advantages:

[0084] · Background subtraction: It can unmix the image even if the background changes. The quality of this unmixing is mainly determined by the lighting quality. Perfect diffused lighting that produces the same background appearance enables perfect unmixing (regardless of the objects in the background).

[0085] · Both the reflection image and the shadow image can be used for reconstruction, especially for objects with poor optical properties:

[0086] · Object is extremely dark: In this case, the reflection image does not contain much information (everything appears black), but due to the light background (in most cases), the outline of the object is still clearly visible, and the reconstruction can at least be simplified to the (optical) object housing.

[0087] · Dark background: In some cases, the background may become dark. The contour image will not work, but if the object is not equally dark, the reflection image still contains the information required for 3D reconstruction.

[0088] · Mirror: The same explanation as for dark objects, mainly specular reflection affects the reflection image.

[0089] · Transparent object: Depending on its transparency, the method can segment and reconstruct transparent objects. It is impossible for the camera to become invisible to the object.

[0090] It should be appreciated that the systems and methods of the present invention described above are not limited to (X-ray) microscopes and can be accomplished by dedicated hardware for scanning (which of course allows the inclusion of other methods). A key factor related to X-ray microscopes is that all coordinate systems should be aligned. This can be achieved by using the same calibration object (alignment column) for the X-ray and camera systems. Such a calibration object can be a simple sphere visible in both systems or a dot matrix grid with the same properties.

[0091] Although the present invention has been specifically illustrated and described with reference to preferred embodiments of the present invention, those skilled in the art should understand that various changes can be made in form and detail without departing from the scope of the present invention covered by the appended claims.

[0092] For example, as another option, the optical data and optical reconstruction can be enhanced by using X-ray data from a microscope or other primary tomography system through image fusion or a Kalman-type voting algorithm. In cases where the confidence level of the optical system is very low, a rough CT scan generated based on X-ray projections can be used to enhance the optical model to create a better model.

Claims

1. A collision avoidance system for a microscopy system, comprising: one or more cameras for capturing images of an object loaded into the microscopy system; a light source for illuminating the object; and a computer that controls the light source to illuminate the object using different illumination conditions, processes image data of the object at different angles with respect to the camera, isolates the object from the background for the different illumination conditions to reconstruct a 3D model of the object, and uses the 3D model to configure the microscopy system for operation.

2. The system according to claim 1, wherein the computer uses the 3D model to avoid a collision between the microscopy system and the object.

3. The system according to claim 1, wherein the microscopy system is an X-ray microscope.

4. The system according to claim 1, wherein the microscopy system is a scanning electron microscope, a focused ion beam system, or an optical microscope.

5. The system according to claim 1, wherein the computer further has a model for configuring the source subsystem and / or the detector subsystem of the microscopy system as well.

6. The system according to claim 1, wherein the computer receives current position data of the source subsystem and / or the detector subsystem.

7. The system according to claim 1, wherein the computer presents a display including a model of the object.

8. The system according to claim 1, further comprising: a light source for illuminating the object under the control of the computer.

9. The system according to claim 8, further comprising: the light source illuminating the object with different colors and / or the object background.

10. The system according to claim 1, wherein the computer generates and presents a user interface on a display, wherein the user interface includes: controls for moving the source stage and / or the object stage and / or the detector stage; and an image area in which a model of the object to be imaged is presented.

11. The system according to claim 10, wherein models of the source subsystem and / or the detector subsystem are also presented in the image area.

12. The system according to claim 10, wherein a user can switch between a view presenting the model and a view presenting the actual image.

13. The system according to claim 1, wherein the computer controls the light source to illuminate the object in one operating mode and to illuminate the background of the object in another operating mode to isolate the object from the background.

14. The system according to claim 13, wherein the computer performs image subtraction to isolate the object image from the object illumination and the background illumination.

15. The system according to claim 13, wherein the computer controls the light source to match the background intensities of the two illumination conditions.

16. A collision avoidance method for a microscopy system, comprising: capturing image data of an object at different angles and under different illumination conditions; isolating the object from the background and reconstructing a 3D model of the object; and using the 3D model to configure the microscopy system for operation.

17. The method according to claim 16, wherein, the computer of the microscopy system uses the 3D model to avoid a collision between the microscopy system and the object.

18. The method according to claim 16, wherein, the microscopy system is an X-ray microscope.

19. The method according to claim 16, wherein, the computer of the microscopy system controls the light source to illuminate the object in one operation mode and to illuminate the background of the object in another operation mode to isolate the object from the background.

20. The method according to claim 19, wherein, the computer performs image subtraction to isolate the object by controlling the light source to match the background intensity of the two illumination conditions.

Images