Automatic system for capturing multispectral images in order to digitalise an object

Abstract

The invention relates to an automatic system for capturing multispectral images (10) in order to digitalise an object (11), including a platform (1) for receiving an object (11) to be scanned; a robot movement unit (4) for moving inside a volume of space where a plurality of regions of the object (11) are scanned; an illumination unit (3) for emitting light with a frequency of a selectable spectral range; a unit for capturing images (5) mounted on the robot movement unit (4); a processing unit (2) for the coordinated controlling of the robot movement unit (4) and the image-capturing unit (5), and for defining a scanning route and a plurality of capturing stops, wherein the processing unit (2) is also configured for labelling and storing the captured images of each region of the object with at least spectral and spatial location information.

Description

[0001]

[0002] AUTOMATIC MULTI-SPECTRAL IMAGING SYSTEM FOR DIGITIZING AN OBJECT

[0003] FIELD OF INVENTION

[0004] This system belongs to the field of object digitization, more specifically it relates to the conservation and restoration of Cultural Heritage (CH) works.

[0005] BACKGROUND OF THE INVENTION

[0006] One of the main problems that conservation and restoration professionals working on flat surfaces of heritage need to solve is access to very small details that cannot be observed with the naked eye.

[0007] To obtain results with high resolution and fidelity in color and textures, existing systems usually work with a combination of active and passive sensors, sometimes using microscopy or associated depth sensors.

[0008] To reduce operating errors, other state-of-the-art systems operate with cameras or scanners associated with robotic arms that require their own calibration procedures.

[0009] Using photogrammetry methods and a certain level of automation, some state-of-the-art systems are designed to correct manufacturing or industrial processing errors, but they don't require precision in textures or color reproduction. Some simply don't attempt to achieve digitization across different light spectra, which is important for working with heritage pieces from a restoration or conservation perspective.

[0010] Other state-of-the-art systems are not automated, and therefore cannot handle the digitization of large collections in a reasonable timeframe. Some other state-of-the-art systems require physical contact with the artworks for processing, which carries a risk of significant damage.

[0011] Working on projects created on flat surfaces allows us to avoid techniques such as the projection of structured light patterns, as it does not apply to reflective, transparent, or translucent surfaces.

[0012] Given the current solutions, it would be desirable to have a high-precision, high-fidelity, low-cost, and easy-to-operate system to obtain multispectral images that allow the creation of digital doubles of objects such as cultural heritage (CH) works, and that faithfully meet the analysis needs of CH conservators and restorers, whose adjustments and processes can be applied automatically.

[0013] BRIEF DESCRIPTION OF THE INVENTION

[0014] The limitations and problems identified in the prior art are addressed by the present invention. In its most general form, the invention relates to an automatic multispectral imaging system for digitizing an object, according to claim 1. The system includes a platform adapted to receive the object, a robotic drive unit for moving within a volume of space to scan a set of regions of the object, an illumination unit for emitting light at a frequency within a selectable spectral range, and an image capture unit mounted on the robotic drive unit. The image capture unit captures a digital image of one or more regions of the object, with focus determined by the spectral range emitted by the illumination unit.A processing unit in the system coordinately controls the robotic displacement unit and the image capture unit, and defines a scanning path with a number of stops for capturing the images, which are also labeled and stored, incorporating spatial and spectral location information.

[0015] The system obtains datasets applicable to photogrammetry and, therefore, valid for the automated creation of both 2D photomosaic composite images and 3D models. Ideally, the datasets should contain information on the geometry and texture of the PC objects, as well as the option to add metadata with additional information that helps trace the history of the piece and subsequent interventions by specialists.

[0016] Conventional orthophotographs applied to PC objects do not, by themselves, reveal the relief of pigments or supports. Relief is an interesting characteristic for restorers and conservators. Advantageously, the proposed system allows for obtaining information about relief. It also avoids direct physical contact with the objects. To achieve this, it uses a data capture strategy based on short-range photogrammetry with passive photographic sensors that always maintain a physical distance from the object thanks to the system's mechanical structure, which effectively prevents any possibility of contact with the artwork.

[0017] Conservation and restoration professionals need to obtain data with high resolution, precision, and fidelity to color, geometry, and appearance, without requiring microscopes or equipment that demands specialized operation. In several proposed embodiments of the invention, the resolution achieved is at least 5 microns.

[0018] It is desirable that overlapping descriptions can be obtained in visible light spectra (from about 380 nm to about 750 nm), near infrared (from about 750 nm to about 1 mm) and ultraviolet (from about 240 nm to about 400 nm) to complete the usefulness of the data needed by PC restorers and conservators.

[0019] Ideally, to avoid human error and spend the least amount of time possible on data collection, it is important to automate all tasks that can be repeated, including route and image processing calculations, allowing for decision-making and control over photographic exposure values, sensor transport, overlap values, and light performance in the different spectra.

[0020] To avoid errors or costly adjustments, intrinsic calibration methods for sensors, luminaires, and optics, as well as extrinsic calibration methods for resolution, position, scaling, and color characterization, must be implemented. These extrinsic calibration issues can be resolved using standard reference marks and colorimetry marks. Typically, these include resolution reference marks such as those based on the IISAF test, spatial and positional reference marks (e.g., April tags), or colorimetry reference marks such as color charts. These reference marks and / or colorimetry marks are used to calibrate the image acquisition unit or the robotic movement unit.

[0021] The proposed system is easily scalable. Implementations are planned with a number of sensors appropriate to the scale and complexity of the object. As an example, a prototype of the system used two photographic sensors and took one hour to collect data from a 40 x 40 cm artwork. The proposed system is compatible with stereophotometry techniques, based on the use of a pair of sensors that minimize occlusions while facilitating referencing and scaling using epipolar geometry. Specific implementations allow for the automatic control of various lighting configurations to improve the accurate reproduction of the object's surfaces and textures. Photogrammetric systems require datasets composed of numerous photographic images to obtain their point clouds free of occlusions. Automated image processing methods for optimization and optimized storage are desirable.

[0022] The image capture unit of the proposed system (which can be expanded by as many units as needed) maintains a fixed distance from the object for each spectrum. Conventional optical-mechanical focusing techniques offered by photographic equipment manufacturers do not always resolve situations involving illuminants with different spectra. Furthermore, adapting the optical elements can alter the resulting geometry of the captured images, which must overlap and coincide at different wavelengths of the spectrum.

[0023] The proposed system automates decision-making for determining exposure values ​​that are essential for achieving accuracy. The system uses algorithms to optimize depth of field (dependent on the selected aperture), edge sharpness (dependent on the shutter speed), and light intensity for each shot (avoiding the need to adjust ISO sensitivity, which introduces noise into the images).

[0024] At the same time, and to avoid an excessive number of images, it is advisable to select an appropriate scanning path, number of captures, and overlap area. For example, with an interface for inputting the parameters desired by restorers and conservators.

[0025] BRIEF DESCRIPTION OF THE DRAWINGS

[0026] To complement the description being made and in order to help a better understanding of the characteristics of the invention, a set of drawings is included as an integral part of said description, in which, for illustrative and non-limiting purposes, the following has been represented: FIG. 1 illustrates a schematic block diagram.

[0027] FIG. 2 illustrates an implementation of the Automatic Multispectral Image Capture System for Digitizing Objects.

[0028] FIG. 3 illustrates a flowchart of the operation of one embodiment of the automatic multispectral photogrammetry system.

[0029] NUMERICAL REFERENCES

[0030] 1 Platform.

[0031] 2 Processing unit.

[0032] 3 Lighting unit.

[0033] 4 Robotic displacement unit.

[0034] 5 Image capture unit that can be composed of several photographic cameras.

[0035] 6 Wheel.

[0036] 8 Interface.

[0037] 10 Automatic multispectral image capture system.

[0038] 11 Object.

[0039] 20 Method for operating the automatic collection system.

[0040] 21 Step to define the system configuration.

[0041] 22 Step to validate the system configuration.

[0042] 23 Step to perform system calibration, with references in scale, shape and position.

[0043] 24 Step to validate system calibration.

[0044] 25 Step to define the work methodology, number of photos, shooting route and overlap ratio.

[0045] 26 Step to perform image optimization, and their distribution, identification and storage.

[0046] 27 Step for obtaining data.

[0047] 27a1 Step to collect spectral range data 1.

[0048] 27b Step to generate a 3D model.

[0049] 27c Step to generate a point cloud.

[0050] 27d Step to generate a texture mesh.

[0051] 27e Step to generate metadata.

[0052] 28. Steps for generating a 3D model. 28. Steps for optimizing the 3D model.

[0053] 28a Step to enter working parameters

[0054] 28b Step to process mesh and texture data.

[0055] 29 Step to validate the 3D model.

[0056] 41 First horizontal lane.

[0057] 42 Second horizontal lane.

[0058] 43 Vertical lane.

[0059] 44 First horizontal engine.

[0060] 45 Second horizontal engine.

[0061] 46 Vertical motor.

[0062] 47 Rotary motor.

[0063] DETAILED DESCRIPTION OF THE INVENTION

[0064] Figure 1 represents a block diagram of system 10. The arrows indicate cooperation between components, either by sending instructions or data. System 10 incorporates a robotic displacement unit 4 that is designed to move relative to regions of the object that are planned to be scanned.

[0065] System 10 incorporates a capture unit 5 for capturing digital images of object regions. To guide or direct the capture unit 5 towards the different regions of the object, it is mounted on the robotic displacement unit 4. This aspect will be shown in FIG. 2.

[0066] The image capture unit 5, which may consist of various cameras or sensors, can take images in coordination with the robotic movement unit 4 and also with a lighting unit 3 included in the system 10. This lighting unit 3 is preferably designed with LEDs and is capable of emitting light at different frequencies. Primarily, light with a spectral range of visible light, infrared light, and ultraviolet light.

[0067] System 10 is controlled by a processing unit 2 that coordinates the movement (e.g., in an orthogonal X,Y,Z reference system) of the sensing unit 5—mounted on the robotic movement unit 4—relative to a specific region of the object. Typically, the robotic movement unit 4 has several motors 44, 45, 46 for this purpose.

[0068] System 10 can also incorporate a user interface 8 for entering instructions on how to perform the object scan by manually predefining certain parameters. For example, to define the characteristics of the captured images (format, resolution, etc.). The interface 8 can also display capture information on a screen to guide the user.

[0069] System 10 can employ a series of different locators of known proportions to capture regions of the object and facilitate obtaining photogrammetric information.

[0070] The System 10 can optimize the focus distance, which differs for each wavelength range. The use of a 46 motor allows the cameras to be moved closer or further away along the vertical axis to achieve the correct focus distance in each case. This distance calibration, when performed, is done before scanning begins, so each wavelength range will have an associated distance correction value to optimize its focus.

[0071] System 10 can employ an original capture algorithm because, in its initial calibration, it considers the number of sensors, their tilt (if applicable), the number of images, their final resolution, the optimal path and overlap for obtaining sufficient detail in the photographs, and the ideal number of homologous points to deliver to the photogrammetry software. The capture algorithm takes into account the size of the artwork and the camera's field of view. With this data, it can automatically calculate the necessary X and Y displacements to maintain the specified overlap. The capture algorithm applies techniques to determine and optimize exposure values: aperture, exposure time, and gain for the various wavelengths, and calculates the ideal setting to maximize the depth of field in each spectral range. This maximizes the focused area and makes homologous points more easily detectable.

[0072] System 10 can, from the raw data obtained from Bayer mosaic images captured by sensors 51, name the images and distribute them according to their geometric position and the light spectrum under which they were captured, and apply compression. For example, compression for final storage and, after applying color correction in two phases—the first automatic and the second directed by a specialist—delivering the data to the photogrammetry software. A color adjustment algorithm (for example, developed in Matlab) allows the removal of patches (colors) from the calibrated pattern that are not present in the artwork for a more precise adjustment. The algorithms typically used always calibrate all 24 patches of the pattern, and the resulting adjustment loses fidelity. The proposed color adjustment algorithm performs a least-squares adjustment using a merit function.The iterative changes are sought that make the patches in the processed image get as close as possible to the representation that is determined as objective; the process is repeated recursively with a Nelder Mead algorithm until the optimal merit function is found that minimizes the difference of the processed image with the pattern.

[0073] System 10 can use a test chart placed on the platform in predetermined positions and illuminated with different spectra (V, UV, IR) using illumination unit 3. System 10 can determine exposure values ​​by comparing the test chart with the dominant color characteristics and tonal range of the object to be scanned, thus obtaining an optimal distribution of image acquisition values ​​to maximize captured information. Existing methods do not determine the maximum and minimum brightness levels (and therefore the recommended contrast) for non-visible spectra.The criterion in the color adjustment algorithm also allows choosing a diaphragm aperture (f) with the highest number among those that preserve the predetermined dynamic range while keeping the shutter speed to a minimum, thus maximizing the depth of field and minimizing the time spent shooting without the risk of motor starts and stops blurring the edges of the image, a key concept for obtaining a maximum number of useful points for photogrammetry.

[0074] Figure 2 represents the implementation of system 10. Figure 2 is particularly suited for digitizing flat objects, that is, quasi-two-dimensional objects, where one of their dimensions is much smaller than the other two. Flat objects can be considered, for example, a painting, a tapestry, and others with greater volume, such as a bas-relief, a high relief, etc.

[0075] The robotic movement unit in this embodiment of system 10 comprises three motors 44-46, which may be stepper motors, and are associated with rails with a maximum travel distance that may be the same or different for each of them. These dimensions depend on the type of object 11 to be scanned, as well as the shape of a platform 1. Two horizontal motors 44, 45 are used respectively on a first horizontal rail 41 and a second horizontal rail 42. Both rails 41, 42 are straight and, preferably, are installed on the platform 1, which is flat in this embodiment and forms a right angle. Preferably, system 10 is used with the platform 1 parallel to the ground; however, minor design variations would allow other arrangements (e.g., vertical or inclined at 45°, 60°, etc.) and different geometries (e.g., curved rails).The second rail 42 is mounted on the first rail 41 at one end and has a wheel 6 at the other end for direct support on platform 1. Similarly, a vertical rail 43 with its associated motor 46 is mounted on the second horizontal rail 42 to vertically move an arm 49 on which a lighting unit with one or more luminaires 31 and a vapor collection unit with digital sensors 51 (two in this embodiment) are installed. A first sensor 51 is positioned perpendicular to platform 1, and a second sensor (not shown) can be positioned at a predetermined angle to the normal. However, other angles are possible; for this purpose, an adjustment bracket can be included to fix other angles. Thus, the sensors 51 and the luminaires 31 move together with the displacement unit.Preferably, the 51 sensors do not incorporate infrared filters since the illumination is spectrally limited to narrow bands and therefore no infrared radiation noise appears in the images in the visible band.

[0076] In other embodiments, the luminaires 31—instead of being mounted on the drive unit—can be fixed to platform 1. Optionally, they can include an angle adjustment mechanism to direct the light beam at a certain angle, for example, selected from a range of 20° to 65°, relative to the normal of the object region to be scanned, in order to avoid direct light reflections due to surface roughness.

[0077] Preferably, the illumination provided by the luminaires 31 should be spectrally selective to avoid the possibility of introducing radiation from a different spectral range into the image of one spectral range.

[0078] The vertical rail 43 can be telescopic. The components are controlled by a processing unit (not shown) which is responsible, among other tasks, for coordinating the movement of a sensor 51 to scan the object 11 and capture images of different regions of it. The action of the motors allows the sensors 51 to follow a scanning path and add scanned regions.

[0079] To further illustrate this implementation, construction details used in a real prototype are described. The prototype has horizontal rails, perpendicular to each other, with a travel length of 50 cm, thus ensuring a useful scanning area of ​​40 cm x 40 cm on the platform. In this implementation, the sensors of the capture unit are 5,472 x 3,648 pixels, equivalent to a resolution of 20 MP. They are fitted with a 50 mm lens that forms an image on a Sony IMX183 sensor. The luminaires 31 are constructed with two LED boards with circuitry for UV, visible, and infrared with wavelengths of 380 nm, visible at 5000K CRI95, and IR at 940 nm. These boards measure 20 x 10 cm, and each board has three rows of ten equidistant LEDs for each spectral range. To prevent buckling of the span caused by the movement of the motors due to the weight of the system, a steel wheel 6 is incorporated that moves on the platform.This platform is made from a rectangular aluminum plate to which the first horizontal rail is attached. This allows the sensors and LEDs to move in three dimensions (width: X, length: Y, height: Z) while maintaining the sensor in an orthogonal, or approximately orthogonal, position relative to the object.

[0080] Figure 3 represents a flowchart with several steps of a method for operating the system 20 according to a particular embodiment. The method 20 for operating the system includes several steps sequentially. One step is to define the system configuration 21, where the number of sensors 21a, optical characteristics 21b, and arrangement 21c of each are indicated; and, on the other hand, the number of luminaires 21d, spectral characteristics 21e, and arrangement 21f of each.

[0081] After step 21, a step to validate the system configuration follows (22). If yes, proceed to step 23 to perform a system calibration. If no, return to step 21 to define the system configuration.

[0082] In the step to perform a calibration of the system 23, several operations are carried out, including a geometric calibration 23a, a colohmethic calibration 23b and a calibration related to uniformity and illumination level 23c.

[0083] After step 23, a step to validate the system calibration follows (24). If yes, continue with a step to define the work methodology (25). If no, return to step 23 to perform a system calibration.

[0084] In step 25 to define the work methodology, the position of the sensors is indicated 25a, the exposure values ​​are indicated 25b, the focus is optimized 25c and the depth of field is defined 25d.

[0085] The following step involves image optimization 26, where a color adjustment 26a and a luminance adjustment 26b are performed.

[0086] The following step involves obtaining digitization data 27, which includes another step for collecting vapor data from spectral ranges 27a1, 27a2, ...

[0087] Using the data from the various spectral ranges 27a1, 27a2, ..., follow a step to generate a 3D model 27b, a step to generate a point cloud 27c, a step to generate a texture mesh 27d and a step to generate metadata 27e.

[0088] The following step involves optimizing the 3D model (28), where the working parameters (28a) are entered and the mesh and texture data (28b) are processed. This is followed by a validation step (29) for the processed data. If the validation is successful, the process (20) ends. If not, it preferably leads to the 3D model generation step (28).

[0089] The following paragraphs provide additional information for a better understanding of the invention.

[0090] Known photogrammetric techniques can be applied, taking into account various factors to improve the result. From general to specific, it is advisable to consider the characteristics of the environment, the object, the procedure to be followed, and the digitization objectives. The main information about the object that can be used to optimize digitization includes:

[0091] - dimensions, in some cases mass would also have to be considered;

[0092] - shape or geometry;

[0093] - the manufacturing material or materials, especially in relation to thicknesses or protrusions;

[0094] - surface and pigmentation characteristics: roughness, texturizing, coatings, hairiness, etc.;

[0095] - others. The procedural information that can be used to optimize digitization can in turn be classified into several categories: photographic capture category, image preprocessing category, and photogrammetric processing category.

[0096] In general, the aim is to avoid gaps in information, whether caused by occlusions, over- or underexposure, focus defects or insufficient depth of field; lack of sharpness or blurriness; it is also necessary to avoid flattening or perspective errors, noise from insufficient sensitivity, excessive or insufficient point overlap, internal sensor calibration errors or external calibration errors (geometric, optical, or color); file naming errors or corruption, unwanted compression, sequential or avoidable iterations in the processes, etc. The choice of capture procedure is crucial (number of cameras, hgs, supports for parts, captures with dynamic parts or sensors), as well as the routes and number of shots, which ideally should be adjusted so that it is sufficient but not much more than necessary to avoid data overload.

[0097] Image preprocessing is essential, especially when working with RAW formats (post-capture processing is highly advantageous but indispensable) and high dynamic range (HDR). Delivering images in optimal condition for photogrammetry software is key to minimizing processing time.

[0098] Photogrammetric calculation programs align images, detect homologous points appearing in different photographs, and create a point cloud valid for constructing the model's geometry. They allow for retouching and modifications, the creation of masks to remove unwanted aspects from the photographs, and references to markers present in the scene. They calculate point meshes in different file formats suitable for 3D programs, and various texture maps. Often, these resulting models have excessively complex geometries, making them difficult to manage, even for previewing details to preserve or correct.

[0099] Furthermore, it is essential to consider automation for all tasks involving repetitive processes, ideally with user-friendly interfaces for both the public and technical staff, and with gentle learning curves. Preferably, this automation would be implemented using free or open-source software. All of this would comprise the system design, which should be appropriate and demonstrate knowledge and mastery of its hardware and software components, as well as ease of application. A thorough workflow analysis is necessary to identify bottlenecks and tasks where time can be saved, even by tenths of a second, given the scale of the project, as significant time savings could be achieved. This planning also entails analyzing the management and performance of logical processors (CPUs) and graphics processors (GPUs), as well as optimizing file management.

[0100] Since each wavelength has a different focusing distance, the camera can be positioned at the corresponding optimal distance before capturing the image for that wavelength. This is also possible with a lens whose variable focal length is controlled by the processing unit; however, this results in a change in the captured field and greater processing complexity.Prior to digitizing the artwork, the following can be established: the focus distance (equivalent to the distance from the sensors to the platform), the exposure values ​​(aperture, shutter speed, gain), the white balance to balance the color of the light with the sensor settings; the path traveled by the sensor support, the number of stops and captures; the naming convention and destination of the photographic files taken; and alternative color adjustments to obtain images with the most suitable appearance according to the criteria of experts and restorers.

Claims

CLAIMS 1. An automatic multispectral imaging system (10) for digitizing an object (11), comprising: - a platform (1) configured to receive an object (11); - a robotic displacement unit (4) configured to move within a volume of space where it scans a plurality of regions of the object (11); - an illumination unit (3) configured to emit light with a frequency of a selectable spectral range from at least a visible light spectral range, an infrared light spectral range and an ultraviolet light spectral range, of the object (11); - an image capture unit (5) mounted on the robotic displacement unit (4), wherein the image capture unit (5) is configured to capture a digital image of a region of the object (11), focused according to the spectral range emitted by the illumination unit (3); - a processing unit (2) configured to coordinately control the robotic displacement unit (4) and the image capture unit (5), and to define a scanning path and a plurality of capture stops, wherein the processing unit (2) is further configured to label and store the captured images of each region of the object with at least spatial and spectral location information.

2. The system (10) according to claim 1, wherein the light emitted by the lighting unit (3) forms a selectable angle with respect to the normal of the object region (11) and adapts its intensity according to a set of chromatic values ​​obtained by the processing unit (2) from an image previously captured by the image capture unit (5).

3. The system (10) according to claim 1 or 2, wherein the robotic displacement unit (4) comprises: - a first rail (41) mounted on the platform (1) in a fixed manner; - a second rail (42) mounted on the first rail (41) in a movable manner by means of a first stepper motor (44); - a third rail (43) mounted on the second rail (42) in a movable manner by means of a second stepper motor (45), wherein the image capture unit (3) is mounted on the third rail (43) by means of a third stepper motor (46); where the processing unit (2) is electrically connected to the three motors (44-46) to individually control each motor (44-46).

4. System (10) according to claim 3, wherein the platform (1) includes a flat, rigid surface for supporting the object (11).

5. System (10) according to any one of claims 4 to 5, wherein the second rail (42) comprises a support wheel (6) for moving on the platform (1) while maintaining the distance of the cameras to the object (11).

6. System (10) according to any one of claims 1 to 5, wherein the image capture unit (5) comprises one or more sensors (51).

7. System (10) according to any one of claims 1 to 6, wherein the data capture path and the capture stops are defined such that there is an overlap zone between captured images corresponding to two different regions.

8. System (10) according to any one of claims 1 to 7, wherein the platform (1) comprises a set of reference marks and / or colorimetry marks for calibrating the image capture unit (5) and / or the robotic displacement unit (4) with the processing unit (2).

Images