Photogrammetric scanner system and imaging assembly

Abstract

A photogrammetric scanner system for capturing an image of a target, the photogrammetric scanner system including a frame, a plurality of image sensors connected to the frame and spaced apart on the frame, and a plurality of light emitting devices connected to the frame and spaced apart on the frame, the light emitting devices configured to project light in an illumination pattern.

Description

[Technical Field]

[0001] The present invention relates to a photogrammetric scanner system and imaging assembly for capturing images of a target, particularly a photogrammetric scanner system for capturing images of a target that can be used to create three-dimensional images and models of the target. [Background technology]

[0002] Any reference to a prior art method, device, or document is not to be construed as constituting evidence or an admission that they formed or form part of the common general knowledge.

[0003] Photogrammetry is the technique of extracting three-dimensional information from photographs. Photogrammetry involves processing a set of overlapping photographs of a target object to generate a 3D computer model of the object.

[0004] A typical photogrammetric reconstruction algorithm searches a set of images for common features and calculates the 3D coordinates of these features by correlating the relative feature positions. The relative positions of the corresponding cameras can also be determined.

[0005] By extrapolating across many features and photographs, a point cloud can be calculated that represents the relative camera coordinates and 3D morphology of the target object. From this point cloud, a 3D computer model consisting of vertices and polygons (triangles or N-gons) can be constructed using artificial neural networks or meshing techniques. To further improve the quality of the final 3D object, a pixel-by-pixel depth map can be calculated for each input image, with a value assigned to each pixel to indicate its relative distance from the sensor. Depth maps from several images can then be combined to estimate further structural information between the calculated 3D feature points before computing the mesh. During each processing stage, various algorithms are available for filtering undesired noise data and applying 3D smoothing, remeshing, and noise reduction algorithms. The final stage generally involves calculating the UV coordinates of the resulting 3D mesh and using the source image to generate a 2D texture for mapping onto the mesh via the UV coordinates.

[0006] Two challenges in photogrammetry are (1) providing optimal source images for feature detection and depth estimation, and (2) ensuring optimal sampling of target objects and sufficient (and reliable) image overlap for unbiased 3D object reconstruction.

[0007] An optimal source image contains only the object to be scanned (no background), is free of noise or lens distortion, is sharp, and has high detail. The object is maximized in the field of view with sufficiently sharp and distinguishable features distributed across its surface, is non-glossy (non-reflective), and is lit to minimize lighting artifacts such as specular reflections and unwanted shadows. To improve feature detection, indirect lighting that produces shadows may be desirable as opposed to flat, direct lighting.

[0008] Optimal object sampling is important to maximize the useful data collected for a given image set. If an object's region is poorly sampled (i.e., not enough image coverage) or if not all of the object's region is covered by enough overlapping images, the reconstruction process will result in a suboptimal object.

[0009] Photogrammetry can be performed using a single camera, often handheld or mounted on a moving object such as an airplane or other vehicle, that takes multiple photographs of an object from different positions. Another approach is to use multiple cameras and lights placed at different positions on a framework aimed at the object, with a system used to synchronously trigger the cameras. A related technique uses overlapping survey images (based on aerial, satellite, or ground vehicles) to reconstruct 3D models of large areas. These multi-camera setups are often heavy, expensive, require a lot of space, and are difficult to assemble and move around. Furthermore, due to the often high quality levels provided by the cameras used in such systems, long processing times can also lead to bottlenecks that affect throughput.

[0010] A drawback of photogrammetry is the difficulty in reconstructing uniform surfaces due to the lack of detail in feature detection and depth map calculation. A way to overcome this limitation is to capture two images per sensor: one with a pattern projected onto the object for geometric reconstruction, and one without the pattern for texture map generation.

[0011] Related 3D surface scanning techniques use structured light. Unlike photogrammetry, which uses computer vision techniques to locate features within a target object, structured light projects a pattern (typically a single line or a series of parallel lines) onto the object being scanned and then observes the pattern's resulting shape. By determining the change in shape, the 3D geometry of the target object can be determined. A common implementation of structured light 3D scanning involves a handheld 3D scanner (pattern projector / imaging) that is swept over the object. The calculated 3D geometry of the target object is refined over time from multiple samples. Summary of the Invention

[0012] In one aspect, the present invention provides a photogrammetric scanner system for capturing images of a target, comprising: The frame and a plurality of image sensors connected to the frame and spaced apart on the frame; a plurality of light emitting devices connected to and spaced apart on the frame and configured to project light in the illumination pattern; A photogrammetry scanner system is provided, comprising:

[0013] Preferably, each image sensor of the multiple image sensors is configured with a wide field of view and a short focal length, which allows the image sensors to be positioned closer to the target object than typical camera photogrammetry rigs, making the photogrammetry scanner system smaller and more portable.

[0014] Preferably, the plurality of image sensors comprise a plurality of cameras. Preferably, the plurality of light emitting devices comprise a plurality of light emitting diodes and / or a plurality of laser diodes for pattern recognition.

[0015] Preferably, the scanner system is configured to perform a coordinated illumination and imaging sequence of the target, wherein a plurality of light-emitting devices project light in an illumination pattern, and one or more images of the target are captured by a plurality of image sensors.

[0016] Preferably, the plurality of image sensors are arranged at predetermined positions on the frame. Preferably, the orientation or angle of each image sensor of the plurality of image sensors is dynamically variable. Preferably, each image sensor of the plurality of image sensors is connected to a servo motor so that the orientation or angle can be dynamically changed.

[0017] Preferably, the plurality of light emitting devices and the plurality of image sensors are configured to coordinate the projection of light from the plurality of light emitting devices with an image capture sequence by the plurality of image sensors. Preferably, the projection of light from the plurality of light emitting devices is synchronized with the image capture sequence of the plurality of image sensors. Preferably, the image capture sequence includes each image sensor of the plurality of image sensors capturing one or more images. Preferably, the image capture sequence includes the plurality of image sensors simultaneously capturing one or more images. Preferably, the image capture sequence includes activating a light emitting device adjacent to an image sensor, followed by capturing an image via the image sensor adjacent to the light emitting device.

[0018] Preferably, the arrangement of the plurality of image sensors and the plurality of light emitting devices includes each image sensor of the plurality of image sensors being adjacent to one light emitting device of the plurality of light emitting devices.

[0019] Preferably, the image sensors of the image sensors and the light emitting devices of the light emitting devices are arranged on each of the arms of the frame, and the arrangement on each arm includes each image sensor adjacent to a light emitting device.

[0020] Preferably, the configuration on each arm is the opposite of the configuration on the immediately adjacent arm, or alternatively, the configuration on each arm is inverted relative to the configuration on the immediately adjacent arm.

[0021] Preferably, no two image sensors of the plurality of image sensors are coplanar.

[0022] Preferably, the frame comprises one or more surfaces to which the plurality of image sensors and the plurality of light emitting devices are connected. Preferably, the one or more surfaces are flat or curved or arcuate.

[0023] Preferably, the frame is a handheld portable frame. Alternatively, the frame is a portable frame.

[0024] Preferably, the frame is modular. Preferably, the frame comprises a plurality of receptacles, each receptacle configured to receive an image sensor and a light emitting device therein. Preferably, the plurality of receptacles are releasably connectable.

[0025] Preferably, the frame comprises a plurality of arms. Preferably, the plurality of arms are spaced apart along the frame support member. Preferably, each of the plurality of arms extends substantially perpendicularly from the frame support member.

[0026] Preferably, each arm includes one or more of the plurality of image sensors thereon. Preferably, each arm is dynamically configurable.

[0027] Preferably, each arm of the plurality of arms is curved. Preferably, the curvature of each arm is substantially equal.

[0028] Preferably, the scanner system includes a plurality of scanner assemblies, each of which comprises one or more imaging assemblies, and each of which further comprises one or more secondary assemblies.

[0029] Preferably, each arm includes an imaging assembly including an image sensor and a light emitting device.

[0030] Preferably, each image sensor of the plurality of image sensors is offset in three dimensions from every other image sensor of the plurality of image sensors.

[0031] Preferably, each image sensor of the plurality of image sensors has coordinates in three-dimensional space that are unique in each of the three dimensions relative to the coordinates of every other image sensor of the plurality of image sensors.

[0032] Preferably, the plurality of image sensors are configured to be sensitive to light in a wavelength range, preferably in a wavelength range (including both visible and non-visible wavelengths) including at least one of ultraviolet light, visible light, and near-infrared light.

[0033] Preferably, one light emitting device of the plurality of light emitting devices is independently controllable relative to another light emitting device of the plurality of light emitting devices. Preferably, the plurality of light emitting devices are configured to emit light in a wavelength range. Preferably, each light emitting device of the plurality of light emitting devices comprises a plurality of lights emitting light of different wavelengths. Preferably, the plurality of lights are independently configurable and / or controllable. Preferably, the plurality of light emitting devices are configured to emit light in a wavelength range comprising at least one of ultraviolet light, visible light, and near infrared light.

[0034] Preferably, the plurality of image sensors are configured to detect wavelengths output by the plurality of light emitting devices.

[0035] Preferably, the light emitting device is configured to project a static or dynamic illumination pattern at a predetermined wavelength. Preferably, the light emitting device is configured to project a static or dynamic illumination pattern at a predetermined wavelength by a) controlling a lighting element sequence, or b) projecting the pattern through a pattern screen and / or lens, or c) using a laser pattern generator (scanned or via an optical grating or holographic lens).

[0036] Preferably, a grating is positioned over the plurality of light emitting devices. Preferably, the grating may be static or dynamic. The grating enables modulated pattern projection for hybrid photogrammetry and / or structured light scanning.

[0037] Preferably, the photogrammetric scanner system further comprises an orientation sensor and / or an acceleration sensor. Preferably, the orientation sensor comprises a gyroscope and / or an accelerometer. Preferably, the acceleration sensor comprises an accelerometer. The orientation sensor and / or acceleration sensor may be used to determine and track the position and orientation of the scanner to assist in aligning a target within the field of view of the image sensor.

[0038] Preferably, each light emitting device of the plurality of light emitting devices is offset from every other light emitting device of the plurality of light emitting devices in each of three dimensions.

[0039] Preferably, each light emitting device of the plurality of light emitting devices has coordinates in three dimensional space that are unique in each of the three dimensions relative to the coordinates of every other light emitting device of the plurality of light emitting devices.

[0040] Preferably, the photogrammetric scanner system further comprises one or more controllers configured to control the plurality of image sensors and / or the plurality of light emitting devices.

[0041] Preferably, the photogrammetric scanner system includes a three-dimensional (3D) depth sensor for calculating accurate point cloud data for the hybrid LIDAR and photogrammetric system to be implemented. Preferably, the 3D depth sensor comprises a LIDAR sensor.

[0042] In another aspect, the present invention provides a photogrammetric scanner system comprising the steps of providing a plurality of image sensors and a plurality of light emitting devices connected to and spaced apart on a frame; activating the photogrammetric scanner system to perform a coordinated illumination and imaging sequence of the target, wherein a plurality of light emitting devices project light in an illumination pattern and one or more images of the target are captured by a plurality of image sensors; The present invention provides a method for photogrammetric scanning, including:

[0043] Preferably, the method includes projecting patterned light, preferably generated by projecting light from a laser diode through a grating or hologram adjacent to the laser diode, thereby resulting in a hybrid optical and photogrammetric system to be implemented.

[0044] In another aspect, the present invention provides an imaging assembly comprising: a body having an image sensor, a light emitting device, and a processing assembly connected thereto; a processing assembly configured to coordinate target illumination and imaging sequences, and an imaging assembly designated as a controller imaging assembly among a network of imaging assemblies each in digital communication with said processing assembly and thus configured to coordinate target illumination and imaging sequences of all imaging assemblies in the network of imaging assemblies; An imaging assembly is provided.

[0045] Preferably, the imaging assembly comprises a housing, the body forming part of the housing.

[0046] Preferably, the processing assembly is configured to detect and communicate with one or more further imaging assemblies to form a network of imaging assemblies.

[0047] Preferably, one further imaging assembly of the one or more further imaging assemblies is detected by the controller imaging assembly when the further imaging assembly is electrically activated. [Brief explanation of the drawings]

[0048] Preferred features, embodiments, and variations of the present invention can be identified from the following detailed description, which provides sufficient information for those skilled in the art to practice the invention. The detailed description should not be considered as limiting the scope of the foregoing summary of the invention in any way. The detailed description makes reference to several drawings, which follow: [Figure 1] FIG. 1 is a front view of a photogrammetric scanner system for capturing images of a target according to an embodiment of the present invention. [Figure 2] FIG. 1 is a side view of a photogrammetric scanner system. [Figure 3] FIG. 1 is a front view of the arm of the photogrammetric scanner system. [Figure 4] FIG. 1 shows a side view of the arm of the photogrammetric scanner system. [Figure 5] 1 shows the image sensor, light emitting device, and controller of the photogrammetric scanner system on a printed circuit board. [Figure 6] 1 shows the image sensor, light emitting device, and controller of the photogrammetric scanner system on a printed circuit board. [Figure 7] FIG. 1 is a functional block diagram of a photogrammetric scanner system. [Figure 8] 1 illustrates an embodiment of a photogrammetric scanner system for imaging a face. [Figure 9] 1 illustrates an embodiment of a photogrammetric scanner system for imaging a face. [Figure 10] 1 illustrates a photogrammetric scanner system reconfigured to be folded for storage and / or transportation. [Figure 11] 1 illustrates a photogrammetric scanner system reconfigured to be folded for storage and / or transportation. [Figure 12] 1 illustrates an imaging assembly according to one embodiment of the present invention. [Figure 13] 1 illustrates an imaging assembly according to one embodiment of the present invention. [Figure 14] 1 illustrates an imaging assembly according to one embodiment of the present invention. [Figure 15] 1 shows a frame for a photogrammetric scanner system. [Figure 16] 1 shows a frame for a photogrammetric scanner system. [Figure 17] 1 illustrates a photogrammetric scanner according to another embodiment of the present invention. [Figure 18] 1 illustrates a photogrammetric scanner according to another embodiment of the present invention. [Figure 19] 1 shows a modular frame for a photogrammetric scanner system. [Figure 20] 1 shows a portion of a photogrammetric scanner system using a modular frame. [Figure 21] 1 shows an enlarged view of a connecting member for connecting receptacles (modules) of a modular frame. DETAILED DESCRIPTION OF THE INVENTION

[0049] 1-5 illustrate a photogrammetric scanner system 10 for capturing images of a target. In particular, the photogrammetric scanner system 10 is for capturing images of a target that can be used to create a three-dimensional image of the target. An example of a target is a human head or face. However, the target may include any part of a human or another object.

[0050] The photogrammetric scanner system 10 includes a frame 100 and a plurality of scanner assemblies comprising a plurality of imaging assemblies 101 connected to the frame 100. The scanner assembly may also include one or more secondary assemblies that perform secondary functions, such as, for example, controlling lighting or imaging sequences.

[0051] The imaging assembly 101 is connected to the frame 100 and spaced apart in an array on the frame 100. More specifically, the imaging assembly 101 includes a plurality of image sensors 120 connected to the frame 100 and spaced apart in an array on the frame 100, and a plurality of light emitting devices 140 connected to the frame 100 and spaced apart on the frame 100, and configured to emit light in an illumination pattern. In some embodiments, each imaging assembly 101 includes at least one image sensor and one light emitting device paired together.

[0052] In use, the plurality of light emitting devices 140 project light in an illumination pattern onto an object, and one or more images of the object are captured by the plurality of image sensors 120 .

[0053] That is, the photogrammetric scanner system 10 is configured to perform a coordinated illumination and imaging sequence of a target, where multiple light-emitting devices 140 project light in an illumination pattern and one or more images of the target are captured by multiple image sensors 120.

[0054] In some embodiments, the coordinated illumination and imaging sequence involves activating one of the plurality of lighting devices 140 adjacent to one of the plurality of image sensors 120, followed by capturing an image via the image sensor adjacent to the lighting device. In other embodiments, the lighting device and image sensor may be activated non-sequentially, such that the lighting device is activated and a non-adjacent image sensor subsequently captures an image.

[0055] In some embodiments, the image sensors have a wide field of view and a short focal length, which allows them to be placed closer to the target than typical camera photogrammetry rigs to optimize their object-to-sensor ratio, i.e., optimize per-sensor coverage of the target and ensure sufficient overlap between images from adjacent sensors.

[0056] The image sensors may take the form of cameras and the light emitting devices may take the form of light emitting diodes, it being understood that any form of light emitting device may be used, including, for example, organic light emitting diodes (OLEDs).

[0057] An embodiment of the present invention uses a 1 / 4 inch image sensor.

[0058] Providing some embodiments of the photogrammetric scanner system 10 with image sensors that have wide fields of view and short focal lengths allows the photogrammetric scanner system 10 to be placed much closer to the target as described above, thereby reducing the need for relatively expensive optics to acquire high-resolution images from greater distances and allowing a higher practical density of sensors in the array than is possible with typical camera photogrammetry rigs. The relatively high sensor density means that a small area of ​​the target can completely fill each sensor, with the sensors being placed to ensure sufficient overlap so that the object is completely imaged.

[0059] It will be appreciated that the exact specifications (eg, focal length and field of view) and density will depend on the typical desired camera-to-object distance and the geometry of the object and scanner rig.

[0060] The illustrated embodiment uses multiple image sensors 120 in an array connected to the surface of the frame 100. The placement of the image sensors in the array (described in more detail below) optimizes coverage and overlap of the target object to maximize the effectiveness of the photogrammetry process.

[0061] Frame 100, in the form of a handheld portable frame, includes a frame support member 105 and a plurality of surfaces in the form of arms 110a-f having a plurality of image sensors 120 and a plurality of light emitting devices 140 connected thereto. Arms 110a-f are connected to frame support member 105 and spaced apart along the length of frame support member 105.

[0062] In a preferred embodiment, the frame 100 is formed from a lightweight plastic or metal material, such as carbon fiber.

[0063] In some embodiments, the frame may comprise a single curved or arcuate surface on which multiple image sensors and multiple light emitting devices are disposed in accordance with the descriptions herein for other embodiments.

[0064] The arms 110a-f and frame support member 105 are arcuate, with the curvature or arc of each arm 110a-f being substantially equal, however, the arms 110a-f and frame support member may also be flat or planar, and may be dynamically configurable between flat and arcuate.

[0065] The illustrated embodiment shows frame support member 105 as arcuate and extending longitudinally in a first direction, and arms 110a-f as arcuate and extending longitudinally from frame support member 105 in a second direction, as previously described. In other words, frame support member 105 extends in a first direction along its first longitudinal axis 106, which extends from first end 107 to second end 108 of frame support member 105, and each arm 110a-f extends in a second direction along its respective second longitudinal axis 111, which extends from first end 112 to second end 113 of each arm 110a-f. First longitudinal axis 106 and second longitudinal axis 111 are substantially perpendicular.

[0066] Although each arm 110a-f is arcuate, it can be considered to extend substantially perpendicularly from the frame support member 105 at the point where each respective arm 110a-f is secured to the frame support member 105.

[0067] The curvature of arms 110a-f and frame support member 105 gives frame 100 a substantially hemispherical (or semi-spherical) shape or configuration.

[0068] The hemispherical arrangement allows each image sensor to sample (image) the target / object from substantially the same distance depending on the target's geometry. This arrangement also allows each image sensor to be located at a unique 3D coordinate (i.e., no image sensor is coplanar), as opposed to the 2D of a cylindrical or planar array configuration. It is envisioned that such an arrangement may result in improved feature localization in 3D space (e.g., triangulation benefits).

[0069] Now, referring to the image sensors and light emitting devices connected to the frame 100, a typical arrangement of the multiple image sensors 120 and the multiple light emitting devices 140 includes each image sensor of the multiple image sensors 120 being adjacent to one light emitting device of the multiple light emitting devices 140.

[0070] 1 and 2, a plurality of image sensors 120 and a plurality of light emitting devices 140 are arranged on each of six arms 110a-f of frame 110. The arrangement on each arm 110a-f includes each image sensor adjacent to a light emitting device.

[0071] Although the illustrated embodiment includes six arms, any number of arms may be provided. For example, in one embodiment, five arms may be provided. In another embodiment, seven arms may be provided.

[0072] The arrangement on each arm 110a-f is opposite or inverted relative to the arrangement on the immediately adjacent arm. For example, in this embodiment, each arm 110a-f includes five image sensors and five light emitting devices, with arm 110a including an alternating arrangement of image sensors and light emitting devices from top to bottom (or end to end) starting with image sensors and ending with light emitting devices, while arm 110b includes alternating light emitting devices and image sensors from top to bottom (or end to end) starting with light emitting devices and ending with image sensors. This pattern is repeated for each of the six arms. More generally, the pattern is repeated alternately with each successive or subsequent arm.

[0073] As previously mentioned, the two image sensors of the photogrammetric scanner system 10 are not coplanar, which ensures that optimal overlap is achieved in the images acquired by all image sensors.

[0074] In the illustrated embodiment, each image sensor of the plurality of image sensors 120 is offset from every other image sensor of the plurality of image sensors 120 in each of three dimensions such that each image sensor of the plurality of image sensors 120 has coordinates in three-dimensional space that are unique in each of the three dimensions relative to the coordinates of every other image sensor of the plurality of image sensors 120.

[0075] Similarly, each light emitting device of the plurality of light emitting devices 140 is offset in each of three dimensions from every other light emitting device of the plurality of light emitting devices 140 such that each light emitting device of the plurality of light emitting devices 140 has a coordinate in three dimensional space that is unique in each of the three dimensions relative to the coordinates of every other light emitting device of the plurality of light emitting devices 140. However, it will be appreciated that in some embodiments, the image sensors are offset from every other image sensors, but the light emitting devices are not offset from every other light emitting device.

[0076] Upon simple touching of the lighting device, each lighting device may be synchronized with or independently controllable relative to another lighting device in the plurality of lighting devices 140 .

[0077] The light emitting devices may be configured to emit light in a range of wavelengths, or each light emitting device may include multiple lights (in the form of light emitting diodes or other suitable light emitting devices) that emit light of different wavelengths, including, but not limited to, ultraviolet, visible, and near-infrared light. Examples of multiple different lights provided can be seen in Figures 3 and 5, which show a first type of light emitting diode 140a configured to emit a first type of light and a second type of light emitting diode 140b configured to emit a second type of light having a wavelength different from the wavelength of the first type of light emitted by the first type of light emitting diode 140a.

[0078] Independently controllable light emitting devices, which may project illumination patterns at different wavelengths, facilitate maximum visibility of targets within each image.

[0079] The independently controllable light emitting devices allow the image sensor to receive reflected light at the target wavelength of the light emitting devices and capture an image of the target object at that wavelength.

[0080] In some embodiments, a bright light (e.g., using a high-brightness LED) is projected very close to the target object. This allows the target object to be illuminated with light at the target wavelength that is brighter than the broad-spectrum light of the surrounding environment. This also means that the shutter speed of the image sensor can be reduced, thereby reducing the incidence of the image sensor focused on objects outside the target area that are less brightly lit than on the brightly lit target.

[0081] To take advantage of these different wavelengths of light provided by the light emitting devices, the multiple image sensors 120 are configured to detect (or be able to image) wavelengths corresponding to the wavelengths output by the multiple light emitting devices 140.

[0082] In use, the plurality of light-emitting devices 140 are configured to project static or dynamic illumination patterns at predetermined wavelengths by a) controlling the illumination element sequence (as described above), or b) projecting the pattern through a pattern screen and / or lens, or c) using a laser pattern generator (scanning or via an optical grating, optical filter or holographic lens).

[0083] A grating (either static or dynamic) in the form of the optical grating described above may be positioned over multiple light emitting devices 140 to influence the illumination pattern and facilitate modulated pattern projection for hybrid photogrammetry and / or structured light scanning.

[0084] In some embodiments, a grating provides patterned light, which is generated by projecting light from a laser diode through a grating (or hologram) adjacent to the laser diode, resulting in a hybrid structured light and photogrammetry system to be implemented.

[0085] In some embodiments, a dynamic grating shapes the light pattern from a light emitting device in a sweeping motion across the surface of a target object. A sequence of images of this changing light pattern on the object's surface can be acquired from a given camera position, and this data is used in a hybrid structured-light / photogrammetry algorithm to improve the accuracy and resolution of the 3D object's geometry.

[0086] In some embodiments, an optical filter (or a series of filters that can be selectively positioned over the image sensor) is provided to selectively transmit or reject wavelengths or ranges. Optical filters are positioned over the image sensor to optimize scanning at specific wavelengths, in addition to or instead of using a multi-wavelength illumination configuration (e.g., multiple light emitting devices emitting light at different wavelengths).

[0087] The use of patterns can be used to provide additional features to photogrammetric reconstruction software to improve the accuracy and detail of the 3D model mesh, or can be used by structured light algorithms in addition to standard photogrammetric processing for the same purpose. In particular, this can be achieved by sequencing the activation of a light emitting device in coordination with the sequence of image sensor captures of a single image or video capture. Alternatively, a grating or holographic lens can be moved over the light emitting device as light is projected from it to move the projected pattern over the target.

[0088] Each arm 110a-f of the photogrammetric scanner system 10 includes a cover panel 114. The image sensors and light emitting devices are disposed between the body of the arm 110a-f and the cover panel 114. The cover panel 114 is transparent to allow light from the light emitting devices to be projected onto the target, and includes an opening (such as opening 115) for each image sensor to allow the image sensor to capture an image. Alternatively, the openings can be replaced with a transparent cover that allows the image sensor to capture an image. Referring to the aforementioned light patterns and use of optical gratings, optical filters, holographic lenses, and pattern screens, these may be formed within the control panel or applied to the surface of the cover panel.

[0089] Each imaging assembly 101 of the photogrammetric scanner system 10 also includes a processing assembly including a controller 180 in the form of a microprocessor connected to and configured to control the plurality of image sensors 120 and / or the plurality of light emitting devices 140. The controller 180 can be seen in Figure 6 and in Figure 7, which shows a functional block diagram of the photogrammetric scanner system 10.

[0090] In this embodiment, the controllers 180 are connected to the image sensors 120 by a camera serial interface (CSI), but may be connected using any other suitable camera data interface. Each controller 180 includes executable software for controlling the image sensors 120.

[0091] The controller software can electronically set sensor imaging parameters such as, for example, resolution, pixel binning, capture mode (still or video), shutter speed and exposure time, gamma, etc., as needed or desired.

[0092] Controller 180 is located on frame 100 in proximity to image sensor 120 (preferably on printed circuit board 121 as shown in Figures 5 and 6) and controls imaging parameters, triggers the camera (to capture still images or video), receives image data via the CSI port, interfaces with local memory to store image data, communicates with other microcontrollers or external computing devices 2 via physical connections or wireless communications (Bluetooth, Wi-Fi, etc.), performs diagnostics, controls lighting, and executes locally stored program code.

[0093] Controller 180 can also be programmed to allow coordinated sequencing and video capture depending on the application, as described below.

[0094] The controller 180 is connected to onboard non-volatile storage 181, where images are stored to allow identification of the images for later processing. To aid in the storage and association of each image set with each physical object, an identification method can be employed that uses imaging of an object descriptor before imaging the object itself. Examples of object descriptions include barcodes, XR codes, text recognition algorithms, or any other symbol designed to identify an object. For example, many hospitals use digital patient identification numbers that are used to link patient information such as notes, measurements, and images within the hospital's patient database system. Alternatively or additionally, facial or object recognition can be used to identify people or target objects to be scanned using single or multiple image sensors.

[0095] This patient ID can be captured before or during a 3D scan of the patient by placing an identifier within the sensor array's field of view to trigger the scan. On-board software can identify the associated code within the captured image, ensuring that each subsequent patient image is associated with the code within the image filename, image header, or other mechanism. This code can be stored in the image header for transfer to the 3D scan file header during photogrammetric 3D reconstruction or within the filename. After each desired imaging sequence is completed, the on-board microcontroller stores and labels the image in a storage location using an intra-controller communication protocol. The photogrammetric scanner system 10 is then ready for the next imaging sequence to be triggered.

[0096] Preferably, the aforementioned hemispherical (or hemispherical) arrangement allows each image sensor to sample the target from substantially the same distance and allows each image sensor to be located at a unique 3D coordinate, as opposed to the 2D of a cylindrical or planar array configuration. However, as described in more detail below, the position and orientation of the image sensors can be reconfigured to optimize for objects of different geometric shapes, and can be fully spherical (by adding more image sensors and arms, or by dynamically repositioning the image sensors and arms along the frame) to simultaneously image 360 ​​degrees of an object, or can be a flat or cylindrical arrangement to image objects of similar geometric shapes.

[0097] The arms of the frame may also be dynamically configurable to create different geometric configurations, allowing a single scanner to be optimized for different target objects. For example, the arms may be configurable between a substantially flat (or planar) configuration and a curved (or arcuate) configuration. In some embodiments, the image sensor may also be configured to branch to scan surrounding geometries, such as the interior or exterior of a building.

[0098] To perform a 3D scan of an object, the photogrammetric scanner system 10 is placed in a position relative to the object and a signal is provided to the controller 180 to initiate the 3D scan. In one embodiment, the image sensor is approximately 27 cm from the central focus, and the target should be placed approximately at the central focus. The signal can be provided by a single input trigger 182, wirelessly by a remote device, or via a timer device (e.g., a clock) or cord. Upon detecting the trigger, the controller 180 1) communicates with the image sensor 120 to set their parameters, acquire images, and transfer the images to a microcontroller for storage in memory 181, and 2) communicates with the lighting control hardware to operate a programmed lighting and image capture sequence, including turning each illumination light-emitting device 140 (and pattern generator) on and off.

[0099] Once the images are captured, they are communicated from the non-volatile storage 181 of the photogrammetric scanner system 10 to an external computer 2 that includes photogrammetric software (such as AliceVision) that is programmed to create a 3D model from the images captured by the photogrammetric scanner system 10. Alternatively, the photogrammetric software may be stored and executed on the non-volatile storage 181 of the imaging assembly to facilitate on-board digital image processing and 3D model processing, with processing performed on a single on-board processor or by clustering on-board processors.

[0100] 12-18 show a photogrammetric scanner system 20 for capturing images of a target. The photogrammetric scanner system 20 is for capturing images of a target that can be used to create a three-dimensional image or model of the target. Examples of targets include a person's head, face, hand, or foot. However, the target can include any part of a person or another object.

[0101] The photogrammetric scanner system 20 includes a frame 200 and a plurality of scanner assemblies comprising a plurality of imaging assemblies 201 connected to the frame 200 .

[0102] The imaging assembly 201 also includes a processing assembly 280 (described in more detail below).

[0103] The scanner assembly may also include one or more secondary assemblies that perform secondary functions, such as controlling illumination or imaging sequences, for example.

[0104] The imaging assemblies 201 are connected or connectable to the frame 200 such that multiple imaging assemblies 201 are spaced apart in an array on the frame 200. More specifically, the imaging assemblies 201 include multiple image sensors 220 (the image sensors 220 shown include both the image sensors and housings to which the image sensors are attached) connected or connectable to the frame 200 and spaced apart in an array on the frame 200, and multiple light emitting devices 240 connected or connectable to the frame 200, spaced apart on the frame 200, and configured to emit light in an illumination pattern. In some embodiments, each imaging assembly 201 includes a pair of at least one image sensor 220 and one light emitting device 240.

[0105] Each imaging assembly 201 includes a body 202 to which at least one image sensor 220 and at least one light emitting device 240 are connected.

[0106] The imaging assembly 201 may also include a grating (either static or dynamic) in the form of an optical grating positioned over the plurality of light-emitting devices 240 to affect the illumination pattern and facilitate modulated pattern projection for hybrid photogrammetry and / or structured light scanning. The grating in the illustrated embodiment takes the form of a diffuser 250 positioned over the light-emitting devices 240 to improve light diffusion and therefore illumination of the subject / target.

[0107] The imaging assembly 201 is shown as being hexagonal, but can be any shape.

[0108] In the illustrated embodiment, the imaging assembly 201 is arranged such that the image sensor 220 is disposed between the body 202 and the light emitting device 240. More specifically, the arrangement is in the following order: body 202, processing assembly 280, image sensor 220, light emitting device 240, and diffuser 250 (if used).

[0109] Body 202, image sensor 220, and light emitting device 240 each include connectors for connecting to connectors on other components (e.g., image sensor 220 includes connectors for connecting to body 202 and light emitting device 240). In the illustrated embodiment, the connectors take the form of multi-pin magnetic connectors 203 that provide connection and power distribution to the necessary components (including processing assembly 280, image sensor 220, and light emitting device 240).

[0110] The imaging assembly 201 may also be disposed within an environmentally sealed housing 204. The body 202 forms a portion of the housing 204.

[0111] The processing assembly 280 includes a system-on-chip with a programmable microprocessor and memory, and is configured to control and operate the image sensor 220, the light emitting device 240, and associated inputs and outputs (I / O).

[0112] Processing assembly 280 may include a wireless transceiver and may be configured to enable wireless synchronous operation and data communication without the need for physical inter-device signaling. Thus, in some embodiments, processing assembly 280 of imaging assembly 220 communicates data exclusively by wireless means or exclusively by a wireless interface. However, processing assemblies 280 may also be physically interconnected for data communication in some embodiments.

[0113] In use, the plurality of light emitting devices 240 project light in an illumination pattern onto a target, and one or more images of the target are captured by the plurality of image sensors 220 .

[0114] That is, the photogrammetric scanner system 20 is configured to perform a coordinated illumination and imaging sequence of the target, where multiple light-emitting devices 240 project light in an illumination pattern and one or more images of the target are captured by multiple image sensors 220.

[0115] In some embodiments, the coordinated lighting and imaging sequence includes activating a light emitting device 240 associated with an image sensor 220 of an imaging assembly 201, followed by capturing an image via the image sensor 220 associated with the light emitting device 240. In other embodiments, the light emitting device 240 and the image sensor 220 may be activated non-sequentially, such that one light emitting device 240 on a first imaging assembly 201 is activated and the image sensor 220 on a second imaging assembly 201 subsequently captures an image.

[0116] In some embodiments, the image sensor 220 has a wide field of view and a short focal length, which allows the image sensor 220 to be placed closer to the target than typical camera photogrammetry rigs to optimize their object-to-sensor ratio, i.e., optimize per-sensor coverage of the target and ensure sufficient overlap between images from adjacent sensors.

[0117] The plurality of image sensors 220 may take the form of a plurality of cameras, and the plurality of light emitting devices 240 may take the form of a plurality of light emitting diodes. It will be appreciated that any form of light emitting device may be used, including, for example, organic light emitting diodes (OLEDs).

[0118] Although embodiments of the present invention use a 1 / 4 inch image sensor, any type of image sensor may be used.

[0119] Providing some embodiments of the photogrammetric scanner system 20 with an image sensor 220 having a wide field of view and a short focal length allows the photogrammetric scanner system 20 to be placed much closer to the target as described above, thereby reducing the need for relatively expensive optics to acquire high-resolution images from greater distances and allowing a higher practical density of sensors in the array than with typical camera photogrammetry rigs. The relatively high sensor density means that a small area of ​​the target can completely fill each sensor, and the sensors are placed to ensure sufficient overlap so that the object is completely imaged.

[0120] It will be appreciated that the exact specifications (eg, focal length and field of view) and density will depend on the typical desired camera-to-object distance and the geometry of the object and scanner rig.

[0121] The frame 200 may be modularly formed.

[0122] In a preferred embodiment, the frame 200 is formed from a lightweight plastic or metal material, such as carbon fiber.

[0123] In some embodiments, frame 200 may comprise a curved or arcuate body 210 having multiple imaging assemblies 201 disposed thereon in accordance with the descriptions herein for other embodiments.

[0124] The frame 200 includes a plurality of receptacles 211 formed in a body 210 of the frame 200 that are shaped to conform to the shape of the imaging assembly 201, thereby allowing the imaging assembly 201 to be connected to the frame 200.

[0125] The receptacles 211 may be releasably connected to make the frame 200 modular and reconfigurable as required for different targets / subjects.

[0126] In its in-use configuration, the illustrated embodiment uses multiple image sensors 120 in an array spaced around the frame 200 based on the location of the receptacle 211 that receives the imaging assembly 201. The placement of the image sensors 220 in the array (described in more detail below) optimizes coverage and overlap of the target object to maximize the effectiveness of the photogrammetry process.

[0127] The body 210 of the frame 200 is generally hemispherical with an arcuate surface, however, the body may also be flat or planar, and may be dynamically configurable between flat and arcuate.

[0128] The hemispherical arrangement allows each image sensor 220 to sample (image) the target / subject from substantially the same distance, depending on the target's geometry. This arrangement also allows each imaging assembly 201 to be located at unique 3D coordinates when measured from a center point (i.e., none of the imaging assemblies 201 are coplanar or non-coplanar), as opposed to 2D as in the case of a cylindrical or planar array configuration. Each imaging assembly 201 may be programmed with designated coordinates based on a known position within the frame 200, or may be configured to determine its coordinates within the frame 200 by communication with any other imaging assemblies 201 within the frame 200. In such an embodiment, at least one imaging assembly 201 may be programmed with coordinates that serve as an initial reference for any new imaging assemblies 201 added to the frame 200.

[0129] Because no two imaging assemblies 201 are coplanar, each imaging assembly 201 is offset from every other imaging assembly 201 .

[0130] Thus, each image sensor 220 is positioned at a unique 3D coordinate when measured from a center point (i.e., none of the image sensors 220 are coplanar or non-coplanar), as opposed to 2D as in the case of a cylindrical or planar array configuration. It is envisioned that such an arrangement may result in improved feature localization in 3D space (e.g., the benefits of triangulation).

[0131] As mentioned above, the two image sensors 220 of the photogrammetric scanner system 20 are not coplanar, which ensures that optimal overlap is achieved in the images acquired by all image sensors 220 .

[0132] Because the two image sensors 220 are not coplanar, each image sensor 220 is offset from every other image sensor 220 of the plurality of image sensors 220 in each of the three dimensions so that each image sensor 220 has coordinates in three-dimensional space that are unique in each of the three dimensions relative to the coordinates of every other image sensor 220.

[0133] Similarly, each light emitting device 240 is offset in each of three dimensions from every other light emitting device 240 of the plurality of light emitting devices 240 such that each light emitting device 240 has coordinates in three dimensional space that are unique in each of the three dimensions relative to the coordinates of every other light emitting device 240. However, it will be appreciated that in some embodiments, while the image sensors 220 are offset from every other image sensors 220, the light emitting devices are not offset from every other light emitting device 240.

[0134] To briefly touch on the lighting devices 240, each lighting device 240 may be synchronized with all other lighting devices 240 in the frame 200 or may be independently controllable with respect to another lighting device 240 by its corresponding processing assembly 280 (described below).

[0135] Light emitting device 240 may be configured to emit light in a range of wavelengths, or one or more light emitting devices 240 may include multiple lights (in the form of light emitting diodes or other suitable light emitting devices) that emit light of different wavelengths, including, but not limited to, ultraviolet, visible, and near-infrared light. An example of where multiple different lights are provided includes a first type of light emitting diode configured to emit a first type of light and a second type of light emitting diode configured to emit a second type of light having a wavelength different from the wavelength of the first type of light emitted by the first type of light emitting diode.

[0136] Independently controllable light emitting devices 240 , which may project illumination patterns at different wavelengths, facilitate maximum visibility of targets within each image captured by the corresponding image sensor 220 .

[0137] The independently controllable light emitting devices 240 enable the corresponding image sensors 220 to receive reflected light at the target wavelength of the light emitting devices 240 and capture an image of the target object at that wavelength.

[0138] In some embodiments, a bright light (e.g., using a high-brightness LED) is projected very close to the target object. This allows the target object to be illuminated with light at the target wavelength that is brighter than the broad-spectrum light of the surrounding environment. This also means that the shutter speed of the image sensor can be reduced, thereby reducing the incidence of the image sensor 220 focusing on objects outside the target area that are less brightly lit than the brightly lit target.

[0139] To take advantage of these different wavelengths of light provided by the light emitting devices 240, the multiple image sensors 220 are configured to detect (or be able to image) wavelengths corresponding to the wavelengths output by the multiple light emitting devices 240.

[0140] In use, the plurality of light-emitting devices 240 are configured to project static or dynamic illumination patterns at predetermined wavelengths by a) controlling the illumination element sequence (as described above), or b) projecting the pattern through a pattern screen and / or lens, or c) using a laser pattern generator (scanning or via an optical grating, optical filter or holographic lens).

[0141] The processing assembly 280 is configured similarly to the processing assembly shown in FIG. 6 and FIG. 7, which shows a functional block diagram of the photogrammetric scanner system 10 .

[0142] In this embodiment, the processing assemblies 280 are connected to the image sensors 220 by a camera serial interface (CSI), although any other suitable camera data interface may be used. Each processing assembly 280 includes executable software to control the associated image sensor 220 and light emitting device 240 of the imaging assembly 201.

[0143] The controller software can electronically set sensor imaging parameters such as, for example, resolution, pixel binning, capture mode (still or video), shutter speed and exposure time, gamma, etc., as needed or desired.

[0144] The processing assembly 280 controls imaging parameters, triggers the camera (to capture still images or video), receives image data via the CSI port, interfaces with local memory to store image data, communicates with other microcontrollers or external computing devices 2 via physical connections or wireless communications (Bluetooth, WI-FI, etc.), performs diagnostics, controls lighting, and executes locally stored program code.

[0145] The processing assembly 280 can also be programmed to allow coordinated sequencing and video capture depending on the application, as will be described below.

[0146] The processing assembly 280 includes an onboard memory 281 in which images are stored to enable identification of the images for later processing. To aid in the storage and association of each image set with each physical object, an identification method can be employed that uses imaging of an object descriptor before imaging the object itself. Examples of object descriptions include barcodes, XR codes, text recognition algorithms, or any other symbol designed to identify an object. For example, many hospitals use digital patient identification numbers that are used to link patient information such as notes, measurements, and images within the hospital's patient database system. Alternatively or additionally, facial or object recognition can be used to identify people or target objects to be scanned using single or multiple image sensors.

[0147] To perform a 3D scan of an object, the photogrammetric scanner system 20 is placed at a position relative to the object, and a signal is provided to the processing assembly 280 to initiate the 3D scan. In one embodiment, the image sensor 220 is approximately 27 cm from the central focus, and the target should be placed approximately at the central focus. The signal can be provided by a single input trigger, wirelessly by a remote device, or via a timer device (e.g., a clock) or cord. Upon detecting the trigger, the controller 280 1) communicates with the image sensor 220 to set their parameters, acquire images, and transfer the images to the microcontroller for storage in memory 281, and 2) communicates with the lighting control hardware to operate a programmed coordinated lighting and image capture sequence, including turning each illumination light-emitting device 240 (and pattern generator) on and off.

[0148] Once the images are captured, they are communicated from the memory 281 of the photogrammetric scanner system 20 to an external computer (such as computer 2 in the previous figures) that includes photogrammetric software (e.g., AliceVision) that is programmed to create a 3D model from the images captured by the photogrammetric scanner system 20. Alternatively, the photogrammetric software may be stored and executed in the memory 281 of the imaging assembly to facilitate on-board digital image processing and 3D model processing, with processing performed on a single on-board processor or by clustering on-board processors.

[0149] Additionally or alternatively, the processing assembly 280 automatically and dynamically reconfigures both the software and hardware of each imaging assembly 201 so that each component functions independently and collectively as desired to achieve 3D scanning functionality.

[0150] In use, once the imaging assembly 201 is supplied with power by connecting the connector on the frame 200, the processing assembly 280 automatically connects to the processing assembly 280 of any other imaging assembly 201 and communicates all necessary information (preferably wirelessly) so as to operate within the same photogrammetric scanner system 20 as the other imaging assembly 201 or secondary assembly.

[0151] In addition to the standard microprocessor operating system installed in the processing assembly 280 in each imaging assembly 201, the photogrammetric scanner system 20 (a collection of individual imaging assemblies 201) has a comprehensive operating system capable of performing necessary functions such as imaging assembly identification and coordination, data transfer, processor and sensor configuration (network and physical), scanning operation sequencing, lighting sequencing, hardware and software fault monitoring and diagnostics, control of additional devices, data transfer to external devices, power and temperature monitoring, and distributed data processing across the network of imaging assemblies 201 (e.g., computer vision functions such as object recognition / segmentation).

[0152] External devices in the form of secondary assemblies such as physical button triggers, displays, motion sensors, etc. can also be integrated into this scanner operating system.

[0153] The photogrammetric scanner system 20 can also be configured to execute multiple, programmed, coordinated lighting and imaging sequences to enable flexible 3D scanning capabilities. For example, a particular imaging assembly can be set to scan at different times or in different orders to capture motion data, or to capture multiple images with different exposure times to combine into a high dynamic range image, or lighting devices can be sequenced to generate structured patterns during a scan to enhance features, etc. Multiple sequences can also be coded and selected to operate during a scan based on user input (i.e., button or automatically).

[0154] To enable dynamic configurability of the photogrammetric scanner system 20, the imaging assemblies 20 can be connected and operated according to a predetermined sequence. When a given imaging assembly is powered on (by connecting it to the necessary power pins), it initializes its operating system and connects to the scanner network (either via a wireless router or using routerless peer-to-peer networking). Each imaging assembly 201 is assigned a network ID. The photogrammetric scanner system 20 can then operate in one of two modes: 1) a mode in which a physical controller is specifically assigned for control and data management functions, or 2) a mode in which one of the imaging assemblies 201 is automatically assigned as the controller (in addition to its normal functions) via the 3D scanner platform operating system so that there is only one controller.

[0155] Upon startup, each imaging assembly 201 queries the network for the presence and ID of a controller. If no controller is present, any imaging assembly 201 nominates itself as the controller on a first-come, first-served basis. To avoid a situation where multiple imaging assemblies 201 can simultaneously self-nominate (due to network delays, etc.), each imaging assembly 201 checks again for the presence of other controllers after a short delay, and then, after negotiation, the redundant controller de-nominates. The controller periodically announces its presence (using a designated network port) to ensure the presence of one controller at all times. After a period of time, if the scanner assembly does not receive repeated announcements from the controller, the scanner assembly is no longer part of the photogrammetric scanner system 20 and the controller can assume self-nomination, starting with an available imaging assembly 201. Each imaging assembly 201 individually contains all the code necessary to function as a controller, so that controller selection is arbitrary and can dynamically adapt to the addition or removal of imaging assemblies 201 from the photogrammetric scanner system 20 during operation. The function of the controller is to interface with external devices, coordinate the individual scanner assemblies, execute the desired programmed 3D scan sequence, aggregate the 3D scan data (such as images or temperature maps), and interface with external network devices (computers) for data transmission and further processing.

[0156] Each imaging assembly 201 (and any secondary assemblies) contains hardware configuration information, the version of the entire 3D scanner operating system, sequence information, hardware configuration setting information, 3D scan data, and a live map of the entire 3D scanner configuration that is dynamically updated if scanner assemblies are added or removed during operation. Furthermore, a physical representation of the 3D scanner can be stored in each imaging assembly 201 as any 6D coordinate (position / orientation) determined by following a physical configuration process involving software processing of construction from motion or by physical signaling. For example, when using a building-block scanner assembly configuration, each interconnect can contain physical information (electronically, physically, using RFID, etc.) regarding its shape and dimensions. When each scanner assembly is added to the receptacle 211 of the frame 200, it captures the necessary physical information and can then communicate this information to all other scanner assemblies in the system (which in turn do the same), thus building a virtual map / representation of the entire 3D scanner physical configuration.

[0157] Software and data types include operating systems, function codes, sequence instruction sets, data transfer protocols, and 3D scan data (images, temperatures, video streams, etc.).

[0158] The operation of each imaging assembly 201 will now be described in more detail.

[0159] First, at startup, the imaging assembly 201 announces its ID to the imaging assembly 201 network.

[0160] Next, a nomination and election process takes place to establish one imaging assembly 201 as the controller (master).

[0161] Notification of scanner assembly configuration information (hardware specifications, capabilities, network ID, etc.) is provided to the controller.

[0162] Notification of the current software version to the controller is also provided. The controller then polls all imaging assemblies 201 to identify the latest version of the software and pulls the latest version.

[0163] The controller updates all software on all imaging assemblies 201 to the latest version as needed.

[0164] Each imaging assembly 201 operates according to software instructions: the sensor listens to the network (or other signal) and initiates processing of a sequence code that may include a single image capture, a video stream capture, or a timed sequence of captures (depending on requirements). This may also include loading configuration data from a separate file before operation.

[0165] Several assemblies may be provided that have functions different from imaging or data capture, such as emitting a target light (laser), acting as a remote trigger via a physical or optical button, displaying information (LED, OLED, e-ink, etc.), direct measurement using TOF or ultrasound, emitting audio signals to the user, etc.

[0166] The collection of images, videos, or data from all connected imaging assemblies (and other assemblies) is then coordinated by the controller.

[0167] As each scan is initiated, the controller sends a trigger signal to all other imaging assemblies 201, initiating their 3D scan sequence. The controller records information about each triggered capture command for the imaging assemblies 201 in a list. Successive capture trigger commands are added to this list. The controller then works through the list, copying scan data from each respective imaging assembly to a central memory location on the controller. This operation is configured to occur in the background of scanner operation and is intended to optimize data transfer from the scanner to another device (without the external device having to poll each imaging assembly 201 for scan data). If a scan trigger is activated during this data transfer procedure, the procedure is paused to allow the trigger signal to be sent, then resumed, and additional scan data is added to the scan list. This occurs automatically until the available scan list is empty.

[0168] The scan sequence list contains the file name and corresponding imaging assembly ID for each scan and other related data. This dynamic list is manipulated (items are added and deleted) by the controller. For redundancy, items copied from the imaging assemblies to the controller data repository are not deleted from each imaging assembly until the entire data repository has been transferred from the controller to an external computer.

[0169] If a controller is removed from the photogrammetric scanner system 20 or fails in any way, the controller will not be able to continue controlling, updating, and storing the scan data. In this case, another imaging assembly 201 is automatically assigned as the replacement controller and takes over the functions of the previous controller. As noted above, each imaging assembly 201 is configured to operate as an imaging assembly as part of a network, or as a controller (except for certain functional assemblies performed by secondary assemblies, such as trigger buttons or displaying only secondary assemblies).

[0170] In some embodiments, multiple imaging assemblies are designated as controllers, or alternatively, no imaging assembly may be designated as a controller, in which case coordination is facilitated by an external device, such as a computer.

[0171] To detect the absence of a controller during operation, some embodiments include a health check function in the network communication protocol, with each imaging assembly 201 periodically announcing its status on a given port. If a status signal is not received by either the controller or all other scanner assemblies in the photogrammetric scanner system 20, a health status request can be broadcast to the associated controller. If no further communication occurs from any one scanner assembly, the scanner assembly is removed from the active assembly list and will not be activated or data requested from it. Because each scanner assembly contains its own internal representation (software, sequence, scan data, etc.) and a system-wide representation (ID, functionality, location information, etc.) of all other active scanner assemblies, the absence of an assembly that is no longer active is detected by each scanner assembly and removed from their system representation.

[0172] Information about off-board transfers of scan data is also communicated by the controller to all other scanner assemblies in the system (which also means that locally stored scan data can be removed from each scanner assembly once it has been transferred from the scanner). This means that if a controller is removed from the frame during operation, the other scanner assemblies are aware of the state of the internal scan data transfers (from each scanner assembly to the controller). In this case, the newly designated controller can rebuild (or copy) the list of scan data and network locations (or scanner assembly IDs) and take over data aggregation operations.

[0173] The photogrammetric scanner systems described herein may also include orientation and / or acceleration sensors to provide information regarding the physical scanner coordinates relative to the real world. These sensors can be used to autonomously identify and communicate correct positioning between the photogrammetric scanner system and the target. The orientation and / or acceleration sensors can be used to determine and track the position and orientation of the scanner to aid in aligning the target within the field of view of the image sensor.

[0174] The orientation sensor may take the form of a gyroscope and / or an accelerometer, and the acceleration sensor may take the form of an accelerometer.

[0175] In some embodiments, a display may be provided in communication with an orientation sensor and / or an acceleration sensor that provides data indicative of the current orientation of the scanner relative to an optimal orientation that may be visually indicated via the display.

[0176] The display may also be used to display information related to the system or to display captured images.

[0177] Other sensors that may be used include an ultrasonic distance sensor, an optical ranging sensor, a magnetometer, a GPS sensor, a temperature sensor (eg, shown as temperature sensor 183 in FIG. 7), and a humidity sensor.

[0178] Distance sensors (e.g., ultrasonic, laser, and lidar) may be provided. The distance sensors may be used to measure the distance between the arm and the target, and in some embodiments, indicate to a user holding the scanner that the scanner is too close or too far from the target being scanned. Alternatively, distance detection may be achieved using computer vision code that detects features of the target (e.g., facial features of a face) and uses that information to determine distance. Computer vision code may also be used to determine distance to target information that may be used for real-time calculation of optimal scanner geometry for automated variable geometry modification of the frame.

[0179] In some embodiments, the frame may comprise multiple arms (as in FIG. 1) with receptacles for receiving the imaging assemblies (as in the frames of FIGS. 15 and 19). The receptacles may be movably connected to the arms or frame to provide dynamic reconfiguration and positioning of the imaging assemblies about the body of the frame to suit different purposes and requirements.

[0180] The physical configuration of sensors in the illustrated embodiment is optimized to enable simultaneous imaging of an object the size of a human head (shown, by way of example, in FIGS. 8 and 9), where sensor spacing is optimized to enable a nearly uniform distribution around the object converging at the center of the object in a hemispherical (or hemispherical) arrangement as shown in FIG. 1. However, this arrangement can be modified to enable imaging of various objects by adding sensors optimized for each object's geometry. For example, a configuration for scanning a human foot would include sensors to surround the bottom of the foot, as well as the sides and lower legs. Such an embodiment is shown in FIGS. 19-21.

[0181] The photogrammetric scanner system 30 for scanning the foot includes a frame 300 and a plurality of imaging assemblies 201 connected to the frame 300 .

[0182] Frame 300 is substantially similar to frame 200 including a receptacle 311 configured to receive imaging assembly 201 .

[0183] Receptacle 311 also includes a connector in the form of a multi-pin magnetic connector 303 that provides connection and power distribution to the necessary components (including processing assembly 280, image sensor 220, and light emitting device 240).

[0184] The receptacles 311 are separable, allowing for modular construction of the frame 300. This is best seen in Figures 19 and 20.

[0185] The receptacles 311 include connectors 312 that matingly engage with corresponding connector members, allowing two or more receptacles 311 to be connected to one another. The connector members may be of various configurations to allow the receptacles 311 to be properly angled relative to one another, or to create specific and / or custom geometries for scanners as needed. For example, a first connector member 313 is shown having an angled portion. In another example, a second connector member 314 is shown in which the member is straight.

[0186] The connector member of receptacle 311 and connector 312 may include reciprocating pins to facilitate power and / or data distribution to receptacle 311 to power imaging assembly 201 .

[0187] Secondary assemblies such as, for example, guide lights, laser pattern devices, and batteries may also be connected to the frame 300 via connectors and corresponding connector members.

[0188] For example, secondary assembly 315 is shown having connector 312 matingly engaging with a connector member that matingly engages connector 312 of receptacle 311 .

[0189] The connector and connecting member preferably engage with a transitional fit to limit movement of the receptacle when connected together.

[0190] In use, the photogrammetric scanner system 30 is placed in a housing having a clear or transparent face so that the foot can be placed against the clear or transparent face while walking and the imaging assembly 201 can image the foot.

[0191] In some embodiments, the photogrammetric scanner may include an additional imaging assembly for imaging the feet and legs (e.g., lower legs).

[0192] Embodiments of the photogrammetric scanner system are capable of scanning the entire body.

[0193] The above-described embodiments with variable or reconfigurable frames allow the system to be reconfigured and optimized for the target geometry.

[0194] In some embodiments, the image sensor is also capable of capturing video (at 30 fps), allowing for the construction of 3D video scans and multiple 3D models from a single video set.

[0195] In some embodiments, each image sensor captures 5 seconds of video that is stored as a Motion JPEG or other format (although the captured video can be of any length). Video capture begins approximately 0.5 seconds before the lighting pattern projected by the light emitting device is activated, allowing the lighting to be used for video frame synchronization. The inventors have found that initialization varies by a few milliseconds between image sensors (due to each image sensor having a microcontroller that is independent of other microcontrollers connected to other image sensors), which has limited impact when capturing images of stationary (or nearly stationary) targets.

[0196] The captured video from each image sensor can be exported to photogrammetry software, which uses lighting patterns to determine the start / first frame based on when light is first detected in each video, thereby excluding frames captured before this determined start frame, allowing the videos / frames from each image sensor to be synchronized.

[0197] In some embodiments, one or more arms of the frame may be crossed (eg, to form an X configuration), with an image sensor positioned along the length of each arm.

[0198] Advantageously, embodiments of the present invention allow the images required for a three-dimensional scan to be taken in less than about 1 ms, which can be important and / or useful in hospitals, particularly children's hospitals, where patients are often unable or unwilling to remain motionless and therefore it is important to keep scan times as short as possible.

[0199] In some embodiments, a single button on the photogrammetric scanner system 10 activates the illumination and imaging sequence of all light emitting devices and image sensors to quickly capture a 3D model of a target (e.g., a patient) or to capture a predetermined sequence of frames for a video.

[0200] In some embodiments, the photogrammetric scanner system includes a beam sensor for automatically initiating the illumination and imaging sequence immediately or after a predetermined time.

[0201] In some embodiments, image sensor 120 can be controlled by controller 180 to capture multiple single images per sensor at different shutter speeds / ISO / exposure settings, to provide raw image data for high dynamic range imaging (HDR), or as raw image data for use with a 3D reconstruction sensor. As explained above, this can be synchronized with an illumination device to allow two or more scans with illumination and aperture set at different levels, for example, to expose dark and light skin and hair simultaneously.

[0202] In some embodiments, a microphone is provided to capture voice commands to trigger scanning operations or other commands to find status information (battery level, on-board scan count, etc.) or to change scanning parameters.

[0203] In some further embodiments, a speaker is provided to notify the user of important information such as the number of scans in memory, battery level, etc. A short sound can also be played through the speaker to provide an audible cue when a scan is captured (e.g., similar to a camera shutter sound).

[0204] Power may be supplied to the photogrammetric scanner system 10 either via a mains power supply or a battery.

[0205] Embodiments provide a lightweight, robust frame designed for potential disinfection and / or sterilization, with an internal processor and imaging sensor housed in a sealed compartment.

[0206] The ability of some embodiments to control illumination to enable capturing 3D models illuminated in the UV and IR regions of the spectrum close to the visible region provides a wider range of applications for photogrammetric scanner systems.

[0207] In some embodiments, it may be advantageous to augment the LED illumination of the light emitting device with one or more lasers capable of illuminating the subject with coherent light.

[0208] Recording video of the reflected light and processing the data after capture can measure changes in the skin and surface tissue for diagnostic purposes. In such an embodiment, all sensors in the photogrammetric scanner system simultaneously and synchronously record short video sequences of the reflected light. Images corresponding to a given time point from each sensor are extracted from the video sequence and processed in a photogrammetric pipeline to generate a single 3D object. The 3D object for all desired frames in the video sequence can be processed, and a surface map can be processed to perform the desired analysis of the reflected light over time. Light polarization can also be incorporated into the laser output and / or sensor input to vary the detected light. One example of this is cross-polarization, designed to minimize specular reflections and enhance target features. In this embodiment, a linear polarizer is attached to the light source at a given orientation, and another linear polarizer is attached to the sensor lens or input at an orthogonal orientation relative to the light source polarizer.

[0209] In accordance with the statute, the invention has been described in language more or less specific to structural or organizational features. The term "comprises" and variations thereof, such as "comprising" and "comprised of," are used in an inclusive sense and do not exclude additional features.

[0210] It is to be understood that the present invention is not limited to the specific features shown or described, as the means described herein include preferred forms of carrying out the invention.

[0211] The invention is, therefore, claimed in any of its forms or modifications within the proper scope of the appended claims as appropriately interpreted by those skilled in the art.

Claims

1. 1. A photogrammetric scanner system for capturing images of a target, comprising: The frame and a plurality of image sensors connected to the frame and spaced apart on the frame; a plurality of light emitting devices connected to and spaced apart on the frame, the light emitting devices configured to project light in an illumination pattern; Equipped with the scanner system is configured to perform a coordinated illumination and imaging sequence of a target, wherein the plurality of light emitting devices project light in an illumination pattern and one or more images of the target are captured by the plurality of image sensors. Photogrammetry scanner system.

2. The photogrammetric scanner system of claim 1 , wherein each image sensor of the plurality of image sensors is configured with a wide field of view and a short focal length.

3. The photogrammetric scanner system of claim 1 or 2, wherein the projection of light from the plurality of light emitting devices is synchronized to the image capture sequence of the plurality of image sensors.

4. 4. The photogrammetric scanner system of claim 1, wherein the image capture sequence includes activating a light emitting device adjacent to an image sensor, followed by capturing an image via the image sensor adjacent to the light emitting device.

5. 5. The photogrammetric scanner system of claim 1, wherein an arrangement of the plurality of image sensors and the plurality of light emitting devices includes each image sensor of the plurality of image sensors adjacent to one light emitting device of the plurality of light emitting devices.

6. The photogrammetric scanner system of claim 1 , wherein no two image sensors of the plurality of image sensors are coplanar.

7. The photogrammetric scanner system of claim 1 , wherein each image sensor of the plurality of image sensors is offset in three dimensions from every other image sensor of the plurality of image sensors.

8. 8. The photogrammetric scanner system of claim 1, wherein each image sensor of the plurality of image sensors has coordinates in three-dimensional space that are unique in each of the three dimensions relative to coordinates of every other image sensor of the plurality of image sensors.

9. 9. The photogrammetric scanner system of claim 1, wherein one light emitting device of the plurality of light emitting devices is independently controllable with respect to another light emitting device of the plurality of light emitting devices.

10. 10. The photogrammetric scanner system of claim 1, wherein each light emitting device of the plurality of light emitting devices comprises a plurality of lights emitting light of different wavelengths.

11. 11. The photogrammetric scanner system of claim 1, wherein the plurality of light emitting devices are configured to emit light in a wavelength range comprising at least one of ultraviolet light, visible light, and near infrared light.

12. The photogrammetric scanner system of claim 1 , wherein a grating is positioned over the plurality of light emitting devices.

13. 13. The photogrammetric scanner system of claim 1, further comprising an orientation sensor and / or an acceleration sensor configured to determine and track the position and orientation of the scanner to assist in aligning the target within the field of view of the image sensor.

14. 14. The photogrammetric scanner system of claim 1, wherein the frame is modular, the frame comprising a plurality of releasably connectable receptacles, each receptacle configured to receive an image sensor and a light emitting device therein.

15. The photogrammetric scanner system of claim 1 , wherein the frame comprises one or more surfaces to which the plurality of image sensors and the plurality of light emitting devices are connected.

16. providing a photogrammetric scanner system having a plurality of image sensors and a plurality of light emitting devices connected to and spaced apart on a frame; activating the photogrammetric scanner system to perform a coordinated illumination and imaging sequence of a target, wherein the plurality of light emitting devices project light in an illumination pattern and one or more images of the target are captured by the plurality of image sensors; A method of photogrammetric scanning including:

17. 1. An imaging assembly comprising: a body having an image sensor, a light emitting device, and a processing assembly connected thereto; the processing assembly is configured to coordinate an illumination and imaging sequence of a target, and the imaging assembly is designated as a controller imaging assembly among a network of imaging assemblies each in digital communication with the processing assembly and thus configured to coordinate the illumination and imaging sequence of the target of one or more imaging assemblies in the network of imaging assemblies; Imaging assembly.

18. The imaging assembly of claim 17 , wherein the imaging assembly comprises a housing, the body forming a portion of the housing.

19. 19. The imaging assembly of claim 17 or 18, wherein the processing assembly is configured to detect and communicate with one or more further imaging assemblies to form a network of the imaging assemblies.

20. 20. The imaging assembly of claim 17, wherein an additional imaging assembly of the one or more additional imaging assemblies is detected by the controller imaging assembly when the additional imaging assembly is electrically activated.

Images