Photogrammetric scanner system and imaging assembly

Abstract

A photogrammetric scanner system for capturing an image of a target. The photogrammetric scanner system includes a frame, a plurality of image sensors, and a plurality of light emitting devices. A plurality of image sensors are connected to and spaced apart from the frame. A plurality of light emitting devices are connected to and spaced apart from the frame and configured to project light in an illumination pattern.

Description

Technical Field

[0001] The present invention relates to a photogrammetric scanner system and an imaging assembly for capturing images of a target. In particular, the photogrammetric scanner system is used to capture images of a target that can be used to create a three-dimensional (3D) image and model of the target. Background Art

[0002] Any reference to methods, apparatus or documents of the relevant art is not to be taken as any evidence or admission that they form (or ever formed) part of the common general knowledge.

[0003] Photogrammetry is the technique of extracting three-dimensional information from photographs. Photogrammetry involves processing a set of overlapping photographs of a target object to produce a 3D computer model of that object.

[0004] Typical photogrammetric reconstruction algorithms search for common features in a set of images and compute the 3D coordinates of these features by correlating the relative feature positions. Similarly, the relative positions of the corresponding cameras can be determined in the same way.

[0005] By extrapolating many features and photos, the relative camera coordinates and the point cloud representing the 3D morphology of the target object can be calculated. From the point cloud, an artificial neural network or meshing technique can be used to construct a three-dimensional computer model consisting of vertices and polygons (triangles or N-gons). In order to further improve the quality of the final 3D object, a pixel-by-pixel depth map can be calculated for each input image, and a value is assigned to each pixel to indicate the relative distance of the pixel from the sensor. Then, the depth maps of several images can be combined to estimate the additional structural information between the calculated 3D feature point clouds before calculating the grid. During each processing stage, various algorithms can be used to filter unwanted noise data and apply 3D smoothing, re-meshing, and noise reduction algorithms. The final stage usually involves calculating UV coordinates for the resulting 3D grid, and generating a two-dimensional (Two Dimensional, 2D) texture using the source image, and mapping the 2D texture to the grid through the UV coordinates.

[0006] Two challenges in photogrammetry are (1) providing optimal source images for feature detection and depth estimation, and (2) ensuring optimal sampling of target objects and providing sufficient (and reliable) image overlap to enable unbiased 3D object reconstruction.

[0007] The optimal source image should contain only the object to be scanned (no background), be free of noise or lens distortion, be sharp and have rich details. The object will be maximized in the field of view, have sufficiently clear distinguishable features distributed on its surface, and be matte (non-reflective) and illuminated in a way that minimizes lighting artifacts (e.g., specular reflections and unwanted shadows). To enhance feature detection, indirect lighting that produces shadows may be more beneficial than flat direct lighting.

[0008] Optimal object sampling is important to maximize the useful data collected from a given set of images. If areas of an object are not adequately sampled (i.e., not enough image coverage) or all areas of an object are not covered by images with sufficient overlap, the reconstruction process will result in a poor quality image of the object.

[0009] Photogrammetry can be performed using a single camera, usually handheld or attached to a moving object (such as an aircraft or other vehicle), which takes multiple photos of the object from different positions. Another approach is to use multiple cameras and light sources, which are arranged at different positions on the frame and are all aimed at the object, while using a system to synchronize the triggering of the cameras. Related technologies use overlapping mapping images (based on aerial, satellite or ground vehicles) to reconstruct 3D models of large areas. These multi-camera setups are usually bulky, expensive, require a lot of space, and are difficult to assemble and move. In addition, because the cameras used in such systems usually produce a high quality level, the long processing time can also cause bottlenecks, thereby affecting the processing volume.

[0010] The weakness of photogrammetry is the difficulty in reconstructing homogeneous surfaces due to the lack of details for feature detection and depth map calculation. A way to overcome this limitation is to take two images per sensor; one with the pattern projected onto the object for geometry reconstruction and the other without the pattern for texture map generation.

[0011] Related 3D surface scanning technology uses structured light methods. Unlike photogrammetry, which uses computer vision technology to find features in the target object, the structured light method projects a pattern onto the scanned object (usually a single line or a series of parallel lines), and then observes the shape changes of the pattern. By determining the changes in the shape of the pattern, the 3D geometry of the target object can be determined. Common implementations of structured light 3D scanning include handheld 3D scanners (pattern projectors / imaging) that scan the object. The calculated 3D geometry of the target object is gradually optimized over time based on multiple samples. Summary of the invention

[0012] In one aspect, the present invention provides a photogrammetric scanner system for capturing an image of a target, comprising:

[0013] frame;

[0014] a plurality of image sensors connected to the frame and spaced apart on the frame; and

[0015] A plurality of light emitting devices are connected to the frame, spaced apart on the frame, and configured to project light in an illumination pattern.

[0016] Preferably, each of the plurality of image sensors is configured to have a wide field of view and a short focal length. This enables the image sensor to be located closer to the target object than a typical camera photogrammetry rig, and the photogrammetry scanner system can be made smaller and more portable.

[0017] Preferably, the plurality of image sensors include a plurality of cameras. Preferably, the plurality of light emitting devices include a plurality of light emitting diodes and / or a plurality of laser diodes for pattern recognition.

[0018] Preferably, the scanner system is configured to perform a coordinated illumination and imaging sequence of a target, wherein a plurality of light emitting devices project light in an illumination pattern and one or more images of the target are captured by a plurality of image sensors.

[0019] Preferably, the plurality of image sensors are located at fixed positions on the frame. Preferably, the direction or angle of each of the plurality of image sensors is dynamically variable. Preferably, each of the plurality of image sensors is connected to a servo motor to enable dynamic change of direction or angle.

[0020] Preferably, the plurality of light emitting devices and the plurality of image sensors are configured to coordinate the projection of light from the plurality of light emitting devices with an image capture sequence by the plurality of image sensors. Preferably, the projection of light from the plurality of light emitting devices is synchronized with the image capture sequence by the plurality of image sensors. Preferably, the image capture sequence comprises each of the plurality of image sensors capturing one or more images. Preferably, the image capture sequence comprises the plurality of image sensors capturing one or more images simultaneously. Preferably, the image capture sequence comprises activating a light emitting device adjacent to an image sensor and subsequently capturing an image by the image sensor adjacent to the light emitting device.

[0021] Preferably, the arrangement of the plurality of image sensors and the plurality of light emitting devices includes each image sensor of the plurality of image sensors being adjacent to one of the plurality of light emitting devices.

[0022] Preferably, there is an arrangement of an image sensor of the plurality of image sensors and a light emitting device of the plurality of light emitting devices on each arm of the plurality of arms of the frame, wherein the arrangement on each arm comprises each image sensor being adjacent to one light emitting device.

[0023] Preferably, the arrangement on each arm is opposite to the arrangement of the immediately adjacent arm. Alternatively, the arrangement on each arm is inverted relative to the arrangement of the immediately adjacent arm.

[0024] Preferably, any two image sensors among the plurality of image sensors are not coplanar.

[0025] Preferably, the frame comprises one or more surfaces, and the plurality of image sensors and the plurality of light emitting devices are connected to the one or more surfaces. Preferably, the one or more surfaces are flat, curved or arc-shaped.

[0026] Preferably, the frame is a hand-held portable frame.Alternatively, the frame is a portable frame.

[0027] Preferably, the frame is modular. Preferably, the frame comprises a plurality of sockets, each socket being configured to accommodate an image sensor and a light emitting device in a respective socket. Preferably, the plurality of sockets are releasably connectable.

[0028] Preferably, the frame comprises a plurality of arms. Preferably, the plurality of arms are spaced apart along a frame support member. Preferably, each arm of the plurality of arms extends substantially perpendicular to the frame support member.

[0029] Preferably, each arm includes one or more of a plurality of image sensors thereon.Preferably, each arm is dynamically configurable.

[0030] Preferably, each arm of the plurality of arms is curved. Preferably, the curvature of each arm is substantially equal.

[0031] Preferably, the scanner system includes a plurality of scanner components. The scanner components include one or more imaging components. The scanner components also include one or more auxiliary components.

[0032] Preferably, each arm comprises an imaging assembly including an image sensor and a light emitting device.

[0033] Preferably, each image sensor of the plurality of image sensors is offset from any other image sensor of the plurality of image sensors in three dimensions.

[0034] Preferably, each image sensor of the plurality of image sensors has coordinates in three-dimensional space that are unique in each of the three dimensions relative to the coordinates of any other image sensor of the plurality of image sensors.

[0035] Preferably, the plurality of image sensors are configured to be sensitive to a range of wavelengths of light.Preferably, the plurality of image sensors are configured to be sensitive to a range of wavelengths of light (including visible and non-visible wavelengths) including at least one of ultraviolet light, visible light and near infrared light.

[0036] Preferably, one of the plurality of light emitting devices is independently controllable relative to another of the plurality of light emitting devices. Preferably, the plurality of light emitting devices are configured to emit light of a certain wavelength range. Preferably, each of the plurality of light emitting devices comprises a plurality of light sources emitting light of different wavelengths. Preferably, the plurality of light sources are independently configurable and / or controllable. Preferably, the plurality of light emitting devices are configured to emit light of a wavelength range including at least one of ultraviolet light, visible light and near infrared light.

[0037] Preferably, the plurality of image sensors are configured to detect wavelengths output by the plurality of light emitting devices.

[0038] Preferably, the light emitting device is configured to project a static or dynamic lighting pattern at a predetermined wavelength. Preferably, the light emitting device is configured to project a static or dynamic lighting pattern at a predetermined wavelength by: a) controlling the sequencing of lighting elements; or b) projecting the pattern via a pattern screen and / or lens; or c) using a laser pattern generator (scanning or through an optical grating, filter or holographic lens).

[0039] Preferably, the grating is located above the plurality of light emitting devices. Preferably, the grating may be static or dynamic. The grating is used to achieve modulation pattern projection in hybrid photogrammetry and / or structured light scanning.

[0040] Preferably, the photogrammetric scanner system further comprises an orientation sensor and / or an acceleration sensor. Preferably, the orientation sensor comprises a gyroscope and / or an accelerometer. Preferably, the acceleration sensor comprises an accelerometer. The orientation sensor and / or the acceleration sensor may be used to determine and track the scanner position and orientation to assist in alignment of the target within the field of view of the image sensor.

[0041] Preferably, each light emitting device in the plurality of light emitting devices is offset from any other light emitting device in the plurality of light emitting devices in each of the three dimensions.

[0042] Preferably, each of the plurality of light emitting devices has coordinates in three-dimensional space, and the coordinates are unique in each of the three dimensions relative to the coordinates of any other light emitting device in the plurality of light emitting devices.

[0043] Preferably, the photogrammetric scanner system further comprises one or more controllers configured to control the plurality of image sensors and / or the plurality of light emitting devices.

[0044] Preferably, the photogrammetric scanner system comprises a 3D depth sensor for calculating accurate point cloud data in order to implement a hybrid laser radar (LIDAR) and photogrammetry system. Preferably, the 3D depth sensor comprises a LIDAR sensor.

[0045] In another aspect, the present invention provides a method for photogrammetric scanning, the method comprising:

[0046] Providing a photogrammetric scanner system having a plurality of image sensors and a plurality of light emitting devices connected to and spaced apart on a frame;

[0047] The photogrammetric scanner system is activated to perform a coordinated illumination and imaging sequence of a target, wherein a plurality of light emitting devices project light in an illumination pattern and one or more images of the target are captured by a plurality of image sensors.

[0048] Preferably, the method comprises projecting patterned light. Preferably, the patterned light is generated by projecting light from a laser diode through a grating or hologram adjacent to the laser diode. This enables a hybrid structured light and photogrammetry system to be achieved.

[0049] In another aspect, the present invention provides an imaging assembly comprising:

[0050] A body, to which an image sensor, a light emitting device, and a processing component are connected; and

[0051] The processing component is configured to coordinate illumination and imaging sequences of a target, wherein the imaging component is designated as a controller imaging component in a network of imaging components, each imaging component is in digital communication with the processing component, and the imaging component is thus configured to coordinate illumination and imaging sequences of a target by all imaging components in the network of imaging components.

[0052] Preferably, the imaging assembly comprises a housing, wherein the body forms part of the housing.

[0053] Preferably, the processing component is configured to form a network of imaging components by detecting and communicating with one or more additional imaging components.

[0054] Preferably, an additional imaging component of the one or more additional imaging components is detected by the controller imaging component when the additional imaging component is electrically activated. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The preferred features, embodiments and variations of the present invention can be derived from the following detailed description, which provides sufficient information for those skilled in the art to implement the present invention. This detailed description should not be construed as limiting the scope of the foregoing invention in any way. A detailed description will be described with reference to the following several drawings.

[0056] Figure 1is a front view of a photogrammetric scanner system for capturing an image of a target according to an embodiment of the present invention.

[0057] Figure 2 is a side view of a photogrammetric scanner system.

[0058] Figure 3 is a front view of the arm of the photogrammetric scanner system.

[0059] Figure 4 A side view of the arm of a photogrammetric scanner system is shown.

[0060] Figure 5 and Figure 6 An image sensor, a light emitting device, and a controller of a photogrammetric scanner system on a printed circuit board are shown.

[0061] Figure 7 is a functional block diagram of a photogrammetry scanner system.

[0062] Figure 8 and Fig. 9 An embodiment of a photogrammetric scanner system imaging a face is shown.

[0063] Fig.10 and Fig.11 The photogrammetric scanner system is shown reconfigured to be folded for storage and / or transport.

[0064] Figure 12 to Figure 14 An imaging assembly according to an embodiment of the present invention is shown.

[0065] Fig.15 and Fig.16 The framework of a photogrammetric scanner system is shown.

[0066] Fig.17 and Fig.18 A photogrammetric scanner according to another embodiment of the present invention is shown.

[0067] Fig.19 A modular framework for a photogrammetric scanner system is shown.

[0068] Fig. 20 A portion of a photogrammetric scanner system using a modular frame is shown.

[0069] Fig.21 An enlarged view of the connecting members used to connect the sockets (modules) of the modular frame is shown. DETAILED DESCRIPTION

[0070] Figures 1 to 5A photogrammetric scanner system 10 is depicted for capturing images of a target. In particular, the photogrammetric scanner system 10 is used to capture images of a target that can be used to create a three-dimensional image of the target. Examples of targets include a person's head or face. However, the target may include any part of a person or another object.

[0071] The photogrammetric scanner system 10 includes a frame 100 and a plurality of scanner assemblies including a plurality of imaging assemblies 101 coupled to the frame 100. The scanner assembly may also include one or more auxiliary components that perform auxiliary functions such as lighting or control of an imaging sequence.

[0072] The imaging assembly 101 is connected to the frame 100 in an array and is spaced apart on the frame 100. More specifically, the imaging assembly 101 includes a plurality of image sensors 120 connected to the frame 100 in an array and spaced apart from the frame 100, and a plurality of light emitting devices 140 connected to the frame 100 and spaced apart on the frame 100, the light emitting devices 140 being configured to emit light in an illumination pattern. In some embodiments, each imaging assembly 101 includes a pair of at least one image sensor and one light emitting device.

[0073] In use, the plurality of light emitting devices 140 project light in an illumination pattern onto a target, and one or more images of the target are captured by the plurality of image sensors 120 .

[0074] That is, the photogrammetric scanner system 10 is configured to perform a coordinated illumination and imaging sequence of a target. The plurality of light emitting devices 140 project light in an illumination pattern, and one or more images of the target are captured by the plurality of image sensors 120 .

[0075] In some embodiments, the coordinated lighting and imaging sequence includes activating a light emitting device in the plurality of light emitting devices 140 adjacent to one of the plurality of image sensors 120, and subsequently capturing an image via the image sensor adjacent to the light emitting device. In other embodiments, the light emitting devices and image sensors may be activated non-sequentially, such that a light emitting device is activated and a non-adjacent image sensor subsequently captures an image.

[0076] In some embodiments, the image sensors have a wide field of view and a short focal length, which allows the image sensors to be placed closer to the target than typical photogrammetry equipment to optimize their object / sensor ratio. That is, optimize each sensor's coverage of the target and ensure sufficient overlap between images from adjacent sensors.

[0077] The plurality of image sensors are in the form of a plurality of cameras, and the plurality of light emitting devices are in the form of a plurality of light emitting diodes. It should be understood that any form of light emitting device may be used, including, for example, organic light emitting diodes (OLEDs).

[0078] Embodiments of the present invention use a 1 / 4 inch image sensor.

[0079] By providing some embodiments of the photogrammetric scanner system 10 with image sensors having a wide field of view and a short focal length, the photogrammetric scanner system 10 can be positioned reasonably close to the target as described above, thereby reducing the need for relatively expensive optical elements for acquiring high-resolution images from a distance, and enabling a higher actual density of sensors in the array than would be the case using typical photogrammetric equipment. A higher sensor density means that a small area of ​​the target can be completely filled with each sensor, and the sensors are arranged to ensure sufficient overlap so that the object is completely imaged.

[0080] It will be appreciated that exact specifications (eg, focal length and field of view) and density will depend on the typical expected camera-to-target distance and the geometry of the target and scanner equipment.

[0081] The illustrated embodiment uses a plurality of image sensors 120 in an array that are attached to the surface of the frame 100. The arrangement of the image sensors in an array (described in further detail below) optimizes coverage and overlap of target objects, thereby maximizing the efficiency of the photogrammetry process.

[0082] The frame 100 in the form of a handheld and portable frame includes a frame support member 105 and a plurality of surfaces in the form of arms 110a-110f to which a plurality of image sensors 120 and a plurality of light emitting devices 140 are connected. The arms 110a-110f are connected to the frame support member 105 and are spaced apart along the length of the frame support member 105.

[0083] In a preferred embodiment, the frame 100 is made of a lightweight plastic or a lightweight metal material (eg, carbon fiber).

[0084] In some embodiments, the frame may include a single curved or arcuate surface having a plurality of image sensors and a plurality of light source devices disposed thereon in accordance with the description herein with respect to other embodiments.

[0085] The arms 110a-110f and frame support member 105 are arcuate, and the curvature or arc of each arm 110a-100f is substantially equal. However, the arms 110a-110f and frame support member can be flat or planar, and can also be dynamically configured between flat and arcuate.

[0086] The illustrated embodiment shows that the frame support member 105 is arcuate (as described above) and extends longitudinally in a first direction, and the arms 110a-110f are arcuate and extend longitudinally in a second direction from the frame support member 105. In other words, the frame support member 105 extends in a first direction along a first longitudinal axis 106 extending from a first end 107 to a second end 108 of the frame support member 105, and each arm 110a-110f extends in a second direction along its respective second longitudinal axis 111 extending from a first end 112 to a second end 113 of each arm 110a-110f. The first longitudinal axis 106 and the second longitudinal axis 111 are substantially perpendicular.

[0087] Although arcuate, each of the arms 110a - 110f may be considered to extend substantially perpendicularly from the frame support member 105 at the point where each respective arm 110a - 110f is fixedly connected to the frame support member 105 .

[0088] The curvature of the arms 110a - 110f and the frame support members 105 provide the frame 100 with a substantially hemispherical (or semi-spherical) shape or arrangement.

[0089] The hemispherical arrangement enables each image sensor to sample (image) the target / object from substantially the same distance depending on the geometry of the target. This arrangement also enables each image sensor to be located at a unique 3D coordinate (i.e., no image sensors are coplanar with each other), rather than 2D as in the case of cylindrical or planar array configurations. It is contemplated that such an arrangement may provide improved feature localization in 3D space (e.g., triangulation advantages).

[0090] Turning now to the image sensors and light emitting devices connected to the frame 100 , a general arrangement of the plurality of image sensors 120 and the plurality of light emitting devices 140 includes each image sensor of the plurality of image sensors 120 being adjacent to one of the plurality of light emitting devices 140 .

[0091] As in Figure 1 and Figure 2As shown, there is an arrangement of some of the plurality of image sensors 120 and some of the plurality of light emitting devices 140 on each of the six arms 110a-110f of the frame 110. The arrangement on each arm 110a-110f includes each image sensor being adjacent to one light emitting device.

[0092] Although the embodiment shown in the figure includes six arms, any number of arms can be provided. For example, in one embodiment, five arms can be provided. In another embodiment, seven arms can be provided.

[0093] The arrangement on each arm 110a-110f is opposite to the arrangement of the next adjacent arm, or is inverted relative to the adjacent arm. For example, in the present embodiment, each arm 110a-110f includes five image sensors and five light emitting devices, wherein arm 110a includes an arrangement of image sensors and light emitting devices alternating from top to bottom (or from one end to the other), starting with image sensors and ending with light emitting devices; and arm 110b includes an arrangement of light emitting devices alternating from top to bottom (or from one end to the other), starting with light emitting devices and ending with image sensors. This pattern is repeated in sequence on each of the six arms. More generally, the pattern is repeated alternating on each consecutive or subsequent arm.

[0094] As mentioned above, no two image sensors of the photogrammetric scanner system 10 are coplanar, and this ensures that optimal overlap is achieved in the images obtained by all image sensors.

[0095] In the illustrated embodiment, each image sensor in the plurality of image sensors 120 is offset from any other image sensor in the plurality of image sensors 120 in each of the three dimensions, such that the coordinates of each image sensor in the plurality of image sensors 120 in three-dimensional space are unique in each of the three dimensions relative to the coordinates of any other image sensor in the plurality of image sensors 120.

[0096] Similarly, each light emitting device in the plurality of light emitting devices 140 is offset in each of the three dimensions from any other light emitting device in the plurality of light emitting devices 140, such that the coordinates of each light emitting device in the plurality of light emitting devices 140 in three-dimensional space are unique in each of the three dimensions relative to the coordinates of any other light emitting device in the plurality of light emitting devices 140. However, it should be understood that in some embodiments, the image sensors are offset from each other, while the light emitting devices are not offset from each other.

[0097] Turning briefly to the lighting devices, each lighting device may be synchronized with or independently controllable relative to another lighting device in the plurality of lighting devices 140 .

[0098] The light emitting devices may be configured to emit a range of wavelengths of light, or each light emitting device may include multiple light sources (e.g., in the form of light emitting diodes or other suitable light emitting devices) that emit light of different wavelengths, including but not limited to ultraviolet light, visible light, and near infrared light. Figure 3 and Figure 5 An example of multiple different light sources provided can be seen in , showing a first type of light emitting diode 140a configured to emit a first type of light and a second type of light emitting diode 140b configured to emit a second type of light, the wavelength of the second type of light being different from the wavelength of the first type of light emitted by the first type of light emitting diode 140a.

[0099] Independently controllable light emitters can also project illumination patterns at different wavelengths, which helps maximize the visibility of the target in each image.

[0100] The independently controllable light emitting devices enable the image sensor to receive reflected light at a target wavelength of the light emitting devices to capture an image of the target object at that wavelength.

[0101] In some embodiments, a bright light is projected very close to the target object (e.g., using a high intensity LED). This enables the target object to be illuminated with light of a target wavelength that is brighter than the surrounding broad spectrum light. This also means that the shutter speed of the image sensor can be reduced, thereby reducing the incidence of the image sensor focusing on objects outside the target area, which are darker than the target illuminated by the bright light.

[0102] In order to utilize these different wavelengths of light provided by the light emitting devices, the plurality of image sensors 120 are configured to detect (or be capable of imaging) wavelengths corresponding to the wavelengths output by the plurality of light emitting devices 140 .

[0103] In use, the plurality of light emitting devices 140 are configured to project static or dynamic lighting patterns of predetermined wavelengths (as described above) by: a) controlling the sequencing of lighting elements; or b) projecting the pattern via a pattern screen and / or lens; or c) using a laser pattern generator (scanning or through an optical grating, filter or holographic lens).

[0104] Gratings (static or dynamic) in the form of optical gratings described above may be positioned above the plurality of light emitting devices 140 to achieve illumination patterns and facilitate modulation pattern projection in hybrid photogrammetry and / or structured light scanning.

[0105] In some embodiments, the grating provides patterned light. The patterned light is generated by projecting light from a laser diode through a grating (or hologram) adjacent to the laser diode. This enables a hybrid structured light and photogrammetry system to be realized.

[0106] In some embodiments, a dynamic grating causes a light pattern from the light emitting device to form in a motion that sweeps across the surface of the target object. A sequence of images of this changing light pattern on the surface of the object can be taken from a fixed camera position, and this data is used in a hybrid structured light / photogrammetry algorithm to improve 3D object geometry accuracy and resolution.

[0107] In some embodiments, a filter (or a series of filters that can be selectively placed on the image sensor) is provided to selectively transmit or block light of a specific wavelength or wavelength range. The filter is positioned above the image sensor to optimize scanning for a specific wavelength, and a multi-wavelength illumination configuration (e.g., multiple light emitting devices emitting light of different wavelengths) is additionally or alternatively used to optimize scanning for a specific wavelength.

[0108] The use of patterns can be used to provide additional features to photogrammetric reconstruction software to improve 3D model mesh accuracy and detail, or used by structured light algorithms in addition to standard photogrammetric processing to achieve the same purpose. In particular, this can be achieved by sequencing the activation of the light emitting devices in coordination with sequencing the image sensor to capture a single image or video. Alternatively, the projected pattern on the target can also be moved by moving a grating or holographic lens through the light emitting device as it projects light.

[0109] Each arm 110a-110f of the photogrammetric scanner system 10 includes a cover plate 114. The image sensor and the light emitting device are located between the body of the arm 110a-110f and the cover plate 114. The cover plate 114 is transparent to allow light from the light emitting device to be projected onto the target, and the cover plate 114 includes an opening (e.g., opening 115) for each image sensor to enable the image sensor to capture an image. Alternatively, the opening may be replaced by a transparent cover that enables the image sensor to capture an image. Regarding the above-mentioned illumination patterns, as well as the use of optical gratings, filters, holographic lenses, and pattern screens, these can be formed on a control panel or applied to the surface of the cover plate.

[0110] Each imaging assembly 101 of the photogrammetric scanner system 10 includes a processing assembly including a controller 180 in the form of a microprocessor connected to the plurality of image sensors 120 and / or the plurality of light emitting devices 140 and configured to control the plurality of image sensors 120 and / or the plurality of light emitting devices 140. The controller 180 may be configured to: Figure 6 and Figure 7 Seen in Figure 6 and Figure 7 A functional block diagram of a photogrammetric scanner system 10 is shown.

[0111] The controller 180 in this embodiment is connected to the image sensor 120 via a camera serial interface (CSI), but any other suitable camera data interface may be used. Each controller 180 includes software executable to control the image sensor 120 .

[0112] Controller software can electronically configure sensor imaging parameters such as resolution, pixel binning, capture mode (still or video), shutter speed, exposure time, and gamma as needed or desired.

[0113] The controller 180 is located on the frame 100 and is close to the image sensor 120 (preferably, as shown in FIG. Figure 5 and 6 As shown, on printed circuit board 121), the controller 180 controls imaging parameters, triggers the camera (to capture still images or video), receives image data via the CSI port, interfaces with local memory to store image data, communicates with other microcontrollers or external computer devices 2 via physical connection or wireless communication (Bluetooth, WI-FI, etc.), performs diagnostics, controls lighting, and runs locally stored program code.

[0114] Controller 180 may also be programmed to enable coordinated sequencing and video capture according to the application as described below.

[0115] The controller 180 is connected to an onboard non-volatile memory 181, and the images are stored in a manner that enables identification of the images during subsequent processing. To help store and associate each image set with each physical object, an identification method can be used that images an object descriptor before imaging the object itself. Examples of object descriptions include bar codes, XR codes, text recognition algorithms, or any other symbol designed to identify an object. For example, many hospitals use digital patient identification numbers that are used to link patient information, such as notes, measurements, and images, within the hospital's patient database system. Alternatively or additionally, facial recognition or object recognition can also be used to identify a person or target object to be scanned using a single or multiple image sensors.

[0116] Prior to or during a 3D scan of the patient, the patient ID can be imaged by placing the identifier within the field of view of the sensor array and triggering the scan. Onboard software can recognize the associated code within the captured image and ensure that each subsequent patient image is associated with the code in the image file name, image header, or other mechanism. The code can be stored in the image header so that it can be transferred to the 3D scan file header or file name during photogrammetric 3D reconstruction. After each desired imaging sequence is completed, the onboard microcontroller stores and tags the image in a storage location using a communication protocol within the controller. At this point, the photogrammetric scanner system 10 is ready to trigger the next imaging sequence.

[0117] Advantageously, the above-described hemispherical (or semi-spherical) arrangement enables each image sensor to sample the target from substantially the same distance, and also ensures that each image sensor is located at a unique 3D coordinate, rather than a 2D coordinate as in the case of a cylindrical or planar array configuration. However, as will be described in more detail below, the position and orientation of the image sensors can be reconfigured to optimize for objects of different geometries, and a spherical arrangement can be implemented (by adding more image sensors and arms, or by dynamically repositioning the image sensors and arms along the frame) to simultaneously image 360 ​​degrees of an object, or a planar or cylindrical arrangement can be implemented to image objects of similar geometry.

[0118] The arms of the frame may also be dynamically configurable to enable a single scanner to be optimized for different target objects and to create arrangements of different shapes. For example, the arms may be configurable between a substantially flat (or planar) arrangement and a curved (or arcuate) arrangement. In some embodiments, the image sensor may also be configured to diverge to scan surrounding geometry, such as inside or outside a building.

[0119] To perform a 3D scan of an object, the photogrammetric scanner system 10 is placed at a certain position relative to the object, and a signal is provided to the controller 180 to initiate the 3D scan. In one embodiment, the image sensor is approximately 27 cm from the central focus, where the target should be approximately located. The signal can be provided by a single input trigger 182, wirelessly provided by a remote device, or provided via a timer device (e.g., a clock) or code. Upon detecting a trigger, the controller 180 operates a programmed lighting and image capture sequence, which involves 1) communicating with the image sensors 120 to set their parameters, acquire images, and transmit the images to the microcontroller for storage in the memory 181; 2) communicating with the lighting control hardware to turn each light emitting device 140 (and pattern generator) on and off.

[0120] Once the images have been captured, they are transferred from the non-volatile storage device 181 of the photogrammetric scanner system 10 to an external computer 2 equipped with photogrammetric software (e.g., AliceVision) programmed to create a 3D model from the images captured by the photogrammetric scanner system 10. Alternatively, the photogrammetric software may be stored and run in the non-volatile storage device 181 of the imaging component to facilitate on-board digital image processing and 3D model processing, such processing being performed on a single on-board processor or by a cluster of on-board processors.

[0121] exist Figures 12 to 18 In FIG. 1 , a photogrammetric scanner system 20 is shown for capturing an image of a target. The photogrammetric scanner system 20 is used to capture an image of a target, which image of the target can be used to create a three-dimensional image or model of the target. Examples of targets include a person's head, face, hands, or feet. However, the target may include any part of a person or another object.

[0122] The photogrammetric scanner system 20 includes a frame 200 and a plurality of scanner assemblies including a plurality of imaging assemblies 201 connected to the frame 200 .

[0123] Imaging assembly 201 also includes processing assembly 280 (described in more detail below).

[0124] The scanner assembly may also include one or more auxiliary components that perform auxiliary functions, such as control of the imaging sequence or illumination.

[0125] The imaging assembly 201 is connected (or connectable) to the frame 200 so that a plurality of imaging assemblies 201 are spaced apart in an array on the frame 200. More specifically, the imaging assembly 201 includes: a plurality of image sensors 220 connected (or connectable) to the frame 200 and spaced apart in an array on the frame 200 (the image sensors 220 shown include image sensors and a housing for mounting the image sensors), and a plurality of light emitting devices 240 connected (or connectable) to the frame 200 and spaced apart on the frame 200, the light emitting devices 240 being configured to emit light in an illumination pattern. In some embodiments, each imaging assembly 201 includes at least one image sensor 220 and one light emitting device 240 in pairs.

[0126] Each imaging assembly 201 includes a body 202 , to which at least one image sensor 220 and at least one light emitting device 240 are connected.

[0127] The imaging assembly 201 may also include a grating (static or dynamic) in the form of an optical grating located above the plurality of light emitting devices 240 to achieve the effect of an illumination pattern and facilitate the projection of a modulated pattern for hybrid photogrammetry and / or structured light scanning. The grating in the illustrated embodiment takes the form of a diffuser 250 located above the light emitting devices 240 to improve light diffusion and thus improve illumination of the object / target.

[0128] The imaging assembly 201 is shown as being hexagonal, but may be of any shape.

[0129] In the illustrated embodiment, the imaging assembly 201 is arranged such that the image sensor 220 is located between the body 202 and the light emitting device 240. More specifically, the arrangement is in the following order: body 202, processing assembly 280, image sensor 220, light emitting device 240, and diffuser 250 (if used).

[0130] The body 202, image sensor 220, and light emitting device 240 each include a connector for connecting to a connector on another component (e.g., the image sensor 220 includes a connector for connecting to the body 202 and the light emitting device 240). In the illustrated embodiment, the connector takes the form of a multi-pin magnetic connector 203 that provides connection to and distributes power to the necessary components (including the processing component 280, the image sensor 220, and the light emitting device 240).

[0131] The imaging assembly 201 may also be located in an environmentally sealed housing 204. The body 202 is a part of the housing 204.

[0132] The processing component 280 includes a system on a chip having a programmable microprocessor and memory. The processing component 280 is configured to control and operate the image sensor 220, the light emitting device 240, and associated inputs and outputs (I / O).

[0133] The processing component 280 may include a wireless transceiver, and the processing component 280 is configured to implement wireless synchronous operation and data communication without the need for physical inter-device signaling. Therefore, in some embodiments, the processing component 280 of the imaging component 220 only communicates data through wireless means or only communicates data through a wireless interface. However, in some embodiments, the processing components 280 may also be physically interconnected for data communication.

[0134] In use, the plurality of light emitting devices 240 project light in an illumination pattern onto a target, and the plurality of image sensors 220 capture one or more images of the target.

[0135] That is, the photogrammetric scanner system 20 is configured to perform a coordinated illumination and imaging sequence of a target, wherein the plurality of light emitting devices 240 project light in an illumination pattern and the plurality of image sensors 220 capture one or more images of the target.

[0136] In some embodiments, the coordinated lighting and imaging sequence includes activating a light emitting device 240 associated with an image sensor 220 of an imaging assembly 201, and subsequently capturing an image by the image sensor 220 associated with the light emitting device 240. In other embodiments, the light emitting device 240 and the image sensor 220 may be activated non-sequentially, such that one light emitting device 240 on a first imaging assembly 201 is activated and an image sensor 220 on a second imaging assembly 201 subsequently captures an image.

[0137] In some embodiments, the image sensors 220 have a wide field of view and a short focal length, which allows the image sensors 220 to be positioned closer to the target than typical camera photogrammetry equipment to optimize their object / sensor ratio, that is, to optimize each sensor's coverage of the target and ensure sufficient overlap between images from adjacent sensors.

[0138] The plurality of image sensors 22 are in the form of a plurality of cameras, and the plurality of light emitting devices 240 are in the form of a plurality of light emitting diodes. It will be appreciated that any form of light emitting device may be used, including, for example, OLEDs.

[0139] Embodiments of the present invention use a 1 / 4 inch image sensor, but any type of image sensor may be used.

[0140] By providing some embodiments of the photogrammetric scanner system 20 with an image sensor 220 having a wide field of view and a short focal length, the photogrammetric scanner system 20 can be positioned reasonably close to the target as described above, thereby reducing the need for relatively expensive optical elements for acquiring high-resolution images from a distance, and enabling a higher actual density of sensors in the array than would be the case using typical photogrammetric equipment. A higher sensor density means that a small area of ​​the target can be completely filled with each sensor, and the sensors are arranged to ensure sufficient overlap so that the object is completely imaged.

[0141] It will be appreciated that exact specifications (eg, focal length and field of view) and density will depend on the typical expected camera-to-target distance and the geometry of the target and scanner equipment.

[0142] The frame 200 may be formed in a modular manner.

[0143] In a preferred embodiment, the frame 200 is made of a lightweight plastic or a lightweight metal material (eg, carbon fiber).

[0144] In some embodiments, the frame 200 may include a curved or arcuate body 210 having a plurality of imaging components 201 disposed thereon as described herein with respect to other embodiments.

[0145] The frame 200 includes a plurality of sockets 211 formed in a body 210 of the frame 200, and the shape of the sockets 211 is consistent with the shape of the imaging assembly 201. This enables the imaging assembly 201 to be connected to the frame 200.

[0146] The sockets 211 may be releasably connected so that the frame 200 can be modularly formed and reconfigured according to the needs of different goals / objects.

[0147] In a use configuration, the illustrated embodiment uses a plurality of image sensors 120 in an array spaced about the frame 200 based on the location of the socket 211 housing the imaging assembly 201. The arrangement of the image sensors 220 in an array (described in further detail below) optimizes coverage and overlap of the target object to maximize the efficiency of the photogrammetry process.

[0148] The body 210 of the frame 200 is substantially hemispherical with curved faces. However, the body may be flat or planar, and may also be dynamically configurable between flat and curved.

[0149] The hemispherical arrangement enables each image sensor 220 to sample (image) the target / object from substantially the same distance depending on the geometry of the target. This arrangement also enables each imaging component 201 to be located at a unique 3D coordinate when measured from a center point (i.e., none of the imaging components 201 are coplanar or non-coplanar), rather than 2D as in the case of a cylindrical or planar array configuration. Each imaging component 201 can be programmed with specified coordinates based on a known position in the frame 200, or alternatively, each imaging component 201 is configured to determine its coordinates within the frame 200 by communicating with any other imaging component 201 within the frame 200. In such embodiments, at least one imaging component 201 can be programmed with coordinates as an initial reference for any new imaging component 201 added to the frame 200.

[0150] Since any two imaging components 201 are not coplanar, each imaging component 201 is offset from any other imaging component 201 .

[0151] Thus, when measured from the center point, each image sensor 220 is located at a unique 3D coordinate (i.e., none of the image sensors 220 are coplanar or non-coplanar), rather than 2D as with a cylindrical or planar array configuration. It is contemplated that such an arrangement may provide improved feature localization in 3D space (e.g., triangulation advantages).

[0152] As described above, no two image sensors 220 of the photogrammetric scanner system 20 are coplanar, and this ensures that optimal overlap is achieved in the images obtained by all image sensors 220 .

[0153] Since any two image sensors 220 are not coplanar, each image sensor 220 is offset from any other image sensor 220 in the multiple image sensors 220 in each of the three dimensions, so that the coordinates of each image sensor 220 in three-dimensional space are unique relative to the coordinates of any other image sensor 220 in each of the three dimensions.

[0154] Similarly, each light emitting device 240 is offset in each of the three dimensions from any other light emitting device 240 in the plurality of light emitting devices 240, such that the coordinates of each light emitting device 240 in three-dimensional space are unique in each of the three dimensions relative to the coordinates of any other light emitting device 240. However, it should be understood that in some embodiments, the image sensors 220 are offset from each other, while the light emitting devices 240 are not offset from each other.

[0155] Turning briefly to the lighting devices 240 , each lighting device 240 may be synchronized with all other lighting devices 240 in the frame 200 , or may be independently controlled relative to another lighting device 240 by its corresponding processing component 280 (described below).

[0156] The light emitting device 240 may be configured to emit a range of wavelengths of light, or one or more of the light emitting devices 240 may include multiple light sources (e.g., in the form of light emitting diodes or other suitable light emitting devices) that emit light of different wavelengths, including, but not limited to, ultraviolet light, visible light, and near infrared light. Examples of multiple different light sources provided include a first type of light emitting diode configured to emit a first type of light and a second type of light emitting diode configured to emit a second type of light, the wavelength of the second type of light being different from the wavelength of the first type of light emitting diode emitting the first type of light.

[0157] The independently controllable light emitting devices 240 may also project illumination patterns at different wavelengths to facilitate maximizing visibility of the target in each image captured by the corresponding image sensor 220 .

[0158] The independently controllable light emitting devices 240 enable the corresponding image sensors 220 to receive reflected light at a target wavelength of the light emitting devices 240 to capture an image of the target object at that wavelength.

[0159] In some embodiments, bright light is projected very close to the target object (e.g., using a high intensity LED). This enables the target object to be illuminated with light of a target wavelength that is brighter than the surrounding broad spectrum light. This also means that the shutter speed of the image sensor can be reduced, thereby reducing the incidence of the image sensor 220 focusing on objects outside the target area, which are darker than the target illuminated by the bright light.

[0160] In order to utilize these different wavelengths of light provided by the light emitting devices 240 , the plurality of image sensors 220 are configured to detect (or be capable of imaging) wavelengths corresponding to the wavelengths output by the plurality of light emitting devices 240 .

[0161] In use, the plurality of light emitting devices 240 are configured to project static or dynamic lighting patterns of predetermined wavelengths (as described above) by: a) controlling the sequencing of lighting elements; or b) projecting the pattern via a pattern screen and / or lens; or c) using a laser pattern generator (scanning or through an optical grating, filter or holographic lens).

[0162] Processing component 280 is similar to Figure 6 and Figure 7 The processing components shown in are configured, Figure 6 and Figure 7 A functional block diagram of a photogrammetric scanner system 10 is shown.

[0163] The processing components 280 in this embodiment are connected to the image sensor 220 via CSI, but any other suitable camera data interface may be used. Each processing component 280 includes software executable to control the image sensor 220 and the light emitting device 240 associated with the imaging component 201 .

[0164] The controller software can electronically configure sensor imaging parameters such as resolution, pixel binning, capture mode (still or video), shutter speed exposure time, and gamma as needed or desired.

[0165] The processing component 280 controls imaging parameters, triggers the camera (to capture still images or video), receives image data via the CSI port, interfaces with local memory to store image data, communicates with other microcontrollers or external computer devices 2 via physical connection or wireless communication (Bluetooth, WI-FI, etc.), performs diagnostics, controls lighting, and runs locally stored program code.

[0166] The processing component 280 may also be programmed to implement coordinated sequencing and video capture according to the application as described below.

[0167] The processing component 280 includes onboard memory 281, and images are stored in a manner that enables identification of the images for subsequent processing. To help store and associate each image set with each physical object, an identification method can be used that images an object descriptor before imaging the object itself. Examples of object descriptions include bar codes, XR codes, text recognition algorithms, or any other symbol designed to identify an object. For example, many hospitals use digital patient identification numbers that are used to link patient information, such as notes, measurements, and images, within the hospital's patient database system. Alternatively or additionally, facial recognition or object recognition can also be used to identify a person or target object to be scanned using a single or multiple image sensors.

[0168] To perform a 3D scan of an object, the photogrammetric scanner system 20 is placed at a certain position relative to the object, and a signal is provided to the processing component 280 to initiate the 3D scan. In one embodiment, the image sensor 220 is approximately 27 cm from the central focus, where the target should be approximately located. The signal can be provided by a single input trigger, wirelessly provided by a remote device, or provided via a timer device (e.g., a clock) or code. Upon detection of a trigger, the processing component 280 operates a programmed coordinated lighting and image capture sequence, which involves 1) communicating with the image sensors 220 to set their parameters, acquire images, and transmit the images to the microcontroller for storage in the memory 281; 2) communicating with the lighting control hardware to turn each light emitting device 240 (and pattern generator) on and off.

[0169] Once the images have been captured, the images are transferred from the memory 281 of the photogrammetric scanner system 20 to an external computer (e.g., computer 2 in the previous illustrations) equipped with photogrammetric software (e.g., AliceVision) programmed to create a 3D model from the images captured by the photogrammetric scanner system 20. Alternatively, the photogrammetric software may be stored and run in the memory 281 of the imaging component to facilitate onboard digital image processing and 3D model processing, such processing being performed on a single onboard processor, or by a cluster of onboard processors.

[0170] In addition or in lieu of the above, the processing component 280 automatically and dynamically reconfigures the software and hardware of the corresponding imaging component 201 to ensure that each component performs independently as needed and performs collectively to achieve 3D scanning functionality.

[0171] In use, when power is provided to the imaging assembly 201 via a connector connected to the frame 200, the processing assembly 280 automatically connects to the processing assembly 280 of any other imaging assembly 201 and transmits all required information (preferably wirelessly) so that the imaging assembly 201 operates in the same manner as other imaging assemblies 201 or auxiliary assemblies within the photogrammetric scanner system 20.

[0172] In addition to the standard microprocessor operating system installed on the processing component 280 for each imaging component 201, the photogrammetric scanner system 20 (the collection of individual imaging components 201) has an overarching operating system that has the ability to perform required functions such as imaging component identification and coordination, data transmission, processor and sensor configuration (network and physical), scanning operation sequencing, lighting sequencing, hardware and software fault monitoring and diagnostics, control of attached equipment, data transmission to external devices, power and temperature monitoring, and distributed data processing across the network of imaging components 201 (e.g., computer vision functions such as object recognition / segmentation).

[0173] External devices in the form of auxiliary components such as physical button triggers, displays, motion sensors, etc. may also be integrated into the scanner operating system.

[0174] The photogrammetric scanner system 20 can also be configured to run multiple programmed coordinated lighting and imaging sequences to achieve flexible 3D scanning capabilities. For example, specific imaging components can be set to scan at different times or in different sequences to capture motion data, or specific imaging components can be set to capture multiple images with different exposure times to combine into a high dynamic range image, or lighting devices can be arranged in sequence to produce structured patterns during scanning to enhance features, etc. Multiple sequences can also be encoded and selected to operate during scanning based on user input (i.e., button or automatically).

[0175] To enable the dynamic configuration capabilities of the photogrammetric scanner system 20, the imaging components 20 can be connected and operated following a predetermined order. When a given imaging component is powered on (by connecting it to the required power pins), the given imaging component initializes the operating system and connects to the scanner network (either through a wireless router or using a peer-to-peer network without a router). A network ID is assigned to each imaging component 201. The photogrammetric scanner system 20 can then operate in one of two modes: 1) a physical controller is dedicatedly assigned for control and data management functions, or 2) one of the imaging components 201 is automatically assigned as a controller (in addition to its usual functions) by the 3D scanner platform operating system so that there is only one controller.

[0176] At startup, each imaging component 201 queries the network for the presence of a controller and its ID. If no controller is present, any imaging component 201 will designate itself as a controller on a first-come, first-served basis. To avoid a situation where multiple imaging components 201 can designate themselves at the same time (due to network delays, etc.), each imaging component 201 checks the presence of other controllers again after a short delay, and then, after negotiation, cancels the redundant controller. The controller (using the designated network port) periodically notifies its presence to ensure that there is always a controller. If, after a period of time, the scanner component does not receive repeated notifications from the controller, the scanner component can assume that the controller is no longer part of the photogrammetric scanner system 20, and the available imaging components 201 begin to self-identify as controllers. Each imaging component 201 individually contains all the code required to act as a controller, so the selection of a controller is arbitrary and can dynamically adapt to the addition or removal of imaging components 201 from the photogrammetric scanner system 20 during operation. The controller's function is to interact with external devices, coordinate the various scanner components, run the required programmed 3D scanning sequence, aggregate 3D scan data (e.g., images or temperature maps), and interact with external networked devices (computers) for data transfer and further processing.

[0177] Each imaging component 201 (and any auxiliary components) includes hardware configuration information, the version of the entire 3D scanner operating system, sequencing information, hardware configuration setting information, 3D scan data, and a real-time map of the entire 3D scanner configuration, which is dynamically updated if scanner components are added or removed during operation. In addition, a physical representation of the 3D scanner can be stored in each imaging component 201 in the form of arbitrary 6D coordinates (position / orientation) determined by following a physical configuration process involving structure-from-motion software processing or determined by physical signaling. For example, if a buildin-block style scanner component configuration is used, each interconnect can contain physical information about its shape and size (whether electronically, physically, or using radio frequency identification (RFID) or similar technology). Each scanner component, when added to a socket 211 of the frame 200, can capture the desired physical information and then transmit that information to all other scanner components in the system (all other scanner components do the same), thereby building a virtual map / representation of the physical configuration of the entire 3D scanner.

[0178] Software and data types include: operating system, function code, sequence instruction set, data transmission protocol, and 3D scanning data (image, temperature, video stream, etc.).

[0179] The operation of each imaging component 201 is described in more detail below.

[0180] First, at startup, one imaging component 201 notifies the network of multiple imaging components 201 of its ID.

[0181] Next, a nomination and election process is performed to establish an imaging component 201 as a controller (master device).

[0182] Provides notification of scanner component configuration information (hardware specifications, capabilities, network ID, etc.) to the controller.

[0183] The controller is also provided with notification of the current software version. The controller then polls all imaging components 201 to identify the latest version of the software and extracts the latest version.

[0184] If necessary, the controller updates all software on all imaging components 201 to the latest version.

[0185] Each imaging component 201 operates according to software instructions; the sensor listens to the network (or other signal) to start processing the sequence code, which may include a single image capture, video stream capture, or a timed capture sequence (depending on the requirements). This may also include loading configuration data from a separate file before operation.

[0186] Some components can be provided that have functions other than imaging or data capture, such as emitting aiming light (laser), acting as a remote trigger with a physical or optical button, displaying information (LED, OLED, e-ink, etc.), direct measurement using Time of Flight (TOF) or ultrasound, emitting sound signals to the user, etc.

[0187] A controller then coordinates the collection of images, video, or data from all connected imaging components (and other components).

[0188] When each scan begins, the controller sends a trigger signal to all other imaging components 201, which begin their 3D scanning sequence. The controller records information about the triggered capture command for each imaging component 201 in a list, adding the continuous capture trigger command to this list. The controller then works through the list and copies the scan data from each corresponding imaging component to a central memory location on the controller. This operation is configured to occur in the background of the scanner operation and is intended to optimize the data transfer from the scanner to another device (without requiring an external device to poll each imaging component 201 for scan data). If a scan trigger is activated during this data transfer process, the process is interrupted to enable the transmission of the trigger signal, whereupon the process resumes and the additional scan data is added to the scan list, which will occur automatically until the available scan list is empty.

[0189] The scan sequence list contains the file name and corresponding imaging component ID for each of the scan data and other related data. This dynamic list is manipulated by the controller (items are added and deleted). For redundancy, items copied from imaging components to the controller data repository are not deleted from each imaging component until after the entire data repository is transferred offboard from the controller to an external computer.

[0190] If at any time a controller is removed from the photogrammetric scanner system 20 or fails in some way, the controller cannot continue to control, update, and accumulate scan data. In this case, another imaging assembly 201 is automatically designated as a replacement controller and takes over the functions of the previous controller. As described above, each imaging assembly 201 is configured to operate as an imaging assembly that is part of a network or as a controller (except for specific functions performed by auxiliary components, such as trigger buttons or display-only auxiliary components).

[0191] In some embodiments, multiple imaging components are designated as controllers. Alternatively, no imaging component may be designated as a controller, but coordination may be performed through an external device (eg, a computer).

[0192] To detect the absence of a controller during operation, some embodiments include a health check function in the network communication protocol, where each imaging component 201 periodically announces its status on one or more given ports. If the status signal is not received by the controller of the photogrammetric scanner system 20 or all other scanner components, a health status request can be broadcast to the relevant controllers. If there is no further communication from any one scanner component, it is removed from the list of active components and it will not be activated or data will not be requested from it. Since each scanner component contains its own internal representation (software, sequences, scan data, etc.) and a system-wide representation (IDs, functions, location information, etc. about all other active scanning components), the absence of a previously activated component is detected on each scanner component and removed from their system representation.

[0193] Information about the off-board transfer of scan data is also communicated by the controller to all other scanner components in the system (which also means that locally stored scan data can be removed from each scanner component once it has been transferred off the scanner). This means that if a controller is removed from the framework during operation, the other scanner components know the status of the internal scan data transfer (from each scanner component to the controller). In this case, the newly designated controller can rebuild (or copy) the list and network location (or scanner component ID) of the scan data and take over the data aggregation operation.

[0194] The photogrammetric scanner systems described herein may also include orientation sensors and / or accelerometers to provide information about the physical scanner coordinates relative to the real world. These sensors may be used to autonomously identify and communicate the correct positioning between the photogrammetric scanner system and the target. The orientation sensors and / or accelerometers may be used to determine and track the scanner position and orientation to aid in alignment of the target within the field of view of the image sensor.

[0195] The orientation sensor may take the form of a gyroscope and / or an accelerometer, and the acceleration sensor may take the form of an accelerometer.

[0196] In some embodiments, a display may be provided in communication with an orientation sensor and / or an acceleration sensor. The orientation sensor and acceleration sensor provide data indicating a current scanner orientation relative to an optimal orientation, which may be visually indicated by the display.

[0197] The display may also be used to display system-related information or to display captured images.

[0198] Other sensors that may be used include ultrasonic distance sensors, optical distance sensors, magnetometers, Global Positioning System (GPS) sensors, temperature sensors (e.g. Figure 7 Shown in the figure are a temperature sensor 183) and a humidity sensor.

[0199] A distance sensor (e.g., ultrasonic, laser, and LIDAR) may also be provided. The distance sensor may be used to measure the distance between the arm and the target, and in some embodiments, the distance sensor indicates to the user of the handheld scanner that the scanner is too close or too far from the target to be scanned. Alternatively, distance detection may be implemented using computer vision code that detects features of the target (e.g., facial features of a human face) and uses that information to determine the distance. The computer vision code may also be used to determine distance information to the target, which may be used to calculate in real time the optimal scanner geometry for an automatic variable geometry variant of the framework.

[0200] In some embodiments, the frame may include multiple arms (e.g. Figure 1 ), the plurality of arms having receptacles for receiving imaging components therein (such as Fig.15 and Fig.19 The socket may be movably connected to the arm or frame to provide dynamic reconfiguration and arrangement of the imaging components around the body of the frame to suit different purposes and requirements.

[0201] The physical configuration of the sensor in the illustrated embodiment is optimized to enable simultaneous imaging of objects the size of a human head (e.g., Figure 8 and Fig. 9 The spacing of the sensors is optimized so that they can be distributed approximately evenly around the object, converge on the center of the object, and be as Figure 1 The hemispherical (or semi-spherical) arrangement shown in FIG. 1 is a hemispherical (or semi-spherical) arrangement. However, this arrangement can be varied to enable imaging of a variety of objects by adding sensors optimized for the geometry of each object. For example, a configuration for scanning a person's foot would include sensors for the bottom of the foot and sensors around the sides of the foot and lower leg. Such an embodiment would be Figures 19 to 21 Shown in.

[0202] The photogrammetric scanner system 30 for scanning a foot includes a frame 300 and a plurality of imaging assemblies 201 connected to the frame 300 .

[0203] The frame 300 is substantially similar to the frame 200 , which includes a receptacle 311 configured to house the imaging assembly 201 .

[0204] The socket 311 also includes a connector in the form of a multi-pin magnetic connector 303 that provides connection to and distributes power to the necessary components, including the processing assembly 280, the image sensor 220, and the light emitting device 240.

[0205] The socket 311 is detachable, thereby enabling the frame 300 to be constructed modularly. Fig.19 and Fig. 20 Best seen in.

[0206] The socket 311 includes a connector 312 that engages with a corresponding connecting member to enable two or more sockets 311 to be connected together. The connecting members can have various configurations to allow the sockets 311 to be at appropriate angles relative to each other, or to create a specific and / or customized geometry for the scanner as needed. For example, a first connecting member 313 is shown that has an angled portion. In another example, a second connecting member 314 is shown that is straight.

[0207] The connector 312 connecting the member and the socket 311 may also include inter-mating pins to distribute power and / or data to the socket 311 to power the imaging assembly 201 .

[0208] Auxiliary components (eg, guide lights, laser pattern devices, and batteries) may also be connected to the frame 300 via connectors and corresponding connection members.

[0209] For example, the auxiliary component 315 is shown as having a connector 312 that is matingly engaged with a connecting member, which can be matingly engaged with the connector 312 of the socket 311.

[0210] The connector and the connecting member preferably cooperate to engage with each other in a transition fit to restrict movement of the socket when connected together.

[0211] In use, the photogrammetric scanner system 30 would be placed in a housing having a clear or transparent surface, so that a walking foot can be placed on the clear or transparent surface and the imaging assembly 201 can image the foot.

[0212] In some embodiments, the photogrammetric scanner may include additional imaging components for imaging the foot and leg (eg, calf).

[0213] Embodiments of the photogrammetric scanner system can scan the entire body.

[0214] The above-described embodiments having a changeable or reconfigurable frame enable the system to be reconfigured to be optimized for a target geometry.

[0215] In some embodiments, the image sensor may also capture video (at 30 fps), allowing 3D video scanning and building multiple 3D models based on a single video set.

[0216] In some embodiments, each image sensor captures 5 seconds of video (although the captured video can be of any length), which is stored as motion-JPEG or other formats. Video capture begins approximately 0.5 seconds before the illumination pattern projected by the light emitting device is activated, so that the illumination can be used for synchronization of the frames of the video. The inventors have found that (since each image sensor has a microcontroller that is independent of other microcontrollers connected to other image sensors) initialization between image sensors can vary by a few milliseconds, which has limited impact when capturing images of stationary (or nearly stationary) objects.

[0217] The captured videos of each image sensor can be exported to photogrammetry software which uses the lighting pattern to determine the start / first frame based on the time when light was first detected in each video. This enables the exclusion of frames captured before the determined start frame and the synchronization of the videos / frames from each image sensor.

[0218] In some embodiments, one or more arms of the frame may intersect (eg, to form an X configuration) with an image sensor positioned along the length of each arm.

[0219] Advantageously, embodiments of the present invention enable the acquisition of images required for a three-dimensional scan in less than approximately 1 ms. This may be important and / or useful in hospitals (particularly children's hospitals) where patients are often unable or unwilling to remain still, and therefore, minimizing the scan time is critical.

[0220] In some embodiments, a single button on the photogrammetric scanner system 10 activates an illumination and imaging sequence of all light devices and image sensors to quickly capture a 3D model of a target (eg, a patient) or to capture a predetermined sequence of video frames.

[0221] In some embodiments, the photogrammetric scanner system includes a beam sensor to automatically activate the illumination and imaging sequence immediately or after a predetermined amount of time.

[0222] In some embodiments, the image sensor 120 can be controlled by the controller 180 to capture multiple individual images of each sensor with different shutter speed / ISO / exposure settings to provide raw image data for High Dynamic Range (HDR) imaging, or as raw image data for a 3D reconstruction sensor. As described above, this is synchronized with the illumination device to enable two or more scans to be performed with the illumination and aperture set to different levels, for example, to expose dark skin and hair and light skin and hair at the same time.

[0223] In some embodiments, a microphone is provided to capture voice commands for triggering a scanning operation, or other commands for finding out status information (battery level, number of onboard scans, etc.) or changing scanning parameters.

[0224] In some further embodiments, a speaker is provided to inform the user of important information such as the number of scans in memory, battery level, etc. A short sound may also be played through the speaker to give an audible prompt when a scan is captured (e.g., similar to a camera shutter sound).

[0225] Power may be provided to the photogrammetric scanner system 10 by mains electricity or by batteries.

[0226] Embodiments provide a lightweight and strong frame designed to have the potential for disinfection and / or sterilization, with the internal processor and imaging sensor enclosed in a sealed compartment.

[0227] The ability of some embodiments to control illumination to enable capture of three-dimensional models illuminated in the ultraviolet (UV) and infrared (IR) regions of the light spectrum, near the visible region, provides a wider range of applications for photogrammetric scanner systems.

[0228] In some embodiments, it may be advantageous to enhance the LED illumination of a light emitting device using one or more lasers that can illuminate an object with coherent light.

[0229] By recording a video of the reflected light and performing post-capture data processing, changes in the skin and surface tissue can be measured for diagnosis. In such an embodiment, all sensors in the photogrammetric scanner system simultaneously and synchronously record a short video sequence of reflected light. The image corresponding to a given time point from each sensor is extracted from the video sequence, and the image is processed in the photogrammetric pipeline to produce a single 3D object. The 3D object for all desired frames in the video sequence can then be processed, and the surface map can be processed to perform the desired analysis of the reflected light over time. Light polarization can also be combined at the laser output and / or sensor input to change the detected light. An example of this is cross polarization, which is designed to minimize specular reflections and enhance target features. In this embodiment, a linear polarizer is attached to the light source in a given direction, and another linear polarizer is attached to the sensor lens or input in an orthogonal direction relative to the light source polarizer.

[0230] According to the procedure, the invention has been described in language more or less specific to structural or methodological features.The term "comprising" and variations thereof, such as "including" and "consisting of" are used throughout the text to broadly include expressions without excluding any additional features.

[0231] It is to be understood that the invention is not limited to the specific features shown or described, since the means herein described are preferred forms of putting the invention into effect.

[0232] Therefore, the present invention is claimed in any of its forms or modifications within the proper scope of the appended claims as appropriately interpreted by those skilled in the art.

Claims

1. A photogrammetric scanner system for capturing an image of a target, comprising: frame; a plurality of image sensors connected to the frame and spaced apart on the frame; as well as a plurality of light emitting devices connected to and spaced apart on the frame and configured to project light in an illumination pattern; Wherein the scanner system is configured to perform a coordinated illumination and imaging sequence of a target, wherein the plurality of light emitting devices project light in the illumination pattern and one or more images of the target are captured by the plurality of image sensors.

2. The photogrammetric scanner system according to claim 1, wherein: Each image sensor of the plurality of image sensors is configured to have a wide field of view and a short focal length.

3. The photogrammetric scanner system according to claim 1 or 2, wherein: The projection of light from the plurality of light emitting devices is synchronized with an image capture sequence of the plurality of image sensors.

4. The photogrammetric scanner system according to any one of claims 1 to 3, wherein: The image capture sequence includes activating a light emitting device adjacent to an image sensor and then capturing an image by the image sensor adjacent to the light emitting device.

5. The photogrammetric scanner system according to any one of claims 1 to 4, wherein: The arrangement of the plurality of image sensors and the plurality of light emitting devices includes each of the plurality of image sensors being adjacent to one of the plurality of light emitting devices.

6. The photogrammetric scanner system according to any one of claims 1 to 5, wherein: Any two image sensors among the plurality of image sensors are not coplanar.

7. The photogrammetric scanner system according to any one of claims 1 to 6, wherein: Each image sensor of the plurality of image sensors is offset from any other image sensor of the plurality of image sensors in three dimensions.

8. The photogrammetric scanner system according to any one of claims 1 to 7, wherein: Each image sensor of the plurality of image sensors has coordinates in a three-dimensional space that are unique in each of the three dimensions relative to coordinates of any other image sensor of the plurality of image sensors.

9. The photogrammetric scanner system according to any one of claims 1 to 8, wherein: One of the plurality of light emitting devices is independently controllable relative to another of the plurality of light emitting devices.

10. The photogrammetric scanner system according to any one of claims 1 to 9, wherein: Each of the plurality of light emitting devices includes a plurality of lights emitting light of different wavelengths.

11. The photogrammetric scanner system according to any one of claims 1 to 10, wherein: The plurality of light emitting devices are configured to emit light of a wavelength range including at least one of ultraviolet light, visible light, and near infrared light.

12. The photogrammetric scanner system according to any one of claims 1 to 11, wherein: The grating is located above the plurality of light emitting devices.

13. The photogrammetric scanner system according to any one of claims 1 to 12, wherein: The photogrammetric scanner system also includes an orientation sensor and / or an acceleration sensor configured to determine and track scanner position and orientation to assist in aligning the target within the field of view of the image sensor.

14. The photogrammetric scanner system according to any one of claims 1 to 13, wherein: The frame is modular and includes a plurality of releasably connectable sockets, each socket being configured to receive an image sensor and a light emitting device therein.

15. The photogrammetric scanner system according to any one of claims 1 to 14, wherein: The frame includes one or more surfaces to which the plurality of image sensors and the plurality of light emitting devices are attached.

16. A method of photogrammetric scanning, the method comprising: providing a photogrammetric scanner system having a plurality of image sensors and a plurality of light emitting devices connected to and spaced apart on a frame; The photogrammetric scanner system is activated to perform a coordinated illumination and imaging sequence of a target, wherein the plurality of light emitting devices project light in an illumination pattern and one or more images of the target are captured by the plurality of image sensors.

17. An imaging assembly, comprising: A body, to which an image sensor, a light emitting device and a processing component are connected; as well as The processing component is configured to coordinate illumination and imaging sequences of a target, wherein the imaging component is designated as a controller imaging component in a network of multiple imaging components, each imaging component digitally communicating with the processing component, and the imaging component is thus configured to coordinate illumination and imaging sequences of the target for one or more imaging components in the network of multiple imaging components.

18. The imaging assembly of claim 17, wherein: The imaging assembly comprises a housing, wherein the body forms a part of the housing.

19. An imaging assembly according to claim 17 or 18, wherein: The processing component is configured to form a network of the imaging components by detecting and communicating with one or more additional imaging components.

20. The imaging assembly according to any one of claims 17 to 19, wherein: An additional imaging component of the one or more additional imaging components is detected by the controller imaging component when the additional imaging component is electrically activated.

Images