Three-dimensional object scanning apparatus and method

By combining projectors and 3D object scanners with LED screens, the problem of environmental lighting impact is solved, and high-quality 3D information and textures are captured under any conditions, which is suitable for a variety of applications.

CN119032380APending Publication Date: 2024-11-26TENCENT AMERICA LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380033704.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-10-27
Filing Date
2023-03-20
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

Existing 3D object scanners are affected by ambient lighting when capturing objects, especially the changes in the color and position of the light source, resulting in the restored surface texture being limited by specific lighting conditions, and it is difficult to deal with objects with patternless surface colors and smooth surface shapes. .

Method used

Using a 3D object scanner combining projector equipment and LED screen, the projector equipment projectes light patterns and LED screen emitted light patterns, uses the imaging device to capture images from multiple perspectives, and controls ambient lighting to be independent of the inherent surface texture of the object. Achieve high-quality digitization of objects of any shape.

Benefits of technology

It realizes high-quality 3D information and texture information of objects captured under any lighting conditions, and can observe and reconstruct virtual object models from any viewpoint. It is suitable for a variety of applications such as virtual reality, augmented reality and 3D printing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119032380A_ABST
    Figure CN119032380A_ABST
Patent Text Reader

Abstract

A three-dimensional (3D) object scanner includes: an object platform configured to host a target object; a projector device configured to project a first light pattern onto the target object; an LED screen surrounding the object platform and configured to emit a second light pattern; and the imaging equipment is configured to obtain a plurality of images of the target object from a plurality of visual angles when the target object and the imaging equipment relatively rotate to a plurality of rotation degrees. The plurality of images includes an image obtained when the projector device projects the first light pattern and an image obtained when the LED screen emits the second light pattern. The plurality of images are used for determining 3D information and texture information of the target object.
Need to check novelty before this filing date? Find Prior Art

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. patent application No. 17 / 975,279, filed on October 27, 2022, the entire contents of which are incorporated herein by reference in their entirety. Technical Field

[0002] The present application relates to the technical field of object scanning, and in particular to a three-dimensional object scanning device and method. Background Art

[0003] 3D object scanners generally aim to recover two aspects of the scanned object, namely, the geometric shape and the surface texture map. However, the surface texture is often affected by the ambient lighting. The color and position of the light source affect the appearance of the object and thus the recovered surface texture. Therefore, the reconstructed surface texture is limited by the specific lighting conditions rather than the inherent surface texture of the object. In addition, 3D object scanners in the related art generally rely on objects with rich surface color patterns and geometric features, but have no control or knowledge of the lighting when capturing the image, and fail if the object presents a patternless surface color and / or a smooth surface shape.

[0004] The disclosed methods and systems are directed to solving one or more of the above-referenced problems as well as other problems. Summary of the invention

[0005] One aspect of the present disclosure provides a three-dimensional (3D) object scanner, comprising: an object platform configured to host a target object; a projector device configured to project a first light pattern onto the target object; an LED screen surrounding the object platform and configured to emit a second light pattern; an imaging device configured to obtain multiple images of the target object from multiple viewing angles when the target object and the imaging device are relatively rotated to multiple rotation degrees. The multiple images include images obtained when the projector device projects the first light pattern and images obtained when the LED screen emits the second light pattern. The multiple images are used to determine 3D information and texture information of the target object.

[0006] Another aspect of the present disclosure provides a 3D object scanning method applied to a 3D object scanner. The method includes: controlling a projector device of the 3D object scanner to project a first light pattern onto a target object, the target object being hosted by an object platform; controlling an LED screen of the 3D object scanner to emit a second light pattern, the LED screen surrounding the object platform; and controlling an imaging device of the 3D object scanner to obtain multiple images of the target object from multiple perspectives when the target object and the imaging device are relatively rotated to multiple rotation degrees. The multiple images include an image obtained when the projector device projects the first light pattern and an image obtained when the LED screen emits the second light pattern. The multiple images are used to determine 3D information and texture information of the target object.

[0007] Another aspect of the present disclosure provides a non-volatile storage medium for storing computer instructions. When the computer instructions are executed by a processor, the processor is caused to execute: controlling a projector device of a three-dimensional 3D object scanner to project a first light pattern onto a target object, the target object being hosted by an object platform; controlling an LED screen of the 3D object scanner to emit a second light pattern, the LED screen surrounding the object platform; and controlling an imaging device of the 3D object scanner to obtain multiple images of the target object from multiple perspectives when the target object and the imaging device are relatively rotated to multiple degrees of rotation. The multiple images include images obtained when the projector device projects the first light pattern and images obtained when the LED screen emits the second light pattern. The multiple images are used to determine the 3D information and texture information of the target object.

[0008] Other aspects of the present disclosure may be understood by those skilled in the art based on the specification, claims and drawings of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic structural diagram of a three-dimensional object scanning device according to an embodiment of the present disclosure is shown;

[0010] Figure 2A and Figure 2B An example of a 3D object scanning device according to some embodiments of the present disclosure is shown;

[0011] Figure 3 An example object platform with a projector device installed according to some embodiments of the present disclosure is shown;

[0012] Figure 4 shows an example light pattern projected by a projector device according to some embodiments of the present disclosure;

[0013] Figure 5shows an example light pattern emitted by an LED screen according to some embodiments of the present disclosure;

[0014] Figure 6 A flowchart showing a calibration process of a 3D object scanning device according to some embodiments of the present disclosure is shown;

[0015] Figure 7 A flowchart of a 3D object scanning process according to some embodiments of the present disclosure is shown;

[0016] Figure 8 A flowchart showing a 3D object reconstruction process according to some embodiments of the present disclosure; and

[0017] Fig. 9 A computer system according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present invention are described below in conjunction with the accompanying drawings. As much as possible, the same reference numerals will be used in all the drawings to refer to the same or similar parts. Obviously, the described embodiments are only some but not all embodiments of the present invention. Other embodiments obtained based on the embodiments of the present invention by those skilled in the art without paying creative work also belong to the protection scope of the present disclosure.

[0019] The present disclosure provides a three-dimensional (3D) object scanning device (also referred to as a 3D object scanner) that can digitize physical objects. Compared with the 3D object scanner in the related art, the disclosed 3D object scanner includes a projector device and an LED screen, which can facilitate the simultaneous restoration of the 3D geometry and inherent surface material of an object of any shape. A virtual object model is generated based on the scanning results of the disclosed 3D object scanner, which can be observed from any viewpoint and under any lighting conditions. The disclosed 3D object scanner is also capable of directly capturing objects under any lighting conditions. The digitized object can be regarded as a high-quality digital twin of a physical object, which has countless applications from virtual museum exhibitions, cultural relics protection, digital content creation to virtual reality (VR), augmented reality (AR), metaverse, 3D printing, etc.

[0020] Figure 1 FIG. 2 shows a schematic diagram of the structure of a three-dimensional object scanning device according to an embodiment of the present disclosure. Figure 1 As shown, the example 3D object scanning device 100 includes an object platform 102 , a projector device 104 , an LED screen 106 , and an imaging device 108 .

[0021] The object platform 102 is configured to host a target object to be scanned. The imaging device 108 is configured to obtain multiple images of the target object from multiple viewing angles when the target object and the imaging device 108 are relatively rotated to multiple degrees of rotation. In one embodiment, the imaging device 108 is rotatable and the object platform 102 is disposed at a fixed position. In another embodiment, the object platform 102 is rotatable and the imaging device 108 is disposed at a fixed position. The images are used to determine 3D information and texture information of the target object.

[0022] The projector device 104 is configured to project a first light pattern onto a target object. The LED screen 106 surrounds the object platform 102 and is configured to emit a second light pattern. The distance between the LED screen 106 and the object platform 102 is greater than the distance between the projector device 104 and the object platform 102. The multiple images captured by the imaging device 108 include: an image obtained when the projector device 104 projects the first light pattern, and an image obtained when the LED screen 106 emits the second light pattern. The projector device 104 can produce a carefully designed shadow pattern on the target object to improve the quality of 3D reconstruction and camera alignment. By controlling the pixel intensity on the LED screen 106, the 3D object scanner 100 can set the ambient lighting of the scanned object. After the object is digitized, a digital model of the object can be rendered under any lighting conditions. In addition, the LED screen 106 can help restore the surface material of the scanned object.

[0023] Figure 2A and Figure 2B 1 shows an example of a 3D object scanning device according to some embodiments of the present disclosure. The target object can be placed at the center of the rotating stage of the object platform 102. The shape of the LED screen 106 can be configured to surround the target object on the object platform 102, providing a light pattern / light source that illuminates the central area where the target object is located from multiple directions. The shape can also surround the projector of the projector device 104. The shape can be, for example, a sphere (such as Figure 2A As shown), cubic shape (as Figure 2B ), geodesic dome shape, hemispherical shape, etc. The disclosed 3D object scanner 100 fully controls the intensity of each pixel on the LED screen 106, thereby facilitating the creation of high-resolution, high-dynamic range ambient lighting maps.

[0024] In some embodiments, the LED screen 106 may include a plurality of LED panels. Hereinafter, in some embodiments, the LED screen 106 may also be referred to as an LED panel. In some embodiments, the LED panels may have substantially the same shape and size, for example Figure 2A The triangular panel shown and Figure 2BWhen LED screen 106 has a 2v geodesic dome, it can include two sets of triangular panels of different sizes. Using LED panels with the same configuration can reduce manufacturing costs. In some other embodiments, LED screen 106 can include one or more custom-shaped LED panels.

[0025] In some embodiments, multiple LED panels of LED screen 106 may be interconnected. The angle formed by two interconnected LED panels may vary depending on the desired shape of LED screen 106. In one example, when LED screen 106 has a cubic shape, the angle formed by two interconnected LED panels belonging to the same face of LED screen 106 may be 180°, and the angle formed by interconnected LED panels belonging to two faces of LED screen 106, respectively, may be 90°. In another example, when LED screen 106 has a dome shape, the angle formed by two interconnected LED panels may be greater than 90° and less than 180°.

[0026] The imaging device 108 is configured to obtain images of the object from the top to the side of the object. In some embodiments, the imaging device 108 may include multiple cameras mounted on a frame. Hereinafter, an image set may refer to images of the object obtained by the imaging device 108 from different perspectives (e.g., from multiple cameras) under the same settings (e.g., the same rotation of the object relative to the rotation axis, the same ambient lighting provided by the LED panel, in the same projector mode, etc.). The frame may include an arc-shaped or approximately arc-shaped portion for mounting the cameras. In some embodiments, the cameras may be evenly distributed. For example, Figure 2B An imaging device 108 having eight cameras is shown, and the angle between every two adjacent cameras is about 10 to 15 degrees. The total angle of the eight cameras is about 90 to 105 degrees. Figure 2A An imaging device 108 having six cameras is shown. The number of cameras may be an integer greater than one.

[0027] As used herein, the camera of the imaging device 108 may refer to any suitable imaging sensor capable of capturing imaging information of a target object, such as a passive imaging sensor (e.g., a single-lens camera, a stereo camera, a depth sensor) or an active imaging device 108 (e.g., a time-of-flight camera). When a time-of-flight camera is used, the 3D object scanner 100 may also include a modulated light source. In some embodiments, the LED screen 106 may be used as a modulated light source. In some embodiments, the modulated light source and the time-of-flight camera may be included in the same device.

[0028] In some embodiments, the frame may include a movable structure configured to mount one or more cameras and drive the one or more cameras to move along a path and stop at multiple positions on the path to obtain images of the target object from multiple perspectives, each stop position corresponding to one of the multiple perspectives. The distances of the multiple stop positions on the path to the object platform 102 may be substantially the same (e.g., the same radial distance to the center of the object platform 102). The path may be an arc. The arc may span an angle of about 90 to 105 degrees (e.g., a polar angle using the center of the object platform 102 as the origin). The movable structure may drive one or more cameras to stop at different positions to have different viewpoints relative to the target object, and when stopped at different angles / viewpoints, one or more cameras may capture images of the target object. The movable structure may include an arc track with a movable seat (the movable seat travels along the arc track), a mechanical arm, or other suitable structure to accommodate and move the one or more cameras to achieve different viewpoints relative to the target object.

[0029] In some embodiments, the imaging device 108 may be located within a housing provided by the LED screen 106, such as Figure 2B In some embodiments, the imaging device 108 may not be located within the housing provided by the LED screen 106. Figure 2A When imaging device 108 is not located within the housing provided by LED screen 106, LED screen 106 does not block the optical path of imaging device 108. In other words, the optical path between imaging device 108 and object platform 102 may pass through a portion of the housing provided by LED screen 106. For example, the housing shape of LED screen 106 may include a portion through which the optical path of imaging device 108 may pass. The portion may be hollow, transparent, or may include any suitable optical device that does not block the optical path.

[0030] In some embodiments, the object platform 102 (e.g., a plate for supporting an object to be scanned) is located at the center of the cross-section of the LED screen 106. In some embodiments, the distance between the center of the object placement surface of the object platform 102 (hereinafter referred to as the object center) and the LED screen 106 may be 1.5-3 times the distance between the object center and the projector of the projector device 104. For example, the distance between the object center and the LED screen 106 is about 1 to 1.25 m; the distance between the object center and the camera of the imaging device 108 is about 0.8 m; and the distance between the object center and the projector of the projector device 104 is about 0.5 m.

[0031] In some embodiments, using the object center as the origin of the spherical coordinate system, the shell shape of LED screen 106 can cover the object center with an azimuth range from 0 to 2π and a polar angle range from 0 to π. In some embodiments, the shell shape of LED screen 106 can cover the object center with an azimuth range from 0 to 2π and a polar angle range from -π / 3 to 4 / 3π.

[0032] Figure 3 An example object platform 102 with a projector device 104 mounted thereon is shown according to some embodiments of the present disclosure. Figure 3 As shown, the object platform 102 may include a turntable 1022. The turntable 1022 may be configured to rotate 360 ​​degrees along a vertical axis 1028 at its center. The rotation of the turntable 1022 may be driven manually or automatically (e.g., using a motor). The turntable 1022 may be fixedly mounted on a fixed base (not shown). The base may be placed on the ground or other stable structure. In operation, the target object 200 may be placed on the top surface of the turntable 1022. In some embodiments, at least a portion of the object platform 102 (e.g., the turntable 1022) may be made of a transparent material such as transparent acrylic, so that a camera at a lower viewing angle may capture some features of the bottom of the object to be scanned. In some embodiments, the turntable 1022 may further include additional brackets to support objects without a flat surface. In some embodiments, the rotation speed of the turntable 1022 may be adjusted, for example, it may be slowed down to reduce jitter and unstable movement.

[0033] The projector device 104 may include projectors 104-1 and 104-2 mounted on the outer edge of the turntable 1022, such as on an extension arm of the turntable 1022, or on a peripheral area of ​​a plate of the turntable 1022. The projectors 104-1 and 104-2 face the vertical axis 1028 so that the light pattern can be projected onto the target object 200. In some embodiments, the projectors 104-1 and 104-2 can be fixed relative to the object. In some embodiments, the horizontal line of the projectors 104-1 and 104-2 is equal to or lower than the horizontal line of the top surface of the turntable 1022. It is understood that although Figure 3 Two projectors are shown, but the projector device 104 may include k projectors, k being an integer greater than 1. In some embodiments, the k projectors may be located on the same horizontal plane and have the same distance from the vertical axis 1028. In some embodiments, the k projectors may be evenly distributed on the circumference of the object platform 102. The angular spacing between adjacent projectors may be the same. For example, when k is 3, the radians of the projectors are 0, 2 / 3π, and 4 / 3π.

[0034] The projectors 104-1 and 104-2 are driven to rotate with the target object 200. The projector device 104 may further include a cable 1042. The cable 1042 may include: a power cable configured to transmit power from a power source to the projector device 104, and a data cable configured to transmit projection data (e.g., light pattern, projector status) between a projector control device (e.g., controller 200) and the projector device 104. The object platform 102 may also include a slip ring 1026, which is configured to transmit power and electrical signals from a fixed entity to a rotating structure, thereby avoiding twisting of the cable when the turntable 1022 rotates. The cable 1042 may pass through a through hole 1024 in the lower portion of the turntable 1022 and be connected to the slip ring 1026. For example, the power cable may be connected to the power source through the slip ring 1026, and the data cable may be connected to the projector control device through the slip ring 1026.

[0035] It will be appreciated that, although the illustrated embodiment shows that the object platform 102 is rotatable and the imaging device 108 is fixed, in other embodiments, the object platform 102 may be fixed and the imaging device 108 may be rotatable. For example, one or more cameras of the imaging device 108 may be mounted on a rotatable portion of the frame. The rotatable portion is configured to drive the camera to rotate a plurality of degrees of rotation around a vertical axis passing through the center of the object platform 102. Slip rings may also be used to prevent twisting of the power / data cables of the camera.

[0036] Return to Figure 1 , the 3D object scanner 100 can be controlled by a controller 200. The controller 200 may include one or more control devices configured to control components of the 3D object scanner 100. These components are one or more of the object platform 102, the projector device 104, the LED screen 106, or the imaging device 108. One control device may correspond to one component or multiple components. The control device may be a separate device connected to its corresponding component through a wired or wireless connection, or a device embedded in its corresponding component, or a combination thereof.

[0037] In some embodiments, the control device corresponding to the object platform 102 may be configured to control the rotation of the turntable 1022, such as starting the rotation, stopping the rotation, and controlling the rotation degree of the turntable 1022. For example, the turntable 1022 may be rotated 10-15 degrees each time and stopped for a certain duration, so that the projector device 104 and / or the LED screen 106 can present one or more light patterns, and the imaging device 108 can obtain an image while presenting the one or more light patterns. In some embodiments, when stopping at each rotation degree, the control device corresponding to the object platform 102 may transmit a signal to the projector device 104 and / or the LED screen 106 to start presenting the light pattern. In some embodiments, when the imaging device 108 has completed obtaining an image at one rotation degree, the control device corresponding to the object platform 102 may receive a signal to rotate the turntable 1022 again.

[0038] In some embodiments, the control device corresponding to the projector device 104 may be configured to control the projection process of the projector device 104, such as controlling a certain projector of the projector device 104 to project a specific light pattern for a specific duration, and controlling one or more projectors to stop projecting. Hereinafter, the light pattern projected by the projector device 104 may be referred to as a first light pattern, and the image obtained when the first light pattern is projected may be referred to as a first image. The first light pattern may include a set of various light patterns projected on the surface of the object for camera registration, shape reconstruction, and / or surface material analysis. Figure 4 2 shows an example light pattern projected by the projector device 104 according to some embodiments of the present disclosure. Figure 4 As shown, the first light pattern may include a gradient pattern, an intensity and color pattern, a Gray code pattern, and / or a noise pattern. The color pattern may include a monochrome pattern. In some embodiments, multiple projectors may project the same pattern (e.g., a noise pattern) simultaneously. In some embodiments, multiple projectors may project the same pattern (e.g., a color pattern) in sequence. In some embodiments, Gray code patterns and noise patterns may be used for calibration; gradient patterns, different intensity and color patterns may be used for material estimation. In some embodiments, a control device corresponding to the projector device 104 may transmit a signal to the imaging device 108 to start acquiring an image when projecting the first light pattern. In some embodiments, when the projection of the light pattern is completed, a control device corresponding to the projector device 104 may transmit a signal to the LED screen 106 to start emitting the LED light pattern.

[0039] In some embodiments, the control device corresponding to the LED screen 106 may be configured to control the display process of the LED screen 106, such as controlling one or more LED panels of the LED screen 106 to emit a specific light pattern for a specific duration, and controlling one or more LED panels to stop displaying. Hereinafter, the light pattern emitted by the LED screen 106 may be referred to as a second light pattern, and the image obtained when the second light pattern is emitted may be referred to as a second image. Figure 5 2 shows an example light pattern emitted by LED screen 106 according to some embodiments of the present disclosure. Figure 5 As shown, the second light pattern may include a gradient pattern, an intensity and color pattern, a Gray code pattern, a noise pattern, and / or a panoramic pattern. In some embodiments, the Gray code pattern may be used for calibration; the gradient pattern and the noise pattern may be used for specular and diffuse separation; different intensity and color patterns may be used for texture map generation; and / or natural images and panoramic images may be used for relighting. In some embodiments, a control device corresponding to the LED screen 106 may transmit a signal to the imaging device 108 to start acquiring an image when projecting the second light pattern.

[0040] In some embodiments, the control device corresponding to the imaging device 108 may be configured to control the imaging process of the imaging device 108, such as capturing and storing images of the target object when presenting the first light pattern and / or the second light pattern. In some embodiments, the control device corresponding to the imaging device 108 may start image capture upon receiving a start signal (e.g., when the object platform 102 rotates to a certain extent, when the first / second light pattern is displayed, when a specific time is reached). In some embodiments, the control device corresponding to the imaging device 108 may automatically perform image capture at set intervals. For example, when there are 20 seconds of light patterns to be emitted by the LED screen 106 at a speed of 3 seconds per pattern, the imaging device 108 may be controlled to capture a set of images every 3 seconds. In the case where the imaging device 108 is movable, the control device corresponding to the imaging device 108 may be further configured to move the camera to a specific position and for a specific duration, such as moving the camera to capture images from different perspectives, or rotating the camera when the object platform 102 is not rotatable.

[0041] In some embodiments, the controller 200 may be configured to control the 3D object scanning device 100 to implement a calibration process and / or an object scanning process. The calibration process and / or the object scanning process may be fully automatic or semi-automatic. During the calibration or scanning process, the object platform 102 or the imaging device 108 may be configured to rotate to a plurality of rotation degrees, and stop at each rotation degree and continue for a specific time interval, the projector device 104 and / or the LED screen 106 may be configured to present one or more specified light patterns at a specified time stamp, and the imaging device 108 may be configured to obtain an image at one or more specified time stamps.

[0042] In some embodiments, the controller 200 may store an image obtained by the imaging device 108 and attribute information corresponding to the image. The attribute information may be used for calibration or reconstruction. The attribute information of the image may include at least one of the following: a camera identifier indicating a camera that captured the image, a camera angle of view indicating the angle of view of the camera when capturing the image, a timestamp indicating the time when the image was captured, a projector identifier indicating a projector that projected a first light pattern when capturing the image, a projector mode identifier indicating a projector that projected a first light pattern when capturing the image, an LED pattern identifier indicating a second light pattern emitted by the LED screen 106 when capturing the image, or a rotation indicating which part of the object the image is capturing, etc.

[0043] The present disclosure also provides a calibration process, which is applicable to the disclosed 3D object scanning device. Figure 6 A flow chart of a calibration process of a 3D object scanning device according to some embodiments of the present disclosure is shown. The calibration process may be implemented by the 3D object scanner 100, the controller device 200, or a combination thereof.

[0044] like Figure 6 As shown, the calibration process 600 may include a camera calibration S602 for the imaging device 108 , a projector calibration S604 for the projector device 104 , and / or an LED screen 106 calibration S606 for the LED screen 106 .

[0045] Camera calibration S602 may be performed to obtain intrinsic and extrinsic parameters of one or more cameras in the imaging device 108. The parameters of the camera may include: focal length, aperture, field of view, resolution, position of the camera, orientation of the camera, etc. The position and orientation of the camera may be recorded relative to a world / global coordinate system established for the 3D object scanner 100. These parameters may be used later in the object reconstruction process.

[0046] Camera calibration S602 may include capturing one or more images of one or more calibration plates placed at different locations in the field of view of one or more cameras of the imaging device 108, and calculating parameters of one or more corresponding cameras based on the captured one or more images. In some embodiments, the calibration plate may be a check plate.

[0047] Projector calibration S604 may be performed to obtain parameters of each projector in the projector device 104. The parameters of the projector may include position, orientation, intensity, etc. Projector calibration S604 may include: when the projector to be calibrated projects a light pattern onto a projector calibration object, obtaining a calibration image of the projector calibration object placed on the objective lens platform, and obtaining parameters of the projector based on the calibration image. In some embodiments, for the same projected calibration light pattern, multiple calibration images of the projector calibration object are obtained from multiple viewpoints, thereby improving the calibration accuracy. In some embodiments, multiple projectors may project different color patterns at the same time, and multiple projectors may be calibrated simultaneously using a color / wavelength multiplexing method. For example, when there are three projectors with no overlapping projection areas, every two projectors may be calibrated for their relative positions, respectively, and then the three projectors are combined into a global coordinate. In some embodiments, the projector calibration object is a diffuse white sphere. In some embodiments, the position of the rotation axis of the object platform 102 or the imaging device 108 in the world coordinate system may also be calculated based on the calibration image.

[0048] The LED screen 106 calibration S606 may be performed to obtain the parameters of the pixels of the LED screen 106, and the LED screen 106 calibration S606 may include coordinate alignment and color calibration. The parameters of the pixels may include position, pixel intensity, dynamic range, etc. The LED screen 106 calibration S606 may include: capturing an image of the LED calibration object when the LED screen 106 emits a calibration light pattern, and obtaining the parameters of the pixels of the LED screen 106 based on the captured image. In some embodiments, the calibration light pattern may include a series of light patterns. By recording the image captured by the camera, the direction and irradiance of the light from each pixel on the LED screen 106 can be estimated. Using the incident light, the 3D position of each pixel on the LED panel is calculated, and then a plane is fitted for each LED panel to obtain its normal direction. Considering the angle between the light and the normal direction of the LED panel, the irradiance change of the pixel relative to the light emission direction can be estimated. In some embodiments, the LED calibration object is a chrome ball. In some embodiments, after calibration, the intensity of the pixel can be used to compensate for the irradiance change of the pixel due to the influence of the direction. In some embodiments, the position of the rotation axis of the object platform 102 or the imaging device 108 in the world coordinate system may also be calculated based on the captured images.

[0049] It is understood that the projector calibration S604 and the LED screen 106 calibration S606 can be performed independently without requiring the same Figure 6 The order shown occurs.

[0050] Figure 7 1 is a flowchart of a 3D object scanning process according to some embodiments of the present disclosure. The scanning process may be implemented by the 3D object scanner 100, the controller device 200, or a combination thereof.

[0051] like Figure 7 As shown, the scanning process S700 may include data acquisition (S702) and object rotation (S706). Data acquisition (S702) may include obtaining image data of the target object when the LED screen 106 and / or the projector device 104 present a specific light pattern. Object rotation (S706) may include rotating the target object relative to the imaging device 108 (e.g., by rotating the object platform 102 or the imaging device 108), and stopping the rotation at multiple rotation degrees. The rotation degree of the target object or the rotation angle of the target object used herein may refer to the rotation degree of the target object along the axis perpendicular to the plane where the target object is placed. The range of the rotation degree may be from 0 to 360 degrees. In some embodiments, the rotation degree interval may be 10 to 15 degrees. That is, the object platform 102 or the imaging device 108 stops once every 10 to 15 degrees of rotation. In some embodiments, the rotation degree interval may be determined based on the specular reflectivity of the target object. A higher specular reflectivity may correspond to a smaller rotation degree interval. In some embodiments, when the target object is placed but has not yet been rotated, the specular reflectivity may be estimated at the beginning of the scanning process. For example, the LED screen 106 may display a series of lighting patterns, and the imaging device 108 may capture corresponding images. The specular reflectance may be estimated based on the intensity variation of the captured images on the object surface, thereby determining the rotation interval of the subsequent one or more object rotation (S706) steps.

[0052] Data collection (S702) may be performed for each of a plurality of object rotation degrees. In some embodiments, object rotation (S706) may be performed after completing data collection (S702) for the current rotation degree.

[0053] In some embodiments, the data acquisition (S702) process may include image acquisition (S7022), LED light pattern change (S7024), and turning on / off the projector device 104 (S7026). In one embodiment, after the target object is placed on the object platform 102 at the current rotation degree, when the projector device 104 is turned on and projects the first light pattern, the imaging device 108 may obtain (S7022) a first image of the target object. In some embodiments, k projectors may be turned on and off in sequence to project the same pattern, and images may be captured each time the projector is turned on to obtain k sets of first images. The projector device 104 may be turned off (S7026). The LED screen 106 may emit (S7024) a second light pattern, and when the LED screen 106 emits the second light pattern, the imaging device 108 may obtain (S7022) a second image. In some embodiments, the LED screen 106 may emit multiple light patterns. When the LED screen 106 changes (S7026) the emitted light pattern, the imaging device 108 may obtain a set of images of the target object corresponding to the current light pattern. In some examples, when the projector device 104 is turned on, the LED screen 106 is turned off, and vice versa.

[0054] In another embodiment, after the target object is placed on the object platform 102 at the current rotation degree, the LED screen 106 can display (S7026) the current LED light pattern, and the imaging device 108 can obtain (S7022) two sets of images of the target object, one set is taken when the projector device 104 is turned on (S7026) and projects the specified projection pattern, and the other set is taken when the projector device 104 is turned off (S7026). When the target object is at the current rotation degree, the LED screen 106 can change (S7026) to display the next LED light pattern, and the imaging device 108 can obtain (S7022) the next two sets of images of the target object, one set is taken when the projector device 104 is turned on (S7026) and projects the specified projection pattern, and the other set is taken when the projector device 104 is turned off (S7026).

[0055] In some embodiments, at S7026, the first light pattern projected by the projector device 104 may include one or more color patterns. The projected color pattern may produce rich features on the corresponding captured image. In some embodiments, in the data acquisition step, the projector has a fixed position relative to the object. Therefore, the pattern projected by the projector is always fixed at an appropriate position on the object, thereby enriching the surface features of the object. The rich surface features greatly improve the accuracy of object pose registration and reconstruction.

[0056] In some embodiments, the scanning process S700 may further include coordinate registration (S704), which may be performed for each of the plurality of object rotations, respectively. In some embodiments, the coordinate registration (S704) may be performed in parallel with the scanning process. In some embodiments, the object rotation (S706) process does not need to wait for the completion of the coordinate registration (S704) of the current rotation to the next rotation. In some embodiments, the coordinate registration (S704) for the plurality of object rotations may be performed after the scanning process.

[0057] In some embodiments, coordinate registration (S704) may include feature extraction and matching (S7042) and estimation of rotation angle and axis (S7044). Feature extraction is performed on each image in the image set. Based on the results of feature extraction, matching is performed on images in an image set captured from different viewing angles with the same light pattern to find pixels corresponding to the same point on the object in the image set. In some embodiments, coordinate registration (S704) may be performed on an image set obtained when the projector device 104 is turned on and the LED screen 106 is turned off. By matching features in images taken across multiple steps (i.e., multiple degrees of rotation), the rotation and position of the camera relative to the target object can be estimated using the camera position relative to the previous step. Using this information, the background LED ambient lighting map (i.e., the second light pattern emitted by the LED screen 106) can be warped to align with the current posture of the target object and obtain constant ambient lighting throughout the 360-degree capture process. For example, when the estimation result obtained using the coordinate registration (S704) of a set of first images indicates that the object has been rotated 15 degrees relative to the axis since the previous step, the LED screen 106 may also rotate the second light pattern used in the previous step by 15 degrees to align with the rotation of the object so that the ambient lighting for the object provided by the LED screen 106 remains unchanged.

[0058] In some embodiments, the object scanning process S700 can be fully automatic. A series of lighting patterns can be displayed on the LED panel, and images can be captured. Determined by the change in the intensity of the image captured on the surface of the object, the rotation angle interval for subsequent capture can be set. Each time after the target object is rotated to a different angle, the projector projects a light pattern on the surface of the object. The rotation axis and angle can be estimated accordingly, and then used to record the LED panel and determine the light pattern displayed after each rotation step. For example, after each rotation step, the light pattern displayed by the LED panel can also be rotated based on the rotation axis and angle (e.g., estimated from the image taken when the projector device 104 is turned on), so that after each rotation step, the object is illuminated by a fixed ambient lighting (e.g., ambient lighting provided by the rotating light pattern emitted by the LED panel). Once this is done, multiple light patterns can be displayed on the LED panel, and the per-pixel light reflectance of the object can be calculated based on the captured image.

[0059] In some embodiments, the scanning process (S700) may be performed multiple times on the same object with different placement positions. For example, the scanning process (S700) may be performed when the object is placed face up, and the scanning process may be performed again when the object is placed face down, so that the 3D information and texture information of the entire surface of the object can be reconstructed.

[0060] In some embodiments, the image of the object under different lighting conditions can be obtained by changing the second light pattern displayed by the LED screen 106. The second light pattern can be as follows: Figure 5 An example of a panoramic pattern is shown. In the prior art, it is difficult to capture an object from different angles under a specific ambient lighting because image capture requires a certain amount of time during which the environment may change. The disclosed 3D object scanner can provide fixed ambient lighting, for example, by obtaining a 360-degree panoramic image and displaying the panoramic image on an LED panel for any desired duration. In addition, the disclosed 3D object scanner can display a rendered virtual environment that is difficult to establish in the real world (e.g., Mars, underwater, or even a fairyland), and obtain an image of a target object corresponding to any desired environment.

[0061] Figure 8 A flowchart of a 3D object reconstruction process according to some embodiments of the present disclosure is shown. The reconstruction process may be implemented by the 3D object scanner 100, the controller device 200, any suitable computing device or a combination thereof.

[0062] The data used for reconstruction may include: the output of the scanning process (e.g., image and attribute information), the LED light pattern used in the scanning process, the projector color pattern used in the scanning process, the camera calibration results (e.g., the non-intrinsic and intrinsic parameters of the camera), the projector calibration results (e.g., the position and orientation of the projector, and the irradiance of the light from the projector), and the LED calibration results (e.g., the LED pixel position and the irradiance of the LED pixel).

[0063] like Figure 8 As shown, the reconstruction process S800 may include sparse point cloud reconstruction / estimation (S802), depth map generation (S806), dense point cloud reconstruction / estimation (S808), dense point cloud fusion and filtering (S810), and mesh generation (S812). The 3D mesh of the target object can be obtained from the mesh generation, and an accurate normal map can be provided. Material estimation (S814) includes estimating the properties of the object, such as roughness, refractive index, subsurface scattering, transparency, etc. Material estimation may include obtaining a reflectivity model of the target object. The reflectivity model may include a bidirectional reflectance distribution function (BRDF). In some embodiments, when the position of the projector is relatively fixed relative to the object, the direction of the incident light is fixed, and the calculation of the BRDF of the target object can be simplified based on the fixed incident light and the pre-calibrated position and intensity of the projector.

[0064] When this is the first cycle (S816, yes), diffuse and specular separation (S818) is performed after material estimation. Diffuse and specular separation (S818) may be performed on a first image captured when the projector device 104 is turned on (e.g., when the LED panel is turned off) based on the projector calibration results (e.g., the position and irradiance of the projector), the 3D shape and normal map of the target object obtained from the mesh generation (S812), and the reflectivity model obtained from the material estimation (S814). Diffuse and specular separation (S818) may also be performed on a second image captured when the LED screen 106 is turned on (e.g., when the projector is turned off) based on the LED calibration results (e.g., the position and irradiance of the pixels of the LED panel), the 3D shape and normal map of the target object obtained from the mesh generation (S812), and the reflectivity model obtained from the material estimation (S814). The initial image (e.g., an image that has not been processed for diffuse and specular separation) reflects the combined result of specular highlights from a light source (e.g., a projector, an LED panel) and diffuse light (e.g., the intrinsic albedo from the target object). Due to different positions and directions of incident light, some parts of the initial image may be affected differently by specular light. The specular light value in the initial image can be determined based on a material estimate of the target object. For example, the reflectivity of a surface portion of a target object with incident light (e.g., incident light caused by LED pixels or light from a projector) can be modeled using a BRDF based on the calibration results of the incident light source (e.g., the direction and irradiance of LED pixels or light from a projector). The specular value can be subtracted from the initial image to obtain a diffuse-only image. In some embodiments, specular highlights may appear only at a specific camera viewing angle, and an initial image captured at a specific viewing angle can be compared with another initial image captured at another viewing angle to extract specular highlights and restore diffuse colors.

[0065] The diffuse-only image can be used for the next round of sparse point cloud reconstruction / estimation (S802), depth map generation (S806), dense point cloud reconstruction / estimation (S808), dense point cloud fusion and filtering (S810), mesh generation (S812), and material estimation (S814). In some embodiments, a second round of steps S802 to S814 is performed on the diffuse-only image obtained from the first image corresponding to the projector to improve the results of the sparse point cloud estimation, mesh generation, and material estimation.

[0066] Since this is not the first cycle (S816, no), a texture map may also be constructed (S820) to obtain the surface reflectance and other appearance parameters of the target object. In some embodiments, the texture map indicates the texture and color of the target object under the white background lighting provided by the LED screen 106.

[0067] The reconstruction process according to the disclosed embodiment can extract accurate surface material information regardless of the geometry of the object. The surface material information may include, for example, reflectivity, refractive index, and BRDF. Existing methods typically assume that the shape of the object is simple enough (e.g., a sphere or a plane) in order to measure its surface reflectivity or BRDF. The disclosed reconstruction process can restore a high-resolution 3D mesh that provides accurate surface normals; and based on images taken when the LED screen 106 illuminates the target object from all directions and generates incident light directions along the normals, the reflectance coefficient can be estimated more accurately.

[0068] In some embodiments, the reconstruction process S800 may further include a rendering and re-lighting step (S822). The 3D model obtained from the second round of mesh generation (S812), the surface texture obtained from the second round of material estimation (S814), and the appearance model obtained from the texture map generation (S820) can be used to render the target object in the virtual scene. In addition, the image obtained from the scanning process can be used for an image-based rendering task to render additional images of the target object from a perspective different from the perspective used in the scanning process and / or under ambient lighting with a light pattern different from the light pattern used in the scanning process. The image-based rendering task can be completed with a deep neural network model, such as a neural radiance field (NeRF).

[0069] The present disclosure also provides a non-volatile storage medium storing computer instructions. When the computer instructions are executed by a processor, the processor is caused to perform the disclosed calibration process, scanning process and / or reconstruction process.

[0070] The disclosed 3D object scanner can facilitate high-quality 3D reconstruction even for objects that are known to be difficult to reconstruct with existing methods. For example, objects without rich surface textures and / or with high specular reflectivity and geometric symmetry pose significant challenges to existing methods. These objects have few surface features, making it difficult to find correspondences between images from different views. In contrast, the disclosed 3D object scanner can use a projector to introduce surface features regardless of the surface texture of the object.

[0071] The disclosed reconstruction process can accurately obtain information about the target object, including surface texture, reflectivity, normal map, and translucency. This information can be used in high-quality rendering applications such as game and movie production.

[0072] Furthermore, since the disclosed object scanner can capture images of objects under different lighting conditions, the disclosed object scanner is completely calibrated using the parameters of the camera (e.g., position, view direction, and focal length). The captured images together with the precise camera information can be used to train neural networks in many applications.

[0073] Fig. 9 An exemplary computer system is shown that implements certain aspects of the 3D object scanner 100, controller 200, and / or other computing devices for the disclosed calibration, scanning, and / or reconstruction processes. Fig. 9 As shown, computer system 900 may include a processor 902 , a storage medium 904 , a monitor 906 , a communication module 908 , a database 910 , and peripheral devices 912 . Fig. 9 The components shown in are illustrative, certain components may be omitted, and other components may be added.

[0074] The processor 902 may include any appropriate one or more processors. In addition, the processor 902 may include multiple cores for multithreading or parallel processing. The processor 902 may be connected to other components via one or more buses or other electrical connections to send data to other components and receive data from other components. The processor 902 may be implemented by using at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA) and programmable logic array (PLA). The processor 902 may also include a main processor and a coprocessor. The main processor may be a central processing unit (CPU), and the coprocessor may be a graphics processing unit (GPU), which is configured to be responsible for rendering and drawing the content that the display screen needs to display. The storage medium 904 may include a memory module, such as a read-only memory (ROM), a random access memory (RAM), a flash memory module, an erasable and rewritable memory, and a large-capacity memory such as a CD-ROM, a USB flash drive and a hard disk. The storage medium 904 may store a computer program for implementing various processes when executed by the processor 902. The communication module 908 may include a network device for establishing a connection through the communication link 240.

[0075] The peripherals 912 may include I / O devices such as a keyboard, a mouse, and audio input / output devices to facilitate user interaction during the calibration, scanning, and / or reconstruction process, such as specifying a light pattern, specifying a time interval for displaying a light pattern, specifying a rotation interval, obtaining a start and stop signal for the current process, identifying image data to be used for reconstruction, etc. The peripherals 912 may also include a connection port compatible with the components of the 3D object scanner 100. The database 910 may include one or more databases for storing specific data and performing specific operations on the stored data, such as database searches. The data may include light patterns, images, and corresponding attribute information obtained during the calibration and scanning process, geometry and material data obtained from the reconstruction process, algorithms for performing the calibration, scanning, and / or reconstruction process, etc. The monitor 906 may be any suitable display technology, a graphical user interface, etc. suitable for displaying an image or video. For example, the monitor 906 may include a liquid crystal display (LCD) screen, an organic light emitting diode (OLED) screen, etc., and may be a touch screen.

[0076] The serial numbers of the above-mentioned embodiments of the present disclosure are only used for descriptive purposes and do not indicate a preference for the embodiments.

[0077] When the integrated unit in the aforementioned embodiment is implemented in the form of a software functional unit and sold or used as an independent product, the integrated unit can be stored in the aforementioned computer-readable storage medium. Based on such an understanding, the technical solution of the present disclosure can be implemented in the form of a software product, or the part that contributes to the relevant technology, or all or part of the technical solution. The computer software product is stored in a storage medium and includes several instructions for instructing one or more computer devices (which can be personal computers, servers, network devices, etc.) to perform all or some steps of the method in the embodiment of the present disclosure.

[0078] In the foregoing embodiments of the present disclosure, the descriptions of the embodiments have different emphases. For parts that are not described in detail in one embodiment, reference may be made to the relevant descriptions of other embodiments.

[0079] In several embodiments provided in the present disclosure, it should be understood that the disclosed client can be implemented in other ways. The above-mentioned device embodiments are merely exemplary. For example, the division of units is merely the division of logical functions, and other division methods can be used in the actual implementation process. For example, multiple units or components can be combined, or can be integrated into another system, or some features can be omitted or not performed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be an indirect coupling or communication connection through some interfaces, units or modules, and can be electrical or other forms.

[0080] Units described as separate parts may or may not be physically separate. Parts shown as units may or may not be physical units and may be located in one location or may be distributed over multiple network units. Some or all of the units may be selected based on actual needs to achieve the purpose of the solution in the embodiment.

[0081] In addition, the functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware, or may be implemented in the form of a software functional unit.

[0082] Although the principles and embodiments of the present disclosure are described in the specification by using specific embodiments, the foregoing description of the embodiments is only intended to help understand the method of the present disclosure and its core idea. At the same time, those of ordinary skill in the art can modify the specific implementation methods and application scope according to the ideas of the present disclosure. In short, the contents of the specification should not be interpreted as limiting the present disclosure.

Claims

1. A three-dimensional 3D object scanner, characterized in that: include: an object platform, configured to host the target object; a projector device configured to project a first light pattern onto the target object; an LED screen surrounding the object platform and configured to emit a second light pattern; An imaging device is configured to obtain multiple images of the target object from multiple viewing angles when the target object and the imaging device are relatively rotated to multiple rotation degrees, the multiple images including an image obtained when the projector device projects the first light pattern and an image obtained when the LED screen emits the second light pattern, wherein the multiple images are used to determine 3D information and texture information of the target object.

2. The 3D object scanner according to claim 1, characterized in that: The LED screen includes a plurality of LED panels.

3. The 3D object scanner according to claim 2, characterized in that: The plurality of LED panels have the same shape.

4. The 3D object scanner according to claim 1, characterized in that: The LED screen has a dome shape or a cube shape.

5. The 3D object scanner according to claim 4, characterized in that: The LED screen is formed by a plurality of interconnected LED panels, the LED panels having the same shape and the same size.

6. The 3D object scanner according to claim 1, characterized in that: The object platform is located at the center of the cross section of the LED screen.

7. The 3D object scanner according to claim 1, characterized in that: The projector device includes a plurality of projectors configured to project one or more first light patterns onto the target object from different directions.

8. The 3D object scanner according to claim 7, characterized in that: Each projector is an elongated member that projects outwardly from an edge of the object platform.

9. The 3D object scanner according to claim 7, characterized in that: The plurality of projectors are fixedly disposed on the object platform.

10. The 3D object scanner according to claim 7, characterized in that: The plurality of projectors are evenly distributed on a circumference of the object platform.

11. The 3D object scanner according to claim 1, characterized in that: The imaging device is located within a housing provided by the LED screen.

12. The 3D object scanner according to claim 1, characterized in that: The imaging device is not located within a housing provided by the LED screen, wherein a light path between the imaging device and the object platform passes unobstructed through a portion of the housing.

13. The 3D object scanner according to claim 1, characterized in that: Further included is a frame for housing the imaging device.

14. The 3D object scanner according to claim 13, characterized in that: The frame includes an arcuate frame portion, and the imaging device includes a plurality of cameras evenly distributed over a span of the arcuate frame portion.

15. The 3D object scanner according to claim 14, characterized in that: The plurality of cameras together span a total angle from 90 degrees to 105 degrees.

16. The 3D object scanner according to claim 13, characterized in that: The imaging device includes a camera, and the frame includes a movable structure configured to move the camera along a path and stop at a plurality of stop positions on the path to obtain images of the target object from the plurality of perspectives, each stop position corresponding to one of the plurality of perspectives.

17. The 3D object scanner according to claim 1, characterized in that: The imaging device is stationary and the object platform is configured to rotate based on the plurality of degrees of rotation.

18. The 3D object scanner according to claim 1, characterized in that: The subject platform is fixed and the imaging device is attached to a frame that rotates based on the plurality of degrees of rotation.

19. A 3D object scanning method applied to a 3D object scanner, characterized in that: include: controlling a projector device of the 3D object scanner to project a first light pattern onto a target object hosted by an object platform; controlling an LED screen of the 3D object scanner to emit a second light pattern, the LED screen surrounding the object platform; as well as controlling an imaging device of the 3D object scanner to obtain a plurality of images of the target object from a plurality of viewing angles when the target object and the imaging device are relatively rotated to a plurality of rotation degrees, The multiple images include images obtained when the projector device projects the first light pattern and images obtained when the LED screen emits the second light pattern, and the multiple images are used to determine 3D information and texture information of the target object.

20. A non-volatile storage medium for storing computer instructions which, when executed by a processor, cause the processor to: controlling a projector device of a three-dimensional 3D object scanner to project a first light pattern onto a target object hosted by an object platform; controlling an LED screen of the 3D object scanner to emit a second light pattern, the LED screen surrounding the object platform; and controlling an imaging device of the 3D object scanner to obtain a plurality of images of the target object from a plurality of viewing angles when the target object and the imaging device are relatively rotated to a plurality of rotation degrees, The multiple images include images obtained when the projector device projects the first light pattern and images obtained when the LED screen emits the second light pattern, and the multiple images are used to determine 3D information and texture information of the target object.