Three-dimensional scanning system and method

Abstract

The present invention is a three-dimensional scanning system and method. The three-dimensional scanning system includes a projector for generating a transmission light projected to an object and a reflection light reflected from the object; a sensor for generating image data based on the reflection light; and a depth processor for generating depth data based on the image data and at least one correction factor representing an amount of deviation between the image data and ideal image data, the amount of deviation being caused by the reflection light passing through a lens of the sensor and then being defocused to impinge on a sensing plane of the sensor to form a blurred circle of confusion.

Description

Technical Field

[0001] This invention relates to a three-dimensional (3D) scanning system and method, and more particularly to an interference-free three-dimensional scanning system and method. Background Technology

[0002] A 3D scanning device is an instrument that measures the three-dimensional shape of an object or scene by analyzing its shape data from the real world, thereby obtaining a depth map, which is an image containing the distance of the object's surface or scene from the viewpoint. The collected 3D data can be used in a variety of applications, such as face recognition, virtual reality, augmented reality, and robot mapping.

[0003] A Time-of-Flight (ToF) scanner is a 3D scanning device that uses time-of-flight technology to determine the distance between a sensor (such as a camera) and an object by measuring the round-trip time of artificial light signals provided by a light source in an image at each point.

[0004] Structured light scanners are another type of 3D scanning device that projects a light pattern provided by a light source onto an object and then uses sensors to capture the distorted reflected light pattern in order to reconstruct the surface shape.

[0005] Due to optical defects in the sensors used in 3D scanning devices, light rays passing through one or more lenses cannot be focused. This non-ideal focusing caused by near objects often interferes with the scanning device's ability to determine the depth of distant objects.

[0006] Since traditional 3D scanning devices cannot effectively resolve the distance between the sensor and the object, there is an urgent need to propose a novel mechanism to overcome the shortcomings of traditional 3D scanning devices. Summary of the Invention

[0007] In view of the above, one of the objectives of this invention is to provide a three-dimensional scanning system and method that can accurately obtain depth data without being affected by the CoC (Crystal of Confusion).

[0008] According to an embodiment of the present invention, a three-dimensional scanning system includes a projector, a sensor, and a depth processor. The projector generates emitted light projected onto an object and reflected light reflected from the object. The sensor generates image data based on the reflected light. The depth processor generates depth data based on the image data and at least one correction factor, the correction factor representing the amount of deviation between the image data and ideal image data, the deviation being caused by the reflected light passing through the sensor's lens and then being out of focus onto the sensor's sensing plane, forming a blurred aperture.

[0009] Preferably, the blur aperture is determined by comparing the image data with the depth data, and the inconsistency or sudden change in the depth value indicates interference from the blur aperture.

[0010] Preferably, at least one pixel near the location of the blurred aperture is used as the ideal image data.

[0011] Preferably, the projector includes a light source, which may be a point light source, a diffused light source, or a planar light source.

[0012] Preferably, the light source comprises a light-emitting diode or a laser diode.

[0013] Preferably, the laser diode comprises edge-emitting laser or vertical cavity surface-emitting laser.

[0014] Preferably, it further includes a controller for controlling the timing of the projector and the sensor.

[0015] Preferably, it further includes: a back-end device that receives the depth data to enable three-dimensional applications.

[0016] Preferably, the blurred aperture becomes larger when the object has high reflectivity or is closer to the sensor.

[0017] Preferably, the image data is generated using multiple measurement windows, during which accumulated charge is collected separately.

[0018] According to an embodiment of the present invention, a three-dimensional scanning method includes: generating emitted light and receiving reflected light to generate image data; determining the position and range of a blurred aperture, the blurred aperture being caused by the reflected light illuminating a sensing plane out of focus; determining at least one correction factor based on the position and range of the blurred aperture, the correction factor representing the amount of deviation between the image data and ideal image data caused by the blurred aperture; and generating depth data based on the image data and the at least one correction factor.

[0019] Preferably, the blur aperture is determined by comparing the image data with the depth data, and the inconsistency or sudden change in the depth value indicates interference from the blur aperture.

[0020] Preferably, at least one pixel near the location of the blurred aperture is used as the ideal image data.

[0021] Preferably, the at least one correction factor is updated and stored for use when generating depth data later.

[0022] Preferably, depth data is repeatedly generated based on the image data and the updated correction factor until the generated depth data is no longer disturbed.

[0023] Preferably, depth data is generated for the first time based solely on the image data before determining the position and extent of the blurred aperture.

[0024] Preferably, after determining the at least one correction factor, depth data is generated a second time based on the image data and the at least one correction factor.

[0025] Preferably, depth data is generated based on at least one correction factor of the image data and the previous frame.

[0026] Preferably, the at least one correction factor is updated and stored for use in the next frame.

[0027] Preferably, the at least one correction factor is determined for each frame, for a fixed or random number of frames, or for a fixed or random time period.

[0028] By means of the above technical solution, the present invention has at least the following advantages: the three-dimensional scanning system and method of the present invention can correctly obtain depth data and will not be affected by the blurring aperture (CoC). Attached Figure Description

[0029] Figure 1 A block diagram showing a three-dimensional scanning system according to an embodiment of the present invention is displayed.

[0030] Figure 2 show Figure 1 A diagram illustrating how a depth processor uses Time-of-Flight (ToF) technology to determine the distance between a sensor and an object.

[0031] Figure 3A This diagram illustrates how, under ideal conditions, reflected light passes through the sensor's lens and is then focused onto the sensor's sensing plane.

[0032] Figure 3B This diagram shows a real-world scene where reflected light passes through the sensor's lens and then shines out of focus onto the sensor's sensing plane.

[0033] Figure 3C A schematic diagram showing reflected light from a distant object and reflected light from a nearby object passing through the sensor's lens and then illuminating the sensor's sensing plane.

[0034] Figure 4 An example of a sensor generating image data is shown.

[0035] Figure 5 This diagram shows a flowchart of a three-dimensional scanning method according to an embodiment of the present invention.

[0036] Figure 6 A flowchart showing another embodiment of the three-dimensional scanning method of the present invention is displayed.

[0037] Figure 7 A flowchart showing another embodiment of the three-dimensional scanning method of the present invention is shown.

[0038] [Explanation of Key Component Symbols]

[0039] 100: 3D scanning system; 10: Object

[0040] 10A: Distant object; 10B: Near object

[0041] 11: Projector 111: Light Source

[0042] 12: Sensor 121: Lens

[0043] 122: Sensor plane 123: Blur aperture

[0044] 124: Circle 13: Controller

[0045] 14: Deepin processor 15: Backend device

[0046] 200: 3D scanning method 300: 3D scanning method

[0047] 400: 3D scanning method 21: Generating image data

[0048] 22: Generate depth data based on image data and correction factors.

[0049] 22B: Depth data generated solely from imagery data

[0050] 22C: Generate depth data based on the image data and the correction factor of the previous frame.

[0051] 23: Determine whether the depth data is disturbed.

[0052] 24: Output depth data 25: Determine the position of the blur aperture

[0053] 26: Determines the range of the blurred aperture. 27: Determines the correction factor.

[0054] d: Distance; Δt: Round trip time

[0055] c: Speed ​​of light; C0~C3: Measurement window

[0056] Q0~Q3: Ideal accumulated charge; Q0'~Q3': Actual accumulated charge Detailed Implementation

[0057] Figure 1 A block diagram showing an embodiment of the three-dimensional (3D) scanning system 100 of the present invention is provided. Although the following embodiment uses time-of-flight (ToF) technology as an example, other three-dimensional scanning technologies, such as structured light scanning technology, may also be used.

[0058] In this embodiment, the 3D scanning system 100 may include a projector 11 for generating emitted light projected onto the object 10 and reflected light reflected from the object 10. The projector 11 may include a light source 111, such as a point light source, a diffused light source, or a planar (or floodlight) light source. In one embodiment, the light source 111 may be a light-emitting diode (LED). In another embodiment, the light source 111 may be a laser diode (LD), such as an edge-emitting laser (EEL) having a laser beam emitted from the surface of a wafer cut into individual chips, or a laser beam emitted perpendicular to the upper surface of a vertical-cavity surface-emitting laser (VCSEL).

[0059] The 3D scanning system 100 of this embodiment may include a sensor 12, such as a camera, which generates image data based on reflected light. In this embodiment, the 3D scanning system 100 may include a controller 13, such as a microcontroller, for controlling the timing of the projector 11 and the sensor 12. The controller 13 can control the timing of the projector 11 generating emitted light and can control the amount of reflected light received by the sensor 12 (i.e., exposure).

[0060] In this embodiment, the 3D scanning system 100 may include a depth processor (or depth decoder) 14, such as an image processor, which generates depth data (e.g., a depth map) based on image data collected by the sensor 12. In one embodiment, the controller 13 and the depth processor 14 may be fabricated on a single integrated circuit. The depth data generated by the depth processor 14 may be further fed to a back-end device 15, such as a general-purpose computer (e.g., a personal computer or mobile device), to enable various 3D applications, such as face recognition, virtual reality, augmented reality, and robot mapping.

[0061] Figure 2 show Figure 1 The depth processor 14 uses Time-of-Flight (ToF) technology to determine the distance d between sensor 12 and object 10, as shown in the diagram. Projector 10 projects emitted light at time t0, and sensor 12 receives reflected light at time t0 + Δt, where Δt represents the round-trip time of the emitted light. Since the emitted and reflected light travel at the speed of light (c), the distance d between sensor 12 and object 10 is (c * Δt / 2).

[0062] Figure 3A This diagram shows a scenario where, under ideal conditions, reflected light passes through the lens 121 of sensor 12 and is then focused onto the sensing plane 122 of sensor 12. Figure 3BThis schematic diagram shows reflected light in a real scene passing through lens 121 of sensor 12 and then out of focus onto sensing plane 122 of sensor 12. The cone of light from lens 121, which is not perfectly focused, causes a circle of confusion (CoC) 123. When the object 10 has high reflectivity or is closer to sensor 12, the circle of confusion 123 typically becomes more pronounced (i.e., larger).

[0063] Figure 3C A schematic diagram showing reflected light from a distant object 10A and reflected light from a near object 10B passing through the lens 121 of sensor 12 and then illuminating the sensing plane 122 of sensor 12. In this (real-world) scene, the blurred aperture caused by the near object 10B would interfere with the depth processor 14's ability to correctly acquire depth data from the distant object 10A within the area shown in circle 124.

[0064] Figure 4 An example of an embodiment in which sensor 12 generates image data is illustrated; for details, please refer to "Introduction to 3D ToF Three-Dimensional Scene Depth Measurement System" by Garrick Dai of Texas Instruments.

[0065] Ideally, multiple (e.g., four) measurement windows C0–C3 are used to generate image data, and ideal accumulated charges Q0–Q3 are collected separately during the measurement windows, such as… Figure 4 As shown. (Depth processor 14) generates depth data based on the ideal accumulated charge Q0~Q3 as follows:

[0066]

[0067] The symbol ∝ represents direct proportion.

[0068] However, in real-world scenarios, due to interference from the blurred aperture (CoC), the actual accumulated charge Q0' to Q3' will be collected. The actual accumulated charge Q0' to Q3' can be represented as follows:

[0069] Q0'=Q0+Q0_CoC

[0070] Q1'=Q1+Q1_CoC

[0071] Q2'=Q2+Q2_CoC

[0072] Q3'=Q3+Q3_CoC

[0073] Where Q0_CoC to Q3_CoC are correction factors, representing the deviation between the actual accumulated charge and the ideal accumulated charge, respectively. (After compensation) the interference-free depth data can be represented as follows:

[0074]

[0075] Figure 5 This flowchart illustrates a three-dimensional (3D) scanning method 200 according to an embodiment of the present invention. In step 21, a projector 11 projects emitted light onto an object 10, and a sensor 12 generates image data based on the reflected light. In step 22, a depth processor 14 generates depth data based on the image data and at least one correction factor, which represents the deviation between the actual image data (e.g., accumulated charge) and the ideal image data (for each pixel of the image data). It is noteworthy that at the beginning of the process, the initial correction factor can be any value.

[0076] In step 23, the depth processor 14 determines whether the generated depth data is affected by a blurred aperture (CoC). If step 23 determines that there is no interference, the depth data is output (step 24), for example, fed to the back-end device 15. If step 23 determines that there is interference, the process proceeds to step 25 to determine the position of the blurred aperture (CoC). Then, in step 26, the range of the blurred aperture (CoC) is determined. In one embodiment, inconsistencies (e.g., areas of inconsistency) or sudden changes in depth values ​​between image data (brightness) and depth data (e.g., a depth map) can indicate the presence of interference.

[0077] In step 27, a modification factor is determined (updated and stored) based on the position and extent of the blurred aperture (CoC). In one embodiment, at least one pixel near the position (within) of the blurred aperture (CoC) is used as ideal image data, and the deviation between the actual image data (e.g., accumulated charge) and the ideal image data is determined as the correction factor. The process then returns to step 22, where depth data is repeatedly generated based on the image data and the updated correction factor until the generated depth data is no longer disturbed (step 23), i.e., the range of disturbance is less than a preset threshold.

[0078] Figure 6 A flowchart illustrating a three-dimensional (3D) scanning method 300 according to another embodiment of the present invention is shown. In step 21, projector 11 projects emitted light onto object 10, and sensor 12 generates image data based on the reflected light. In step 22B, depth processor 14 generates depth data for the first time based solely on the image data.

[0079] In step 25, the position of the blur aperture (CoC) is determined. In step 26, the range of the blur aperture (CoC) is determined. In step 27, a modification factor is determined (and stored) based on the position and range of the blur aperture (CoC). Next, in step 22, the depth processor 14 generates depth data based on the image data and the modification factor. Finally, in step 24, the depth data is output, for example, fed to the back-end device 15.

[0080] Figure 7 A flowchart illustrating a three-dimensional (3D) scanning method 400 according to another embodiment of the present invention is shown. In step 21, projector 11 projects emitted light onto object 10, and sensor 12 generates image data based on the reflected light. In step 22C, depth processor 14 generates depth data based on the image data and a correction factor of the previous frame. Then, in step 24, the depth data is output, for example, fed to back-end device 15.

[0081] Next, in step 25, the position of the blur aperture (CoC) is determined. In step 26, the range of the blur aperture (CoC) is determined. In step 27, based on the position and range of the blur aperture (CoC), the modification factor for the next frame is determined (and updated and stored), and the process returns to step 21 for the next frame. It is worth noting that this can be performed for each frame. Figure 7 The process can be executed on multiple frames (fixed or random number). Figure 7 The process can be executed for periods of (fixed or random length). Figure 7 The process.

[0082] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the patent application of the present invention; any equivalent changes or modifications made without departing from the spirit disclosed in the invention should be included within the scope of the patent application.

Claims

1. A three-dimensional scanning system, characterized in that, Include: A projector is used to generate emitted light that is projected onto an object, as well as reflected light that is reflected from the object. The sensor generates image data based on the reflected light; and A depth processor generates depth data based on the image data and at least one correction factor, the correction factor representing the amount of deviation between the image data and ideal image data, the deviation being caused by the reflected light passing through the lens of the sensor and then shining out of focus onto the sensing plane of the sensor to form a blurred aperture.

2. The three-dimensional scanning system according to claim 1, characterized in that, The blur aperture is determined by comparing the image data with the depth data; inconsistencies or sudden changes in depth values ​​indicate interference from the blur aperture.

3. The three-dimensional scanning system according to claim 1, characterized in that, At least one pixel near the location of the blurred aperture is used as the ideal image data.

4. The three-dimensional scanning system according to claim 1, characterized in that, The projector includes a light source, which may be a point light source, a diffused light source, or a planar light source.

5. The three-dimensional scanning system according to claim 4, characterized in that, The light source contains either a light-emitting diode or a laser diode.

6. The three-dimensional scanning system according to claim 5, characterized in that, The laser diode contains edge-emitting lasers or vertical cavity surface-emitting lasers.

7. The three-dimensional scanning system according to claim 1, characterized in that, It also includes: The controller is used to control the timing of the projector and the sensor.

8. The three-dimensional scanning system according to claim 1, characterized in that, It also includes: The backend device receives this depth data to enable 3D applications.

9. The three-dimensional scanning system according to claim 1, characterized in that, The blurred aperture becomes larger when the object has high reflectivity or is close to the sensor.

10. The three-dimensional scanning system according to claim 1, characterized in that, The image data was generated using multiple measurement windows, during which accumulated charge was collected separately.

11. A three-dimensional scanning method, characterized in that, Include: It generates emitted light and receives reflected light to produce image data; The position and range of the blurred aperture are determined by the reflected light shining out of focus onto the sensing plane; At least one correction factor is determined based on the position and extent of the blurred aperture, the correction factor representing the amount of deviation between the image data and the ideal image data caused by the blurred aperture; and Depth data is generated based on the image data and the at least one correction factor.

12. The three-dimensional scanning method according to claim 11, characterized in that, The blur aperture is determined by comparing the image data with the depth data; inconsistencies or sudden changes in depth values ​​indicate interference from the blur aperture.

13. The three-dimensional scanning method according to claim 11, characterized in that, At least one pixel near the location of the blurred aperture is used as the ideal image data.

14. The three-dimensional scanning method according to claim 11, characterized in that, The at least one correction factor is updated and stored for use when generating depth data later.

15. The three-dimensional scanning method according to claim 14, characterized in that, Depth data is repeatedly generated based on the image data and updated correction factors until the generated depth data is no longer disturbed.

16. The three-dimensional scanning method according to claim 11, characterized in that, Before determining the location and extent of the blurred aperture, depth data is generated for the first time based solely on the image data.

17. The three-dimensional scanning method according to claim 16, characterized in that, After determining the at least one correction factor, depth data is generated a second time based on the image data and the at least one correction factor.

18. The three-dimensional scanning method according to claim 11, characterized in that, Depth data is generated based on the image data and at least one correction factor from the previous frame.

19. The three-dimensional scanning method according to claim 18, characterized in that, At least one correction factor is updated and stored for use in the next frame.

20. The three-dimensional scanning method according to claim 19, characterized in that, The at least one correction factor is determined for each frame, for a fixed or random number of frames, or for a fixed or random time period.

Images