Three-dimensional structure sensing system

Abstract

The invention provides a three-dimensional structure sensing system comprising an image sensor used for receiving reflected light generated when an object is irradiated by emitted light and converting the reflected light into image data to represent an image of the object; the depth processing unit generates depth data according to the image data. Whether the depth data is affected by the reflecting surface is determined according to the image data and the depth data.

Description

Technical Field

[0001] The present invention relates to a sensing system, and more particularly to a three-dimensional or stereoscopic structure sensing system. Background Art

[0002] One of the most promising trends in the mobile phone industry is the development of 3D facial recognition applications, which can provide users with greater security and convenience. To achieve this goal, different 3D sensor technologies are needed to capture and process the user's facial features. Among the various 3D sensor technologies, active light source sensing systems such as time of flight (ToF) and structured light (SL) are very popular because they can project light patterns onto objects and measure the distance and shape of objects based on the reflected light. These systems can also be used for other purposes, such as terrain recognition for robots and self-driving cars, which require accurate and reliable 3D information of the environment for navigation and obstacle avoidance.

[0003] One of the challenges in designing sensing systems is ensuring that sensors can accurately detect the presence and properties of different materials. However, certain materials, such as mirrors and metals, can interfere with the performance of sensors and produce false readings. This can have serious consequences for applications that rely on sensing systems for safety, efficiency, or quality control.

[0004] Therefore, it is urgent to understand how mirrors and metals affect the sensors of sensing systems and how to mitigate their negative effects. Summary of the invention

[0005] In view of the above, one of the objectives of the embodiments of the present invention is to provide a three-dimensional structure sensing system and method for determining whether the generated depth data is affected by a reflective surface.

[0006] According to an embodiment of the present invention, a three-dimensional structure sensing system includes an image sensor and a depth processing unit. The image sensor is used to receive reflected light generated by an object being irradiated by emitted light, and then convert the reflected light into image data to represent the image of the object. The depth processing unit generates depth data based on the image data. Based on the image data and the depth data, it is determined whether the depth data is affected by a reflective surface.

[0007] In one embodiment of the present invention, it further comprises:

[0008] A projector generates the emitted light to illuminate the object.

[0009] In one embodiment of the present invention, the projector and the image sensor adopt time-of-flight or structured light technology.

[0010] In one embodiment of the present invention, it further comprises:

[0011] The controller is used to control the light emission timing of the projector and the exposure of the image sensor.

[0012] In one embodiment of the present invention, the emitted light includes ambient light.

[0013] In one embodiment of the present invention, the emitted light includes visible light, infrared light or ultraviolet light.

[0014] In one embodiment of the present invention, the depth processing unit includes a depth processor.

[0015] In one embodiment of the present invention, it further comprises:

[0016] The backend device receives the depth data from the depth processor.

[0017] In one embodiment of the present invention, the deep processing unit includes a software component disposed in a backend device.

[0018] In one embodiment of the present invention, the brightness of the image data and the depth data are negatively correlated with each other.

[0019] In one embodiment of the present invention, the brightness of the image data corresponding to the depth data is compared with a default associated threshold.

[0020] In one embodiment of the present invention, if the brightness is greater than a critical value associated with the corresponding depth data, the depth data is output.

[0021] In one embodiment of the present invention, if the brightness is not greater than a critical value associated with the corresponding depth data, the depth data is modified.

[0022] In one embodiment of the present invention, the modified depth data is set as a default value.

[0023] In one embodiment of the present invention, if the brightness is not greater than a critical value associated with the corresponding depth data, the entire image is skipped.

[0024] In one embodiment of the present invention, the reflective surface includes a mirror surface or a metal surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A block diagram showing a three-dimensional (or stereoscopic) structure sensing system according to an embodiment of the present invention;

[0026] Figure 2 A flowchart showing a three-dimensional (or stereoscopic) structure sensing method, which can be applied to Figure 1 3D structure sensing system;

[0027] Figure 3A schematic diagram illustrating a reflective surface which reflects emitted light so that a virtual image is seen on the reflective surface;

[0028] Figure 4 Display the relationship between brightness and depth data;

[0029] Figure 5A Schematic diagram illustrating a (mis)reconstructed configuration, where an object is placed in front of a reflective surface and a virtual image is produced;

[0030] Figure 5B Schematic diagram illustrating the (correct) reconstruction structure, where the virtual image has been occluded;

[0031] Figure 6 A block diagram of a three-dimensional (or stereoscopic) structure sensing system according to another embodiment of the present invention is shown.

[0032]

Explanation of symbols

[0033] 100: 3D structure sensing system

[0034] 600: 3D structure sensing system

[0035] 11: Projector

[0036] 12: Image sensor

[0037] 13: Controller

[0038] 14: Depth Processing Unit

[0039] 15: Backend device

[0040] 200: Three-dimensional structure sensing method

[0041] 21: Generate emission light

[0042] 22: Convert reflected light into image data

[0043] 23: Generate depth data based on image data

[0044] 24: Whether the brightness is greater than the associated critical value

[0045] 25: Output depth data

[0046] 26: Modify depth data

[0047] 30:Object

[0048] 31: Reflective surface

[0049] 32: virtual image

[0050] 51:Object

[0051] 52: Reflective surface

[0052] 53: virtual image DETAILED DESCRIPTION

[0053] Figure 1 A block diagram showing a three-dimensional (or stereoscopic) structure sensing system 100 according to an embodiment of the present invention is shown. Figure 2 A flowchart showing a three-dimensional (or stereoscopic) structure sensing method 200 is applicable to Figure 1 A three-dimensional structure sensing system 100 is provided. Figure 1 The blocks and steps of the second figure can be implemented using hardware, software or a combination thereof. The three-dimensional structure sensing system 100 can be applicable to time-of-flight (ToF) and structured light (SL) applications, but is not limited thereto.

[0054] In this embodiment, the three-dimensional structure sensing system 100 may include a projector 11 as an active light source for emitting light (i.e., generating emitted light), and then irradiating the emitted light onto the object to be measured (step 21). In another alternative embodiment, the projector 11 may be omitted and a passive light source (e.g., ambient light) may be used instead. Generally speaking, the emitted light of the (active / passive) light source may be electromagnetic waves of various wavelengths such as visible light, infrared light, or ultraviolet light.

[0055] The 3D structure sensing system 100 of this embodiment may include an image sensor 12 for receiving reflected light and converting the reflected light into image data to represent the image of the object to be measured (step 22). In one embodiment, the projector 11 and the image sensor 12 use time-of-flight (ToF) or structured light (SL) technology.

[0056] In this embodiment, the 3D structure sensing system 100 may include a controller 13 for controlling the light emission timing of the projector 11 via a strobe signal and controlling the exposure of the image sensor 12 (ie, the amount of light reaching the image sensor 12 ) via an exposure signal.

[0057] The three-dimensional structure sensing system 100 of this embodiment may include a depth processing unit 14 to generate depth data according to the image data (step 23). The generated depth data represents the distance between the surface of the object to be measured and the image sensor 12. In this embodiment, the depth processing unit 14 may include a depth processor (e.g., an integrated circuit) disposed near the image sensor 12.

[0058] According to one of the features of this embodiment, the depth processing unit 14 determines whether the generated depth data is affected by the spatial distribution and / or temporal distribution characteristics of the reflective surface (eg, mirror surface or metal surface) according to the image data and the depth data. Figure 3A schematic diagram of a reflective surface 31 is shown, which reflects the emitted light (of the projector 11), so that a virtual image 32 is seen on the reflective surface 31 (eg, a mirror). The virtual image 32 affects the generated depth data, so that the object 30 is reconstructed incorrectly.

[0059] Figure 4 Displays the relationship between brightness and depth information (of image data). Figure 4 As shown, the brightness (of the image data) and the depth data are negatively correlated. If the brightness corresponding to a depth data is greater than a (preset) threshold, it is acceptable, otherwise it is unacceptable. It is worth noting that the critical curve (or straight line) is not necessarily smooth, and objects with different reflectivity may have different critical curves (or straight lines).

[0060] In this embodiment, the brightness (of the image data) corresponding to a depth data is compared with a (preset) associated critical value (step 24). If the brightness is greater than the critical value associated with the corresponding depth data, it is an acceptable brightness, so the generated depth data is output (step 25) to the backend device 15 (such as a computer) for further processing or application. If the brightness is not greater than the critical value associated with the corresponding depth data, it is unacceptable brightness, so the depth data is modified before outputting the generated depth data to the backend device (step 26). In one embodiment, the depth data corresponding to the unacceptable brightness is set to a default value (e.g., "0"), which can be recognized by the backend device 15. In another embodiment, the entire image corresponding to the unacceptable brightness is skipped.

[0061] Figure 5A Schematic diagram illustrating a (mis)reconstructed configuration, where an object 51 is placed in front of a reflecting surface 52 and a virtual image 53 is generated. Figure 5B Schematic diagram illustrating the (correct) reconstruction structure, where ( Figure 5A The virtual image 53 has been blocked by the above-mentioned embodiment.

[0062] Figure 6 A block diagram of a three-dimensional (or stereoscopic) structure sensing system 600 according to another embodiment of the present invention is shown. Figure 6 The three-dimensional structure sensing system 600 is similar to Figure 1 The differences between the three-dimensional structure sensing system 100 are described as follows.

[0063] In the three-dimensional structure sensing system 600, the depth processing unit 14 (and its software components) is disposed in the backend device 15, wherein depth data is generated (step 23), and then, based on the image data and the depth data, the brightness of the corresponding depth data (image data) is compared with a (preset) associated threshold value to determine whether the generated depth data is affected by a reflective surface (e.g., a mirror).

[0064] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A three-dimensional structure sensing system, characterized in that: Include: An image sensor is used to receive reflected light generated by an object being irradiated by the emitted light and then convert the reflected light into image data to represent an image of the object; and A depth processing unit, generating depth data according to the image data; Whether the depth data is affected by a reflective surface is determined according to the image data and the depth data.

2. The three-dimensional structure sensing system according to claim 1, characterized in that: Also includes: A projector generates the emitted light to illuminate the object.

3. The three-dimensional structure sensing system according to claim 2, characterized in that: The projector and the image sensor adopt time-of-flight or structured light technology.

4. The three-dimensional structure sensing system according to claim 2, characterized in that: Also includes: The controller is used to control the light emission timing of the projector and the exposure of the image sensor.

5. The three-dimensional structure sensing system according to claim 1, characterized in that: The emitted light includes ambient light.

6. The three-dimensional structure sensing system according to claim 1, characterized in that: The emitted light includes visible light, infrared light or ultraviolet light.

7. The three-dimensional structure sensing system according to claim 1, characterized in that: The depth processing unit includes a depth processor.

8. The three-dimensional structure sensing system according to claim 7, characterized in that: Also includes: The backend device receives the depth data from the depth processor.

9. The three-dimensional structure sensing system according to claim 1, characterized in that: The deep processing unit includes a software component and is disposed in a backend device.

10. The three-dimensional structure sensing system according to claim 1, characterized in that: The brightness of the image data and the depth data are negatively correlated with each other.

11. The three-dimensional structure sensing system according to claim 1, characterized in that: The brightness of the image data corresponding to the depth data is compared with a default associated threshold.

12. The three-dimensional structure sensing system according to claim 11, characterized in that: If the brightness is greater than a critical value associated with the corresponding depth data, the depth data is output.

13. The three-dimensional structure sensing system according to claim 11, characterized in that: If the brightness is not greater than a critical value associated with the corresponding depth data, the depth data is modified.

14. The three-dimensional structure sensing system according to claim 13, characterized in that: The modified depth data is set as the default value.

15. The three-dimensional structure sensing system according to claim 13, characterized in that: If the brightness is not greater than the threshold associated with the corresponding depth data, the entire image is skipped.

16. The three-dimensional structure sensing system according to claim 1, characterized in that: The reflective surface includes a mirror surface or a metal surface.

Images