TOF (Time of Flight) and structured light fused sensing device, sensing method thereof and electronic equipment

Abstract

The embodiment of the invention provides a TOF (Time of Flight) and structured light fused sensing device, a sensing method of the TOF and structured light fused sensing device and electronic equipment. The sensing device comprises a projector, a TOF receiving module, at least one structured light receiving module and a controller. The projector works in a time division multiplexing mode under the control of the controller, and can selectively project a dot matrix pattern and a floodlight pattern; the TOF receiving module is used for working in the projection time period of the floodlight pattern so as to receive the floodlight pattern and generate first depth information; the structured light receiving module is used for working in the projection time period of the dot matrix pattern so as to receive the dot matrix pattern and generate second depth information; the controller executes time division multiplexing control, and by synchronously switching the projection mode of the projector and the working state of the corresponding receiving module, the working period of the TOF receiving module coincides with the projection period of the floodlight pattern in time, and the working period of the structured light receiving module coincides with the projection period of the dot matrix pattern in time.

Description

Technical Field

[0001] This application relates to the field of three-dimensional vision sensing technology. More specifically, this application relates to a sensing device and sensing method and electronic device that fuses TOF and structured light. Background Technology

[0002] In the field of 3D visual sensing, Time-of-Flight (TOF) and structured light are two mainstream active depth sensing technologies. TOF technology calculates distance by measuring the time of flight of light waves, and its advantages are long operating distance and fast response speed; while structured light technology reconstructs 3D contours by analyzing the deformation of specific patterns, and usually has higher accuracy at close range.

[0003] To combine the advantages of these two technologies, existing technologies have proposed sensing systems that combine Time-of-Flight (TOF) and structured light. However, these systems suffer from a fundamental architectural flaw: they employ a simple hardware stacking scheme, placing a complete TOF sensing unit (containing an independent floodlight projector and a TOF receiver module) and a complete structured light sensing unit (containing an independent dot matrix projector and a structured light receiver module) side-by-side. This dual-system parallel architecture leads to a series of inherent drawbacks: First, the system must be configured with two independent projectors and at least two receiver modules, resulting in severe hardware redundancy and significantly increasing material costs and assembly complexity; second, the two emission optical systems require their own independent optical paths and structural spaces, making it difficult to compress the device size and resulting in a bulky overall structure; finally, this large hardware architecture severely limits its integrated application in consumer electronics terminals with stringent spatial requirements, such as smartphones and AR / VR devices.

[0004] Therefore, there is an urgent need in this field for a TOF and structured light fusion sensing solution that can effectively reduce the number of hardware components, lower costs, and achieve device miniaturization. Summary of the Invention

[0005] The purpose of this application is to provide a new technical solution for a sensing device and sensing method and electronic device that integrates TOF and structured light, so as to solve the problems of large size and high hardware cost of existing integrated sensing solutions.

[0006] In a first aspect, embodiments of this application provide a sensing device that fuses Time-of-Flight (TOF) and structured light, the sensing device comprising: A projector, operating in a time-division multiplexing manner under the control of a controller, is configured to selectively project a dot pattern for structured light sensing and a floodlight pattern for TOF sensing. A TOF receiving module is configured to operate during the projection period of the floodlight pattern to receive the floodlight pattern and generate first depth information; At least one structured light receiving module is configured to operate during the projection period of the dot pattern to receive the dot pattern and generate second depth information; and, The controller, which is communicatively connected to the projector, the TOF receiving module, and the structured light receiving module, is configured to execute the time-division multiplexing control. By synchronously switching the projection mode of the projector and the working state of the corresponding receiving module, the working period of the TOF receiving module coincides with the projection period of the floodlight pattern, and the working period of the structured light receiving module coincides with the projection period of the dot matrix pattern.

[0007] Optionally, the number of structured light receiving modules is one, constituting a monocular structured light sensing system; The controller is configured to: synchronously trigger the projector to project the dot pattern during the working period of the structured light receiving module, and generate the second depth information based on the received dot pattern by the structured light receiving module.

[0008] Optionally, the number of structured light receiving modules is two, forming a binocular structured light sensing system; The controller is configured to: synchronously trigger the projector to project the dot pattern during the working period of the two structured light receiving modules, and generate the second depth information by the two structured light receiving modules based on the dot pattern they receive through parallax calculation.

[0009] Optionally, the disparity calculation includes: The two structured light receiving modules acquire dot matrix patterns modulated by the target object from different perspectives to obtain two images; The disparity value is calculated by comparing the positional differences of corresponding points in the two images; and, The second depth information is determined based on the disparity value.

[0010] Optionally, the controller is configured to, through the time-division multiplexing control, ensure that at any given time, only one of the TOF receiving module and the floodlight pattern, and the structured light receiving module and the dot matrix pattern are in operation.

[0011] Optionally, the projector is a single projector with switchable modes, which can selectively project the dot pattern and the floodlight pattern under the control of the controller by changing its internal optical path.

[0012] Optionally, the projector includes a light source and an optical projection component disposed in the light output path of the light source; The projector also includes a switchable transparent sheet and a driving mechanism. The controller controls the driving mechanism to move the transparent sheet into or out of the optical path between the light source and the optical projection component, thereby switching the pattern output by the projector. Specifically, when the transparent sheet moves out of the light path, the dot matrix pattern is formed; when the transparent sheet moves into the light path, the floodlight pattern is formed.

[0013] Optionally, the optical projection assembly includes a collimating optical element and a diffractive optical element arranged sequentially along the optical path; or, the optical projection assembly is a collimating-diffractive integrated optical element or a metasurface lens.

[0014] Secondly, embodiments of this application provide a sensing method that fuses Time-of-Flight (TOF) and structured light, the sensing method comprising: The following steps are executed alternately using time-sharing multiplexing control: In the first time period, the projector is controlled to project a floodlight pattern, and a TOF receiving module is simultaneously controlled to work to obtain the first depth information; In the second time period, the projector is controlled to project a dot matrix pattern, and at least one structured light receiving module is simultaneously controlled to work to obtain second depth information; The first time period and the second time period alternate.

[0015] Thirdly, embodiments of this application provide an electronic device, which includes the TOF and structured light fusion sensing device described in the second aspect, and the electronic device is a smartphone, AR glasses, robot or smart door lock.

[0016] The beneficial effects of this application are as follows: The TOF and structured light fusion sensing device provided in this application embodiment achieves effective fusion of two depth sensing technologies by employing a single projector and time-division multiplexing it under the control of a controller to alternately project dot matrix patterns and floodlight patterns, in conjunction with a receiving module operating in corresponding time periods. The optical design of this application overcomes the hardware redundancy problem caused by using two independent projection systems in existing technologies, significantly reducing the number of components. This not only effectively compresses the device size but also greatly reduces manufacturing costs, making it possible to integrate high-performance 3D vision functions into space-constrained electronic devices.

[0017] Other features and advantages of this specification will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of this specification and, together with their description, serve to explain the principles of this specification.

[0019] Figure 1 This is one of the structural schematic diagrams of the sensing device that fuses TOF and structured light according to an embodiment of this application; Figure 2 This is the second schematic diagram of the sensor device that fuses TOF and structured light according to the embodiments of this application.

[0020] Explanation of reference numerals in the attached figures: 1. Projector; 2. TOF receiver module; 3. Structured light receiver module; 4. Controller; 11. Light source; 12. Transparent sheet; 13. Collimating optical element; 14. Diffractive optical element. Detailed Implementation

[0021] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0022] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the scope of this application and its application or use.

[0023] Technologies and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such technologies and equipment should be considered part of the specification.

[0024] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0025] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0026] The following description, in conjunction with the accompanying drawings, details the TOF and structured light fusion sensing device, sensing method, and electronic device provided in the embodiments of this application.

[0027] According to one embodiment of this application, a sensing device integrating TOF and structured light is provided, see [link to relevant documentation]. Figure 1 and Figure 2The sensing device includes: a projector 1, a Time-of-Flight (TOF) receiving module 2, at least one structured light receiving module 3, and a controller 4. The projector 1 operates in a time-division multiplexing mode under the control of the controller 4, configured to selectively project a dot matrix pattern for structured light sensing and a floodlight pattern for TOF sensing. The TOF receiving module 2 operates during the projection period of the floodlight pattern to receive the floodlight pattern and generate first depth information. The at least one structured light receiving module operates during the projection period of the dot matrix pattern to receive the dot matrix pattern and generate second depth information. The controller 4 is communicatively connected to the projector 1, the TOF receiving module 2, and the structured light receiving module 3, and is configured to execute the time-division multiplexing control. By synchronously switching the projection mode of the projector 1 and the operating state of the corresponding receiving module, the operating period of the TOF receiving module 2 coincides with the projection period of the floodlight pattern, and the operating period of the structured light receiving module 3 coincides with the projection period of the dot matrix pattern.

[0028] The TOF and structured light fusion sensing device provided in this application is primarily aimed at various electronic devices with urgent needs for 3D perception capabilities and strict requirements for device integration and cost control. Specifically, this sensing device can be integrated as a 3D vision module into mobile terminals such as smartphones and tablets to achieve high-precision facial recognition, secure payment, and gesture interaction functions; it can be applied to augmented reality / virtual reality (AR / VR) devices, providing them with reliable spatial perception and scene reconstruction capabilities; it can serve as a vision system for robots and intelligent devices, supporting key functions such as autonomous navigation, precise obstacle avoidance, and intelligent grasping; and it can be integrated into smart door locks, security monitoring, and other devices to achieve highly secure liveness detection and identity authentication. Through innovative hardware integration design and time-division multiplexing control logic, the sensing device provided in this application ensures high-performance 3D perception while meeting the requirements of modern electronic devices for compact and cost-effective sensing system structures.

[0029] According to one embodiment of this application, a sensing device integrating Time-of-Flight (TOF) and structured light is provided. See also... Figure 1 and Figure 2 The sensing device adopts an integrated architecture, comprising: a projector 1, a TOF receiving module 2, at least one structured light receiving module 3, and a controller 4 that coordinates the operation of each component. The components work collaboratively through the following configuration: the controller 4 establishes communication connections with the projector 1, the TOF receiving module 2, and the structured light receiving module 3 respectively; it controls the projector to alternately output different lighting patterns through a time-division multiplexing mechanism; and it synchronously schedules the corresponding receiving modules to collect data during corresponding time periods, thereby achieving deep fusion of two 3D sensing technologies and reuse of hardware resources.

[0030] The TOF and structured light fusion sensing device provided in this application embodiment achieves the fusion of two three-dimensional sensing technologies through an innovative system architecture. The core components and operating methods of this sensing device are described in detail below with reference to the accompanying drawings.

[0031] The TOF and structured light fusion sensing device provided in this application embodiment has a hardware architecture comprising four modules: a projector 1, which serves as the only light source emitting unit in the device and has the dual function of switching between dot matrix patterns and floodlight patterns; a TOF receiving module 2, which is used to receive floodlight signals and perform TOF depth calculation; a structured light receiving module 3, which is responsible for acquiring dot matrix patterns and performing structured light depth analysis, and its specific configuration and number can be flexibly set according to the needs of the application scenario; and a controller 4, which serves as the control core of the entire system and is responsible for coordinating the collaborative work of each module.

[0032] Regarding the working mechanism, the sensing device provided in this application embodiment adopts a specially designed time-division multiplexing control mechanism. The controller 4, through precise time sequence management, divides the sensing device's working cycle into two alternating periods: during the TOF working period, the controller 4 synchronously controls the projector 1 to output a floodlight pattern and simultaneously activates the TOF receiving module 2 to acquire and process signals, generating the first depth information; during the structured light working period, the controller 4 synchronously controls the projector 1 to switch to dot matrix pattern output and correspondingly activates the structured light receiving module 3 to acquire the second depth information. These two working periods alternate on the time axis and do not overlap, avoiding signal interference between different sensing modes and achieving efficient reuse of hardware resources.

[0033] The TOF and structured light fusion sensing device provided in this application embodiment achieves effective fusion of two depth sensing technologies by employing a single projector and time-division multiplexing it under the control of a controller to alternately project dot matrix patterns and floodlight patterns, in conjunction with a receiving module operating in corresponding time periods. The optical design of this application overcomes the hardware redundancy problem caused by using two independent projection systems in existing technologies, significantly reducing the number of components. This not only effectively compresses the device size but also greatly reduces manufacturing costs, making it possible to integrate high-performance 3D vision functions into space-constrained electronic devices.

[0034] See some examples in this application. Figure 1 The structured light receiving module 3 is one in number, forming a monocular structured light sensing system; the controller 4 is configured to: synchronously trigger the projector 1 to project the dot pattern during the working period of the structured light receiving module 3, and generate the second depth information by the structured light receiving module 3 based on the received dot pattern.

[0035] This example of the application provides a specific embodiment of the sensing device. For example... Figure 1 As shown, in this specific embodiment, the number of structured light receiving modules 3 is one, thus constituting a monocular structured light sensing system. This configuration scheme optimizes hardware costs by simplifying the receiver architecture while maintaining the accuracy of structured light technology.

[0036] See Figure 1 The sensor device structure shown allows for a compact side-by-side arrangement of components, such as projector 1, TOF receiver module 2, and structured light receiver module 3 arranged sequentially. In practical implementation, the placement of the TOF receiver module 2 can be flexibly adjusted according to actual needs such as optical design, heat dissipation, and circuit wiring. For example, it can be positioned between projector 1 and structured light receiver module 3, or placed in other suitable locations based on system integration requirements. This layout optimizes the optical path design of each module, reduces mutual optical interference, and provides a structural basis for device miniaturization.

[0037] In this application, the controller 4 performs synchronous control according to preset control logic within a working period specifically designated for structured light measurement: on the one hand, it triggers the projector 1 to output a preset dot matrix pattern, and on the other hand, it simultaneously activates the structured light receiving module 3 to enter the working state. This synchronous control ensures that the structured light receiving module 3 can completely acquire the dot matrix pattern modulated by the target object, providing reliable input data for subsequent depth calculation.

[0038] After acquiring the modulated dot matrix pattern, the structured light receiving module 3 calculates the scene's depth information by analyzing the deformation and displacement of feature points relative to a reference position, thus generating the second depth information. This process is based on the principle of monocular vision and does not rely on multi-view parallax calculation, reflecting the core characteristics of monocular structured light technology.

[0039] This implementation combines a single structured light receiving module 3 with a time-division multiplexing control mechanism to achieve a seamless integration of TOF and structured light sensing modes. This solution not only significantly reduces the number of hardware components, lowering system complexity and manufacturing costs, but also effectively controls the device size through a compact layout, providing a practical technical solution for integrating high-performance 3D vision functionality into space-constrained devices such as mobile terminals.

[0040] See some examples in this application. Figure 2The number of structured light receiving modules 3 is two, forming a binocular structured light sensing system; the controller 4 is configured to: synchronously trigger the projector 1 to project the dot pattern during the working period of the two structured light receiving modules 3, and generate the second depth information by the two structured light receiving modules 3 based on the dot pattern they receive through parallax calculation.

[0041] This example from the application provides another specific implementation of the sensing device. For example... Figure 2 As shown, in this specific embodiment, there are two structured light receiving modules 3, thus forming a binocular structured light sensing system. This configuration achieves higher precision depth perception through the collaborative operation of the two receiving modules.

[0042] In this configuration, the controller 4 performs synchronous control within the dedicated working period for structured light measurement, following a time-division multiplexing control mechanism. Specifically, the controller 4 simultaneously sends control signals to the projector 1 and the two structured light receiving modules 3: on the one hand, it triggers the projector 1 to output a preset dot matrix pattern; on the other hand, it simultaneously activates the two structured light receiving modules 3 to enter the data acquisition state. This synchronous control mechanism ensures that the two structured light receiving modules 3 can acquire the dot matrix pattern modulated by the target object from different spatial perspectives at the same time, providing accurate and reliable input data for subsequent parallax calculations.

[0043] See Figure 2 This specific implementation makes full use of the principle of binocular stereo vision, and acquires depth information through the spatial distribution of dual receiving modules. While maintaining the simplicity of the hardware architecture, it significantly improves the accuracy and reliability of three-dimensional perception.

[0044] See Figure 2 The sensor structure shown employs a side-by-side arrangement of components: the projector 1 and the TOF receiving module 2 are located in the middle, while the two structured light receiving modules 3 are positioned on either side. This specific positional relationship facilitates the optimization of the optical path design of each module and reduces mutual optical interference.

[0045] In some examples of this application, the disparity calculation includes the following steps: Step 100: The two structured light receiving modules 3 acquire dot matrix patterns modulated by the target object from different perspectives to obtain two images; Step 200: Calculate the disparity value by comparing the positional differences of corresponding points in the two images; and, Step 300: Determine the second depth information based on the disparity value.

[0046] This example details the specific implementation process of disparity calculation in a binocular structured light system. Based on stereo vision principles, this calculation process extracts depth information through the following three steps: Step 1: Simultaneous Acquisition of Dual-View Images. The two structured light receiving modules 3 can simultaneously acquire dot matrix patterns modulated by the surface topography of the target object from different spatial positions. Since there is a defined baseline distance between the two structured light receiving modules 3, the two images they acquire simultaneously contain parallax information, which provides a data foundation for subsequent depth calculation.

[0047] Step 2: Disparity Map Generation. A stereo matching algorithm is used to establish pixel-level correspondences between the two images, identifying corresponding pixels representing the same object point in both images. By calculating the horizontal positional deviation of these corresponding points in the image coordinate system, the system can calculate the disparity value pixel by pixel, ultimately generating a complete disparity map.

[0048] Step 3: Depth Information Conversion. For example, based on the principle of triangulation, the disparity values ​​obtained in Step 2 are converted into actual distance information. Specifically, based on pre-calibrated system parameters (including baseline distance, focal length, etc.), each pixel value in the disparity map can be converted into a corresponding depth value through geometric calculations, thereby generating high-precision second depth information.

[0049] This step-by-step parallax calculation method fully leverages the technological advantages of binocular stereo vision. Through image matching and geometric calculation, it achieves a reliable conversion from two-dimensional image information to three-dimensional depth data, ensuring the accuracy advantage of the binocular structured light system in depth perception.

[0050] See Figure 2 This binocular implementation combines the architectural advantages of the dual structured light receiver module 3 with a time-division multiplexing control mechanism, significantly improving the sensing performance of the device while inheriting the hardware simplification and cost advantages of a single projector. Compared to a monocular solution, this configuration improves the absolute accuracy of depth measurement and the reliability of 3D reconstruction through stereo vision principles. This feature makes the solution suitable for applications with stringent accuracy requirements, such as high-precision 3D inspection in industrial vision, fine modeling of complex objects, and robot guidance requiring stable depth perception.

[0051] In some examples of this application, the controller 4 is configured such that, through the time-division multiplexing control, only one of the TOF receiving module 2 and the floodlight pattern, and the structured light receiving module 3 and the dot matrix pattern are in working state at any given time.

[0052] This example of the application reveals the mechanism by which the controller 4 achieves coordinated system operation. The time-division multiplexing control is specifically configured as an exclusive operating mode: within any given time segment, the sensing device allows only one of the following combinations to be operational: either the combination of the TOF receiving module 2 with the floodlight pattern, or the combination of the structured light receiving module 3 with the dot matrix pattern.

[0053] This control mechanism is implemented through time-series management. The controller 4 divides the working timeline of the sensing device into continuous time periods, limiting the sensing device to operate in only a single sensing mode within each time period: specifically, activating the TOF sensing channel, in which case the projector 1 outputs a floodlight pattern and the TOF receiving module 2 operates synchronously; or activating the structured light sensing channel, in which case the projector 1 outputs a dot matrix pattern and the structured light receiving module 3 operates synchronously. The two working states alternate on the timeline without overlapping, ensuring isolation between different sensing modes from a temporal perspective.

[0054] The time-division control mechanism adopted in this application yields significant technical benefits, mainly in the following three aspects: First, the mechanism eliminates potential optical interference and signal crosstalk between Time-of-Flight (TOF) and structured light measurements through time isolation, ensuring the data purity of each sensing mode; second, the mechanism supports time-division multiplexing of a single projector's core optical hardware between the two depth sensing technologies, achieving intensive utilization of key hardware resources; finally, precise timing scheduling effectively reduces overall power consumption while maintaining system performance indicators. This control mechanism constitutes the core technical foundation for the synergistic development of hardware architecture simplification and system performance optimization in this application.

[0055] In some examples of this application, the projector 1 is a single projector with switchable modes, which can selectively project the dot pattern and the floodlight pattern under the control of the controller by changing its internal optical path.

[0056] The projector 1 provided in this application embodiment has two basic operating modes, and different optical patterns can be output by changing the internal optical path.

[0057] See some examples in this application. Figure 1 and Figure 2The projector 1 includes a light source 11 and an optical projection component disposed on the light output path of the light source 11; the projector also includes a switchable transparent sheet 12 and a driving mechanism, wherein the controller 4 controls the driving mechanism to move the transparent sheet 12 into or out of the light path between the light source 11 and the optical projection component, thereby switching the pattern output by the projector; wherein, when the transparent sheet 12 moves out of the light path, the dot matrix pattern is formed; when the transparent sheet 12 moves into the light path, the floodlight pattern is formed.

[0058] See Figure 1 and Figure 2 The optical projection component is fixedly disposed on the light output path of the light source 11, maintaining a stable relative position with the light source 11. The optical projection component performs two key optical functions: first, it collimates the diverging beam emitted by the light source 11, converting it into a collimated beam; then, it diffracts and replicates the collimated beam, using the principle of diffraction optics to decompose and replicate the incident beam into multiple coherent beams, thereby expanding the field of view of the output beam and forming the desired projection pattern.

[0059] The switchable transparent sheet 12 can be positioned on the light propagation path between the light source 11 and the optical projection assembly. Its key design feature is that it can be controlled to move in and out of the light path. The driving mechanism (such as an electromagnetically actuated slicer) is mechanically connected to the transparent sheet 12 and can control the transparent sheet 12 in two working states in the light path: moving out of the light path to ensure unobstructed light propagation, or moving into the light path to intervene in and modulate the propagation characteristics of the light beam.

[0060] This carefully designed optical structure layout forms the physical basis for realizing the dual-mode projection function. By controlling the position of the transparent sheet 12 in the optical path, the optical characteristics of the system are modulated, thereby achieving different projection effects.

[0061] The projector provided in this application embodiment introduces a switchable transparent sheet 12 between the light source 11 and the optical projection assembly. By moving the transparent sheet 12 into or out of the optical path between the light source 11 and the optical projection assembly, the optical path between the light source 11 and the optical projection assembly is changed. This allows the same light source 11 and the same optical projection assembly to output high-quality dot matrix patterns and uniform floodlight patterns respectively, without changing the axial physical distance between the light source 11 and the optical projection assembly. This mode-switching mechanism based on optical path modulation rather than physical displacement eliminates the reliance on precision displacement mechanisms such as piezoelectric ceramics and motors in existing technologies. It fundamentally solves the defects of existing solutions caused by mechanical displacement, such as complex structure, high manufacturing cost, stringent control precision requirements, and insufficient long-term reliability. It provides a simplified, stable, and more industrially viable two-in-one projection solution.

[0062] When the transparent sheet 12 moves out of the optical path, the light-emitting surface of the light source 11 is located at the object-side focal plane of the optical projection assembly; when the transparent sheet 12 moves into the optical path, the light-emitting surface of the light source 11 is equivalently located at the defocus position of the optical projection assembly.

[0063] The working principles of dot matrix mode and floodlight mode are described below.

[0064] (1) Working principle of dot matrix mode: When the sensing device needs to acquire high-precision depth information, the projector 1 operates in dot matrix mode. In this mode, the driving mechanism completely removes the transparent sheet 12 from the optical path between the light source 11 and the optical projection assembly. At this time, the emitting surface of the light source 11 is at the object-side focal plane of the optical projection assembly. After collimation and diffraction processing by the optical projection assembly, the diverging beam emitted by the light source 11 forms a clear speckle pattern with high contrast and a large divergence angle in the far-field interference, which is the desired dot matrix pattern.

[0065] (2) Working principle of floodlight mode: When the sensing device requires uniform illumination, the projector 1 switches to floodlight mode. In this mode, the drive mechanism moves the transparent sheet 12 into the optical path between the light source 11 and the optical projection assembly, see [link to relevant documentation]. Figure 1 and Figure 2 The introduction of the transparent sheet 12 alters the light path propagation characteristics based on the law of refraction, making the light source 11 optically equivalent to being located at the virtual focal position of the optical projection component. The light beam emitted by the light source 11 forms a diverging beam after passing through the optical projection component. Light spots of different orders fully overlap in the far field and eventually merge into a uniform illumination spot with uniform illuminance and a large divergence angle, i.e., the desired floodlight pattern.

[0066] By switching between the two working modes mentioned above, the projector 1 of this application realizes dual-mode output based on optical path modulation, and can complete the integration of dot matrix projection and floodlight illumination functions without changing the axial physical position of the optical elements.

[0067] In some examples of this application, the optical projection assembly includes a collimating optical element 13 and a diffractive optical element 14 arranged sequentially along the optical path; or, the optical projection assembly is a collimating-diffractive integrated optical element or a metasurface lens.

[0068] In one example, see Figure 1 and Figure 2 The optical projection assembly includes a collimating optical element 13 and a diffractive optical element 14 arranged sequentially along the optical path. The optical projection assembly adopts a discrete optical configuration, consisting of independent collimating optical elements 13 and diffractive optical elements 14 arranged sequentially along the optical path. The advantage of this discrete design is that the collimating optical element 13 is specifically used to collimate and shape the light beam emitted from the light source 11, converting it into a parallel beam with good directionality; the diffractive optical element 14 is specifically used to perform phase modulation and wavefront replication on the collimated beam, achieving angular expansion and pattern generation of the beam through diffraction. This specialized optical architecture ensures excellent optical performance.

[0069] In one example, the optical projection component can also be a collimating-diffraction integrated optical element or a metasurface lens. That is, the optical projection component can employ highly integrated optical elements. Specifically, it takes two forms: a collimating-diffraction integrated optical element or a metasurface lens.

[0070] The integrated collimation-diffraction optical element combines collimation and diffraction functions onto a single optical substrate, achieving both beam collimation and diffraction replication through precise optical surface design. This integrated design significantly reduces the number of optical elements and system size, while avoiding alignment errors during the assembly of multiple optical elements, thus improving system stability and production efficiency.

[0071] Metasurface lenses, as another implementation scheme, employ nanoscale subwavelength structures to achieve precise control of light waves. Through a carefully designed array of nanostructures on the surface, they can simultaneously achieve beam collimation and diffraction modulation within an extremely thin scale. This approach not only achieves extreme thinness of the system but also provides greater design freedom, facilitating the implementation of complex optical field manipulation functions.

[0072] Both of these highly integrated optical solutions effectively simplify the structure of the projector, reduce assembly complexity, and provide an ideal technical path for the miniaturization and mass production of the projector.

[0073] In some examples of this application, the surface of the transparent sheet 12 is coated with an antireflective film.

[0074] In this example of the application, an antireflective coating (or anti-reflective film) is designed to be coated on one or both surfaces of the transparent sheet 12. The antireflective coating can effectively suppress the reflection loss of light at the two interfaces of the transparent sheet 12 through precisely controlled interference effects.

[0075] The antireflective coating works on the principle of destructive interference in thin films. Specifically, when light passes through the surface of the antireflective coating, reflected light of a specific wavelength is significantly weakened due to destructive interference, thereby significantly improving light transmittance. This technical process enables the transparent sheet 12 to minimize the loss of incident light energy when it is moved into the corresponding optical path, ensuring that the projector can maintain sufficient output light power in floodlight mode.

[0076] Meanwhile, the application of antireflective coatings also brings additional optical optimization effects: on the one hand, it reduces stray light caused by multiple reflections and improves the signal-to-noise ratio of the output pattern; on the other hand, it reduces the energy consumption of the projector and improves the efficiency of light energy utilization.

[0077] The projector provided in this application possesses excellent output characteristics, with its dot pattern exhibiting a wide divergence angle and its floodlight pattern displaying a high level of uniformity. These characteristics collectively validate the effectiveness and technical advantages of the optical architecture of this application: the wide divergence angle ensures the coverage of 3D perception, while the high uniformity provides a high-quality illumination foundation for applications such as 2D imaging and face detection. This design enables the 3D projector to simultaneously meet the differentiated needs of depth perception and 2D vision, exhibiting good system adaptability and comprehensive performance.

[0078] This application details different implementation architectures of the sensing device through the following two typical embodiments.

[0079] Example 1 This embodiment provides a sensing device that fuses monocular Time-of-Flight (TOF) with structured light. See also Figure 1 The sensing device adopts an integrated design, including: a single projector 1, a single TOF receiver module 2, a single structured light receiver module 3, and a controller 4.

[0080] The projector 1 operates under the time-division multiplexing control of the controller 4, and can selectively output two optical patterns: a dot pattern for structured light sensing and a floodlight pattern for TOF sensing. The TOF receiving module 2 operates specifically during the floodlight pattern projection period, responsible for collecting light signals and generating the first depth information; the structured light receiving module 3 operates during the dot pattern projection period, generating the second depth information by collecting the dot pattern.

[0081] The controller 4, acting as the control core, enables the coordinated operation of all components through time-series management. Its control mechanism ensures that during the TOF measurement period, the projector 1 outputs a floodlight pattern while the TOF receiving module 2 operates synchronously; similarly, during the structured light measurement period, the projector 1 outputs a dot matrix pattern while the structured light receiving module 3 operates synchronously. These two operating periods alternate on the time axis without overlapping, fundamentally avoiding signal interference.

[0082] The projector 1 in this embodiment 1 adopts a single projector architecture with switchable optical path, including a light source 11, an optical projection assembly, a switchable transparent sheet 12, and a driving mechanism. The controller 4 controls the position state of the transparent sheet 12 through the driving mechanism: when the transparent sheet 12 moves out of the optical path, a high-quality dot matrix pattern is output; when the transparent sheet 12 moves into the optical path, a uniform floodlight pattern is output. This mode switching mechanism based on optical path modulation can achieve dual-mode output without changing the physical position of the optical elements.

[0083] This embodiment 1 achieves a highly simplified hardware architecture by combining monocular structured light configuration with time-division multiplexing control, while ensuring 3D perception performance. This solution significantly reduces the number of components and system complexity, providing a preferred implementation scheme for integrating high-performance 3D vision functionality into space-constrained mobile devices.

[0084] Example 2 This embodiment provides a sensing device that fuses binocular Time-of-Flight (TOF) and structured light. See also... Figure 2 The sensing device has an expanded architecture based on Embodiment 1, including: a single projector 1, a single TOF receiving module 2, two structured light receiving modules 3, and a controller 4.

[0085] The projector 1 operates under the time-division multiplexing control of the controller 4, and can selectively output two optical patterns: a dot pattern for structured light perception and a floodlight pattern for TOF perception. The TOF receiving module 2 operates specifically during the floodlight pattern projection period, responsible for collecting light signals and generating the first depth information; the two structured light receiving modules 3 operate synchronously during the dot pattern projection period, forming a binocular stereo vision system.

[0086] The controller 4, acting as the control core, enables the coordinated operation of all components through time-series management. Its control mechanism ensures that during the TOF measurement period, the projector 1 outputs a floodlight pattern while the TOF receiving module 2 operates synchronously; during the structured light measurement period, the projector 1 outputs a dot matrix pattern while the two structured light receiving modules 3 are activated synchronously. The two structured light receiving modules 3 synchronously acquire dot matrix patterns from different spatial perspectives, providing stereo image pairs for subsequent parallax calculations.

[0087] This binocular structured light system acquires high-precision depth information through disparity calculation. The specific processing steps include: First, two structured light receiving modules 3 respectively acquire dot matrix patterns modulated by the target object to obtain two images with disparity; then, the positional differences of corresponding points in the two images are calculated using a stereo matching algorithm to generate a disparity map; finally, based on the principle of triangulation, the disparity values ​​are converted into precise depth information to generate the second depth information.

[0088] This embodiment 2, through the combination of binocular structured light configuration and time-division multiplexing control, significantly improves the accuracy and reliability of depth measurement while maintaining a simple hardware architecture. This solution fully leverages the technical advantages of stereo vision and is suitable for applications requiring high measurement accuracy, such as industrial inspection and high-precision 3D reconstruction.

[0089] According to another embodiment of this application, a sensing method integrating TOF and structured light is provided, the sensing method comprising alternately executing the following steps through time-division multiplexing control: In the first time period, the projector 1 is controlled to project a floodlight pattern, and a TOF receiving module 2 is simultaneously controlled to work to obtain the first depth information; In the second time period, the projector 1 is controlled to project a dot matrix pattern, and at least one structured light receiving module 3 is controlled to work simultaneously to obtain second depth information; The first time period and the second time period alternate.

[0090] The sensing method provided in this application embodiment enables the collaborative operation of two depth sensing technologies based on a time-division multiplexing control mechanism. This sensing method divides the system's working cycle into alternating first and second time periods through time-series management, performing corresponding sensing operations within each time period.

[0091] The sensing method specifically includes the following alternating steps: In the first time period, the controller 4 synchronously executes projection control and reception control, drives the projector 1 to output a floodlight pattern, and simultaneously starts the TOF receiving module 2 to acquire and process signals, thereby obtaining the first depth information; In the second time period, the controller 4 synchronously controls the projector 1 to switch to dot matrix pattern output, and correspondingly starts at least one structured light receiving module 3 to work, and obtains the second depth information by acquiring and analyzing the dot matrix pattern.

[0092] The core of this sensing method lies in its timing control mechanism. The first and second time periods are alternated on the timeline without overlapping, ensuring that the system performs only a single type of depth sensing task at any given time. This timing arrangement not only guarantees that each sensing mode can obtain optimal signal conditions, but also fundamentally avoids mutual interference between different modes.

[0093] The sensing method provided in this application achieves efficient utilization of hardware resources through a time-division multiplexing mechanism, enabling a single projector 1 to be used for two different depth sensing needs. Simultaneously, synchronous control ensures the coordinated operation of each component within its corresponding time period.

[0094] According to yet another embodiment of this application, an electronic device is provided, including a sensing device that fuses TOF and structured light as described in any of the preceding claims, wherein the electronic device is a smartphone, AR glasses, a robot, or a smart door lock.

[0095] The specific implementation of the electronic device in this application can refer to the various embodiments of the sensing device that fuses TOF and structured light described above. Therefore, it has at least all the beneficial effects brought about by the technical solutions of the above embodiments, and will not be described in detail here.

[0096] The above embodiments mainly describe the differences between the various embodiments. As long as the different optimization features between the various embodiments are not contradictory, they can be combined to form a better embodiment. For the sake of brevity, they will not be elaborated here.

[0097] While specific embodiments of this application have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of this application. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this application. The scope of this application is defined by the appended claims.

Claims

1. A sensing device integrating TOF and structured light, characterized in that, include: A projector (1), under the control of a controller (4), operates in a time-division multiplexing manner and is configured to selectively project a dot pattern for structured light sensing and a floodlight pattern for TOF sensing. A TOF receiving module (2) is used to operate during the projection period of the floodlight pattern to receive the floodlight pattern and generate first depth information; At least one structured light receiving module (3) is used to operate during the projection period of the dot pattern to receive the dot pattern and generate second depth information; as well as, The controller (4), which is communicatively connected to the projector (1), the TOF receiving module (2) and the structured light receiving module (3), is configured to perform the time-division multiplexing control. By synchronously switching the projection mode of the projector (1) and the working state of the corresponding receiving module, the working period of the TOF receiving module (2) coincides with the projection period of the floodlight pattern, and the working period of the structured light receiving module (3) coincides with the projection period of the dot matrix pattern.

2. The sensing device according to claim 1, characterized in that, The number of structured light receiving modules (3) is one, which constitutes a monocular structured light sensing system; The controller (4) is configured to: synchronously trigger the projector (1) to project the dot pattern during the working period of the structured light receiving module (3), and generate the second depth information by the structured light receiving module (3) based on the received dot pattern.

3. The sensing device according to claim 1, characterized in that, The number of structured light receiving modules (3) is two, forming a binocular structured light sensing system; The controller (4) is configured to: synchronously trigger the projector (1) to project the dot pattern during the working period of the two structured light receiving modules (3), and generate the second depth information by the two structured light receiving modules (3) based on the dot pattern they receive through parallax calculation.

4. The sensing device according to claim 3, characterized in that, The disparity calculation includes: The two structured light receiving modules (3) respectively acquire the dot matrix pattern modulated by the target object from different perspectives to obtain two images; The disparity value is calculated by comparing the positional differences of corresponding points in the two images; and, The second depth information is determined based on the disparity value.

5. The sensing device according to claim 1, characterized in that, The controller (4) is configured to, through the time-division multiplexing control, ensure that at any given time, only one of the TOF receiving module (2) and the floodlight pattern, and the structured light receiving module (3) and the dot matrix pattern are in working state.

6. The sensing device according to any one of claims 1 to 5, characterized in that, The projector (1) is a single projector with switchable modes, which can selectively project the dot pattern and the floodlight pattern under the control of the controller by changing the internal optical path.

7. The sensing device according to claim 6, characterized in that, The projector (1) includes a light source (11) and an optical projection component disposed on the light output path of the light source (11); The projector also includes a switchable transparent sheet (12) and a driving mechanism. The controller (4) controls the driving mechanism to move the transparent sheet (12) into or out of the light path between the light source (11) and the optical projection component, thereby switching the pattern output by the projector. When the transparent sheet (12) moves out of the light path, the dot matrix pattern is formed; when the transparent sheet (12) moves into the light path, the floodlight pattern is formed.

8. The sensing device according to claim 7, characterized in that, The optical projection assembly includes a collimating optical element (13) and a diffractive optical element (14) arranged sequentially along the optical path; or, the optical projection assembly is a collimating-diffractive integrated optical element or a metasurface lens.

9. A sensing method integrating TOF and structured light, characterized in that, include: The following steps are executed alternately using time-sharing multiplexing control: In the first time period, the projector (1) is controlled to project a floodlight pattern, and a TOF receiving module (2) is simultaneously controlled to work to obtain the first depth information; During the second time period, the projector (1) is controlled to project a dot matrix pattern, and at least one structured light receiving module (3) is simultaneously controlled to work to obtain second depth information; The first time period and the second time period alternate.

10. An electronic device, characterized in that, The sensing device comprising the fusion of TOF and structured light as described in any one of claims 1 to 8, wherein the electronic device is a smartphone, AR glasses, a robot, or a smart door lock.

Images