A three-dimensional information measuring apparatus, measuring method, and electronic device

Abstract

The application provides a three-dimensional information measuring device, a measuring method and an electronic device, which are used for expanding the measuring depth of the three-dimensional information measuring device. The three-dimensional information measuring device comprises a light source module, an event sensing module and a processing module. The light source module is used for projecting at least two speckle images to a target object, and any two speckle images in the at least two speckle images correspond to different measuring depths. The event sensing module is used for collecting speckle images after the at least two speckle images are reflected by the target object, and obtaining a plurality of event images. The processing module is used for determining three-dimensional information of the target object according to the plurality of event images.

Description

Technical Field

[0001] This application relates to the field of optical technology, and in particular to a three-dimensional information measurement device, measurement method and electronic device. Background Technology

[0002] Three-dimensional (3D) reconstruction refers to establishing mathematical models of three-dimensional objects suitable for computer representation and processing. It forms the basis for processing, manipulating, and analyzing the properties of 3D objects in a computer environment and is a key technology for creating virtual reality representations of the objective world within a computer. 3D reconstruction can also be called 3D information reconstruction, and it has widespread demand in terminal devices. For example, 3D information reconstruction is widely used in motion-sensing devices, mobile phones, and autonomous vehicles. In motion-sensing devices, 3D information reconstruction obtains the 3D information of objects, thereby achieving a better gaming experience. Summary of the Invention

[0003] This application provides a three-dimensional information measurement device, measurement method, and electronic device. The three-dimensional information measurement device can project at least two speckle images corresponding to different measurement depths, thereby expanding the measurement depth.

[0004] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0005] In a first aspect, a three-dimensional information measurement device is provided, comprising: a light source module, an event sensing module, and a processing module; the light source module is used to project at least two speckle images onto a target object, wherein any two speckle images correspond to different measurement depths; the event sensing module is used to acquire speckle images after the at least two speckle images are reflected by the target object, thereby obtaining multiple event images; the processing module is used to determine the three-dimensional information of the target object based on the multiple event images.

[0006] In the above technical solution, the light source module in the 3D information measurement device can be used to project at least two speckle images onto a target object, wherein any two speckle images correspond to different measurement depths; the event sensing module can be used to acquire speckle images after the at least two speckle images are reflected by the target object, obtaining multiple event images; the processing module can be used to determine the 3D information of the target object based on the multiple event images. In this solution, because the at least two speckle images correspond to different measurement depths, target objects within different depth ranges can be measured using at least two speckle images, thereby completing the 3D information measurement of target objects within different measurement depths and expanding the depth measurement range of the 3D information measurement device.

[0007] In one possible implementation of the first aspect, the light source module includes a light projector for projecting the at least two speckle images at different focal lengths at different times. In this possible implementation, since the focal length of the speckle image is directly proportional to the measurement depth (i.e., the smaller the focal length of the speckle image, the smaller the corresponding measurement depth), the light projector projects the at least two speckle images at different focal lengths. By changing the focal length of the at least two speckle images, they correspond to different measurement depths, thus obtaining at least two speckle images corresponding to different measurement depths. When these at least two speckle images are used to measure target objects at different depths, the measurement range is expanded.

[0008] In one possible implementation of the first aspect, the light projector includes: a light emitter, a focal length adjustable collimating lens, and a diffractive optical element; the light emitter is used to emit a first light beam; the focal length adjustable collimating lens is used to adjust the divergence angle of the first light beam and to adjust the focal length of the first light beam to obtain at least two light beams with different measurement depths; the diffractive optical element is used to modulate the at least two light beams to obtain at least two speckle images. In the above possible implementation, the light projector projects the at least two speckle images with different focal lengths, and the focal length of each speckle image in the at least two speckle images is changed by the focal length adjustable collimating lens, so that the at least two speckle images correspond to different measurement depths. Target objects within different depth ranges can be measured using at least two speckle images, thus expanding the measurement range.

[0009] In one possible implementation of the first aspect, the light source module includes a distributed plurality of light projectors; the plurality of light projectors are used to project the at least two speckle images at different times and at different measurement depths. In the above possible implementation, when multiple projectors are used to project at least two speckle images, the measurement depth corresponding to each speckle image in the at least two speckle images is different. By projecting at least two speckle images at different times and at different measurement depths, the target object in different depth ranges can be measured, thus expanding the depth measurement range of the three-dimensional information measurement device.

[0010] In one possible implementation of the first aspect, each of the plurality of light projectors includes: a light emitter, a collimating lens with a fixed focal length, and a diffractive optical element; wherein the diffractive optical elements in different light projectors correspond to different measurement depths; the light emitter is used to emit a second beam; the collimating lens with a fixed focal length is used to adjust the divergence angle of the second beam; the diffractive optical element is used to modulate the second beam and set the measurement depth of the second beam to obtain a speckle image corresponding to the measurement depth, the speckle image being any one of the at least two speckle images. In the above possible implementation, by changing the diffractive optical element to change the measurement depth of the speckle image, when at least two speckle images are projected using multiple projectors, the measurement depth corresponding to each of the at least two speckle images is different. By projecting at least two speckle images at different times and different measurement depths, the target object within different depth ranges is measured, thus expanding the depth measurement range of the three-dimensional information measurement device.

[0011] In one possible implementation of the first aspect, the event sensing module is further configured to determine at least one event image pair based on the plurality of event images, wherein the distance between the two event images in the event image pair is less than a preset distance; the processing module is further configured to determine the three-dimensional information of the target object based on the at least one event image pair. In the above possible implementation, the event sensing module determines at least one event image pair from the plurality of event images, which facilitates the calculation of the subsequent processing module.

[0012] In one possible implementation of the first aspect, the at least two speckle images are projected sequentially according to their corresponding depth measurements. This possible implementation increases the diversity of the projection methods of the light source module.

[0013] In one possible implementation of the first aspect, the light source module is further configured to wait for a preset time after projecting any one of the at least two speckle images. In the above possible implementation, when the light source module projects any one of the at least two speckle images onto the target object, the preset time allows for noticeable pixel changes on the surface of the target object, facilitating the acquisition by the event sensing module.

[0014] In one possible implementation of the first aspect, the three-dimensional information of the target object is determined based on the at least one event image pair and the at least two speckle images. In the above possible implementation, determining the three-dimensional information of the target object based on the at least one event image pair and the at least two speckle images allows for the acquisition of three-dimensional information of the target object within different depth ranges, thus expanding the measurement range.

[0015] Secondly, a three-dimensional information measurement method is provided, the method comprising: projecting at least two speckle images onto a target object, wherein any two speckle images correspond to different measurement depths; acquiring speckle images after the at least two speckle images are reflected by the target object to obtain multiple event images; and determining the three-dimensional information of the target object based on the multiple event images.

[0016] In one possible implementation of the second aspect, the projection of at least two speckle images includes: projecting the at least two speckle images at different times with different focal lengths.

[0017] In one possible implementation of the second aspect, the method further includes: generating a first beam; adjusting the divergence angle of the first beam and adjusting the first beam to different focal lengths respectively to obtain at least two beams with different measurement depths; and modulating the at least two beams respectively to obtain the at least two speckle images.

[0018] In one possible implementation of the second aspect, the projection of at least two speckle images includes: projecting the at least two speckle images at different times and at different measurement depths.

[0019] In one possible implementation of the second aspect, at least two second beams are generated; the divergence angles of the at least two second beams are adjusted respectively; the at least two second beams are modulated respectively, and the measurement depths of the at least two second beams are set to obtain at least two speckle images with different measurement depths.

[0020] In one possible implementation of the second aspect, the method further includes: determining at least one event image pair based on the plurality of event images, wherein the distance between two event images in the event image pair is less than a preset distance; and determining the three-dimensional information of the target object based on the at least one event image pair.

[0021] In one possible implementation of the second aspect, the at least two speckle images are projected sequentially according to the magnitude relationship of the corresponding measured depth.

[0022] In one possible implementation of the second aspect, the method further includes: waiting for a preset time after projecting any one of the at least two speckle images.

[0023] In one possible implementation of the second aspect, any two of the at least two speckle images have the same brightness, shape, and density.

[0024] In one possible implementation of the second aspect, the method further includes: determining the three-dimensional information of the target object based on the at least one event image pair and the at least two speckle images.

[0025] Thirdly, an electronic device is provided, comprising: a three-dimensional information measuring device, wherein the three-dimensional information measuring device is provided by the first aspect or any possible implementation thereof.

[0026] Understandably, the three-dimensional information measurement method and electronic device provided above include all the contents of the three-dimensional information measurement device provided above. Therefore, the beneficial effects that it can achieve can be referred to the beneficial effects of the three-dimensional information measurement device provided above, and will not be repeated here. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the structure of an imaging system;

[0028] Figure 2 This is a schematic diagram of another imaging system.

[0029] Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0030] Figure 4 This is a schematic diagram of the structure of a three-dimensional information measurement device provided in an embodiment of this application;

[0031] Figure 5 This is a schematic diagram of the structure of a light projector provided in an embodiment of this application;

[0032] Figure 6 A schematic diagram showing the output signal of a light projector at different times, the focal length of a speckle image, and the measured depth, provided for embodiments of this application;

[0033] Figure 7 This is a schematic diagram of another light projector provided in an embodiment of this application;

[0034] Figure 8 A schematic diagram illustrating the output signals and depth measurement of multiple light projectors at different times, as provided in an embodiment of this application;

[0035] Figure 9 This is a flowchart illustrating a three-dimensional information measurement method provided in an embodiment of this application. Detailed Implementation

[0036] In this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. Furthermore, embodiments of this application use terms such as "first" and "second" to distinguish identical or similar items with essentially the same function and effect. For example, the first threshold and the second threshold are merely to distinguish different thresholds and do not limit their order. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order.

[0037] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0038] Before introducing the embodiments of this application, the relevant content of three-dimensional (3D) reconstruction will be introduced and explained first.

[0039] 3D reconstruction refers to establishing a mathematical model of a 3D object suitable for computer representation and processing. It is the foundation for processing, manipulating, and analyzing the properties of 3D objects in a computer environment, and a key technology for creating virtual reality representations of the objective world within a computer. 3D reconstruction can also be called 3D information reconstruction, and it has widespread demand in terminal devices. For example, 3D information reconstruction is widely used in motion-sensing devices, mobile phones, and autonomous vehicles. In motion-sensing devices, 3D information reconstruction obtains the 3D information of objects, thereby achieving a better gaming experience. 3D information is typically reconstructed using the following two methods.

[0040] Option 1: Reconstruct 3D information using an imaging system that includes a projector and a camera. Figure 1This is a schematic diagram of an imaging system, which includes a projector with an optical lens, a camera with an infrared sensor, and a processor. Specifically, the projector projects an infrared diffuse spot onto a 3D object through the optical lens. The camera collects light reflected from the 3D object at a fixed frame rate and generates an image. The camera sends the image to the processor, which calculates the depth of various locations on the 3D object based on the distortion degree of the diffuse spot in the image, thereby reconstructing the 3D information. Figure 1 The infrared sensor and processor of the structured light system are not shown in the diagram. However, due to the bandwidth interface limitation of the infrared sensor, i.e., the infrared sensor collects reflected light at a fixed frame rate, resulting in a low frame rate for the camera; on the other hand, the fixed structure projected by the projector leads to a limited measurement depth range corresponding to the projected diffuse spot, further resulting in a small measurement depth range for the imaging system.

[0041] Option 2: Reconstruct 3D information using an imaging system that includes a projector and an event camera. Figure 2 This is a schematic diagram of an imaging system, which includes: an event frame generation module with a projector and an event camera, and a processor with a digital correlation algorithm module and a depth reconstruction module. Specifically, the projector alternately projects random speckle and black fields onto the object to be scanned, causing changes in brightness on the object's surface. When the object's surface changes from dark to light or from light to dark, the event camera senses the change in light intensity, generating an event signal and outputting it as image frames. Image frames containing the object to be scanned are used as deformation frames, while image frames not containing the object are used as reference frames. The digital correlation algorithm module calculates the pixel offset (including horizontal and vertical pixel offsets) between the reference frame and the deformation frame. The depth reconstruction module calculates the depth at various locations on the object to be scanned based on the pixel offsets, the spatial positions of the event camera and projector, and calibrated parameters, thus completing 3D information reconstruction. However, the fixed structure of the projector in this scheme results in a limited measurement depth range corresponding to the projected random speckle, further limiting the measurement depth range of this structured light system.

[0042] Before introducing the embodiments of this application, we will first introduce and explain the relevant knowledge of structured light.

[0043] Structured light is a system consisting of a projector and a camera. Specifically, the projector projects specific light information onto the surface of the object being measured. The camera then captures the light reflected from the object's surface. Based on the changes in the light signal caused by the object, the system calculates the object's position and depth, thus reconstructing the object's entire three-dimensional space. Since the projection method of the light information in structured light is actively controlled by the projector, it can also be called active structured light.

[0044] Classification of optical information: The optical information projected by the projector can include point structured light, line structured light, and area structured light. Among them, point structured light uses laser dots projected by the light emitter as a pattern and is mainly used for distance measurement; line structured light uses lines projected by the light emitter as a pattern and is mainly used to obtain the cross-sectional contour of the object being measured; area structured light uses a two-dimensional point array composed of speckles projected by the light emitter or vertical bars including time-coded patterns as a pattern and is mainly used to obtain the three-dimensional information of the object being measured.

[0045] Event cameras are biomimetic sensors with microsecond-level response times that record changes in pixel brightness; in other words, they output the changes in pixel brightness. Traditional cameras output a complete image. Compared to traditional cameras, event cameras offer advantages such as low latency, high dynamic range (HDR), no motion blur, extremely low power consumption, and low data bandwidth.

[0046] This application provides a three-dimensional information measurement device, which may include a light source module, an event sensing module, and a processing module. This device can be integrated into an electronic device for measuring target objects within different depth ranges, thus expanding the measurement range. The electronic device may include a server and a processor, or it may include a terminal device. The following description uses an example where the electronic device includes a terminal device.

[0047] This terminal device includes, but is not limited to, personal computers, server computers, mobile devices (such as mobile phones, tablets, media players, etc.), wearable devices, in-vehicle devices, consumer terminal devices, mobile robots, and drones. The specific structure of this terminal device is described below.

[0048] Figure 3 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application, using a mobile phone as an example for illustration. Figure 3 As shown, the terminal device may include: a memory 101, a processor 102, a sensor assembly 103, a multimedia assembly 104, a power supply 105, and an input / output interface 106.

[0049] The memory 101 can be used to store data, software programs, and software modules; it mainly includes a program storage area and a data storage area. The program storage area can store the operating system and application programs required for at least one function, such as sound playback or image playback. The data storage area can store data created according to the use of the terminal device, such as audio data, image data, or spreadsheet data. In addition, the terminal device may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0050] The processor 102 is the control center of the terminal device, connecting various parts of the device through various interfaces and lines. It executes software programs and / or software modules stored in the memory 101, and calls data stored in the memory 101 to perform various functions and process data, thereby providing overall monitoring of the terminal device. Optionally, the processor 102 may include one or more processing units. For example, the processor 102 may include a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. In this embodiment, the processor 102 may include the processing modules described herein.

[0051] Sensor assembly 103 includes one or more sensors for providing various aspects of the terminal device's status assessment. Sensor assembly 103 may include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor. Sensor assembly 103 can detect the terminal device's acceleration / deceleration, orientation, on / off state, relative positioning of components, or temperature changes, etc. Furthermore, sensor assembly 103 may also include a light sensor for detecting ambient light. In this embodiment, sensor assembly 103 may also include a bionic sensor and a light sensor. The bionic sensor can be used to sense changes in pixel brightness on the surface of the measured object, and the light sensor can be used to emit a light source, etc.

[0052] Multimedia component 104 provides a screen that serves as an output interface between the terminal device and the user. This screen can be a touch panel, and when it is a touch panel, it can be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. Furthermore, multimedia component 104 includes at least one camera, for example, a front-facing camera and / or a rear-facing camera. When the terminal device is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera can receive external multimedia data. Each front-facing and rear-facing camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

[0053] It should be noted that the event sensing module described below may include the bionic sensor in sensor assembly 103 and the camera in multimedia assembly 104. This event sensing module can be used to acquire information about changes in pixel brightness. The light source module described below may include the light sensor in sensor assembly 103 and the camera in multimedia assembly 104. This light source module can emit speckle images.

[0054] Power supply 105 is used to provide power to the various components of the terminal device. Power supply 105 may include a power management system, one or more power supplies, or other components associated with the generation, management and distribution of power by the terminal device.

[0055] Input / output interface 106 provides an interface between processor 102 and peripheral interface modules, such as keyboards, mice, or universal serial bus (USB) devices.

[0056] Although not shown, the terminal device may also include audio components and communication components, such as a microphone for the audio component and a wireless fidelity (WiFi) module or a Bluetooth module for the communication component. These will not be elaborated further in the embodiments of this application. Those skilled in the art will understand that... Figure 3 The terminal device structure shown does not constitute a limitation on the terminal device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0057] Figure 4This is a schematic diagram of a three-dimensional information measurement device provided in an embodiment of this application. The three-dimensional information measurement device includes: a light source module 01, an event sensing module 02, and a processing module 03. The light source module 01 can be used to project at least two speckle images onto a target object 04, wherein any two speckle images correspond to different measurement depths. The event sensing module 02 can be used to acquire speckle images after the at least two speckle images are reflected by the target object 04, obtaining multiple event images. The processing module 03 can be used to determine the three-dimensional information of the target object 04 based on the multiple event images.

[0058] In this embodiment, any one of the at least two speckle images can be a two-dimensional dot array composed of speckles (hereinafter referred to as a speckle array) or vertical bars with time-series coding. The following embodiments use a speckle array as an example for illustration.

[0059] Furthermore, the at least two speckle images may include two or more speckle images; for example, the speckle images may include two speckle images or four speckle images. The measured depths corresponding to any two of the at least two speckle images are different.

[0060] In addition, the depth is measured as the straight-line distance between the light source module 01 and the target object 04.

[0061] In this embodiment, the light source module 01 is further configured to wait for a preset time after projecting any one of the at least two speckle images. For example, the at least two speckle images include a first speckle image and a second speckle image. The light source module 01 is further configured to wait for a preset time after projecting the first speckle image before projecting the second speckle image. During the projection of the first speckle image, the pixels on the surface of the target object 04 change from dark to bright. After the projection of the first speckle image stops (i.e., after waiting for the preset time), the pixels on the surface of the target object 04 change from bright to dark.

[0062] The preset duration can be determined based on actual needs and the experience of relevant technical personnel; however, this application embodiment does not impose specific limitations on it.

[0063] Additionally, the event sensing module 02 may include an event camera, which may include a bionic sensor and a camera. A description of the bionic sensor and camera can be found in [reference needed]. Figure 3 The relevant descriptions of the bionic sensor and camera in the original text will not be repeated here.

[0064] Secondly, for each of the at least two speckle images, the event sensing module 02 is used to acquire the speckle image after reflection from the target object 04, thereby obtaining one or more event images, thus obtaining multiple event images corresponding to the at least two speckle images. Specifically, when the pixel brightness on the surface of the target object 04 changes from dark to bright or from bright to dark, the event sensing module 02 can be used to acquire the speckle image after reflection from the target object 04, thereby obtaining multiple event images.

[0065] Secondly, each of the multiple event images includes event image information. This information may include location coordinates (X, Y), a timestamp T, and event image polarity. The location coordinates (X, Y) include the location coordinates (X, Y) of each speckle in the speckle array. The timestamp T is the time when the event sensing module 02 acquired the event image. The event image polarity indicates whether the event image is a positive or negative event image; a positive event image can be represented as P+, and a negative event image as P-. Specifically, a positive event image is an event image generated when the pixels on the surface of the target object 04 change from dark to light, and a negative event image is an event image generated when the pixels on the surface of the target object 04 change from light to dark.

[0066] Furthermore, the light source module 01 may include a single light projector or multiple distributed light projectors. The structure of the light projectors also differs depending on the number of light projectors included in the light source module 01. The following section discusses... Figure 6 and Figure 7 The following descriptions will cover two scenarios: the light source module 01 includes a single light projector, and the light source module 01 includes multiple distributed light projectors. In the following embodiments, the at least two speckle images include a first speckle image and a second speckle image as an example.

[0067] In a first possible embodiment, the light source module 01 includes a light projector for projecting the at least two speckle images at different times and with different focal lengths. In one example, such as Figure 5 As shown, the light projector includes: a light emitter, a collimating lens with adjustable focal length, and diffractive optical elements.

[0068] The light emitter can be used to emit a first beam, which is the base beam for generating the speckle pattern. This first beam typically has a large divergence angle. The divergence angle indicates the degree of divergence of the beam; a large divergence angle indicates that the beam is more divergent, making it difficult to focus the beam onto a single point. In one possible example, the light emitter can be a laser.

[0069] The focal length adjustable collimating lens can be used to adjust the divergence angle of the first beam, that is, to adjust the first beam to a beam with a smaller divergence angle, so that the beam after adjusting the divergence angle can be matched with the size of the diffractive optical element, thereby enabling the speckle image to be successfully focused to the desired position. The focal length adjustable collimating lens is also used to adjust the first beam to different focal lengths to obtain at least two beams with different measurement depths. The focal length is proportional to the measurement depth; when the beam has a smaller focal length, the measurement depth corresponding to that beam is smaller. For example, the focal length adjustable collimating lens is used to adjust the first beam to a first focal length F1 to obtain a first beam with a measurement depth D1, and to adjust the first beam to a second focal length F2 to obtain a second beam with a measurement depth D2, where D1 and D2 are different.

[0070] The diffractive optical element is used to modulate the at least two light beams respectively, thereby obtaining the at least two speckle images. For example, the diffractive optical element can modulate the first light beam and the second light beam respectively, so that both the first light beam and the second light beam have characteristic information, such as speckle shape, speckle size, array size, and light intensity, thereby obtaining the first speckle image and the second speckle image. Specifically, the diffractive optical element modulates the light beam through a micro / nano structured lens, thereby giving the light beam characteristic information.

[0071] For example, Figure 6 This diagram illustrates the output signal of the light projector at different times, the focal length of the speckle image, and the measured depth. Figure 6 (a) in the diagram is a schematic diagram of the output signal versus time. Figure 6 (b) in the diagram is a schematic diagram of the focal length versus time for the corresponding speckle image. Figure 6 (c) in the diagram is a schematic representation of different measurement depths corresponding to speckle images with different focal lengths. For example... Figure 6 As shown in (a), the projector generates an output signal at time t1, which corresponds to the projection of the first speckle image; at time t2, the projector generates an output signal, which corresponds to the projection of the second speckle image, and so on until at time tn, the projector generates an output signal, which corresponds to the projection of the nth speckle image, and the time interval between each two adjacent output signals is T. Figure 6 As shown in (b), the focal length corresponding to the first speckle image is F1, and the corresponding measurement depth is D1; ​​the focal length corresponding to the second speckle image is F2, and the corresponding measurement depth is D2; the focal length corresponding to the nth speckle image is Fn. Figure 6 As shown in (c), the measurement depth corresponding to the speckle image with focal length F1 is D1, the measurement depth corresponding to the speckle image with focal length F2 is D2, and the measurement depth corresponding to the speckle image with focal length Fn is Dn, where D1 is greater than D2 and D2 is greater than Dn. Figure 6 No speckle image is shown.

[0072] Since the measurement depth (also known as the focusing distance) of the speckle image is determined by the focal length of the collimating lens and the diffractive optical element, in this embodiment, the light projector projects at least two speckle images at different times with different focal lengths. By changing the focal length of the adjustable collimating lens, the measurement depth of different speckle images is changed. By projecting at least two speckle images at different times with different focal lengths, the target object in different depth ranges is measured, thus expanding the depth measurement range of the three-dimensional information measurement device.

[0073] In a second possible embodiment, the light source module 01 includes a distributed plurality of light projectors for projecting the at least two speckle images at different times and at different measurement depths. Each of the plurality of light projectors includes a light emitter, a collimating lens with a fixed focal length, and a diffractive optical element. The measurement depths corresponding to the diffractive optical elements in different light projectors are different; that is, the micro / nano-structured lenses in the diffractive optical elements of different light projectors are different, resulting in different measurement depths corresponding to the diffractive optical elements.

[0074] For example, Figure 7 This is a schematic diagram of a light projector, which includes a light emitter, a collimating lens with a fixed focal length, and a diffractive optical element.

[0075] The light emitter can be used to emit a second beam, which is the base beam for generating the speckle image. This second beam typically has a large divergence angle.

[0076] A collimating lens with a fixed focal length is used to change the divergence angle of the second beam, that is, to adjust the second beam to a beam with a smaller divergence angle.

[0077] The diffractive optical element can be used to modulate the second beam and set the measurement depth of the second beam to obtain a speckle image corresponding to the measurement depth. The speckle image is any one of the at least two speckle images.

[0078] In this embodiment, the measurement depths corresponding to the diffractive optical elements in different light projectors are different. That is, the micro-nano structures of the lenses in the diffractive optical elements in different light projectors are different, resulting in different measurement depths corresponding to the diffractive optical elements. However, the diffractive optical elements in different light projectors only change the corresponding measurement depth; the beam modulation effect remains unchanged. In other words, the beams passing through different diffractive optical elements have the same characteristic information.

[0079] For example, Figure 8This is a schematic diagram showing the output signals and measured depth of the multiple light projectors at different times. Figure 8 (a) in the diagram is a schematic diagram of the output signal of the projector versus time. Figure 8 (b) in the diagram illustrates different measurement depths corresponding to different speckle images. Figure 8 As shown in (a), at time t1, projector L1 generates an output signal corresponding to the projection of the first speckle image; at time t2, projector L2 generates an output signal corresponding to the projection of the second speckle image; and at time tn, projector Ln generates an output signal corresponding to the output of the nth speckle image. The time interval between any two adjacent output signals is T. Figure 8 As shown in (b), the measurement depth corresponding to the first speckle image is D1, the measurement depth corresponding to the second speckle image is D2, and the measurement depth corresponding to the nth speckle image is Dn. Figure 8 (b) takes D1 greater than D2 and D2 greater than Dn as an example.

[0080] In this embodiment, since the diffractive optical elements in different light projectors are different, the measurement depth of the speckle image is changed by changing the diffractive optical elements. Thus, when at least two speckle images are projected by multiple projectors, the measurement depth corresponding to each speckle image in the at least two speckle images is different. By projecting at least two speckle images at different times and at different measurement depths, the target object in different depth ranges is measured, thereby expanding the depth measurement range of the three-dimensional information measurement device.

[0081] Optionally, the at least two speckle patterns are projected sequentially according to their corresponding measurement depths. For example, the at least two speckle patterns can be projected in descending order of their corresponding measurement depths, or in ascending order of their corresponding measurement depths. Figure 6 and Figure 8 The following example illustrates how at least two speckle patterns are projected according to the relationship between their corresponding measurement depths, from largest to smallest.

[0082] In practical applications, the above Figure 6 and Figure 7 The light source module 01 shown may include, in addition to the light emitter, collimating lens and diffractive optical element mentioned above, a substrate, which can be used to encapsulate the light projector.

[0083] Furthermore, any two speckle images in the at least two speckle images have the same brightness, shape, and density. The feature information of each speckle image in the at least two speckle images can be determined by the diffractive optical element. Therefore, the feature information of any two speckle images in the at least two speckle images can be the same when emitted in the projector; for example, any two speckle images in the at least two speckle images have the same brightness, shape, and density. However, when the at least two speckle images are projected onto a target object at different measurement depths, the feature information of the reflected speckle images formed after reflection from the target object is different. For example, the density of the reflected speckle image corresponding to the speckle image with a smaller measurement depth is less than the density of the reflected speckle image corresponding to the speckle image with a larger measurement depth.

[0084] Furthermore, the event perception module 02 is also used to determine at least one event image pair based on the plurality of event images, wherein the distance between the two event images in the event image pair is less than a preset distance; the processing module 03 is also used to determine the three-dimensional information of the target object based on the at least one event image pair.

[0085] In one possible embodiment, the event sensing module 02 traverses multiple event images within a preset time T1 (where the preset time T1 is greater than the projection interval T). These multiple event images can be generated from a single speckle image or from multiple speckle images. The multiple event images include a first event image and a second event image. If the event image information of the first event image includes (X1, Y1), T1, and P+, and the event image information of the second event image includes (X2, Y2), T2, and P-, and the absolute value of the difference between X1 and X2 is less than d1, and the absolute value of the difference between Y1 and Y2 is less than d2, then the first event image and the second event image are an event image pair. Here, (X1, Y1) and (X2, Y2) are the position coordinates of any speckle in the first and second event images, respectively. In this embodiment, the event sensing module 02, by traversing the multiple event images, determines at least one event image pair among them, improving the quality of the event images and facilitating the calculations of the subsequent processing module 03.

[0086] Optionally, the event perception module 02 is also used to delete time images (also known as isolated event images) from the multiple event images other than the event image itself. By deleting isolated event images from the multiple event images, the quality of the event images is improved. When performing calculations on the multiple event images with the isolated event images deleted, the computational efficiency of the processing module 03 is improved, as well as the accuracy of the three-dimensional information of the target object 04.

[0087] Optionally, before projecting at least two speckle images, the light source module 01 can be calibrated to obtain its parameter information. For example, this parameter information may include the position of the light source module 01. Specific calibration methods can be found in existing technologies and will not be elaborated here.

[0088] Similarly, before the event sensing module 02 acquires the speckle image after reflection from the target object from the at least two speckle images, parameter calibration can be performed on the event sensing module 02 to obtain its parameter information and perform distortion correction. This parameter information may include internal and external parameters of the event sensing module 02. Internal parameters may include the focal length and pixel size of the event sensing module 02. External parameters may include the position and rotation direction of the event sensing module 02. Specific calibration methods can be found in existing technologies and will not be elaborated here.

[0089] Furthermore, the event perception module 02 is also used to acquire reference images corresponding to at least two speckle images. The reference images are the images acquired by the event perception module 02 when the at least two speckle images are projected onto a plane at the corresponding measurement depth and where there is no target object.

[0090] Furthermore, the processing module 02 is also used to: determine the three-dimensional information of the target object 04 within different measurement depths based on at least one event image pair among the plurality of event images, the reference image corresponding to the at least two speckle images, the calibration parameters of the event perception module 02, and the calibration parameters of the light source module 01. Specifically, taking a target object 04 within a measurement depth as an example, the processing module 03 extracts the local binary pattern (LBP) of each event image in the at least one event image pair and the LBP in the reference image corresponding to the speckle image. The LBP of each event image in the at least one event image pair includes the temporal image information of the temporal image. A matching cost is calculated using an improved Jaccard similarity coefficient distance to obtain a depth image, and the three-dimensional information of the target object is determined based on the information of each pixel in the depth image. The process by which the processing module 03 determines the three-dimensional information of the object based on the at least one event image pair, the reference image corresponding to the speckle image, the calibration parameters of the event perception module, and the calibration parameters of the light source module is similar to existing technologies, and can be referred to existing technologies for details, which will not be elaborated here.

[0091] In the three-dimensional information measurement device provided in this application embodiment, the light source module can be used to project at least two speckle images onto a target object, wherein any two speckle images correspond to different measurement depths; the event sensing module can be used to acquire speckle images after the at least two speckle images are reflected by the target object, obtaining multiple event images; the processing module can be used to determine the three-dimensional information of the target object based on the multiple event images. In this scheme, since the at least two speckle images correspond to different measurement depths, target objects within different depth ranges can be measured using at least two speckle images to complete the three-dimensional information measurement of target objects within different measurement depths, thereby expanding the depth measurement range of the three-dimensional information measurement device.

[0092] Based on the three-dimensional information measurement device mentioned above, Figure 9 This is a flowchart illustrating a three-dimensional information measurement method provided in an embodiment of this application. The method may include the following steps.

[0093] S901: Project at least two speckle images onto the target object, wherein any two speckle images correspond to different measurement depths.

[0094] In this context, any one of the at least two speckle images can be a two-dimensional dot array composed of speckles, or a vertical bar with temporal coding.

[0095] In addition, the at least two speckle images may include two or more speckle images, wherein any two speckle images correspond to different measurement depths.

[0096] Furthermore, the projection method of the at least two speckle images can include two projection methods, which will be described below.

[0097] In a first possible embodiment, the at least two speckle images are projected at different times with different focal lengths. In this embodiment, the at least two speckle images are projected at different times with different focal lengths, meaning that any two speckle images have different focal lengths, resulting in different measurement depths corresponding to these two speckle images. This allows for the measurement of target objects within different depth ranges using the at least two speckle images, thereby expanding the measurement depth.

[0098] In a second possible embodiment, the at least two speckle images are projected at different times and at different measurement depths. In this embodiment, the at least two speckle images are projected at different times and at different measurement depths, meaning that any two of the at least two speckle images correspond to different measurement depths. This allows for the measurement of target objects within different depth ranges using the at least two speckle images, thereby expanding the measurement depth.

[0099] Optionally, the at least two speckle patterns are projected sequentially according to their corresponding measurement depths. For example, the at least two speckle patterns can be projected in descending order of their corresponding measurement depths, or in ascending order of their corresponding measurement depths.

[0100] Furthermore, any two speckle images in the at least two speckle images have the same feature information; for example, any two speckle images in the at least two speckle images have the same brightness, shape, and density.

[0101] S902: Acquire speckle images after the at least two speckle images are reflected by the target object, and obtain multiple event images.

[0102] Specifically, when the pixel brightness on the surface of the target object changes, the speckle image after reflection from the target object is acquired, thus obtaining multiple event images. Specifically, when the pixel brightness on the surface of the target object changes from dark to bright, or from bright to dark, the speckle image after reflection from the target object is acquired, thus obtaining multiple event images.

[0103] Secondly, each of the multiple event images includes event image information. This information may include location coordinates (X, Y), a timestamp T, and event image polarity. The location coordinates (X, Y) include the location coordinates (X, Y) of each speckle in the speckle array. The timestamp T is the time when the event sensing module 02 acquired the event image. The event image polarity indicates whether the event image is a positive or negative event image; a positive event image can be represented as P+, and a negative event image can be represented as P-.

[0104] Furthermore, the method further includes: determining at least one event image pair based on the plurality of event images, wherein the distance between the two event images in the event image pair is less than a preset distance; and determining the three-dimensional information of the target object based on the at least one event image pair.

[0105] For example, multiple event images within a preset time T1 (where the preset time T1 is greater than the projection interval T) are traversed. These multiple event images include a first event image and a second event image. If the event image information of the first event image includes (X1, Y1), T1, and P+, and the event image information of the second event image includes (X2, Y2), T2, and P-, and the absolute value of the difference between X1 and X2 is less than d1, and the absolute value of the difference between Y1 and Y2 is less than d2, then the first event image and the second event image are an event image pair. Here, (X1, Y1) and (X2, Y2) are the position coordinates of any speckle in the first event image and the second event image, respectively. This event perception module, by traversing these multiple event images, determines at least one event image pair among them, improving the quality of the event images and facilitating the calculations of subsequent processing modules.

[0106] Furthermore, the method further includes deleting time images (also known as isolated event images) from the plurality of event images other than the event image itself. By deleting isolated event images from the plurality of event images, the quality of the event images is improved. When performing calculations on the plurality of event images after deleting isolated event images, the computational efficiency of the processing module is improved, as well as the accuracy of the three-dimensional information of the target object is increased.

[0107] S903: Determine the three-dimensional information of the target object based on the multiple event images.

[0108] Optionally, the three-dimensional information of the target object is determined based on at least one event image pair from the plurality of event images and a reference image corresponding to the at least two speckle images. The reference image is an image acquired when the at least two speckle images are projected onto a plane at the corresponding measurement depth, where there is no target object.

[0109] Specifically, the local binary pattern (LBP) of each image in the at least one event image pair and the LBP of the reference image corresponding to the speckle image are extracted. The LBP of each event image in the at least one event image pair includes the temporal image information of that time image. A matching cost is calculated using an improved Jaccard similarity coefficient distance to obtain a depth image. The 3D information of the target object is determined based on the information of each pixel in the depth image. The process of determining the 3D information of the object is similar to existing techniques and can be referred to therein; it will not be elaborated here.

[0110] Furthermore, the method also includes: after projecting any one of the at least two speckle images, waiting for a preset time. Specifically, the at least two speckle images include a first speckle image and a second speckle image. After projecting the first speckle image, waiting for the preset time, and then projecting the second speckle image. Wherein, after projecting the first speckle image, the pixels on the surface of the target object change from dark to bright, and the resulting event image is a positive event image. After stopping the projection of the first speckle image (i.e., waiting for the preset time), the pixels on the surface of the target object change from bright to dark, and the resulting event image is a negative event image.

[0111] The preset duration can be determined based on actual needs and the experience of relevant technical personnel; however, this application embodiment does not impose specific limitations on it.

[0112] This application provides a method for measuring three-dimensional information. It involves projecting at least two speckle images onto a target object, where any two speckle images correspond to different measurement depths. The method then acquires speckle images reflected from the target object, resulting in multiple event images. Based on these event images, the three-dimensional information of the target object is determined. Because the at least two speckle images correspond to different measurement depths, the method allows for the measurement of target objects at different depth ranges using at least two speckle images, thereby expanding the depth measurement range of the three-dimensional information measuring device.

[0113] In another aspect, this application provides an electronic device including a three-dimensional information measuring device that performs the relevant steps in the above-described method embodiments. This electronic device can be... Figure 3 The controller provided in [the document / platform].

[0114] It should be noted that the above detailed description of the three-dimensional information measurement device can be applied to the corresponding embodiments of the three-dimensional information measurement method and the electronic device, and the embodiments of this application will not be repeated here.

[0115] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A three-dimensional information measurement device, characterized in that, The device includes: a light source module, an event sensing module, and a processing module; The light source module is used to project at least two speckle images onto the target object, wherein any two speckle images correspond to different measurement depths. The event sensing module is used to acquire speckle images after the at least two speckle images are reflected by the target object, thereby obtaining multiple event images; the event sensing module is also used to determine at least one event image pair based on the multiple event images, wherein the distance between the two event images in the event image pair is less than a preset distance; The processing module is used to determine the three-dimensional information of the target object based on the at least one event image pair.

2. The apparatus according to claim 1, characterized in that, The light source module includes a light projector; The light projector is used to project the at least two speckle images at different times and at different focal lengths.

3. The apparatus according to claim 2, characterized in that, The light projector includes: a light emitter, a collimating lens with adjustable focal length, and a diffractive optical element; The light emitter is used to emit a first light beam; The collimating lens with adjustable focal length is used to adjust the divergence angle of the first beam and to adjust the focal length of the first beam to obtain at least two beams with different measurement depths. The diffractive optical element is used to modulate the at least two light beams respectively to obtain the at least two speckle images.

4. The apparatus according to claim 1, characterized in that, The light source module includes multiple distributed light projectors; The plurality of light projectors are used to project the at least two speckle images at different times and at different measurement depths.

5. The apparatus according to claim 4, characterized in that, Each of the plurality of light projectors includes: a light emitter, a collimating lens with a fixed focal length, and a diffractive optical element; wherein the diffractive optical elements in different light projectors correspond to different measurement depths; The light emitter is used to emit a second beam of light; The collimating lens with a fixed focal length is used to adjust the divergence angle of the second beam; The diffractive optical element is used to modulate the second beam and set the measurement depth of the second beam to obtain a speckle image corresponding to the measurement depth, wherein the speckle image is any one of the at least two speckle images.

6. The apparatus according to any one of claims 1-5, characterized in that, The at least two speckle images are projected sequentially according to the magnitude relationship of the corresponding measured depths.

7. The apparatus according to any one of claims 1-6, characterized in that, The light source module is also used to wait for a preset time after projecting any one of the at least two speckle images.

8. The apparatus according to any one of claims 1-7, characterized in that, The brightness, shape, and density of any two of the at least two speckle images are the same.

9. The apparatus according to any one of claims 6-8, characterized in that, The processing module is also used for: The three-dimensional information of the target object is determined based on the at least one event image pair and the at least two speckle images.

10. A three-dimensional information measurement method, characterized in that, The method includes: At least two speckle images are projected onto a target object, wherein any two speckle images correspond to different depth measurements. Acquire speckle images after reflection from the target object from the at least two speckle images to obtain multiple event images; determine at least one event image pair based on the multiple event images, wherein the distance between the two event images in the event image pair is less than a preset distance; The three-dimensional information of the target object is determined based on the at least one event image pair.

11. The method according to claim 10, characterized in that, The projection of at least two speckle images includes: The at least two speckle images are projected at different times with different focal lengths.

12. The method according to claim 11, characterized in that, The method further includes: Generate the first beam; Adjust the divergence angle of the first beam and adjust the focal length of the first beam to obtain at least two beams with different measurement depths. The at least two light beams are modulated respectively to obtain the at least two speckle images.

13. The method according to claim 10, characterized in that, The projection of at least two speckle images includes: The at least two speckle images are projected at different times and at different measurement depths.

14. The method according to claim 13, characterized in that, The method further includes: Generate at least two second beams; Adjust the divergence angle of each of the at least two second beams; The at least two second beams are modulated respectively, and the measurement depths of the at least two second beams are set to obtain at least two speckle images with different measurement depths.

15. The method according to any one of claims 10-14, characterized in that, The at least two speckle images are projected sequentially according to the magnitude relationship of the corresponding measured depths.

16. The method according to any one of claims 10-15, characterized in that, The method further includes: waiting for a preset time after projecting any one of the at least two speckle images.

17. The method according to any one of claims 10-16, characterized in that, The brightness, shape, and density of any two of the at least two speckle images are the same.

18. The method according to any one of claims 15-17, characterized in that, The method further includes: The three-dimensional information of the target object is determined based on the at least one event image pair and the at least two speckle images.

19. An electronic device, characterized in that, The electronic device includes: a three-dimensional information measuring device, as described in any one of claims 1-9.

Images