Three-dimensional ranging method and device

Abstract

The present disclosure provides a three-dimensional ranging method and apparatus. A three-dimensional ranging apparatus includes: a light source unit configured to emit light pulses to irradiate a scene to be measured; an optical transmission unit configured to control the transmission of the reflected light after the light pulses are reflected by an object in the scene to be measured; a photosensor unit configured to receive the light after passing through the optical transmission unit to perform imaging; and a processor unit configured to control the light source unit, the optical transmission unit, and the photosensor unit, and determine the scene distance information of the scene to be measured based on the imaging result of the photosensor unit, wherein the light pulses at least include a first light pulse and a second light pulse, and the ratio of a first processed pulse envelope after the first pulse envelope of the first light pulse is processed by the optical transmission unit to a second processed pulse envelope after the second pulse envelope of the second light pulse is processed by the optical transmission unit is a monotonic function varying with time.

Description

Technical Field

[0001] The present disclosure relates to the field of optical ranging, and more particularly, to a three-dimensional ranging method and a three-dimensional ranging device. Background Art

[0002] With the emergence of application scenarios such as autonomous driving, 3D audio and video, games, smartphone navigation, and intelligent robots, it has become increasingly important to determine the depth of a scene in real time and accurately.

[0003] Currently, there are various methods for measuring the depth of a scene. In traditional triangulation ranging, as the ranging distance increases, the distance resolution deteriorates continuously. With the development of laser technology, it is common to use lasers to measure the depth of a scene. One method is to emit a modulated optical signal to the scene to be measured, receive the light reflected by an object in the scene to be measured, and then determine the distance of the object in the scene to be measured by demodulating the received light. Since this is a point-to-point measurement method, a large number of scans are required to obtain the depth information of the scene, and its spatial resolution is limited. Another method uses light with a predetermined illumination pattern to illuminate the scene to be measured and obtains the depth information of the scene to be measured using pre-acquired calibration information. In addition, another method is time-of-flight ranging, which emits a modulated signal and uses four sensors associated with a single photosensitive pixel at four different phases of the modulated signal to obtain the relative phase shift of the returned signal with respect to the transmitted signal, thereby determining the depth information.

[0004] These existing ranging methods generally require dedicated hardware configurations, the ranging devices are large and bulky, and the spatial resolution of ranging is low, or the ranging field of view is narrow, or the test distance is short. Summary of the Invention

[0005] In view of the above problems, the present disclosure is proposed. The present disclosure provides a three-dimensional ranging method and a three-dimensional ranging device.

[0006] According to an aspect of the present disclosure, there is provided a three-dimensional ranging device, including: a light source unit configured to emit light pulses to irradiate a scene to be measured; an optical transmission unit configured to control the transmission of the reflected light after the light pulses are reflected by an object in the scene to be measured; a photosensor unit configured to receive the light after passing through the optical transmission unit to perform imaging; and a processor unit configured to control the light source unit, the optical transmission unit, and the photosensor unit, and determine the scene distance information of the scene to be measured based on the imaging result of the photosensor unit, wherein the light pulses at least include a first light pulse and a second light pulse, and the ratio of a first processed pulse envelope of the first pulse envelope of the first light pulse after being processed by the optical transmission unit to a second processed pulse envelope of the second pulse envelope of the second light pulse after being processed by the optical transmission unit is a monotonic function that varies with time.

[0007] In addition, for the three-dimensional ranging device according to an embodiment of the present disclosure, the light source unit is configured to emit light pulses of different wavelengths, different polarizations, and different spatial structures and / or temporal structures simultaneously or sequentially.

[0008] In addition, for the three-dimensional ranging device according to an embodiment of the present disclosure, the photosensor unit is configured to perform imaging pixel by pixel or region by region simultaneously or sequentially.

[0009] In addition, for the three-dimensional ranging device according to an embodiment of the present disclosure, the photosensor unit acquires a first scene image corresponding to the first light pulse, a second scene image corresponding to the second light pulse, and a background scene image of the scene to be measured, and the processor unit acquires the scene distance information of the scene to be measured based on the background scene image, the first scene image, and the second scene image.

[0010] In addition, for the three-dimensional ranging device according to an embodiment of the present disclosure, the background scene image is a background scene image obtained by imaging the scene to be measured in a band other than the first light pulse and the second light pulse band, and / or a background scene image obtained by imaging the scene to be measured in the first light pulse and the second light pulse band without the first light pulse and without the second light pulse.

[0011] In addition, for the three-dimensional ranging device according to an embodiment of the present disclosure, the processor unit generates a target region image composed of multiple sub-regions based on the first scene image, the second scene image, and the background scene image, wherein the sub-regions include simple graphic elements and / or superpixel regions, and generates the scene distance information of the target region based on the first scene image, the second scene image, and the target region image.

[0012] In addition, for the three-dimensional ranging device according to an embodiment of the present disclosure, the target area image is generated by using a deep neural network.

[0013] In addition, for the three-dimensional ranging device according to an embodiment of the present disclosure, based on the first scene image, the second scene image, and the background scene image, the deep neural network is pre-optimized for sub-region segmentation and scene distance information generation.

[0014] In addition, for the three-dimensional ranging device according to an embodiment of the present disclosure, by using the real-time scene images that have been collected, and then using simulation to generate sub-region data calibration of the virtual 3D world corresponding to the real-time scene images, and at the same time using the pre-calibrated real-world images and sub-region data calibration, and / or using the scene images and data collected by at least one other such three-dimensional ranging device for calibration, the deep neural network is updated in real time.

[0015] In addition, for the three-dimensional ranging device according to an embodiment of the present disclosure, the output of the deep neural network is calibrated by the data of the simulated virtual 3D world into simple graphic elements and / or super-pixel sub-regions containing three-dimensional information, and the simple graphic elements and / or super-pixel sub-regions are used to generate the scene distance information of the target area.

[0016] In addition, for the three-dimensional ranging device according to an embodiment of the present disclosure, it further includes: a beam splitter unit configured to direct the reflected light reflected by the object in the scene to be measured to the optical transmission unit, and direct the light reflected by the object in the scene to be measured to the photosensor unit, where the photosensor unit includes at least a first photosensor sub-unit and a second photosensor sub-unit, the first photosensor sub-unit is configured to perform imaging on the reflected light, and the second photosensor sub-unit is configured to perform imaging on the natural light reflected light; wherein the first photosensor sub-unit further includes at least an uneven light pulse scene image generated for unevenly distributed light pulses in space, and based on the background scene image, at least the first scene image, the second scene image, the target area image, and / or the uneven light pulse scene image, the scene distance information is generated.

[0017] In addition, for the three-dimensional ranging device according to an embodiment of the present disclosure, the three-dimensional ranging device is installed on a vehicle, and the light source unit is configured by the left headlight and / or the right headlight of the vehicle.

[0018] In addition, a three-dimensional ranging device according to an embodiment of the present disclosure, wherein the optical transmission unit includes a first optical transmission sub-unit and a second optical transmission sub-unit, the photosensor unit includes a first photosensor sub-unit and a second photosensor sub-unit, the three-dimensional ranging device further includes a first beam splitter sub-unit and a second beam splitter sub-unit, and the first optical transmission sub-unit, the first beam splitter sub-unit, and the first photosensor sub-unit form a first sub-optical path for imaging the optical pulse; the second optical transmission sub-unit, the second beam splitter sub-unit, and the second photosensor sub-unit form a second sub-optical path for imaging the visible light, and the processor unit controls alternate imaging or simultaneous imaging via the first sub-optical path and / or the second sub-optical path. Among them, based on at least the background scene image, at least the first scene image and the second scene image, and the target area image, the scene distance information is generated.

[0019] In addition, the three-dimensional ranging device according to an embodiment of the present disclosure further includes: an amplifier unit, configured after the light source unit to amplify the optical pulse, or configured after the first optical transmission sub-unit or the first beam splitter sub-unit to amplify the reflected light.

[0020] In addition, the three-dimensional ranging device according to an embodiment of the present disclosure, wherein the processor unit is further configured to output the scene distance information and the scene image of the scene to be measured, and the scene image includes a geometric image and a flow image.

[0021] According to another aspect of the present disclosure, a three-dimensional ranging method is provided, including: emitting an optical pulse to irradiate a scene to be measured; controlling the transmission of the reflected light after the optical pulse is reflected by an object in the scene to be measured; receiving the light after passing through the optical transmission unit to perform imaging; and determining the scene distance information of the scene to be measured based on the result of the imaging, wherein the optical pulse includes at least a first optical pulse and a second optical pulse, and the ratio of the first processed pulse envelope after the first pulse envelope of the first optical pulse is processed by the optical transmission unit to the second processed pulse envelope after the second pulse envelope of the second optical pulse is processed by the optical transmission unit is a monotonically varying function of time.

[0022] In addition, the three-dimensional ranging method according to an embodiment of the present disclosure, wherein the three-dimensional ranging method includes: simultaneously or sequentially emitting optical pulses with different wavelengths, different polarizations, and different spatial and / or temporal structures.

[0023] In addition, the three-dimensional ranging method according to an embodiment of the present disclosure, wherein the three-dimensional ranging method includes: simultaneously or sequentially performing imaging pixel by pixel or region by region.

[0024] In addition, according to the three-dimensional ranging method of an embodiment of the present disclosure, the three-dimensional ranging method includes: obtaining a first scene image corresponding to a first light pulse, a second scene image corresponding to a second light pulse, and a background scene image of the scene to be measured; and obtaining scene distance information of the scene to be measured based on the background scene image, the first scene image, and the second scene image.

[0025] In addition, according to the three-dimensional ranging method of an embodiment of the present disclosure, the background scene image is a background scene image obtained by imaging the scene to be measured in a band other than the first light pulse and the second light pulse band, and / or a background scene image obtained by imaging the scene to be measured in the first light pulse and the second light pulse band without the first light pulse and without the second light pulse.

[0026] In addition, according to the three-dimensional ranging method of an embodiment of the present disclosure, the three-dimensional ranging method includes: generating a target region image composed of a plurality of sub-regions based on the first scene image, the second scene image, and the background scene image, and generating scene distance information of the target region based on the first scene image, the second scene image, and the target region image.

[0027] In addition, according to the three-dimensional ranging method of an embodiment of the present disclosure, the three-dimensional ranging method further includes: pre-optimizing a deep neural network based on the first scene image, the second scene image, and the background scene image for sub-region segmentation and scene distance information generation.

[0028] In addition, according to the three-dimensional ranging method of an embodiment of the present disclosure, the three-dimensional ranging method further includes: using the already acquired real-time scene image, and then using the sub-region data calibration of the virtual 3D world corresponding to the real-time scene image generated by simulation, and at the same time using the pre-calibrated real-world image and sub-region data calibration, and / or using the scene images and data collected by at least one other such three-dimensional ranging device for calibration to update the deep neural network in real time.

[0029] In addition, according to the three-dimensional ranging method of an embodiment of the present disclosure, the output of the deep neural network is calibrated by the data of the simulated virtual 3D world into simple primitives and / or superpixel sub-regions containing three-dimensional information, and the simple primitives and / or superpixel sub-regions are used to generate scene distance information of the target region.

[0030] In addition, according to the three-dimensional ranging method of an embodiment of the present disclosure, the three-dimensional ranging method further includes: guiding the reflected light reflected by an object in the scene to be measured to the optical transmission unit, and guiding the light reflected by the object in the scene to be measured to the photosensor unit, where the photosensor unit includes at least a first photosensor subunit and a second photosensor subunit, the first photosensor subunit is configured to perform imaging on the reflected light, and the second photosensor subunit is configured to perform imaging on the natural light reflected light, where the first photosensor subunit at least further includes an uneven light pulse scene image generated for unevenly distributed light pulses in space, and based on the background scene image, at least the first scene image and the second scene image, the target area image, and the uneven light pulse scene image, the scene distance information is generated.

[0031] In addition, according to the three-dimensional ranging method of an embodiment of the present disclosure, the optical transmission unit includes a first optical transmission subunit and a second optical transmission subunit, the photosensor unit includes a first photosensor subunit and a second photosensor subunit, the three-dimensional ranging device further includes a first beam splitter subunit and a second beam splitter subunit, the first optical transmission subunit, the first beam splitter subunit, and the first photosensor subunit form a first sub-optical path for imaging the light pulse; the second optical transmission subunit, the second beam splitter subunit, and the second photosensor subunit form a second sub-optical path for imaging the visible light, where the three-dimensional ranging method further includes: controlling to perform imaging alternately or simultaneously via the first sub-optical path and the second sub-optical path, and based on at least the background scene image, at least the first scene image and the second scene image, and the target area image, the scene distance information is generated.

[0032] In addition, according to the three-dimensional ranging method of an embodiment of the present disclosure, the three-dimensional ranging method further includes: outputting the scene distance information and the scene image of the scene to be measured, where the scene image includes a geometric image and a flowing light image.

[0033] As will be described in detail below, the three-dimensional ranging method and device according to an embodiment of the present disclosure achieve accurate and real-time acquisition of depth information without scanning and narrow field-of-view limitations by using a standard CCD or CMOS image sensor and imaging through controllable illumination and sensor exposure. In addition, since no additional mechanical components are used, and devices such as CCDs or CMOSs used can be mass-produced, the reliability and stability of the system are increased while the cost is reduced.

[0034] It is to be understood that both the foregoing general description and the following detailed description are exemplary and are intended to provide further explanation of the claimed technology. Brief Description of the Drawings

[0035] The above and other objects, features, and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail with reference to the accompanying drawings. The drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. They are used together with the embodiments of the present disclosure to explain the present disclosure and do not constitute a limitation to the present disclosure. In the drawings, the same reference numerals generally represent the same components or steps.

[0036] Figure 1 is a schematic diagram outlining the application scenario of the three-dimensional ranging method and device according to an embodiment of the present disclosure;

[0037] Figure 2 is a flowchart outlining the three-dimensional ranging method according to an embodiment of the present disclosure;

[0038] Figure 3 is a schematic diagram further illustrating the application scenario of the three-dimensional ranging method and device according to an embodiment of the present disclosure;

[0039] Figure 4 is a schematic diagram further illustrating the application scenario of the three-dimensional ranging method and device according to an embodiment of the present disclosure;

[0040] Figure 5 is a schematic diagram further illustrating the application scenario of the three-dimensional ranging method and device according to an embodiment of the present disclosure; and

[0041] Figure 6 is a flowchart further illustrating the three-dimensional ranging method according to an embodiment of the present disclosure. Detailed Description of the Embodiments

[0042] In order to make the purpose, technical solution, and advantages of the present disclosure more apparent, exemplary embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure. It should be understood that the present disclosure is not limited by the exemplary embodiments described here.

[0043] First, refer to Figure 1 to schematically describe the application scenario of the present disclosure. Figure 1 is a schematic diagram outlining the application scenario of the three-dimensional ranging method and device according to an embodiment of the present disclosure.

[0044] As Figure 1As shown, the three-dimensional ranging device 10 according to an embodiment of the present disclosure performs ranging on the scene 1040 to be measured. In an embodiment of the present disclosure, the three-dimensional ranging device 10 is configured, for example, in an autonomous driving system. The three-dimensional ranging device 10 measures the relative distances of objects in the driving scene of the vehicle (e.g., streets, highways, etc.), and the obtained scene distance information will be used for functions such as positioning for driverless driving, drivable area detection, lane marking detection, obstacle detection, dynamic object tracking, and obstacle classification and recognition. In another embodiment of the present disclosure, the three-dimensional ranging device 10 is configured, for example, in an AR / VR audio-visual game system. By measuring the scene distance information of the user's environment through the three-dimensional ranging device 10, the position of the user in the three-dimensional space can be accurately located, enhancing the real experience in the game. In another embodiment of the present disclosure, the three-dimensional ranging device 10 is configured, for example, in an intelligent robot system. By measuring the scene distance information of the working environment where the robot is located through the three-dimensional ranging device 10, the modeling of the working environment where the robot is located and the intelligent path planning of the robot can be realized.

[0045] As Figure 1 Schematically shown, the three-dimensional ranging device 10 according to an embodiment of the present disclosure includes a light source unit 101, an optical transmission unit 102, a photosensor unit 103, and a processor unit 104.

[0046] The light source unit 101 is configured to emit light pulses λ1, λ2 to irradiate the scene 1040 to be measured. In an embodiment of the present disclosure, according to the actual specific application scenario, the light source unit 101 can be configured to simultaneously or sequentially emit light pulses (e.g., structured light) and / or time structures (frequency-modulated continuous wave (FMCW)) of different wavelengths, different polarizations, and different spatial structures under the control of the processor unit 104. In an embodiment of the present disclosure, the three-dimensional ranging device 10 can be configured on an automobile, and the light source unit 101 is configured by the left headlight and / or the right headlight of the automobile.

[0047] The optical transmission unit 102 is configured to control the transmission of the reflected light after the light pulse is reflected by an object in the scene to be measured. In an embodiment of the present disclosure, according to the actual specific application scenario, the optical transmission unit 102 can be configured to allow light pulses of a specific wavelength and polarization to pass through and process the envelope of the passed light pulses under the control of the processor unit 104. In an embodiment of the present disclosure, the optical transmission unit 102 can be implemented as an optical gate, for example.

[0048] The photosensor unit 103 is configured to receive the light after passing through the optical transmission unit 102 to perform imaging. In an embodiment of the present disclosure, according to the actual specific application scenario, the photosensor unit 103 may be configured to perform pixel-by-pixel or region-by-region imaging simultaneously or sequentially under the control of the processor unit 104. In an embodiment of the present disclosure, for example, the photosensor unit 103 may respectively arrange RGBL filters for every four pixels (the RBG filter corresponds to the ordinary visible light spectrum, and L corresponds to the laser spectrum), so as to record visible light and laser images simultaneously. Alternatively, the photosensor unit 103 may include photosensor sub-units for visible light and laser respectively.

[0049] The processor unit 104 is configured to control the light source unit 101, the optical transmission unit 102, and the photosensor unit 103, and determine the scene distance information of the scene 1400 to be measured based on the imaging result of the photosensor unit 103.

[0050] As Figure 1 Schematically shown, the optical pulse at least includes a first optical pulse λ1 and a second optical pulse λ2, and the ratio of the first processed pulse Λ1 envelope after the first pulse envelope of the first optical pulse λ1 is processed by the optical transmission unit 102 to the second processed pulse Λ2 envelope after the second pulse envelope of the second optical pulse λ2 is processed by the optical transmission unit 102 is a monotonic function of time. The first processed pulse Λ1 is, for example, a monotonic decreasing ramp of the optical pulse envelope with time, while the second processed pulse Λ2 is, for example, a square wave whose optical pulse envelope does not change with time. Alternatively, the first processed pulse Λ1 may also be a rising or falling ramp of the optical pulse envelope with time, and the second processed pulse Λ2 is a different rising or falling ramp. That is to say, in the three-dimensional ranging method according to the embodiment of the present disclosure, it is required that the ratio of the first pulse envelope of the first optical pulse λ1 to the second pulse envelope of the second optical pulse λ2 is a monotonic function of time. This monotonic function relationship between the first pulse envelope of the first optical pulse λ1 and the second pulse envelope of the second optical pulse λ2 will be recorded for the subsequent determination of the scene distance information by the processor unit 104.

[0051] The principle of determining the scene distance information using at least two optical pulses with a monotonic function relationship between the envelopes is described as follows.

[0052] At time t = 0, a first optical pulse is emitted. The duration of the first optical pulse is Δ1, and the optical pulse envelope of the first optical pulse is f1(t). That is, t = 0 is the start time of the first light emission, and Δ1 is the end time of the first light emission. Assume that there are two objects in the scene to be measured, namely object 1 at a relatively far distance and object 2 at a relatively near distance, and assume that the surface reflectivities of the objects are R1 and R2 respectively. For object 1, starting from time T1, the first optical pulse reflected by object 1 begins to return. (T1 + t11) is the start time of the first exposure, and (T1 + t12) is the end time of the first exposure. For object 2, starting from time T2, the first optical pulse reflected by object 2 begins to return. (T2 + t21) is the start time of the first exposure, and (T2 + t22) is the end time of the first exposure. The difference between the start time and the end time of the first exposure is the first exposure time τ1 for the first optical pulse. In addition, for object 1, the distances for the emission and reflection of the first optical pulse are r11 and r12 respectively; for object 2, the distances for the emission and reflection of the first optical pulse are r21 and r22 respectively.

[0053] Similarly, at time t = 0, a second optical pulse is emitted. The duration of the second optical pulse is Δ2, and the optical pulse envelope of the second optical pulse is f2(t). That is, t = 0 is the start time of the second light emission, and Δ2 is the end time of the first light emission. It should be understood that showing the first optical pulse and the second optical pulse as both being emitted at time t = 0 is only schematic, and in fact, the first optical pulse and the second optical pulse can be emitted simultaneously or in a non - simultaneous sequential manner. For object 1, starting from time T3, the second optical pulse reflected by object 1 begins to return. (T3 + t31) is the start time of the first exposure, and (T3 + t32) is the end time of the second exposure. For object 2, starting from time T4, the second optical pulse reflected by object 2 begins to return. (T4 + t41) is the start time of the second exposure, and (T4 + t42) is the end time of the second exposure. The difference between the start time and the end time of the second exposure is the second exposure time τ2 for the second optical pulse, and the second exposure time τ2 of the second optical pulse can be equal to the first exposure time τ1 of the first optical pulse.

[0054] Thus, the exposure amounts 1 and 2 of the first optical pulse for pixel 1 on object 1 and pixel 2 on object 2 can be expressed as:

[0055]

[0056]

[0057] The exposure amounts 3 and 4 of the second optical pulse for pixel 1 on object 1 and pixel 2 on object 2 can be expressed as:

[0058]

[0059]

[0060] Where C1 and C2 are constants respectively, related to the space represented by pixels 1 and 2 and independent of time. It is easy to understand that the image output values obtained by imaging pixels 1 and 2 are proportional to their respective exposure amounts.

[0061] In an embodiment of the present disclosure, the first exposure time is controlled to satisfy a first predetermined duration such that at least a portion of the first light pulse reflected by each point in the scene to be measured can be used to acquire the first scene image during the first exposure time, and the second exposure time is controlled to satisfy a second predetermined duration such that at least a portion of the second light pulse reflected by each point in the scene to be measured can be used to acquire the second scene image during the second exposure time.

[0062] For a pixel 1 or 2, in the ideal case without considering background light exposure, the exposure amount ratio g of the two exposures by the first light pulse and the second light pulse is expressed as:

[0063]

[0064]

[0065] If background light exposure is considered, then the exposure amount ratio g of the two exposures by the first light pulse and the second light pulse is expressed as:

[0066]

[0067]

[0068] T1 to T4 are all related to the distance D, and t11, t12, t31, t32, t21, t22, t41, t42, τ1 and τ2 are controllable parameters. Then, only by controlling f1(t) / f2(t) to satisfy a monotonically varying function, g(D) becomes a monotonic function of the distance D. Therefore, for a specific pixel, by measuring the two exposure amounts of this pixel, the distance information D of this pixel can be determined through the ratio of the two exposure amounts.

[0069] Therefore, when the ratio of the first post-processed pulse envelope after the first pulse envelope of the first optical pulse is processed by the optical transmission unit and the second post-processed pulse envelope after the second pulse envelope of the second optical pulse is processed by the optical transmission unit is a monotonic function of time, the photosensor unit 103 acquires a first scene image M2 corresponding to the first optical pulse λ1, a second scene image M3 corresponding to the second optical pulse λ2, and a background scene image of the scene to be measured 1400 (including M1 and M4 as described below). The processor unit 104 acquires the scene distance information of the scene to be measured 1400 based on the background scene images (M1 and M4), the first scene image M2, and the second scene image M3.

[0070] Specifically, the background scene image is a background scene image obtained by imaging the scene to be measured in a band other than the first optical pulse and the second optical pulse (that is, regardless of whether a laser pulse is emitted, the photosensor unit 103 is controlled not to perform imaging in the laser pulse band and only perform imaging in the natural light band to obtain the background scene image M4), and / or a background scene image obtained by imaging the scene to be measured in the first optical pulse and the second optical pulse band without the first optical pulse and the second optical pulse (that is, in the case of no laser pulse emission, the photosensor unit 103 is controlled to perform imaging in the laser pulse band and not perform imaging in the natural light band to obtain the background scene image M1).

[0071] In an embodiment of the present disclosure, the processor unit 104 generates a target region image M5 composed of multiple sub-regions based on the first scene image M2, the second scene image M3, and the background scene images (M1 and M4), and generates the scene distance information of the target region based on the first scene image M2, the second scene image M3, and the target region image M5. In this embodiment, the processor unit 104 uses a pre-trained neural network to perform sub-region segmentation on the target region in the scene to be measured based on the first scene image M2, the second scene image M3, and the background scene images (M1 and M4), and automatically generates the scene distance information.

[0072] In one embodiment of the present disclosure, the output of the deep neural network is calibrated with data of a simulated virtual 3D world into simple primitive elements and / or superpixel sub-regions containing three-dimensional information, and the simple primitive elements and / or superpixel sub-regions are used to generate the scene distance information of the target region. Generally, the output target (data calibration) of a neural network for general image recognition is the bounding box (boundary) of an object and the name of the object represented by the bounding box, such as an apple, a tree, a person, a bicycle, a car, and so on. However, the output in this embodiment is simple primitive elements: triangles, rectangles, circles, and so on. In other words, in the processing of the three-dimensional ranging device, the target object is recognized / simplified into "simple primitive elements" (including bright spots and dimensions, the so-called "simple primitive elements"), and both the original image and the simple primitive elements are part of the generated scene distance information of the target region.

[0073] Further, during the entire processing, the deep neural network is updated in real time by using the real-time scene images that have been collected, then using the sub-region data calibration of the virtual 3D world corresponding to the real-time scene images generated by simulation, and at the same time using the pre-calibrated real-world images and sub-region data calibration, and / or using the scene images and data calibration collected by at least one other such three-dimensional ranging device.

[0074] Figure 2 It is a flowchart outlining a three-dimensional ranging method according to an embodiment of the present disclosure. Figure 2 is with reference to Figure 1 The basic flowchart of the three-dimensional ranging device according to the embodiment of the present disclosure outlined.

[0075] As Figure 2 shown, the three-dimensional ranging method according to an embodiment of the present disclosure includes the following steps.

[0076] In step S201, an optical pulse is emitted to irradiate the scene to be measured.

[0077] In an embodiment of the present disclosure, according to the actual specific application scenario, optical pulses of different wavelengths, different polarizations, and different spatial structures (e.g., structured light) and / or time structures (frequency-modulated continuous wave (FMCW)) can be emitted simultaneously or sequentially.

[0078] In step S202, the transmission of the reflected light after the optical pulse is reflected by an object in the scene to be measured is controlled.

[0079] In an embodiment of the present disclosure, according to the actual specific application scenario, optical pulses of a specific wavelength and polarization are allowed to pass through, and the envelope of the passed optical pulse is processed.

[0080] In step S203, light is received after the transmission to perform imaging.

[0081] In embodiments of the present disclosure, according to actual specific application scenarios, pixel-by-pixel or region-by-region imaging can be performed simultaneously or sequentially. In embodiments of the present disclosure, for example, the photosensor unit 103 can be arranged with an RGBL filter for every four pixels respectively (the RBG filter corresponds to the ordinary visible light spectrum, and L corresponds to the laser spectrum), so as to record visible light and laser images simultaneously. Alternatively, the photosensor unit 103 can include photosensor sub-units for visible light and laser respectively.

[0082] In step S204, based on the imaging result, the scene distance information of the to-be-measured scene is determined.

[0083] In embodiments of the present disclosure, the optical pulse at least includes a first optical pulse and a second optical pulse, and the ratio of the first processed pulse envelope of the first pulse envelope of the first optical pulse after being processed by the optical transmission unit to the second processed pulse envelope of the second pulse envelope of the second optical pulse after being processed by the optical transmission unit is a monotonically varying function of time.

[0084] According to the basic ranging principle described above with reference to Figure 1 In step S203, a first scene image M2 corresponding to the first optical pulse, a second scene image M3 corresponding to the second optical pulse, and background scene images (M1 and M4) of the to-be-measured scene are acquired. In step S204, based on the background scene images, the first scene image, and the second scene image, the scene distance information of the to-be-measured scene is acquired.

[0085] More specifically, in embodiments of the present disclosure, in step S204, by using a pre-optimized deep neural network, a target region image M5 composed of multiple sub-regions is generated based on the first scene image M2, the second scene image M3, and the background scene images (M1 and M4), and the scene distance information of the target region is generated based on the first scene image M2, the second scene image M3, and the target region image M5.

[0086] Next, with further reference to Figures 3 to 5 Specific application scenarios of the three-dimensional ranging method and device according to embodiments of the present disclosure are described.

[0087] Figure 3 is a schematic diagram further illustrating the application scenario of the three-dimensional ranging method and device according to embodiments of the present disclosure. As Figure 3As shown, the three-dimensional ranging device 10 according to an embodiment of the present disclosure further includes a beam splitter unit 105 configured to direct the reflected light reflected by the object 1041 in the scene to be measured to the optical transmission unit 102, and direct the light reflected by the object 1041 in the scene to be measured to the photosensor unit 103. The photosensor unit 103 includes at least a first photosensor sub-unit 1031 and a second photosensor sub-unit 1032. The first photosensor sub-unit 1031 is configured to perform imaging on the reflected light in the laser band. For example, in the case where no laser pulse is emitted, the first photosensor sub-unit 1031 performs imaging on the laser pulse band and does not perform imaging on the natural light band to obtain the background scene image M1; and in the case where a laser pulse is emitted, the first photosensor sub-unit 1031 obtains a first scene image M2 corresponding to the first light pulse λ1 and a second scene image M3 corresponding to the second light pulse λ2. The second photosensor sub-unit 1032 is configured to perform imaging on the natural light reflected light. For example, regardless of whether a laser pulse is emitted or not, the second photosensor sub-unit 1032 does not perform imaging on the laser pulse band and only performs imaging on the natural light band to obtain the background scene image M4. In addition, the first photosensor sub-unit 1031 at least further includes an uneven light pulse scene image M6 generated by generating unevenly distributed light pulses in space.

[0088] The processor unit 104 configured with a deep neural network performs sub-region segmentation on the target region based on the background scene images (M1 and M4), at least the first scene image M2 and the second scene image M3, generates the target region image M5, and obtains the scene distance information. In an embodiment of the present disclosure, the scene distance information is presented as a 3D distance point cloud map R(i,j) = F(M1,M2,M3,M4,M5,M6). The three-dimensional ranging device 10 according to an embodiment of the present disclosure outputs a 2D viewable image and a 3D distance point cloud map.

[0089] Figure 4 is a schematic diagram further illustrating an application scenario of the three-dimensional ranging method and device according to an embodiment of the present disclosure. As Figure 4As shown, the optical transmission unit 102 of the three-dimensional ranging device 10 according to an embodiment of the present disclosure further includes a first optical transmission subunit 1021 and a second optical transmission subunit 1022. The first optical transmission subunit 1021 and the second optical transmission subunit 1022 may be configured with different light passing functions to perform different processing on the passed laser pulse envelopes. When performing laser band imaging, light pulses of corresponding bands are allowed to pass, and when performing visible light band imaging, visible light of corresponding bands is allowed to pass. The photosensor unit 103 includes a first photosensor subunit 1031 and a second photosensor subunit 1032. The first photosensor subunit 1031 and the second photosensor subunit 1032 may alternately perform exposure to improve the spatial pixel matching accuracy. In addition, the first photosensor subunit 1031 and the second photosensor subunit 1032 may simultaneously perform exposure to improve the ranging accuracy of dynamic objects. In addition, the beam splitter unit 105 of the three-dimensional ranging device 10 further includes a first beam splitter subunit 1051 and a second beam splitter subunit 1052. The first beam splitter subunit 1051 and the second beam splitter subunit 1052 may be used to separate laser and visible light, and may controllably separate laser pulses of different wavelengths, polarizations, and angles. It is easy to understand that the number and configuration positions of the above components are not restrictive.

[0090] The first optical transmission subunit 1021, the first beam splitter subunit 1051, and the first photosensor subunit 1031 form a first sub-optical path for imaging the light pulses; the second optical transmission subunit 1022, the second beam splitter subunit 1052, and the second photosensor subunit 1032 form a second sub-optical path for imaging the visible light. The processor unit 104 controls alternate imaging or simultaneous imaging via the first sub-optical path and the second sub-optical path. The processor unit 104 configured with a deep neural network generates the scene distance information based on at least the background scene images (M1 and M4), at least the first scene image M2 and the second scene image M3, and the target area image M5.

[0091] Figure 5 is a schematic diagram further illustrating an application scenario of the three-dimensional ranging method and device according to an embodiment of the present disclosure. As Figure 5 shown, the three-dimensional ranging device 10 according to an embodiment of the present disclosure is further configured with an amplifier unit 106 (including a first amplifier subunit 1061 and a second amplifier subunit 1062), which may be configured after the light source unit 101 to amplify the light pulses, or may be configured after the first optical transmission subunit 1021 or the beam splitter unit 105 to amplify the reflected light.

[0092] Figure 6It is a flowchart further illustrating a three-dimensional ranging method according to an embodiment of the present disclosure.

[0093] As Figure 6 shown, the three-dimensional ranging method according to another embodiment of the present disclosure includes the following steps.

[0094] In step S601, the deep neural network is pre-optimized for sub-region segmentation and scene distance information generation.

[0095] That is to say, the three-dimensional ranging method according to another embodiment of the present disclosure needs to perform training on the deep neural network for ranging.

[0096] In step S602, an optical pulse is emitted to irradiate the scene to be measured.

[0097] In the embodiments of the present disclosure, according to the actual specific application scenario, optical pulses of different wavelengths, different polarizations, and different spatial structures (e.g., structured light) and / or temporal structures (frequency-modulated continuous wave (FMCW)) can be emitted simultaneously or sequentially.

[0098] In step S603, the transmission of the reflected light after the optical pulse is reflected by an object in the scene to be measured is controlled.

[0099] In the embodiments of the present disclosure, according to the actual specific application scenario, optical pulses of a specific wavelength and polarization are allowed to pass through, and the envelope of the passed optical pulse is processed. Specifically, for example, the configuration described with reference to Figures 3 to 5 can be adopted.

[0100] In step S604, light is received after the transmission to perform imaging.

[0101] In the embodiments of the present disclosure, according to the actual specific application scenario, pixel-by-pixel or region-by-region imaging can be performed simultaneously or sequentially. In the embodiments of the present disclosure, for example, the photosensor unit 103 can respectively arrange RGBL filters (the RBG filter corresponds to the ordinary visible light spectrum, and L corresponds to the laser spectrum) for every four pixels, so as to record visible light and laser images simultaneously. Alternatively, the photosensor unit 103 can include photosensor sub-units for visible light and laser respectively.

[0102] In step S605, based on the imaging result, the scene distance information of the scene to be measured is determined.

[0103] In an embodiment of the present disclosure, the optical pulse at least includes a first optical pulse and a second optical pulse, and a ratio of a first processed pulse envelope after the first pulse envelope of the first optical pulse is processed by the optical transmission unit to a second processed pulse envelope after the second pulse envelope of the second optical pulse is processed by the optical transmission unit is a monotonic function varying with time.

[0104] According to the basic ranging principle described above with reference to Figure 1 In step S604, a first scene image M2 corresponding to the first optical pulse, a second scene image M3 corresponding to the second optical pulse, and background scene images (M1 and M4) of the scene to be measured are acquired. In step S204, based on the background scene images, the first scene image, and the second scene image, scene distance information of the scene to be measured is acquired.

[0105] More specifically, in an embodiment of the present disclosure, in step S605, a depth neural network pre-optimized in step S601 is utilized to generate a target region image M5 composed of multiple sub-regions based on the first scene image M2, the second scene image M3, and the background scene images (M1 and M4), and based on the first scene image M2, the second scene image M3, and the target region image M5, scene distance information of the target region is generated.

[0106] In step S606, the depth neural network is updated in real time.

[0107] More specifically, in an embodiment of the present disclosure, real-time scene images that have been acquired are utilized, and sub-region data calibration of a virtual 3D world corresponding to the real-time scene images is generated by simulation, and at the same time, calibration of real-world images and sub-region data that have been pre-calibrated is utilized, and / or calibration of scene images and data collected by at least one other such three-dimensional ranging device is utilized to update the depth neural network in real time.

[0108] In step S607, the scene distance information and scene images of the scene to be measured are output.

[0109] In one embodiment of the present disclosure, the output of the deep neural network is calibrated with data of a simulated virtual 3D world into simple primitive elements and / or superpixel sub-regions containing three-dimensional information, and the simple primitive elements and / or superpixel sub-regions are used to generate the scene distance information of the target region. Generally, the output target (data calibration) of a neural network generally used for image recognition is the bounding box (boundary) of an object and the name of the object represented by the bounding box, such as apple, tree, person, bicycle, car, etc. However, the output in this embodiment is simple primitive elements: triangles, rectangles, circles, etc. In other words, in the processing of the three-dimensional ranging device, the target object is recognized / simplified into "simple primitive elements" (including bright points and sizes), and both the original image and the simple primitive elements are part of the scene distance information of the generated target region. The three-dimensional ranging method according to the embodiment of the present disclosure outputs a 2D viewable image and a 3D distance point cloud image.

[0110] As described above, with reference to the accompanying drawings, a three-dimensional ranging method and apparatus according to an embodiment of the present disclosure are described. It uses a standard CCD or CMOS image sensor, and through controllable laser illumination and sensor exposure imaging, without the need for scanning and narrow field-of-view limitations, precise and real-time depth information acquisition is achieved by using a deep neural network.

[0111] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A professional technician can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0112] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, advantages, effects, etc. mentioned in the present disclosure are only examples and not limitations. It cannot be considered that these advantages, advantages, effects, etc. are essential for each embodiment of the present disclosure. In addition, the above-disclosed specific details are only for the purpose of illustration and easy understanding, rather than limitations. The above details do not limit the present disclosure to necessarily adopt the above specific details to implement.

[0113] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that the connections, arrangements, and configurations must be made in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any way. Words such as "including", "comprising", "having", etc. are open-ended terms, meaning "including but not limited to", and can be used interchangeably with each other. The words "or" and "and" used herein refer to the phrase "and / or" and can be used interchangeably with it, unless the context clearly indicates otherwise. The phrase "such as" used herein refers to the phrase "such as but not limited to" and can be used interchangeably with it.

[0114] In addition, as used herein, the "or" used in the listing of items starting with "at least one" indicates a disjunctive listing. For example, the listing of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Further, the term "exemplary" does not mean that the examples described are preferred or better than other examples.

[0115] It should also be noted that in the systems and methods of this disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent solutions of this disclosure.

[0116] Various changes, substitutions, and alterations to the technologies described herein can be made without departing from the teachings defined by the appended claims. In addition, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and acts described above. Current or later-developed processes, machines, manufactures, compositions of events, means, methods, or acts that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or acts within their scope.

[0117] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

[0118] The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit embodiments of the present disclosure to the form disclosed herein. Although several example aspects and embodiments have been discussed above, those skilled in the art will recognize some of their variations, modifications, alterations, additions, and subcombinations.

Claims

1. A three-dimensional ranging device, comprising: A light source unit configured to emit light pulses to irradiate a scene to be measured; An optical transmission unit configured to control the transmission of the reflected light after the light pulses are reflected by an object in the scene to be measured; A photoreceptor unit configured to receive the light after passing through the optical transmission unit to perform imaging; And A processor unit configured to control the light source unit, the optical transmission unit, and the photoreceptor unit, and determine the scene distance information of the scene to be measured based on the imaging result of the photoreceptor unit, wherein the light pulses at least include a first light pulse and a second light pulse, and the ratio of the first processed pulse envelope of the first pulse envelope of the first light pulse after being processed by the optical transmission unit to the second processed pulse envelope of the second pulse envelope of the second light pulse after being processed by the optical transmission unit is a monotonically varying function with time.

2. The three-dimensional ranging device according to claim 1, wherein, The light source unit is configured to emit light pulses of different wavelengths, different polarizations, and different spatial structures and / or temporal structures simultaneously or sequentially.

3. The three-dimensional ranging device according to claim 1 or 2, wherein, The photoreceptor unit is configured to perform imaging pixel by pixel or region by region simultaneously or sequentially.

4. The three-dimensional ranging device according to claim 1, wherein, The photoreceptor unit acquires a first scene image corresponding to the first light pulse, a second scene image corresponding to the second light pulse, and a background scene image of the scene to be measured, and the processor unit acquires the scene distance information of the scene to be measured based on the background scene image, the first scene image, and the second scene image.

5. The three-dimensional ranging device according to claim 4, wherein, The background scene image is a background scene image obtained by imaging the scene to be measured in a band other than the first light pulse and the second light pulse band, and / or a background scene image obtained by imaging the scene to be measured in the first light pulse and the second light pulse band without the first light pulse and without the second light pulse.

6. The three-dimensional ranging device according to claim 4 or 5, wherein, The processor unit generates a target region image composed of multiple sub-regions based on the first scene image, the second scene image, and the background scene image, wherein the sub-regions include simple graphic elements and / or superpixel regions, and generates the scene distance information of the target region based on the first scene image, the second scene image, and the target region image.

7. The three-dimensional ranging device according to claim 6, wherein, The target region image is generated using a deep neural network.

8. The three-dimensional ranging device according to claim 7, wherein, Based on the first scene image, the second scene image, and the background scene image, the deep neural network is pre-optimized for sub-region segmentation and scene distance information generation.

9. The three-dimensional ranging device according to claim 8, wherein, Using the real-time scene images that have been acquired, and then using the simulation to calibrate the sub-region data of the virtual 3D world corresponding to the real-time scene images, and at the same time using the pre-calibrated real-world images and sub-region data for calibration, and / or using the scene images and data collected by at least one other such three-dimensional ranging device for calibration, to update the deep neural network in real time.

10. The three-dimensional ranging device according to claim 9, wherein, The output of the deep neural network is calibrated by the data of the simulated virtual 3D world as simple graphic elements and / or superpixel sub-regions containing three-dimensional information, and the simple graphic elements and / or superpixel sub-regions are used to generate the scene distance information of the target region.

11. The three-dimensional ranging device according to claim 6, further comprising: A beam splitter unit configured to direct the reflected light reflected by an object in the to-be-measured scene to the optical transmission unit, and direct the light reflected by the object in the to-be-measured scene to the photosensor unit, wherein the photosensor unit includes at least a first photosensor sub-unit and a second photosensor sub-unit, the first photosensor sub-unit is configured to perform imaging on the reflected light, and the second photosensor sub-unit is configured to perform imaging on the natural light reflected light; wherein the first photosensor sub-unit further includes an uneven light pulse scene image generated for unevenly distributed light pulses in space, and Based on the background scene image, at least the first scene image and the second scene image, the target area image, and / or the uneven light pulse scene image, the scene distance information is generated.

12. The three-dimensional ranging device according to any one of claims 7 to 11, wherein, The three-dimensional ranging device is installed on a vehicle, and the light source unit is configured by the left headlight and / or the right headlight of the vehicle.

13. The three-dimensional ranging device according to claim 6, wherein, The optical transmission unit includes a first optical transmission sub-unit and / or a second optical transmission sub-unit, and the photosensor unit includes a first photosensor sub-unit and / or a second photosensor sub-unit, The three-dimensional ranging device further includes a first beam splitter sub-unit and / or a second beam splitter sub-unit, The first optical transmission sub-unit, the first beam splitter sub-unit, and the first photosensor sub-unit form a first sub-optical path for imaging the light pulse; The second optical transmission sub-unit, the second beam splitter sub-unit, and the second photosensor sub-unit form a second sub-optical path for imaging visible light, The processor unit controls alternate imaging or simultaneous imaging via the first sub-optical path and / or the second sub-optical path, wherein, based on at least the background scene image, at least the first scene image and the second scene image, and the target area image, the scene distance information is generated.

14. The three-dimensional ranging device according to claim 13, further comprising: An amplifier unit configured after the light source unit to amplify the light pulse, or configured after the first optical transmission sub-unit or the first beam splitter sub-unit to amplify the reflected light.

15. The three-dimensional ranging device according to claim 13 or 14, wherein, The processor unit is further configured to output the scene distance information and the scene image of the to-be-measured scene, and the scene image includes a geometric image and a streamer image.

16. A three-dimensional ranging method, comprising: Emitting a light pulse to irradiate a to-be-measured scene; Controlling the transmission of the reflected light after the light pulse is reflected by an object in the to-be-measured scene; Receiving the light after the transmission to perform imaging; And Based on the result of the imaging, determining the scene distance information of the to-be-measured scene, wherein the light pulse includes at least a first light pulse and a second light pulse, and the ratio of the first processed pulse envelope of the first pulse envelope of the first light pulse after being processed by the optical transmission unit to the second processed pulse envelope of the second pulse envelope of the second light pulse after being processed by the optical transmission unit is a monotonically varying function of time.

17. The three-dimensional ranging method according to claim 16, wherein, The three-dimensional ranging method includes: To simultaneously or sequentially emit optical pulses with different wavelengths, different polarizations, and different spatial and / or temporal structures.

18. The three-dimensional ranging method according to claim 16 or 17, wherein, The three-dimensional ranging method includes: Performing imaging pixel by pixel or region by region simultaneously or sequentially.

19. The three-dimensional ranging method according to claim 16, wherein, The three-dimensional ranging method includes: Obtaining a first scene image corresponding to the first optical pulse, a second scene image corresponding to the second optical pulse, and a background scene image of the scene to be measured; and Based on the background scene image, the first scene image, and the second scene image, obtaining the scene distance information of the scene to be measured.

20. The three-dimensional ranging method according to claim 19, wherein, The background scene image is a background scene image obtained by imaging the scene to be measured in a band other than the first optical pulse and the second optical pulse band, and / or a background scene image obtained by imaging the scene to be measured in the first optical pulse and the second optical pulse band without the first optical pulse and without the second optical pulse.

21. The three-dimensional ranging method according to claim 19 or 20, wherein, The three-dimensional ranging method includes: Generating a target region image composed of multiple sub-regions based on the first scene image, the second scene image, and the background scene image, wherein the sub-regions include simple primitive elements and / or superpixel regions, and Based on the first scene image, the second scene image, and the target region image, generating the scene distance information of the target region.

22. The three-dimensional ranging method according to claim 21, wherein, The three-dimensional ranging method further includes: Pre-optimizing a deep neural network based on the first scene image, the second scene image, and the background scene image for sub-region segmentation and scene distance information generation.

23. The three-dimensional ranging method according to claim 22, wherein, The three-dimensional ranging method further includes: Using the already acquired real-time scene image, and then using the sub-region data calibration of the virtual 3D world corresponding to the real-time scene image generated by simulation, and at the same time using the pre-calibrated real-world image and sub-region data calibration, and / or using the scene image and data calibration collected by at least one other three-dimensional ranging device to update the deep neural network in real time.

Images