Laser projector, camera assembly and method and device for generating depth image

Abstract

The invention discloses a laser projector, a camera assembly and a method and device for generating a depth image. The laser projector comprises a first light-emitting part and at least one second light-emitting part, the method comprises the following steps: acquiring at least two laser images; respectively coding each laser image in the at least two laser images to obtain a coding result of each laser image; the coding results of all the laser images are integrated to obtain coding data of the measured object; acquiring preset reference coded data, and calculating the matching cost of the coded data of the measured object and the reference coded data under all possible parallax errors; and calculating a disparity map based on the matching cost, and obtaining a depth image based on the disparity map. According to the laser projector, the camera assembly and the method and device for generating the depth image, the equipment cost and space can be saved, the calculation amount of a subsequent depth estimation algorithm can be reduced, and algorithm optimization acceleration is facilitated.

Description

Technical Field

[0001] The present application relates to the field of imaging technology, and more specifically to a laser projector, a camera assembly, and a method and device for generating a depth image. Background Art

[0002] Structured light is a system structure composed of a projector and a camera. The projector projects specific light information onto the surface of an object and the background, which is then collected by the camera. The position and depth of the object are calculated based on the changes in the light signal caused by the object, and the entire three-dimensional space is restored.

[0003] A laser projector is a projector that can project laser information onto an object. The mainstream solution is that a single projector projects a speckle pattern, and a single infrared camera is used to collect images for processing and recognition. In traditional technology, in order to improve recognition accuracy, it is generally necessary to increase the number of projectors, collect multiple speckle patterns projected by multiple projectors, analyze and generate depth information to improve accuracy, but this method has disadvantages: on the one hand, the production cost is high, and the imaging device is relatively large, which is not conducive to the miniaturization design of the imaging device; on the other hand, the multi-projector solution requires the calculation of the depth of different baseline projector patterns separately and then data fusion, the algorithm is complex and baseline errors are prone to occur. Summary of the invention

[0004] The present application is proposed to solve at least one of the above problems. According to one aspect of the present application, a laser projector is provided, the laser projector comprising: a laser light source assembly, comprising a first light-emitting portion and at least one second light-emitting portion, the first light-emitting portion and the second light-emitting portion emitting laser light of the same wavelength, the second light-emitting portion comprising two parts spaced apart from each other along a first direction, the two parts being respectively located on both sides of the first light-emitting portion; a control assembly, the control assembly being coupled to the laser light source assembly, for controlling different light-emitting portions to emit laser light at different times; a diffractive optical element, for diffracting the laser light emitted by different light-emitting portions to form at least two different projection patterns; wherein the first light-emitting portion has a first size in the first direction and a second size in the second direction, each of the two parts of the second light-emitting portion has a third size in the first direction and a fourth size in the second direction, wherein the third size is half of the first size, and the fourth size is equal to the second size.

[0005] In one embodiment of the present application, the laser light source assembly also includes at least one third light-emitting portion, the first light-emitting portion and the third light-emitting portion emit laser light of the same wavelength, the third light-emitting portion includes two parts spaced apart from each other along the second direction, and the two parts are respectively located on both sides of the first light-emitting portion; wherein each of the two parts of the third light-emitting portion has a fifth size along the first direction and a sixth size along the second direction, wherein the fifth size is equal to the first size, and the sixth size is half of the second size.

[0006] In one embodiment of the present application, the laser projector further includes a collimating lens for collimating the laser, wherein the diffraction optical element is used to diffract the collimated laser to form the projection pattern.

[0007] In one embodiment of the present application, the first direction is the length direction of the first light-emitting portion, and the second direction is the width direction of the first light-emitting portion, and both the length direction and the width direction are perpendicular to the optical axis of the collimator.

[0008] In one embodiment of the present application, the first direction is the width direction of the first light-emitting portion, and the second direction is the length direction of the first light-emitting portion, and both the length direction and the width direction are perpendicular to the optical axis of the collimator.

[0009] In one embodiment of the present application, a distance between the two parts of the second light-emitting portion along the first direction is 2N times the third size, where N is a positive integer.

[0010] In one embodiment of the present application, a distance between the two parts of the third light-emitting portion along the second direction is 2N times the sixth dimension, where N is a positive integer.

[0011] In one embodiment of the present application, N is any one of 1, 2 and 3.

[0012] According to another aspect of the present application, a camera assembly is provided, which includes: the above-mentioned laser projector, the laser projector is used to emit laser to the object to be measured to form the at least two different projection patterns; an image sensor, used to collect the at least two different projection patterns to obtain at least two laser images; a processor, used to process the at least two laser images to obtain a depth image, wherein the processor processes the at least two laser images to obtain a depth image, including: encoding each of the at least two laser images separately to obtain an encoding result of each of the laser images; integrating the encoding results of all the laser images to obtain the encoding data of the object to be measured; obtaining preset reference encoding data, and calculating the matching cost between the encoding data of the object to be measured and the reference encoding data under all possible parallaxes; calculating a disparity map based on the matching cost, and obtaining a depth image based on the disparity map.

[0013] In one embodiment of the present application, the processor encodes each of the at least two laser images separately to obtain the encoding result of each of the laser images, including: allocating variables of corresponding types according to the total number of laser images and the number of encoding bits for each point in each of the laser images; integrating the encoding results of each corresponding point in each of the laser images into one of the variables to obtain the encoding data of the object under test.

[0014] In one embodiment of the present application, the processor calculates the disparity map based on the matching cost, including: constructing a matching cost space based on the matching cost; optimizing the matching cost space using an aggregation algorithm to obtain an optimized matching cost space; and calculating the disparity map in the optimized matching cost space.

[0015] In one embodiment of the present application, the processor calculates the disparity map, including: for each point in the matching cost space, traversing the respective matching costs of the point under all possible disparities, finding the disparity corresponding to the extreme matching cost, interpolating the disparity corresponding to the extreme matching cost, and obtaining a final disparity value of the point, wherein the final disparity value is a floating point value; and obtaining the disparity map based on the final disparity value of each point in the matching cost space.

[0016] In one embodiment of the present application, the processor is further configured to: perform left-right consistency verification on the disparity map to remove error data in the disparity map; and obtain a depth image based on the disparity map that has undergone left-right consistency verification.

[0017] In one embodiment of the present application, the processor includes the control component.

[0018] According to another aspect of the present application, an electronic device is provided, comprising: a housing; and the above-mentioned camera assembly, wherein the camera assembly is arranged to the housing and exposed from the housing to obtain a depth image.

[0019] According to another aspect of the present application, a method for generating a depth image is provided, the method comprising: acquiring at least two laser images, wherein the at least two laser images are obtained by an image sensor collecting at least two different projection patterns, and the at least two different projection patterns are formed by the above-mentioned laser projector emitting laser light to the object to be measured; encoding each of the at least two laser images separately to obtain an encoding result of each of the laser images; integrating the encoding results of all the laser images to obtain encoding data of the object to be measured; acquiring preset reference encoding data, and calculating the matching cost between the encoding data of the object to be measured and the reference encoding data under all possible parallaxes; calculating a disparity map based on the matching cost, and obtaining a depth image based on the disparity map.

[0020] In one embodiment of the present application, encoding each of the at least two laser images separately to obtain the encoding result of each of the laser images includes: allocating variables of corresponding types according to the total number of laser images and the number of encoding bits for each point in each of the laser images; integrating the encoding results of each corresponding point in each of the laser images into one of the variables to obtain the encoding data of the object under test.

[0021] In one embodiment of the present application, calculating the disparity map based on the matching cost includes: constructing a matching cost space based on the matching cost; optimizing the matching cost space using an aggregation algorithm to obtain an optimized matching cost space; and calculating the disparity map in the optimized matching cost space.

[0022] In one embodiment of the present application, calculating the disparity map includes: for each point in the matching cost space, traversing the respective matching costs of the point under all possible disparities, finding the disparity corresponding to the extreme matching cost, interpolating the disparity corresponding to the extreme matching cost, and obtaining a final disparity value of the point, wherein the final disparity value is a floating point value; and obtaining the disparity map based on the final disparity value of each point in the matching cost space.

[0023] In one embodiment of the present application, the method further includes: performing left-right consistency verification on the disparity map to remove error data in the disparity map; and obtaining a depth image based on the disparity map that has undergone the left-right consistency verification.

[0024] According to another aspect of the present application, a device for generating a depth image is provided, the device comprising: an acquisition module, used to acquire at least two laser images, wherein the at least two laser images are obtained by an image sensor collecting at least two different projection patterns, and the at least two different projection patterns are formed by the above-mentioned laser projector emitting laser light to the object to be measured; an encoding module, used to encode each of the at least two laser images separately to obtain an encoding result of each of the laser images; an integration module, used to integrate the encoding results of all the laser images to obtain encoding data of the object to be measured; a first calculation module, used to obtain preset reference encoding data, and calculate the matching cost between the encoding data of the object to be measured and the reference encoding data under all possible parallaxes; a second calculation module, used to calculate a disparity map based on the matching cost, and obtain a depth image based on the disparity map.

[0025] In one embodiment of the present application, the encoding module encodes each of the at least two laser images separately to obtain an encoding result for each of the laser images, including: allocating variables of corresponding types according to the total number of laser images and the number of encoding bits for each point in each of the laser images; integrating the encoding results of each corresponding point in each of the laser images into one of the variables to obtain the encoding data of the object under test.

[0026] In one embodiment of the present application, the second calculation module calculates the disparity map based on the matching cost, including: constructing a matching cost space based on the matching cost; optimizing the matching cost space using an aggregation algorithm to obtain an optimized matching cost space; and calculating the disparity map in the optimized matching cost space.

[0027] In one embodiment of the present application, the second calculation module calculates the disparity map, including: for each point in the matching cost space, traversing the respective matching costs of the point under all possible disparities, finding the disparity corresponding to the extreme matching cost, interpolating the disparity corresponding to the extreme matching cost, and obtaining a final disparity value of the point, wherein the final disparity value is a floating point value; and obtaining the disparity map based on the final disparity value of each point in the matching cost space.

[0028] In one embodiment of the present application, the second calculation module is further used to: perform left-right consistency verification on the disparity map to remove error data in the disparity map; and obtain a depth image based on the disparity map that has undergone left-right consistency verification.

[0029] According to another aspect of the present application, a device for generating a depth image is provided, the device comprising a memory and a processor, the memory storing a computer program executed by the processor, and when the computer program is executed by the processor, the processor executes the above-mentioned method for generating a depth image.

[0030] According to another aspect of the present application, a storage medium is provided, on which a computer program executed by a processor is stored. When the computer program is executed by the processor, the processor executes the above-mentioned method for generating a depth image.

[0031] According to yet another aspect of the present application, a computer program is further provided. When the computer program is executed by a processor, the processor executes the above-mentioned method for generating a depth image.

[0032] The laser projector of the present application can realize the solution of forming multiple projection patterns with a single projector. Compared with the single-projector single-pattern solution, it can provide richer texture information within a unit space, so that the subsequent depth estimation results based on the projection pattern are more accurate. Compared with the multi-projector multi-pattern solution, it can save equipment costs and space, and can also reduce the computational complexity of the subsequent depth estimation algorithm, which is conducive to algorithm optimization and acceleration. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0034] Figure 1 A schematic structural diagram of a laser projector according to an embodiment of the present application is shown.

[0035] Figure 2 A schematic diagram showing a projection pattern when the first light-emitting unit in a laser projector according to an embodiment of the present application emits laser light.

[0036] Figure 3 Show Figure 2 Schematic diagram of the zero-order pattern corresponding to the projected pattern shown.

[0037] Figure 4 A schematic diagram showing the arrangement of light sources of the first light-emitting portion in a laser projector according to an embodiment of the present application is shown.

[0038] Figure 5A schematic diagram showing a projection pattern when the second light emitting unit in a laser projector according to an embodiment of the present application emits laser.

[0039] Figure 6 Show Figure 5 Schematic diagram of the zero-order pattern corresponding to the projected pattern shown.

[0040] Figure 7 A schematic diagram showing the arrangement of light sources of the second light-emitting portion in a laser projector according to an embodiment of the present application is shown.

[0041] Figure 8 A schematic diagram showing the emission of laser by a first light emitting unit in a laser projector according to an embodiment of the present application.

[0042] Fig. 9 A schematic diagram showing the emission of laser by the second light emitting unit in the laser projector according to an embodiment of the present application.

[0043] Fig.10 A schematic diagram showing the arrangement of the light-emitting part of a laser projector according to an embodiment of the present application is shown.

[0044] Fig.11 A schematic structural diagram of a laser projector according to another embodiment of the present application is shown.

[0045] Fig.12 A schematic diagram showing a projection pattern when the third light emitting unit in the laser projector according to an embodiment of the present application emits laser.

[0046] Fig.13 Show Fig.12 Schematic diagram of the zero-order pattern corresponding to the projected pattern shown.

[0047] Fig.14 A schematic diagram showing the arrangement of light sources of the third light-emitting portion in the laser projector according to an embodiment of the present application is shown.

[0048] Fig.15 A schematic diagram showing the emission of laser by the third light emitting unit in the laser projector according to an embodiment of the present application.

[0049] Fig.16 A schematic diagram showing the arrangement of the light-emitting part of a laser projector according to an embodiment of the present application is shown.

[0050] Fig.17 A schematic structural block diagram of a camera assembly according to an embodiment of the present application is shown.

[0051] Fig.18 A schematic flowchart of a method for generating a depth image according to an embodiment of the present application is shown.

[0052] Fig.19A schematic structural block diagram of an apparatus for generating a depth image according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0054] Figure 1 FIG. 1 shows a schematic structural diagram of a laser projector 100 according to an embodiment of the present application. Figure 1 As shown, the laser projector 100 includes a laser light source assembly 110, a control assembly (not shown) and a diffractive optical element (DOE) 120. The laser light source assembly 110 includes at least two different light emitting parts (in Figure 1 The first light-emitting unit A and the second light-emitting unit (B1, B2) are shown in the figure, wherein the second light-emitting unit (B1, B2) includes two parts, namely B1 and B2), and each light-emitting unit emits laser light of the same wavelength. The control component is coupled to the laser light source component 110, and is used to control different light-emitting units to emit laser light at different times. The diffraction optical element 120 is used to diffract the laser light emitted by different light-emitting units to form at least two different projection patterns (i.e., speckle patterns). Because at least two different light-emitting units are included, each light-emitting unit can emit laser light to form a projection pattern, so the laser projector 100 can ultimately form at least two projection patterns.

[0055] The laser light source assembly 110 may be disposed on a substrate 130. The substrate 130 may be a PCB substrate, a copper substrate, an aluminum substrate, a BT substrate, etc. The diffractive optical element 120 may be disposed on a bracket 140. Optionally, the laser projector 100 further includes a collimator 150 for collimating the laser, wherein the diffractive optical element 120 is used to diffract the collimated laser to form a projection pattern. The collimator 150 may also be disposed on the upper bracket 140.

[0056] Specifically, the laser light source assembly 110 includes a first light emitting portion A and at least one second light emitting portion (B1, B2), wherein the first light emitting portion A and the second light emitting portion (B1, B2) emit laser light of the same wavelength. The second light emitting portion (B1, B2) includes two portions B1 and B2 spaced apart from each other along a first direction, and the two portions are located on both sides of the first light emitting portion A, respectively.

[0057] exist Figure 1In the embodiment shown, the first direction is the length direction of the first light-emitting portion A (such as the x-axis direction), and the second direction is the width direction of the first light-emitting portion A (such as the y-axis direction), and both the length direction and the width direction are perpendicular to the optical axis (the z-axis direction, which is also the height direction of A). Therefore, the two parts B1 and B2 of the second light-emitting portion (B1, B2) are located on the left and right sides of the first light-emitting portion A, respectively.

[0058] In another embodiment, the first direction is the width direction of the first light-emitting portion A (such as the y-axis direction), and the second direction is the length direction of the first light-emitting portion A (such as the x-axis direction), and both the length direction and the width direction are perpendicular to the optical axis (z-axis direction, which is also the height direction of A). Therefore, the two parts B1 and B2 of the second light-emitting portion (B1, B2) are respectively located on the front and back sides of the first light-emitting portion A. In yet another embodiment (to be described later), the first direction may also be the height direction of the first light-emitting portion A (such as the z-axis direction). In this yet another embodiment, the two parts B1 and B2 of the second light-emitting portion (B1, B2) are respectively located on the upper and lower sides of the first light-emitting portion A (refer to Fig.11 The positions of C1 and C2 are shown).

[0059] For simplicity, Figure 1 In the figure, only one second light-emitting unit (B1, B2) is shown. In other examples, more than one second light-emitting unit (B1, B2) may be included. In this case, the two parts B1 and B2 of each of the plurality of second light-emitting units (B1, B2) are sequentially arranged to diverge outward on the left and right sides of the first light-emitting unit A. For example, if two second light-emitting units (B1, B2) are included, the left part of the first second light-emitting unit (B1, B2) is located on the left side of the first light-emitting unit A, and the right part of the first second light-emitting unit (B1, B2) is located on the right side of the first light-emitting unit A; the left part of the second second light-emitting unit is located on the left side of the left part B1 of the first second light-emitting unit (B1, B2), and the right part of the second second light-emitting unit is located on the right side of the right part B2 of the first second light-emitting unit (B1, B2). In the following, for the sake of simplicity, the case where one second light-emitting unit (B1, B2) is mainly described.

[0060] Among them, Figure 1 As shown, the size of the first light emitting portion A along the first direction is the first size L 1 , the size along the second direction is the second size is H 1 The size of each of the two parts B1 and B2 of the second light emitting portion (B1, B2) along the first direction is the third size L 2 , the dimension along the second direction is the fourth dimension H 2 , where the third dimension L 2 The first size is L 1 One-half of the fourth dimension H 2With the second dimension H 1 are equal. That is, L 2 =L 1 / 2, H 2 =H 1 .

[0061] exist Figure 1 In the embodiment shown, the control component can control different light emitting parts to emit lasers at different times to form different projection patterns. For example, the control component first controls the first light emitting part A to emit lasers, and at this time a first projection pattern can be formed, such as Figure 2 As shown. The zero-order pattern corresponding to the first projection pattern is as follows Figure 3 As shown, the corresponding light source arrangement is as follows Figure 4 Afterwards, the control component controls the second light emitting portion (B1, B2) to emit laser light, and a second projection pattern can be formed at this time, as shown in FIG. Figure 5 The zero-order pattern corresponding to the second projection pattern is shown as Figure 6 As shown, the corresponding light source arrangement is as follows Figure 7 shown.

[0062] Therefore, in an embodiment of the present application, one laser projector 100 can project multiple projection patterns. Compared with the single-projector single-pattern solution of "one projector projects one projection pattern", the single-projector multi-pattern solution of the present application provides richer texture information in a unit space, so that the subsequent depth estimation results based on the projection pattern are more accurate. Specifically, the single-projector multi-pattern solution of the present application can upgrade the spatial matching in the parallax matching process to space + time matching. In the single-projector single-pattern solution, in order to ensure the matching reliability of a single speckle image, the matching window often needs to have a larger size to provide sufficient matching information, but the large-size matching box often causes the depth data to produce an obvious smoothing effect, and it is impossible to restore finer depth details. In the single-projector multi-pattern solution, since multiple images provide image information at the same time, the total matching information is equal to the single image information*n, where n is the number of images. Therefore, under the premise of ensuring the matching reliability, the geometric size of the matching box on each image can be greatly reduced, which is more conducive to restoring the detailed depth information.

[0063] Compared with the multi-projector multi-pattern solution of "multiple projectors projecting multiple projection patterns, and each projector requires a corresponding diffractive optical element and bracket", the single-projector multi-pattern solution of the present application does not require multiple groups of diffractive optical elements and brackets, but only requires one group of diffractive optical elements and brackets, which has a compact structure and saves equipment costs and space. On the other hand, in the multi-projector multi-pattern solution, multiple projectors do not share a triangulation baseline with each other. When estimating the depth, it is necessary to calculate the depth of projector patterns with different baselines separately and then perform data fusion, which will consume a lot of calculations. However, the single-projector multi-pattern solution of the present application shares the same diffractive optical element and thus the same triangulation baseline. The parallax values ​​of different patterns of pixels at the same position are also the same. In terms of algorithm, it is not necessary to calculate the depth of projector patterns with different baselines separately and then perform data fusion as in the multi-projector solution, which reduces the impact of baseline errors and is conducive to the subsequent optimization and acceleration of the algorithm.

[0064] The following describes why the single-projector multi-pattern solution of the present application can share the same diffractive optical element.

[0065] like Figure 8 As shown, the length and width of the first light emitting portion A are L 1 and H 1 , the focal length of the collimator is f. According to the grating equation:

[0066]

[0067] We can get:

[0068]

[0069]

[0070] Among them, P h and P v are the minimum lateral period and vertical period of the microstructure of the DOE device, respectively, and λ is the wavelength of the laser emitted by the first light-emitting part A.

[0071] like Fig. 9 As shown, the second light-emitting portion includes two light-emitting areas B1 and B2 separated by a certain distance, and the distance between the boundaries of the light-emitting areas B1 and B2 (that is, the distance between the two parts of the second light-emitting portion along the first direction) is L d The light-emitting areas B1 and B2 have the same size, and their length and width are L 2 and H 2 , the focal length of the collimator is f. There is a first light emitting unit A between B1 and B2 (there can be a certain interval between A and B1 and B2 or they can be set close to each other). In order to make the scattered speckle array formed by B1 and B2 after the collimator and DOE just overlap, it is necessary to make L 2 =Ld / 2N, where N is a positive integer. Preferably, N is recommended to be 1 to 3.

[0072] According to the grating equation:

[0073]

[0074] We can get:

[0075]

[0076]

[0077] Among them, P h and P v are the minimum lateral period and vertical period of the DOE device microstructure, respectively, and L = 2L 2 +L d =2(n+1)L 2 Here, due to B1 0 and B1 1 The spacing between them is 2*L 2 , so the half-width is atan(2L 2 / 2f), hence the half angle used in formula (3).

[0078] like Fig.10 As shown, in order to make both the first light-emitting part A and the second light-emitting part (B1, B2) light sources match the DOE device, the DOE microstructure needs to satisfy both the first light-emitting part A and the second light-emitting part (B1, B2), that is, the DOE devices suitable for the two chips of the first light-emitting part A and the second light-emitting part (B1, B2) have equal periods in the horizontal and vertical directions.

[0079] Specifically, let the minimum vertical periods of the two structures be equal, that is, formula (2) = formula (4):

[0080]

[0081] Thus, H 2 =H 1 The width dimension relationship between the first light emitting portion A and the second light emitting portion (B1, B2) satisfies H 2 =H 1 , the vertical periods of the two DOE microstructures are equal, and the same DOE device can be used.

[0082] Let the minimum lateral periods of the two structures be equal, that is, formula (1) = formula (3):

[0083]

[0084] Thus, L 2 =L1 / 2.

[0085] Therefore, when the size relationship between the first light-emitting part A and the second light-emitting part (B1, B2) satisfies H 2 =H 1 and L 2 =L 1 / 2, the same DOE device can be used. As mentioned above, the size relationship between the first light-emitting part A and the second light-emitting part (B1, B2) of the present application satisfies such a relationship, so the first light-emitting part A and the second light-emitting part (B1, B2) share the same diffractive optical element, and thus share the same triangulation baseline.

[0086] In the embodiment of the present application, the first light emitting part A can be realized by a vertical cavity surface emitting laser (Vcsel), and each of the two parts B1 and B2 of the second light emitting part (B1, B2) can be realized by a vertical cavity surface emitting laser (Vcsel). Therefore, the number of lasers used in the laser projector 100 of the present application can be 3, 5, 7, ..., that is, an odd number greater than or equal to 3.

[0087] In the embodiment of the present application, the diffractive optical element 120 can use a collimated light diffractive optical element and a divergent light diffractive optical element to achieve the purpose of projecting different scattered spot arrays from the same DOE device. Its material can be PC, PET, UV integrated, glass, transparent aluminum oxide material. In addition, the diffractive optical element 120 can be packaged separately to achieve higher reliability.

[0088] In the embodiment of the present application, the laser light source assembly 110 can be designed as a surface mount device (SMD) and can be applied as a surface mount technology (SMT) patch; it can also be designed as a flexible circuit board (FPC) and directly connected to the electrical system through a connector. The control assembly controls the different Vcsel operations by controlling the driving circuit, and then uses DOE to perform superposition and replication to form a pattern that meets the algorithm requirements.

[0089] exist Figure 1 In the example shown, when the laser light source assembly 110 includes a first light-emitting portion A and a second light-emitting portion (B1, B2), it can form two projection patterns. When the laser light source assembly 110 includes a first light-emitting portion A and two second light-emitting portions (B1, B2), it can form three projection patterns. That is, one light-emitting portion forms one projection pattern, and the number of light-emitting portions can be set according to needs to form a corresponding number of projection patterns.

[0090] The above exemplifies a laser projector 100 according to an embodiment of the present application. Based on the above description, the laser projector 100 according to the embodiment of the present application can realize a solution of forming multiple projection patterns with a single projector. Compared with a single-projector single-pattern solution, it can provide richer texture information in a unit space, so that the subsequent depth estimation result based on the projection pattern is more accurate. Compared with a multi-projector multi-pattern solution, it can save equipment cost and space, and can also reduce the calculation amount of the subsequent depth estimation algorithm, which is conducive to algorithm optimization and acceleration.

[0091] The following describes a laser projector 200 according to another embodiment of the present application. The laser projector 200 of this embodiment is added with a third light emitting unit (C1, C2) relative to the laser projector 100 of the previous embodiment. Fig.11 As shown. Fig.11 In the illustrated embodiment, the laser projector 200 includes a laser light source assembly 210, a control assembly (not shown), and a diffractive optical element 220. The laser light source assembly 210 includes at least three different light emitting parts. Fig.11 The first light-emitting unit A, the second light-emitting unit (B1, B2) and the third light-emitting unit (C1, C2) are shown in the figure, wherein the second light-emitting unit (B1, B2) includes two parts, namely B1 and B2, and the third light-emitting unit (C1, C2) includes two parts, namely C1 and C2). Each light-emitting unit emits laser light of the same wavelength. The control component is coupled to the laser light source component 210, and is used to control different light-emitting units to emit laser light at different times. The diffraction optical element (DOE) 220 is used to diffract the laser light emitted by different light-emitting units to form at least two different projection patterns (i.e., speckle patterns). Because at least three different light-emitting units are included, each of which can emit laser light to form a projection pattern, the laser projector 200 can ultimately form at least three projection patterns.

[0092] The laser light source assembly 210 may be disposed on a substrate 230. The substrate 230 may be a PCB substrate, a copper substrate, an aluminum substrate, a BT substrate, etc. The diffractive optical element 220 may be disposed on a bracket 240. Optionally, the laser projector 200 further includes a collimator 250 for collimating the laser, wherein the diffractive optical element 220 is used to diffract the collimated laser to form a projection pattern. The collimator 250 may also be disposed on the upper bracket 240.

[0093] Specifically, the laser light source assembly 210 includes a first light-emitting portion A, at least one second light-emitting portion (B1, B2) and at least one third light-emitting portion (C1, C2). The first light-emitting portion A, the second light-emitting portion (B1, B2) and the third light-emitting portion (C1, C2) all emit laser light of the same wavelength. The second light-emitting portion (B1, B2) includes two portions spaced apart from each other along a first direction, and the two portions are respectively located on both sides of the first light-emitting portion A. The third light-emitting portion (C1, C2) includes two portions spaced apart from each other along a third direction or a second direction, and the two portions are respectively located on both sides of the first light-emitting portion A.

[0094] exist Fig.11 In the illustrated embodiment, the first direction is the length direction of the first light-emitting portion A (such as the x-axis direction), the second direction is the width direction of the first light-emitting portion A (such as the y-axis direction), and the third direction is the height direction of the first light-emitting portion A (such as the z-axis direction). Both the length direction and the width direction are perpendicular to the optical axis (such as the z-axis direction). Therefore, the two parts of the second light-emitting portion (B1, B2) are respectively located on the left and right sides of the first light-emitting portion A. The two parts of the third light-emitting portion (C1, C2) are respectively located on the upper and lower sides of the first light-emitting portion A (at this time, the third light-emitting portion (C1, C2) includes two parts C1 and C2 spaced apart from each other along the third direction. If they are spaced apart from each other along the second direction, C1 and C2 are on the front and back sides of A, not shown).

[0095] In another embodiment (to be described later), the first direction may also be the width direction of the first light-emitting portion A, and the second direction may be the length direction of the first light-emitting portion A, and both the length direction and the width direction are perpendicular to the optical axis of the collimator. In this other embodiment, the two parts of the second light-emitting portion (B1, B2) are respectively located at the front and rear sides of the first light-emitting portion A, and the two parts of the third light-emitting portion (C1, C2) are respectively located at the left and right sides or the upper and lower sides of the first light-emitting portion A.

[0096] For simplicity, Fig.11In the figure, only one second light-emitting unit (B1, B2) and one third light-emitting unit (C1, C2) are shown. In other examples, more than one second light-emitting unit (B1, B2) and more than one third light-emitting unit (C1, C2) may be included. At this time, two parts of each of the multiple second light-emitting units (B1, B2) are arranged divergently outward on the left and right sides of the first light-emitting unit A in sequence; two parts of each of the multiple third light-emitting units (C1, C2) are arranged divergently outward on the upper and lower sides of the first light-emitting unit A in sequence. For example, if two second light-emitting units (B1, B2) are included, the left part of the first second light-emitting unit (B1, B2) is located on the left side of the first light-emitting unit A, and the right part of the first second light-emitting unit (B1, B2) is located on the right side of the first light-emitting unit A; the left part of the second second light-emitting unit is located on the left side of the left part B1 of the first second light-emitting unit (B1, B2), and the right part of the second second light-emitting unit is located on the right side of the right part B2 of the first second light-emitting unit (B1, B2). Similarly, for example, if two third light-emitting units (C1, C2) are included, the upper portion of the first third light-emitting unit (C1, C2) is located on the upper side of the first light-emitting unit A, and the lower portion of the first third light-emitting unit (C1, C2) is located on the lower side of the first light-emitting unit A; the upper portion of the second third light-emitting unit is located on the upper side of the upper portion C1 of the first third light-emitting unit (C1, C2), and the lower portion of the second third light-emitting unit is located on the lower side of the lower portion C2 of the first third light-emitting unit (C1, C2). In the following, for the sake of simplicity, the case of including one second light-emitting unit (B1, B2) and one third light-emitting unit (C1, C2) is mainly described.

[0097] Among them, Figure 1 As shown, the size of the first light emitting portion A along the first direction is the first size L 1 , the size along the second direction is the second size is H 1 The size of each of the two parts of the second light emitting portion (B1, B2) along the first direction is the third size L 2 , the dimension along the second direction is the fourth dimension H 2 , where the third dimension L 2 The first size is L 1 One-half of the fourth dimension H 2 With the second dimension H 1 are equal. That is, L 2 =L 1 / 2, H 2 =H 1 The size of each of the two parts of the third light emitting portion (C1, C2) along the first direction is the fifth size L 3 , the dimension along the second direction is the sixth dimension H 3 , where the fifth dimension L 3 With the first size L1 Equal, sixth size H 3 The second dimension H 1 That is, L 3 =L 1 , H 3 =H 1 / 2. Due to L 2 =L 1 / 2, H 2 =H 1 , therefore, L 3 =2L 2 =L 1 , H 3 =H 2 / 2=H 1 / 2.

[0098] exist Fig.11 In the embodiment shown, the control component can control different light emitting parts to emit lasers at different times to form different projection patterns. For example, the control component first controls the first light emitting part A to emit lasers, and at this time a first projection pattern can be formed, such as Figure 2 As shown. The zero-order pattern corresponding to the first projection pattern is as follows Figure 3 As shown, the corresponding light source arrangement is as follows Figure 4 Afterwards, the control component controls the second light emitting portion (B1, B2) to emit laser light, and a second projection pattern can be formed at this time, as shown in FIG. Figure 5 The zero-order pattern corresponding to the second projection pattern is shown as Figure 6 As shown, the corresponding light source arrangement is as follows Figure 7 Afterwards, the control component controls the third light emitting portion (C1, C2) to emit laser light, and a third projection pattern can be formed at this time, as shown in FIG. Fig.12 The zero-order pattern corresponding to the third projection pattern is shown as Fig.13 As shown, the corresponding light source arrangement is as follows Fig.14 shown.

[0099] Therefore, in an embodiment of the present application, one laser projector can project multiple projection patterns. Compared with the single-projector single-pattern solution of "one projector projects one projection pattern", the single-projector multi-pattern solution of the present application provides richer texture information in a unit space, so that the subsequent depth estimation results based on the projection pattern are more accurate. Specifically, the single-projector multi-pattern solution of the present application can upgrade the spatial matching in the parallax matching process to space + time matching. In the single-projector single-pattern solution, in order to ensure the matching reliability of a single speckle image, the matching window often needs to have a larger size to provide sufficient matching information, but the large-sized matching box often causes the depth data to produce an obvious smoothing effect, and it is impossible to restore finer depth details. In the single-projector multi-pattern solution, since multiple images provide image information at the same time, the total matching information is equal to the single image information*n, where n is the number of images. Therefore, under the premise of ensuring the matching reliability, the geometric size of the matching box on each image can be greatly reduced, which is more conducive to restoring detailed depth information.

[0100] Compared with the multi-projector multi-pattern solution of "multiple projectors projecting multiple projection patterns, and each projector requires a corresponding diffractive optical element and bracket", the single-projector multi-pattern solution of the present application does not require multiple groups of diffractive optical elements and brackets, but only requires one group of diffractive optical elements and brackets, which has a compact structure and saves equipment costs and space. On the other hand, in the multi-projector multi-pattern solution, multiple projectors do not share a triangulation baseline with each other. When estimating the depth, it is necessary to calculate the depth of projector patterns with different baselines separately and then perform data fusion, which will consume a lot of calculations. However, the single-projector multi-pattern solution of the present application shares the same diffractive optical element and thus the same triangulation baseline. The parallax values ​​of different patterns of pixels at the same position are also the same. In terms of algorithm, it is not necessary to calculate the depth of projector patterns with different baselines separately and then perform data fusion as in the multi-projector solution, which reduces the impact of baseline errors and is conducive to the subsequent optimization and acceleration of the algorithm.

[0101] The above has described the reason why the first light-emitting unit A and the second light-emitting unit (B1, B2) can share the same diffractive optical element. The following describes the reason why the first light-emitting unit A, the second light-emitting unit (B1, B2) and the third light-emitting unit (C1, C2) can share the same diffractive optical element.

[0102] like Fig.15 As shown, the third light-emitting portion includes two light-emitting areas C1 and C2 separated by a certain distance, and the distance between the boundaries of the light-emitting areas C1 and C2 (that is, the distance between the two parts of the third light-emitting portion along the second direction) is H. d The light-emitting areas C1 and C2 have the same size, and their length and width are L 3 and H 3, the focal length of the collimator is f. There is a first light emitting part A between C1 and C2 (there can be a certain interval between A and C1 and C2 or they can be set close to each other). In order to make the scattered speckle array formed by C1 and C2 after the collimator and DOE just overlap, it is necessary to make H 3 =H d / 2N, where N is a positive integer. Preferably, N is recommended to be 1 to 3.

[0103] According to the grating equation:

[0104]

[0105] We can get:

[0106]

[0107]

[0108] Among them, P h and P v are the minimum lateral period and vertical period of the DOE device microstructure, respectively, and H = 2H 3 +H d =2(n+1)L 2 .

[0109] like Fig.16 As shown, in order to make the three light sources of the first light-emitting part A, the second light-emitting part (B1, B2), and the third light-emitting part (C1, C2) match the DOE device, the DOE microstructure needs to simultaneously satisfy the first light-emitting part A, the second light-emitting part (B1, B2) and the third light-emitting part (C1, C2), that is, the DOE devices suitable for the three chips of the first light-emitting part A, the second light-emitting part (B1, B2) and the third light-emitting part (C1, C2) have equal periods in the horizontal and vertical directions.

[0110] Specifically, let the minimum vertical periods of the three structures be equal, that is, formula (2) = formula (4) = formula (6):

[0111]

[0112] Thus we can get: H 3 =H 2 / 2=H 1 / 2.

[0113] Let the minimum lateral periods of the two structures be equal, that is, formula (1) = formula (3) = formula (5):

[0114]

[0115] Thus we can get: L 3 =2L 2 =L1 .

[0116] Therefore, when the size relationship between the first light-emitting part A, the second light-emitting part (B1, B2) and the third light-emitting part (C1, C2) satisfies H 3 =H 2 / 2=H 1 / 2 and L 3 =2L 2 =L 1 As mentioned above, the size relationship between the first light-emitting part A, the second light-emitting part (B1, B2) and the third light-emitting part (C1, C2) of the present application satisfies such a relationship, so the first light-emitting part A, the second light-emitting part (B1, B2) and the third light-emitting part (C1, C2) share the same diffractive optical element, and thus share the same triangulation baseline.

[0117] In the embodiment of the present application, the first light emitting part A can be realized by a vertical cavity surface emitting laser (Vcsel), each of the two parts B1 and B2 of the second light emitting part (B1, B2) can be realized by a vertical cavity surface emitting laser (Vcsel), and each of the two parts C1 and C2 of the third light emitting part (C1, C2) can be realized by a vertical cavity surface emitting laser (Vcsel). Therefore, the number of lasers used in the laser projector 200 of the present application can be 5, 7, ..., that is, an odd number greater than or equal to 5.

[0118] In the embodiment of the present application, the diffractive optical element 220 can use a collimated light diffractive optical element and a divergent light diffractive optical element to achieve the purpose of projecting different scattered spot arrays from the same DOE device. Its material can be PC, PET, UV integrated, glass, transparent aluminum oxide material. In addition, the diffractive optical element 220 can be packaged separately to achieve higher reliability.

[0119] In the embodiment of the present application, the laser light source assembly 210 can be designed as a surface mount device (SMD) and can be applied as a surface mount technology (SMT) patch; it can also be designed as a flexible circuit board (FPC) and directly connected to the electrical equipment through a connector when applied. The control assembly controls the different Vcsel operations by controlling the driving circuit, and then uses DOE to perform superposition and replication to form a pattern that meets the algorithm requirements.

[0120] exist Fig.11In the example shown, when the laser light source assembly 210 includes a first light-emitting portion A and a second light-emitting portion (B1, B2) and a third light-emitting portion (C1, C2), it can form three projection patterns. When the laser light source assembly includes a first light-emitting portion A, two second light-emitting portions (B1, B2) and a third light-emitting portion (C1, C2), it can form four projection patterns. When the laser light source assembly includes a first light-emitting portion A, two second light-emitting portions (B1, B2) and two third light-emitting portions (C1, C2), it can form five projection patterns. That is, one light-emitting portion forms one projection pattern, and the number of light-emitting portions can be set as required to form a corresponding number of projection patterns.

[0121] The above exemplifies a laser projector 200 according to another embodiment of the present application. Based on the above description, the laser projector 200 according to the embodiment of the present application can realize a solution of forming multiple projection patterns with a single projector. Compared with the single-projector single-pattern solution, it can provide richer texture information in a unit space, so that the subsequent depth estimation results based on the projection pattern are more accurate. Compared with the multi-projector multi-pattern solution, it can save equipment costs and space, and can also reduce the amount of calculation of the subsequent depth estimation algorithm, which is conducive to algorithm optimization acceleration. The more light-emitting parts in the laser light source assembly, the more projection patterns can be formed, and the more accurate the depth estimation results. However, as the number of light-emitting parts increases, the cost can increase accordingly, and the volume of the equipment is also occupied. Therefore, a suitable number of light-emitting parts can be set according to the trade-offs required.

[0122] In the above combination Figure 1 and Fig.11 In the two described embodiments, the first direction is the length direction of the first light-emitting portion A, and the second direction is the width direction of the first light-emitting portion A. In such an embodiment, the two parts of the second light-emitting portion are respectively located on the left and right sides of the first light-emitting portion, and the third light-emitting portion is respectively located on the front and rear sides or the upper and lower sides of the first light-emitting portion.

[0123] In other embodiments, the first direction may also be the width direction of the first light-emitting portion A, and the second direction may be the length direction of the first light-emitting portion A. In such an embodiment, the two parts of the second light-emitting portion are respectively located at the front and rear sides of the first light-emitting portion, and the third light-emitting portion is respectively located at the left and right sides or the upper and lower sides of the first light-emitting portion. Compared with the above-mentioned embodiments, the only difference is the position, and the principle is the same, so it will not be repeated here for the sake of brevity.

[0124] Combine the following Fig.17 The camera assembly 1700 according to an embodiment of the present application is described. Fig.17As shown, the camera assembly 1700 may include a laser projector 1710, an image sensor 1720, and a processor 1730. The laser projector 1710 may be the laser projector 100 or 200 according to the embodiment of the present application described above, which is a laser projector that forms multiple projection patterns through a single projector. The laser projector 1710 is used to emit laser light to the object to be measured to form at least two different projection patterns. The image sensor 1720 is used to collect at least two different projection patterns to obtain at least two laser images; the processor 1730 is used to process at least two laser images to obtain a depth image. The processor 1730 processes at least two laser images to obtain a depth image, including: encoding each of the at least two laser images separately to obtain an encoding result of each laser image; integrating the encoding results of all laser images to obtain the encoding data of the object to be measured; obtaining preset reference encoding data, calculating the matching cost of the encoding data of the object to be measured and the reference encoding data under all possible parallaxes; calculating the disparity map based on the matching cost, and obtaining the depth image based on the disparity map.

[0125] In the embodiment of the present application, since a single laser projector 1710 can form at least two projection patterns, the image sensor 1720 can capture at least two laser images. In addition, since the multiple light-emitting parts in a single laser projector 1710 share the same triangulation baseline, the disparity values ​​of different patterns of pixels at the same position are also the same. Therefore, in terms of algorithm, it is not necessary to calculate the depth of different baseline projector patterns separately and then perform data fusion as in the multi-projector solution. Instead, the encoding results of all laser patterns can be integrated in the encoding stage, and then only one cost matching and disparity map calculation operation is performed. That is, the solution of the present application only requires one depth calculation for multiple different projector patterns of the same baseline, and does not require multiple depth calculations followed by data fusion, thereby greatly reducing the amount of calculation.

[0126] The operation of depth calculation by the processor 1730 is described in more detail below by taking the laser projector 1710 projecting n projection patterns as an example.

[0127] Before projecting laser light onto the object to be measured to obtain a projection pattern (i.e., a speckle image), a reference code data needs to be obtained through a preset operation. Specifically, the laser projector 1710 first projects a laser beam at a specified distance Z. 0Project the laser downward toward the reference object (generally, the reference plane can be selected) to obtain n speckle images. Then, encode the n plane speckle images separately. Taking the more classic census encoding as an example, traverse each point in the speckle image and draw a rectangle (size is, for example, 3×3, or other sizes) with the point as the center. Each point in the rectangle except the center point is compared with the center point. The grayscale value greater than the center point is recorded as 1, and less than the center point is recorded as 0, and a coding sequence of only 0 and 1 with a length of 8 is obtained. Replace the grayscale value of the point with this census sequence, and the census encoding of a single speckle image is completed. The encoded data has stronger anti-interference ability. After that, according to the number of speckle images n, different types of variables can be allocated to integrate the encoding results. If n=2, int16 type data can be used to store the first speckle pattern encoding result in the lower 8 bits and the second speckle pattern encoding result in the upper 8 bits. If n=3 or 4, int32 type data can be used to store the first speckle pattern coding result in the lower 8 bits, and the storage location of the second, third or fourth speckle pattern coding results increases by 8 bits in sequence. If n continues to increase, the corresponding data type bit number should also increase accordingly. That is, according to the total number of laser images and the number of coding bits for each point in each laser image, the corresponding type of variable can be allocated, and the coding results of each corresponding point in each laser image can be integrated into one variable to obtain the coding data of the object under test. Afterwards, the step coding integration result is stored in the device as the reference coding data R for standby use.

[0128] When the camera assembly 1700 is working, n speckle images of the object to be measured are collected in real time, and the aforementioned encoding operation and encoding integration operation are repeated to obtain the encoding data O of the object to be measured.

[0129] After obtaining the coded data O of the object to be measured, the matching cost of the coded data O of the object to be measured and the reference coded data R under all possible parallaxes is calculated. The "all possible parallaxes" here are related to the working distance of the camera assembly 1700. For example, the working distance of the camera assembly 1700 is 20 cm to 2 m, and this distance range will correspond to a parallax range when converted to the image. All parallaxes in the parallax range are all possible parallaxes.

[0130] Among them, the method of calculating the matching cost can be specifically: traverse each point Po in the coded data O of the object under test, draw an L*L rectangle with it as the center, and then find the corresponding point Pr of Po in R corresponding to the current disparity, and also draw an L*L rectangle with Pr as the center, perform an XOR operation on the coded values ​​at the same position in the two rectangles, obtain the Hamming distance and accumulate it as the matching cost of Po and Pr under the current disparity.

[0131] Calculate the matching cost of the coded data O of the object under test and the reference coded data R under all possible parallaxes to form a matching cost space. This step is the most computationally intensive part of the stereo matching process. Since the O and R data constructed by multiple pattern codes have n times more information per point than the single pattern code, the single-projector multi-pattern solution can obtain a completely close matching capability with a rectangle that is equivalent to 1 / n of the L*L matching rectangle area in the single-projector single-pattern solution. As the size of the matching rectangle is reduced, the interference caused by the local deformation of the object surface is reduced, so the ability to restore the details of the object is stronger.

[0132] After forming a matching cost space based on the matching cost, a disparity map can be calculated in the matching cost space. Optionally, the matching cost space can be further optimized using aggregation algorithms such as semi-global matching (SGM) to obtain an optimized matching cost space, and the disparity map is calculated in the optimized matching cost space. Specifically, the disparity map is calculated in the matching cost space, and specifically, for each point in the matching cost space, the matching costs of the point under all possible disparities are traversed, the disparity corresponding to the extreme matching cost is found, and the disparity corresponding to the extreme matching cost is interpolated to obtain the final disparity value of the point, and the final disparity value is a floating point value (a value with a decimal point); based on the final disparity value of each point in the matching cost space, a disparity map is obtained.

[0133] The reason why the disparity corresponding to the extreme matching cost is interpolated is that all possible disparities preset when constructing the matching cost space are integers (although described as all possible disparities, there are actually infinite disparities that cannot be listed one by one, and can only be expressed by sampling using integers). Therefore, the disparity corresponding to the extreme value in the matching cost space is also an integer, and integers cannot accurately express the true disparity value. Assuming that the true disparity values ​​of several consecutive points are 0.9, 1.0, 1.1, and 1.2, the disparity corresponding to the extreme value in the matching cost space is the integer 1. If it is directly converted to depth without interpolation, the depth values ​​of the several points are exactly the same. Therefore, interpolation can restore the decimal part of the disparity of each point as much as possible, thereby showing the subtle changes in depth and obtaining a more accurate depth image.

[0134] Here, the extreme matching cost to be found can be the minimum matching cost or the maximum matching cost, which depends on the specific calculation method for obtaining the matching cost. In the example described above, the matching cost is calculated in the following way: traverse each point Po in the coded data O of the object to be measured, draw an L*L rectangle with it as the center, and then find the corresponding point Pr in R corresponding to the current disparity of Po, and draw an L*L rectangle with Pr as the center. Perform an XOR operation on the coded values ​​at the same position in the two rectangles, obtain the Hamming distance and accumulate it as the matching cost of Po and Pr under the current disparity. Under this calculation method, the extreme matching cost to be found should be the minimum matching cost. In other calculation methods, the extreme matching cost to be found may also be the maximum matching cost.

[0135] After obtaining the disparity map, the depth image can be obtained by the depth disparity formula. Optionally, the disparity map can also be verified for left-right consistency to remove the error data in the disparity map, and the depth image can be obtained based on the disparity map that has been verified for left-right consistency, which can further improve the accuracy of the generated depth image. Among them, the disparity value of each point in the disparity map can be converted into a depth value by the depth disparity formula to form a depth map. The formula is as follows:

[0136]

[0137] Among them, Z 0 is the shooting distance of the reference coded data R, f is the internal reference focal length of the image sensor 1720, b is the baseline length from the laser projector 1710 to the image sensor 1720, d is the parallax value, and Z is the depth value.

[0138] The above describes in detail the process of depth calculation by the processor 1730. The depth calculation relies on at least two projection patterns projected by the laser projector 1710 according to the embodiment of the present application as described above, wherein different light-emitting parts are controlled by a control component to emit light at different times to generate projection patterns respectively, and the processor 1730 may include the control component to control the generation of the projection pattern.

[0139] Based on the above description, the camera assembly 1700 according to the embodiment of the present application can implement a solution of forming multiple projection patterns with a single projector. Compared with the single-projector single-pattern solution, it can provide richer texture information within a unit space, so that the subsequent depth estimation results based on the projection pattern are more accurate. Compared with the multi-projector multi-pattern solution, it can save equipment costs and space, and can also reduce the computational complexity of the depth estimation algorithm, which is conducive to algorithm optimization and acceleration.

[0140] According to another aspect of the present application, an electronic device is also provided, which may include a housing and the camera assembly according to the embodiment of the present application described above, wherein the camera assembly is arranged on the housing and exposed from the housing to obtain a depth image. The structure and operation of the camera assembly have been described in detail above, and those skilled in the art may understand the structure and operation of the camera assembly in the electronic device in combination with the above, and for the sake of brevity, they will not be described here in detail.

[0141] Combine the following Fig.18 A method 1800 for generating a depth image according to another aspect of the present application is described. Fig.18 As shown, the method 1800 for generating a depth image may include the following steps:

[0142] In step S1810, at least two laser images are acquired, wherein the at least two laser images are obtained by an image sensor capturing at least two different projection patterns, and the at least two different projection patterns are formed by a single laser projector according to an embodiment of the present application emitting laser to the object to be measured.

[0143] In step S1820, each of the at least two laser images is encoded respectively to obtain an encoding result of each laser image.

[0144] In step S1830, the encoding results of all laser images are integrated to obtain the encoding data of the object under test.

[0145] In step S1840, preset reference coding data is obtained, and matching costs between the coding data of the object under test and the reference coding data under all possible parallaxes are calculated.

[0146] In step S1850 , a disparity map is calculated based on the matching cost, and a depth image is obtained based on the disparity map.

[0147] The method 1800 for generating a depth image according to an embodiment of the present application is an operation performed by the processor in the camera assembly described above. The operations performed by the processor have been described in detail above. Those skilled in the art can understand the operations and principles of the method 1800 for generating a depth image in combination with the above. For the sake of brevity, the details will not be repeated here, and only some main operations will be described.

[0148] In an embodiment of the present application, in step S1820, each of the at least two laser images is encoded separately to obtain the encoding result of each laser image, which may include: allocating variables of corresponding types according to the total number of laser images and the number of encoding bits for each point in each laser image; integrating the encoding results of each corresponding point in each laser image into a variable to obtain the encoding data of the object under test.

[0149] In an embodiment of the present application, calculating the disparity map based on the matching cost in step S1850 may include: constructing a matching cost space based on the matching cost; optimizing the matching cost space using an aggregation algorithm to obtain an optimized matching cost space; and calculating the disparity map in the optimized matching cost space.

[0150] In an embodiment of the present application, calculating the disparity map in step S1850 may include: for each point in the matching cost space, traversing the matching costs of the point under all possible disparities, finding the disparity corresponding to the extreme matching cost, interpolating the disparity corresponding to the extreme matching cost, and obtaining the final disparity value of the point, where the final disparity value is a floating point value; and obtaining a disparity map based on the final disparity value of each point in the matching cost space.

[0151] In an embodiment of the present application, method 1800 may further include the following steps: performing left-right consistency verification on the disparity map to remove error data in the disparity map; and obtaining a depth image based on the disparity map that has undergone left-right consistency verification.

[0152] Based on the above description, the method 1800 for generating a depth image according to an embodiment of the present application generates a depth image based on multiple projection patterns of a common triangulation baseline, which can realize the calculation of the depth image with less computational effort, and there is no need to calculate the depth of different baseline projector patterns separately and then perform data fusion like the multi-projector solution.

[0153] Combine the following Fig.19 The apparatus 1900 for generating a depth image provided according to another aspect of the present application is described. Fig.19 As shown, the device 1900 for generating a depth image includes an acquisition module 1910, an encoding module 1920, an integration module 1930, a first calculation module 1940 and a second calculation module 1950. The acquisition module 1910 is used to acquire at least two laser images, wherein the at least two laser images are obtained by an image sensor collecting at least two different projection patterns, and the at least two different projection patterns are formed by the laser projector according to the embodiment of the present application emitting lasers to the object to be measured. The encoding module 1920 is used to encode each of the at least two laser images respectively to obtain the encoding result of each laser image. The integration module 1930 is used to integrate the encoding results of all laser images to obtain the encoding data of the object to be measured. The first calculation module 1940 is used to obtain preset reference encoding data and calculate the matching cost between the encoding data of the object to be measured and the reference encoding data under all possible parallaxes. The second calculation module 1950 is used to calculate the disparity map based on the matching cost, and obtain the depth image based on the disparity map.

[0154] According to an embodiment of the present application, the device 1900 for generating a depth image is used to execute the method 1800 for generating a depth image described above. The method 1800 has been described in detail above. Those skilled in the art can understand the structure and operation of the device 1900 for generating a depth image in combination with the above. For the sake of brevity, the details are not repeated here, and only some main operations are described.

[0155] In an embodiment of the present application, the encoding module 1920 encodes each of the at least two laser images separately to obtain the encoding result of each laser image, which may include: allocating variables of corresponding types according to the total number of laser images and the number of encoding bits for each point in each laser image; integrating the encoding results of each corresponding point in each laser image into a variable to obtain the encoding data of the object under test.

[0156] In an embodiment of the present application, the second calculation module 1950 calculates the disparity map based on the matching cost, which may include: constructing a matching cost space based on the matching cost; optimizing the matching cost space using an aggregation algorithm to obtain an optimized matching cost space; and calculating the disparity map in the optimized matching cost space.

[0157] In an embodiment of the present application, calculating the disparity map in the second calculation module 1950 may include: for each point in the matching cost space, traversing the matching costs of the point under all possible disparities, finding the disparity corresponding to the extreme matching cost, interpolating the disparity corresponding to the extreme matching cost, and obtaining the final disparity value of the point, where the final disparity value is a floating point value; based on the final disparity value of each point in the matching cost space, obtaining a disparity map.

[0158] In an embodiment of the present application, the second calculation module 1950 may also be used to: perform left-right consistency verification on the disparity map to remove error data in the disparity map; and obtain a depth image based on the disparity map that has undergone left-right consistency verification.

[0159] Based on the above description, the device 1900 for generating a depth image according to an embodiment of the present application generates a depth image based on multiple projection patterns of a common triangulation baseline, which can realize the calculation of the depth image with less computational effort, and there is no need to calculate the depth of different baseline projector patterns separately and then perform data fusion like a multi-projector solution.

[0160] In addition, according to an embodiment of the present application, a storage medium is also provided, on which program instructions are stored, and when the program instructions are executed by a computer or a processor, the corresponding steps of the method for generating a depth image of the embodiment of the present application are executed. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0161] In addition, according to an embodiment of the present application, a computer program is also provided, which can be stored in a cloud or local storage medium. When the computer program is executed by a computer or a processor, it is used to execute the corresponding steps of the method for generating a depth image in the embodiment of the present application.

[0162] Based on the above description, the laser projector, camera assembly, and electronic device according to the embodiments of the present application can realize a solution of forming multiple projection patterns with a single projector. Compared with the single-projector single-pattern solution, it can provide richer texture information in a unit space, so that the subsequent depth estimation results based on the projection pattern are more accurate. Compared with the multi-projector multi-pattern solution, it can save equipment costs and space, and can also reduce the amount of calculation of the subsequent depth estimation algorithm, which is conducive to algorithm optimization and acceleration. The method, device, storage medium, and computer program according to the embodiments of the present application generate depth images based on multiple projection patterns with a common triangulation baseline, and can realize the calculation of depth images with less calculation, without having to calculate the depth of different baseline projector patterns separately and then perform data fusion like the multi-projector solution.

[0163] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0164] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0165] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0166] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0167] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0168] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0169] In addition, those skilled in the art will appreciate that, although some embodiments herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0170] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules in the article analysis device according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0171] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0172] The above is only a specific embodiment or description of a specific embodiment of the present invention, and the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A laser projector, characterized in that: The laser projector comprises: A laser light source assembly, comprising a first light emitting portion and at least one second light emitting portion, wherein the first light emitting portion and the second light emitting portion emit laser light of the same wavelength, and the second light emitting portion comprises two portions spaced apart from each other along a first direction, and the two portions are respectively located on both sides of the first light emitting portion; A control component, coupled to the laser light source component, for controlling different light emitting parts to emit lasers at different times; a diffractive optical element, used for diffracting lasers emitted by different light-emitting parts to form at least two different projection patterns; Among them, the size of the first light-emitting portion along the first direction is a first size, and the size along the second direction is a second size, the size of each of the two parts of the second light-emitting portion along the first direction is a third size, and the size along the second direction is a fourth size, wherein the third size is half of the first size, and the fourth size is equal to the second size.

2. The laser projector according to claim 1, characterized in that: The laser light source assembly further includes at least one third light emitting portion, the first light emitting portion and the third light emitting portion emit laser light of the same wavelength, the third light emitting portion includes two portions spaced apart from each other along the second direction, the two portions are respectively located on both sides of the first light emitting portion; Among them, each of the two parts of the third light-emitting portion has a fifth size along the first direction and a sixth size along the second direction, wherein the fifth size is equal to the first size, and the sixth size is half of the second size.

3. The laser projector according to claim 1 or 2, characterized in that: The laser projector also includes a collimating lens for collimating the laser, wherein the diffraction optical element is used to diffract the collimated laser to form the projection pattern.

4. The laser projector according to claim 3, characterized in that: The first direction is the length direction of the first light-emitting portion, and the second direction is the width direction of the first light-emitting portion. Both the length direction and the width direction are perpendicular to the optical axis of the collimator.

5. The laser projector according to claim 3, characterized in that: The first direction is the width direction of the first light-emitting portion, and the second direction is the length direction of the first light-emitting portion. Both the length direction and the width direction are perpendicular to the optical axis of the collimator.

6. The laser projector according to claim 1, characterized in that: A distance between the two parts of the second light emitting portion along the first direction is 2N times the third size, where N is a positive integer.

7. The laser projector according to claim 2, characterized in that: A distance between the two parts of the third light emitting portion along the second direction is 2N times the sixth dimension, where N is a positive integer.

8. The laser projector according to claim 7, characterized in that: The N is any one of 1, 2 and 3.

9. A camera assembly, characterized in that: The camera assembly comprises: The laser projector according to any one of claims 1 to 8, wherein the laser projector is used to emit laser light to the object to be measured to form the at least two different projection patterns; An image sensor, used for collecting the at least two different projection patterns to obtain at least two laser images; A processor, configured to process the at least two laser images to obtain a depth image, wherein the processor processes the at least two laser images to obtain a depth image, comprising: Encoding each of the at least two laser images respectively to obtain an encoding result of each of the laser images; Integrating the coding results of all the laser images to obtain the coding data of the object to be measured; Obtaining preset reference coding data, and calculating matching costs between the coding data of the object under test and the reference coding data under all possible parallaxes; A disparity map is calculated based on the matching cost, and a depth image is obtained based on the disparity map.

10. The camera assembly according to claim 9, characterized in that: The processor encodes each of the at least two laser images respectively to obtain an encoding result of each of the laser images, including: Allocate variables of corresponding types according to the total number of laser images and the number of encoding bits for each point in each laser image; The coding results of each corresponding point in each of the laser images are integrated into one of the variables to obtain the coding data of the object to be measured.

11. The camera assembly according to claim 9, characterized in that: The processor calculates a disparity map based on the matching cost, including: forming a matching cost space based on the matching cost; Optimizing the matching cost space by using an aggregation algorithm to obtain an optimized matching cost space; A disparity map is calculated in the optimized matching cost space.

12. The camera assembly according to claim 11, characterized in that: The processor calculates the disparity map, including: For each point in the matching cost space, traverse the matching costs of the point under all possible disparities, find the disparity corresponding to the extreme matching cost, interpolate the disparity corresponding to the extreme matching cost, and obtain a final disparity value of the point, where the final disparity value is a floating point value; The disparity map is obtained based on the final disparity value of each point in the matching cost space.

13. The camera assembly according to any one of claims 9 to 12, characterized in that: The processor is further configured to: Performing left-right consistency verification on the disparity map to eliminate error data in the disparity map; A depth image is obtained based on the disparity map verified by the left-right consistency.

14. The camera assembly according to any one of claims 9 to 12, characterized in that: The processor includes the control component.

15. An electronic device, characterized in that: include: shell; The camera assembly according to any one of claims 9 to 14, wherein the camera assembly is disposed to the housing and exposed from the housing to obtain a depth image.

16. A method for generating a depth image, characterized in that The method comprises: Acquire at least two laser images, wherein the at least two laser images are obtained by collecting at least two different projection patterns by an image sensor, and the at least two different projection patterns are formed by the laser projector according to any one of claims 1 to 8 emitting laser light to the object to be measured; Encoding each of the at least two laser images respectively to obtain an encoding result of each of the laser images; Integrating the coding results of all the laser images to obtain the coding data of the object to be measured; Obtaining preset reference coding data, and calculating matching costs between the coding data of the object under test and the reference coding data under all possible parallaxes; A disparity map is calculated based on the matching cost, and a depth image is obtained based on the disparity map.

17. The method according to claim 16, characterized in that The step of encoding each of the at least two laser images to obtain an encoding result of each of the laser images comprises: Allocate variables of corresponding types according to the total number of laser images and the number of encoding bits for each point in each laser image; The coding results of each corresponding point in each of the laser images are integrated into one of the variables to obtain the coding data of the object to be measured.

18. The method according to claim 16, characterized in that The calculating the disparity map based on the matching cost includes: forming a matching cost space based on the matching cost; Optimizing the matching cost space by using an aggregation algorithm to obtain an optimized matching cost space; A disparity map is calculated in the optimized matching cost space.

19. The method according to claim 18, characterized in that The calculating the disparity map comprises: For each point in the matching cost space, traverse the matching costs of the point under all possible disparities, find the disparity corresponding to the extreme matching cost, interpolate the disparity corresponding to the extreme matching cost, and obtain a final disparity value of the point, where the final disparity value is a floating point value; The disparity map is obtained based on the final disparity value of each point in the matching cost space.

20. The method according to any one of claims 16 to 19, characterized in that The method further comprises: Performing left-right consistency verification on the disparity map to eliminate error data in the disparity map; A depth image is obtained based on the disparity map verified by the left-right consistency.

21. A device for generating a depth image, characterized in that The device comprises: An acquisition module, used for acquiring at least two laser images, wherein the at least two laser images are obtained by an image sensor collecting at least two different projection patterns, and the at least two different projection patterns are formed by the laser projector according to any one of claims 1 to 8 emitting laser light to the object to be measured; An encoding module, used to encode each of the at least two laser images respectively to obtain an encoding result of each of the laser images; An integration module, used for integrating the coding results of all the laser images to obtain the coding data of the object to be measured; A first calculation module, used to obtain preset reference coding data, and calculate the matching cost between the coding data of the object under test and the reference coding data under all possible parallaxes; The second calculation module is used to calculate a disparity map based on the matching cost, and obtain a depth image based on the disparity map.

22. The device according to claim 21, characterized in that The encoding module encodes each of the at least two laser images respectively to obtain an encoding result of each of the laser images, including: Allocate variables of corresponding types according to the total number of laser images and the number of encoding bits for each point in each laser image; The coding results of each corresponding point in each of the laser images are integrated into one of the variables to obtain the coding data of the object to be measured.

23. The device according to claim 21, characterized in that The second calculation module calculates the disparity map based on the matching cost, including: forming a matching cost space based on the matching cost; Optimizing the matching cost space by using an aggregation algorithm to obtain an optimized matching cost space; A disparity map is calculated in the optimized matching cost space.

24. The device according to claim 23, characterized in that The second calculation module calculates the disparity map, including: For each point in the matching cost space, traverse the matching costs of the point under all possible disparities, find the disparity corresponding to the extreme matching cost, interpolate the disparity corresponding to the extreme matching cost, and obtain a final disparity value of the point, where the final disparity value is a floating point value; The disparity map is obtained based on the final disparity value of each point in the matching cost space.

25. The device according to any one of claims 21 to 24, characterized in that The second calculation module is also used for: Performing left-right consistency verification on the disparity map to eliminate error data in the disparity map; A depth image is obtained based on the disparity map verified by the left-right consistency.

26. A device for generating a depth image, characterized in that The device includes a memory and a processor, wherein a computer program executed by the processor is stored in the memory, and when the computer program is executed by the processor, the processor executes the method for generating a depth image according to any one of claims 16 to 20.

27. A storage medium, characterized in that: The storage medium stores a computer program executed by a processor, and when the computer program is executed by the processor, the processor executes the method for generating a depth image according to any one of claims 16 to 20.

28. A computer program, characterized in that When the computer program is executed by a processor, the processor is caused to perform the method for generating a depth image according to any one of claims 16 to 20.

Images