Visual system, depth detection method thereof, robot and readable storage medium

Abstract

The invention discloses a visual system, a depth detection method thereof, a robot and a readable storage medium. The depth detection method comprises the following steps: determining a first depth value of each pixel of the reflected light based on a time-of-flight method; determining a second depth value of at least part of pixels of the reflected light based on a triangulation method; and determining a final depth value of each pixel according to the first depth value, the second depth value and a weight impact factor, wherein the weight impact factor is related to an optical parameter of the reflected light and / or a physical parameter of the visual system.

Description

Technical Field

[0001] The present application relates to the field of optical technology, and in particular to a visual system and a depth detection method thereof, a robot and a readable storage medium. Background Art

[0002] Structured light vision technology is widely used in the fields of three-dimensional model reconstruction, object surface contour measurement, especially robot obstacle avoidance, due to its advantages such as non-contact, high controllability, high close-range measurement accuracy, fast measurement speed, wide measurement range and high real-time performance.

[0003] TOF (Time of Flight) technology is a commonly used structured light vision technology. The principle of TOF technology is to continuously emit light pulses to the target object, and detect the time interval from the emission to the reception of the light pulse or the phase difference caused by the round trip of the light pulse based on the light pulse reflected by the target object, so as to measure the distance, three-dimensional profile and other parameters of the target object. However, the environment where the target object is located usually has the problem of multi-path reflection interference, causing the actually received light pulse to include the reflected light of other objects, resulting in poor accuracy of the detected depth value. Summary of the invention

[0004] The visual system and its depth detection method, robot and readable storage medium provided in the embodiments of the present application can solve or partially solve the above-mentioned deficiencies in the prior art or other deficiencies in the prior art.

[0005] According to a first aspect of the present application, a depth detection method of a visual system is provided, wherein the visual system comprises a light emitting component and a light receiving component, wherein the light emitting component is used to emit light to a target object, and the light receiving component is used to receive reflected light after the emitted light is reflected by the target object;

[0006] Wherein, the depth detection method comprises:

[0007] Determine a first depth value of each pixel of the reflected light based on a time-of-flight method;

[0008] Determining a second depth value of at least some pixels of the reflected light based on triangulation; and

[0009] A final depth value of each pixel is determined according to the first depth value, the second depth value and a weight influence factor, where the weight influence factor is related to an optical parameter of the reflected light and / or a physical parameter of the visual system.

[0010] According to one embodiment of the present application, the weight influencing factor includes a ranging error determined based on a physical parameter of the visual system, the time-of-flight method is an indirect time-of-flight method, and the pixel of the reflected light includes a target pixel;

[0011] Wherein, determining the final depth value of each pixel according to the first depth value, the second depth value and the weight influence factor includes:

[0012] determining, according to the ranging error, a first weight corresponding to the first depth value of the target pixel and a second weight corresponding to a third depth value of the target pixel, wherein the third depth value is determined based at least on the second depth value of the at least part of the pixels, and the first weight and the second weight are greater than 0 and less than 1; and

[0013] A weighted sum is performed on the first weight, the first depth value, the second weight, and the third depth value to obtain the final depth value of the target pixel.

[0014] According to an embodiment of the present application, determining, according to the ranging error, a first weight corresponding to the first depth value of the target pixel and a second weight corresponding to the third depth value of the target pixel includes:

[0015] In response to the ranging error being not less than a first preset threshold and not greater than a second preset threshold, determining the first weight and the second weight according to the ranging error;

[0016] Wherein, determining a final depth value of each pixel according to the first depth value, the second depth value and the ranging error further comprises:

[0017] In response to the ranging error being less than the first preset threshold, setting the second depth value of the target pixel as the final depth value of the target pixel; and

[0018] In response to the ranging error being greater than the second preset threshold, the first depth value of the target pixel is set as the final depth value of the target pixel.

[0019] According to an embodiment of the present application, performing weighted summation on the first weight, the first depth value, the second weight, and the third depth value includes:

[0020] A = 1-1 / a;

[0021] B = 1 / a;

[0022] d=A*d itof +B*d 线结构光 ;

[0023] Wherein, d is the final depth value of the target pixel, A is the first weight, B is the second weight, a is the ranging error, and d itof is the first depth value of the target pixel, d 线结构光 is the third depth value of the target pixel.

[0024] According to one embodiment of the present application, the weight influencing factor includes a ranging error determined based on a physical parameter of the visual system, the time-of-flight method is an indirect time-of-flight method, and the pixel of the reflected light includes a target pixel;

[0025] Wherein, determining the final depth value of each pixel according to the first depth value, the second depth value and the weight influence factor includes:

[0026] In response to the ranging error being less than a preset threshold, setting a third depth value of the target pixel as the final depth value, the third depth value being determined based at least on the second depth value of the at least part of the pixels; and

[0027] In response to the ranging error being not less than the preset threshold, setting the first depth value of the target pixel as the final depth value.

[0028] According to one embodiment of the present application, the emitted light is single-linear modulated structured light, multi-linear modulated structured light or scattered-speckle modulated structured light, the target pixel is any pixel of the reflected light, and the third depth value is equal to the second depth value of the target pixel.

[0029] According to one embodiment of the present application, the emitted light is the scattered speckle modulated structured light;

[0030] Wherein, determining the second depth value of at least part of the pixels of the reflected light based on the triangulation method comprises:

[0031] Determine the parallax of the scattered spots of the reflected light based on the monocular structured light principle; and

[0032] The second depth value is determined based on triangulation according to the parallax, a baseline length between the light emitting component and the light receiving component, and a focal length of the light receiving component.

[0033] According to an embodiment of the present application, the emitted light is single-light-band structured light or multi-light-band structured light, the target pixel is a pixel at the center of the light band of the reflected light, and the third depth value is equal to the second depth value.

[0034] According to an embodiment of the present application, the emitted light is single-light-band structured light or multi-light-band structured light, the target pixel is a pixel at a non-light-band center of the reflected light, and the at least part of the pixels are pixels at a light-band center of the reflected light;

[0035] Wherein, the depth detection method of the visual system further includes:

[0036] calculating a first depth difference between the first depth value of the target pixel and the first depth value of an auxiliary pixel, the auxiliary pixel being a pixel in the at least some pixels that is located in the same column as the target pixel; and

[0037] Determine a second depth estimation value of the target pixel according to the first depth difference and a second depth value of the auxiliary pixel;

[0038] The third depth value is equal to the second depth estimation value.

[0039] According to one embodiment of the present application, the physical parameters of the visual system include the actual physical size of the picture element of the light receiving component and the baseline length between the light emitting component and the light receiving component.

[0040] According to an embodiment of the present application, the pixel of the reflected light includes a target pixel, the time-of-flight method is an indirect time-of-flight method, the weight influencing factor includes a light spot offset of the target pixel, the light spot offset is determined based on an optical parameter of the target pixel and an optical parameter of a preset calibration pixel, and the optical parameter includes the number of rows or columns;

[0041] Wherein, determining the final depth value of each pixel according to the first depth value, the second depth value and the weight influence factor includes:

[0042] In response to the light spot offset being less than a preset offset, setting the first depth value of the target pixel to the final depth value; and

[0043] In response to the light spot offset being not less than the preset offset, the second depth value of the target pixel is set as the final depth value.

[0044] According to an embodiment of the present application, the pixel of the reflected light includes a target pixel, the time-of-flight method is an indirect time-of-flight method, and the weight influencing factor includes an optical parameter of the target pixel, and the optical parameter includes a light spot width;

[0045] Wherein, determining the final depth value of each pixel according to the first depth value, the second depth value and the weight influence factor includes:

[0046] In response to the light spot width being smaller than a preset width, setting the second depth value of the target pixel as the final depth value; and

[0047] In response to the light spot width being not less than the preset width, the first depth value of the target pixel is set as the final depth value.

[0048] According to an embodiment of the present application, the emitted light is single-linear modulated structured light or multi-linear modulated structured light, and the target pixel is any pixel of the reflected light.

[0049] According to one embodiment of the present application, the weight influencing factor includes a ranging error determined based on physical parameters of the visual system, the time-of-flight method is a direct time-of-flight method, and the pixel of the reflected light includes a target pixel;

[0050] Wherein, determining the first depth value of each pixel of the reflected light based on the time-of-flight method comprises: determining the first depth value of each pulse wave peak corresponding to each pixel based on the direct time-of-flight method;

[0051] Wherein, determining the final depth value of each pixel according to the first depth value, the second depth value and the weight influence factor includes:

[0052] Filtering a candidate depth value from the first depth values ​​of each pulse wave peak corresponding to the target pixel according to the ranging error and a fourth depth value of the target pixel, wherein the fourth depth value is determined based on at least the second depth value of at least part of the pixels; and

[0053] The final depth value of the target pixel is selected from the candidate depth values ​​according to the fourth depth value of the target pixel.

[0054] According to an embodiment of the present application, selecting a candidate depth value from the first depth values ​​of each pulse wave peak corresponding to the target pixel according to the ranging error and the fourth depth value of the target pixel includes:

[0055] In response to the first depth value of the pulse peak being at d′ 线结构光 Within the range of ±a, the first depth value of the pulse peak is set as the candidate depth value;

[0056] Among them, d′ 线结构光 is the fourth depth value of the target pixel, and a is the ranging error.

[0057] According to an embodiment of the present application, filtering the final depth value of the target pixel from the candidate depth values ​​according to the second depth value of at least part of the pixels includes:

[0058] Calculating a second depth difference between each of the candidate depth values ​​and the fourth depth value of the target pixel; and

[0059] The candidate depth value corresponding to the second depth difference value with the smallest absolute value among the plurality of second depth differences is set as the final depth value of the target pixel.

[0060] According to one embodiment of the present application, the emitted light is single-linear modulated structured light, multi-linear modulated structured light or scattered-speckle modulated structured light, the target pixel is any pixel of the reflected light, and the fourth depth value is equal to the second depth value of the target pixel.

[0061] According to one embodiment of the present application, the emitted light is the scattered speckle modulated structured light;

[0062] Wherein, determining the second depth value of at least part of the pixels of the reflected light based on the triangulation method comprises:

[0063] Determine the parallax of the scattered spots of the reflected light based on the monocular structured light principle; and

[0064] The second depth value is determined based on triangulation according to the parallax, a baseline length between the light emitting component and the light receiving component, and a focal length of the light receiving component.

[0065] According to an embodiment of the present application, the emitted light is single-band structured light or multi-band structured light, the target pixel is a pixel at the center of the light band of the reflected light, and the fourth depth value is equal to the second depth value of the target pixel.

[0066] According to an embodiment of the present application, the emitted light is single-light-band structured light or multi-light-band structured light, the target pixel is a pixel at a non-light-band center of the reflected light, and the at least part of the pixels are pixels at a light-band center of the reflected light;

[0067] Wherein, the depth detection method of the visual system further includes:

[0068] Calculating a third depth difference between each of the first depth values ​​of the target pixel and the first depth value corresponding to an auxiliary pixel, the auxiliary pixel being a pixel in the at least some of the pixels that is located in the same column as the target pixel;

[0069] Determine a target depth difference according to the plurality of third depth differences; and

[0070] Determine a second depth estimation value of the target pixel according to the target depth difference and a second depth value of the auxiliary pixel;

[0071] The fourth depth value is equal to the second depth estimation value.

[0072] According to the second aspect of the present application, a visual system is provided, comprising:

[0073] A light emitting component emits light toward a target object;

[0074] a light receiving component, receiving reflected light after the emitted light is reflected by the target object; and

[0075] a controller comprising at least one processor and a memory in communication with the at least one processor;

[0076] Among them, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the depth detection method of the visual system described in the first aspect of the present application.

[0077] According to an embodiment of the present application, the first depth value and the second depth value are determined by the same processor.

[0078] The robot provided according to the third aspect of the present application comprises a robot body and the visual system described in the second aspect of the present application, wherein the visual system is arranged on the robot body

[0079] The computer-readable storage medium provided in accordance with the fourth aspect of the present application stores a computer program, which, when executed by a processor, implements the depth detection method of the visual system described in the first aspect of the present application.

[0080] The visual system and its depth detection method, robot and readable storage medium provided in the embodiments of the present application use the time-of-flight method to determine the first depth value of each pixel of the reflected light, use the triangulation method to determine the second depth value of all or part of the pixels of the reflected light, and determine the final depth value based on the first depth value, the second depth value and the weight influence factor. This not only gives full play to the respective advantages of the time-of-flight method and the triangulation method and reduces the impact of multipath interference, but also avoids the problem of obvious data splicing marks and inconsistent data accuracy in different ranges caused by selecting the first depth value and the second depth value based on the distance.

[0081] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Other features, purposes and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments made with reference to the following drawings. The drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0083] In the attached picture:

[0084] Figure 1 is a schematic flow chart of a depth detection method of a visual system according to one embodiment of the present application;

[0085] Figure 2 is a flow chart of another depth detection method of a visual system according to an embodiment of the present application;

[0086] Figure 3 is a flow chart of another depth detection method of a visual system according to an embodiment of the present application;

[0087] Figure 4 is a block diagram of a visual system according to an embodiment of the present application; and

[0088] Figure 5 It is a block diagram of an electronic device used to implement the depth detection method of the visual system according to the embodiment of the present application.

[0089] Reference numerals:

[0090] 110, light emitting component; 120, light receiving component; 130, controller;

[0091] 200. electronic device; 201. computing unit; 202. memory (ROM);

[0092] 203, memory (RAM); 204, bus; 205, I / O interface;

[0093] 206, input unit; 207, output unit; 208, storage unit;

[0094] 209. Communication unit. DETAILED DESCRIPTION

[0095] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0096] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0097] Taking the visual system used in obstacle avoidance robots as an example, in the related technology, the visual system usually includes a laser projector, an infrared camera and a controller. The laser projector emits laser light to the target obstacle, and the laser reflected by the target obstacle is received by the infrared camera. The controller calculates the distance, three-dimensional contour and other parameters of the obstacle based on the time interval from laser emission to laser reception or the phase difference generated by the laser's round trip. However, during the robot's obstacle avoidance process, there is a problem of multi-path reflection interference in the surrounding environment, causing the laser actually received by the infrared camera to include the reflected light of other objects, which cannot meet the requirements of high-precision avoidance, mapping and positioning.

[0098] In order to solve the above problems, the relevant technology adopts a variety of methods to reduce the impact of multipath interference. For example, although the multi-frequency fusion method can reduce the impact of multipath interference, the multi-level fusion method has too much calculation and slow response speed. For another example, although the Spot-TOF method can also reduce the impact of multipath interference, it is difficult to take into account both resolution and detection accuracy, especially the low accuracy of long-distance detection. For another example, the time-of-flight method + triangulation method first calculates the depth value according to the time-of-flight method and the triangulation method respectively, and then calculates the average of the above two depth values, and finally compares it with the preset threshold. If it is greater than the preset threshold, it is determined that the target object is in the medium and long distance range, and the final depth value adopts the depth value calculated according to the time-of-flight method; if it is not greater than the preset threshold, it is determined that the target object is in the close range, and the final depth value adopts the depth value calculated according to the triangulation method. However, since the ranging range of the triangulation method is very small, generally only a dozen centimeters, the above-mentioned time-of-flight method + triangulation method can only solve the multipath interference problem in the close range, the correction range is limited, and the versatility is poor. In addition, the error orders of magnitude between the time-of-flight method and the triangulation method differ greatly. Using the time-of-flight method for medium and long distances and the triangulation method for close distances will cause obvious traces of data splicing. In other words, the accuracy of data in different ranges is inconsistent, and slight faults will appear at the data splicing points.

[0099] Based on this, in order to solve the above problems, Figure 4 As shown, the embodiment of the present application provides a visual system, which includes a light emitting component 110 and a light receiving component 120, wherein the light emitting component 110 is used to emit light to a target object, and the light receiving component 120 is used to receive reflected light after the emitted light is reflected by the target object. Figure 1 As shown, the embodiment of the present application also provides a depth detection method of a visual system, and the depth detection method 1000 of the visual system includes the following steps:

[0100] S100, determining a first depth value of each pixel of reflected light based on a time-of-flight method;

[0101] S200, determining a second depth value of at least part of the pixels of the reflected light based on a triangulation method;

[0102] S300: Determine a final depth value of each pixel according to the first depth value, the second depth value, and a weight influence factor, where the weight influence factor is related to an optical parameter of the reflected light and / or a physical parameter of the visual system.

[0103] The implementation manner of the present application determines the first depth value of each pixel of the reflected light by using the time-of-flight method, determines the second depth value of all or part of the pixels of the reflected light by using the triangulation method, and determines the final depth value according to the first depth value, the second depth value and the weight influence factor. It not only gives full play to the respective advantages of the time-of-flight method and the triangulation method and reduces the influence of multipath interference, but also can use the ranging error to avoid the problem of obvious data splicing marks and inconsistent data accuracy in different ranges caused by selecting the first depth value and the second depth value according to the distance.

[0104] It should be noted that the weight influencing factor can be a parameter related to the optical parameters of the reflected light pixels such as the number of rows, columns or spot width, or a parameter related to the physical parameters of the visual system, or a parameter related to both the optical parameters of the reflected light and the physical parameters of the visual system.

[0105] In some embodiments, the weight influencing factor may include a ranging error determined based on physical parameters of the visual system, and the ranging error is related to the actual physical size of the pixel of the light receiving component 120 and the baseline length between the light emitting component 110 and the light receiving component 120. As an example, the ranging error can be determined based on the actual physical size of the pixel of the light receiving component 120, the baseline length, the intrinsic reference focal length of the visual system, and the angle between the emitted light emitted by the light emitting component 110 and the baseline. Of course, the ranging error can also be determined according to other methods, which is not limited in this application.

[0106] For example, the ranging error a:

[0107]

[0108] Wherein, a is the distance measurement error, f is the intrinsic focal length of the visual system, s is the actual physical size of the pixel of the light receiving component 120, and d is x is the baseline length between the light emitting component 110 and the light receiving component 120, β is the angle between the emitted light and the baseline, u i is the coordinate of the current point, that is, the position of the pixel of the reflected light at the current distance, uo The principal point coordinates are the positions of the pixels reflecting light at zero distance.

[0109] It should be noted that the time of flight, or TOF, is divided into two types: ITOF (Indirect Time of Flight) and DTOF (Direct Time of Flight). There is a phase difference between the emitted light emitted by the optical emitting component 110 and the reflected light received by the optical receiving component 120. ITOF calculates the distance, three-dimensional contour and other parameters of the target object based on the phase difference, while DTOF directly calculates the distance, three-dimensional contour and other parameters of the target object based on the time interval between emission and reception. Since the principles of ITOF and DTOF are different, the method of calculating the final depth value based on the first depth value determined by ITOF or DTOF is also different.

[0110] The following takes the weight influencing factor including the ranging error as an example to introduce the depth detection method corresponding to different types of emission light when determining the first depth value based on the ITOF method.

[0111] like Figure 2 As shown, in the case where the time-of-flight method used in step S100 is the ITOF method, step S300 may include: S310, determining a first weight corresponding to the first depth value of the target pixel and a second weight corresponding to the third depth value of the target pixel according to the ranging error, the third depth value is determined based on at least the second depth value of at least part of the pixels, and the first weight and the second weight are greater than 0 and less than 1; S320, weighted summing the first weight, the first depth value, the second weight and the third depth value to obtain a final depth value of the target pixel. The target pixel is one of the pixels of the reflected light.

[0112] In some embodiments, if the light emitting component 110 emits a single linear modulated structured light of a certain wavelength, such as 850 nm, to the target object, the depth detection method may include: determining the first depth value of each pixel of the reflected light based on the ITOF method; since the emitted light is a single linear modulated structured light, the second depth value of each pixel of the reflected light can be directly determined by the triangulation method. Taking any pixel of the reflected light as the target pixel as an example, its first depth value is d itof The second depth value is d 线结构光 , the first depth value d can be determined based on the ranging error a itof The first weight A corresponding to the third depth value and the second weight B corresponding to the third depth value are the second depth values ​​d 线结构光 ; For example, A = 1-1 / a, B = 1 / a; then the first weight A, the first depth value d itof , the second weight and the third depth value d 线结构光Perform weighted summation to obtain the final depth value d of the target pixel. For example, d = A*d itof +B*d 线结构光 =(1-1 / a)d itof +1 / a*d 线结构光 It should be noted that the final depth values ​​of other pixels of reflected light can also be determined by the above steps. In addition, those skilled in the art should understand that A=1-1 / a and B=1 / a are merely exemplary and do not constitute a limitation of the present application. Those skilled in the art can set the first weight and the second weight to other values ​​related to the ranging error according to design requirements and other factors.

[0113] In the case where the emitted light is a single linear modulated structured light, in addition to the above method to determine the final depth value of the target pixel, other methods can also be used to determine the final depth value. For example, the depth detection method may include: determining the first depth value of each pixel of the reflected light based on the ITOF method; since the emitted light is a single linear modulated structured light, the second depth value of each pixel of the reflected light can be directly determined by the triangulation method. Taking any pixel of the reflected light as the target pixel as an example, its first depth value is d itof The second depth value is d 线结构光 If the distance measurement error is less than the preset threshold, the first depth value d can be determined. itof The corresponding first weight A is 0%, and the third depth value is the second depth value d 线结构光 The corresponding second weight B is 100%, so the second depth value d 线结构光 That is, the final depth value of the target pixel; if the ranging error is not less than the preset threshold, the first depth value d can be determined. itof The corresponding first weight A is 100%, and the third depth value is the second depth value d 线结构光 The corresponding second weight B is 0%, so the first depth value d itof That is, the final depth value of the target pixel. It should be noted that the final depth values ​​of other pixels of the reflected light can also be determined by the above steps. In addition, the above preset threshold can be determined according to the error curves of the ITOF method and the DTOF method. For example, the preset threshold can be the error corresponding to the intersection of the error curve of the ITOF method and the error curve of the DTOF method.

[0114] For another example, the depth detection method may include: determining a first depth value of each pixel of the reflected light based on the ITOF method; and directly determining a second depth value of each pixel of the reflected light based on the triangulation method. Taking any pixel of the reflected light as the target pixel as an example, its first depth value is d itof The second depth value is d 线结构光 If the ranging error is less than the first preset threshold, the first depth value d can be determined.itof The corresponding first weight A is 0%, and the third depth value is the second depth value d 线结构光 The corresponding second weight B is 100%, so the second depth value d 线结构光 That is, the final depth value of the target pixel; if the ranging error is greater than the second preset threshold, the first depth value d can be determined. itof The corresponding first weight A is 100%, and the third depth value is the second depth value d 线结构光 The corresponding second weight B is 0%, so the first depth value d itof That is, the final depth value of the target pixel; if the ranging error is not less than the first preset threshold and not greater than the second preset threshold, the first depth value d can be determined. itof The corresponding first weight A is 1-1 / a, and the third depth value is the second depth value d 线结构光 The corresponding second weight B is 1 / a, so the final depth value d of the target pixel is (1-1 / a)d itof +1 / a*d 线结构光 It should be noted that the final depth values ​​of other pixels of the reflected light can also be determined by the above steps, the first preset threshold and the second preset threshold can also be determined according to the error curves of the ITOF method and the DTOF method, and the first weight and the second weight can also be set to other values ​​related to the ranging error according to design requirements and other factors.

[0115] As can be seen from the above, in the case where the emitted light is a single linear modulated structured light, the final depth value of the pixel of the reflected light after the single linear modulated structured light is reflected by the target object can be determined by, but is not limited to, any one of the above three methods. In addition, it should be noted that if the light emitting component 110 emits multi-linear modulated structured light to the target object, then any one of the above three methods can also be used to determine the final depth value of the pixel of the reflected light after each linear modulated structured light in the multi-linear modulated structured light is reflected by the target object. In other words, the depth detection method corresponding to the single linear modulated structured light is also applicable to the multi-linear modulated structured light.

[0116] In some embodiments, if the light emitting component 110 emits a speckle modulated structured light of a certain wavelength, such as 850 nm, to the target object, the depth detection method may include: determining the first depth value of each pixel of the reflected light, i.e., the speckle, based on the ITOF method; since the emitted light is the speckle modulated structured light, the second depth value of each pixel of the reflected light may be determined by the triangulation method. Taking any pixel of the reflected light as the target pixel as an example, its first depth value is d itof The second depth value is d 线结构光 , the first depth value d can be determined based on the ranging error a itofThe first weight A corresponding to the third depth value and the second weight B corresponding to the third depth value are the second depth values ​​d 线结构光 ; For example, A = 1-1 / a, B = 1 / a; then the first weight A, the first depth value d itof , the second weight and the third depth value d 线结构光 Perform weighted summation to obtain the final depth value d of the target pixel. For example, d = A*d itof +B*d 线结构光 =(1-1 / a)d itof +1 / a*d 线结构光 . It should be noted that the final depth values ​​of other pixels of reflected light can also be determined by the above steps. In addition, the final depth value of the pixel of reflected light after the speckle modulated structured light is reflected by the target object, that is, the scattered speckle, can also be determined by comparing the ranging error with the threshold. In addition, the second depth value of each pixel of the reflected light, that is, the scattered speckle, can be determined by the following method: determining the parallax of the scattered speckle of the reflected light based on the principle of monocular structured light; determining the second depth value based on the parallax, the baseline length between the light emitting component 110 and the light receiving component 120, and the focal length of the light receiving component 120 based on the triangulation method. For example,

[0117]

[0118] Among them, d is the parallax of the scattered speckle, f is the intrinsic focal length of the visual system, b is the baseline length between the light emitting component 110 and the light receiving component 120, z0 is the distance between the reference baseline and the light receiving component 120, and z is the distance between the position where the emitted light irradiates the target object and the light emitting component 110.

[0119] In some embodiments, if the light emitting component 110 emits a single light band structured light of a certain wavelength, such as 850 mm, to the target object, the depth detection method may include: determining the first depth value of each pixel of the reflected light based on the ITOF method; since the emitted light is a single light band structured light, only the pixels at the center of the light band of the reflected light can directly use the triangulation method. In other words, the triangulation method can only directly determine the second depth value of the pixels at the center of the light band of the reflected light, and the second depth values ​​of the pixels in other areas of the reflected light cannot be directly determined. Therefore, the pixels of the light band of the reflected light can be divided into two parts: pixels located at the center of the light band and pixels located outside the center of the light band. For the pixels at the center of the light band of the reflected light, their first depth value and second depth value can be directly determined, so that if any pixel at the center of the light band of the reflected light is used as the target pixel, then the first depth value of the target pixel determined based on the ITOF method is d itof The second depth value determined based on triangulation is d 线结构光 Therefore, the first depth value d can be determined based on the distance measurement error a.itof The first weight A corresponding to the third depth value and the second weight B corresponding to the third depth value are the second depth values ​​d 线结构光 ; For example, A = 1-1 / a, B = 1 / a; then the first weight A, the first depth value d itof , the second weight and the third depth value d 线结构光 Perform weighted summation to obtain the final depth value d of the target pixel. For example, d = A*d itof +B*d 线结构光 =(1-1 / a)d itof +1 / a*d 线结构光 . For pixels in other areas of the light band of reflected light, the first depth value can be directly determined based on the ITOF method, while the second depth value cannot be directly determined based on the triangulation method, and thus needs to be estimated with the help of the second depth value of the pixel at the center of the light band in the same column. Furthermore, since the difference between the first depth value and the second depth value of the pixel at the center of the light band and the pixels in the same column other than the center of the light band are substantially equal, if any pixel in other areas other than the center of the light band of reflected light is used as the target pixel, the first depth difference d between the first depth value of the target pixel and the first depth value of the auxiliary pixel can be calculated first. diff The auxiliary pixel is the pixel at the center of the light band of the reflected light and is located in the same column as the target pixel; the second depth estimation value of the target pixel is determined according to the first depth difference and the second depth value of the auxiliary pixel. Since the auxiliary pixel is located at the center of the light band of the reflected light, its second depth value can be directly determined by triangulation, so the second depth estimation value d′ of the target pixel 线结构光 It can be equal to the second depth value d of the auxiliary pixel 线结构光 The difference with the first depth d iff The sum of d′ 线结构光 =d 线结构光 +d diff Next, the first depth value d of the target pixel is determined according to the ranging error a. itof The first weight A corresponding to the third depth value and the second weight B corresponding to the third depth value are the second depth estimation value d′ of the target pixel. 线结构光 ; For example, A = 1-1 / a, B = 1 / a; then the first weight A, the first depth value d itof , the second weight and the third depth value d′ 线结构光 Perform weighted summation to obtain the final depth value d of the target pixel. For example, d = A*d itof +B*d′ 线结构光 =(1-1 / a)d itof +1 / a*d′ 线结构光 .

[0120] In the case where the emitted light is a single light band structured light, in addition to the above method, other methods can be used to determine the final depth value of the target pixel. For example, for the pixel at the center of the light band of the reflected light, its first depth value and second depth value can be directly determined. Therefore, if any pixel at the center of the light band of the reflected light is used as the target pixel, then the first depth value d of the target pixel is determined based on the ITOF method. itof , based on triangulation, determine its second depth value d 线结构光 If the distance measurement error is less than the preset threshold, the first depth value d can be determined. itof The corresponding first weight A is 0%, and the third depth value is the second depth value d 线结构光 The corresponding second weight B is 100%, so the second depth value d 线结构光 That is, the final depth value of the target pixel; if the ranging error is not less than the preset threshold, the first depth value d can be determined. itof The corresponding first weight A is 100%, and the third depth value is the second depth value d 线结构光 The corresponding second weight B is 0%, so the first depth value d itof That is the final depth value of the target pixel. For pixels in other areas of the light band of reflected light, the first depth value can be directly determined based on the ITOF method, but the second depth value cannot be directly determined based on the triangulation method, so it is necessary to use the second depth value of the pixel in the center of the light band in the same column to estimate. If any pixel in other areas outside the center of the light band of reflected light is used as the target pixel, the first depth difference d between the first depth value of the target pixel and the first depth value of the auxiliary pixel can be calculated first. diff The auxiliary pixel is the pixel at the center of the light band of the reflected light and is located in the same column as the target pixel; the second depth estimation value of the target pixel is determined according to the first depth difference and the second depth value of the auxiliary pixel. Since the auxiliary pixel is located at the center of the light band of the reflected light, its second depth value can be directly determined by triangulation, so the second depth estimation value d′ of the target pixel 线结构光 It can be equal to the second depth value d of the auxiliary pixel 线结构光 The difference with the first depth d diff The sum of d′ 线结构光 =d 线结构光 +d diff If the distance measurement error is less than the preset threshold, the first depth value d of the target pixel can be determined. itof The corresponding first weight A is 0%, and the third depth value is the second depth estimation value d′ 线结构光 The corresponding second weight B is 100%, so the second depth estimation value d′ 线结构光 That is, the final depth value of the target pixel; if the ranging error is not less than the preset threshold, the first depth value d of the target pixel can be determined.itof The corresponding first weight A is 100%, and the third depth value is the second depth estimation value d′ 线结构光 The corresponding second weight B is 0%, so the first depth value d of the target pixel itof This is the final depth value of the target pixel.

[0121] For another example, the depth detection method may include: determining a first depth value of each pixel of the reflected light based on the ITOF method; and directly determining a second depth value of each pixel at the center of the light band of the reflected light based on the triangulation method. If any pixel at the center of the light band of the reflected light is taken as the target pixel, then the first depth value d of the target pixel is determined based on the ITOF method. itof , based on triangulation, determine its second depth value d 线结构光 If the distance measurement error is less than the first preset threshold, the first depth value d of the target pixel can be determined. itof The corresponding first weight A is 0%, and the third depth value is the second depth value d 线结构光 The corresponding second weight B is 100%, so the second depth value d of the target pixel 线结构光 That is, the final depth value of the target pixel; if the ranging error is greater than the second preset threshold, the first depth value d of the target pixel can be determined. itof The corresponding first weight A is 100%, and the third depth value is the second depth value d 线结构光 The corresponding second weight B is 0%, so the first depth value d of the target pixel itof That is, the final depth value of the target pixel; if the ranging error is not less than the first preset threshold and not greater than the second preset threshold, the first depth value d of the target pixel can be determined. itof The corresponding first weight A is 1-1 / a, and the third depth value is the second depth value d 线结构光 The corresponding second weight B is 1 / a, so the final depth value d of the target pixel is (1-1 / a)d itof +1 / a*d 线结构光 For pixels not in the center of the reflected light band, the first depth value can be directly determined based on the ITOF method, but the second depth value cannot be directly determined based on the triangulation method, so it is necessary to estimate it with the help of the second depth value of the pixel in the center of the light band in the same column. Similar to the above, if any pixel in the area other than the center of the reflected light band is used as the target pixel, the first depth difference d between the first depth value of the target pixel and the first depth value of the auxiliary pixel can be calculated first. diff The auxiliary pixel is a pixel at the center of the light band of the reflected light and is located in the same column as the target pixel; a second depth estimation value d′ of the target pixel is determined according to the first depth difference and the second depth value of the auxiliary pixel. 线结构光 is d 线结构光 +d diffIf the distance measurement error is less than the first preset threshold, the first depth value d of the target pixel can be determined. itof The corresponding first weight A is 0%, and the third depth value is the second depth estimation value d′ 线结构光 The corresponding second weight B is 100%, so the second depth estimation value d′ of the target pixel 线结构光 That is, the final depth value of the target pixel; if the ranging error is greater than the second preset threshold, the first depth value d of the target pixel can be determined. itof The corresponding first weight A is 100%, and the third depth value is the second depth estimation value d′ 线结构光 The corresponding second weight B is 0%, so the first depth value d of the target pixel itof That is, the final depth value of the target pixel; if the ranging error is not less than the first preset threshold and not greater than the second preset threshold, the first depth value d of the target pixel can be determined. itof The corresponding first weight A is 1-1 / a, and the third depth value is the second depth estimation value d′ 线结构光 The corresponding second weight B is 1 / a, so the final depth value d of the target pixel is (1-1 / a)d itof +1 / a*d′ 线结构光 .

[0122] As can be seen from the above, in the case where the emitted light is a single light band structured light, the final depth value of the pixel of the reflected light after the single light band structured light is reflected by the target object can be determined by, but not limited to, any one of the above three methods. In addition, it should be noted that if the light emitting component 110 emits multiple light band structured light to the target object, then any one of the above three methods can also be used to determine the final depth value of the pixel of the reflected light after each light band structured light in the multiple light band structured light is reflected by the target object. In other words, the depth detection method corresponding to the single light band structured light is also applicable to the multiple light band structured light.

[0123] As can be seen from the above, the weight influencing factor can also be a parameter related to the optical parameters of the reflected light pixel, such as the number of rows, the number of columns, or the width of the light spot. In some embodiments, when the emitted light is a single linear modulated structured light or a multi-linear modulated structured light, the weight influencing factor may include the light spot offset of the target pixel, and the light spot offset may be determined based on the optical parameters of the target pixel and the optical parameters of the preset calibration pixel, for example, the light spot offset is equal to the difference between the number of rows of the target pixel and the number of rows of the preset calibration pixel, or the light spot offset is equal to the difference between the number of columns of the target pixel and the number of columns of the preset calibration pixel. If the light spot offset is less than the preset offset, the first depth value d can be determined. itof The corresponding first weight A is 100%, and the second depth value d 线结构光 The corresponding second weight B is 0%, so the first depth value d itofIf the light spot offset is not less than the preset offset, the first depth value d can be determined. itof The corresponding first weight A is 0%, and the second depth value d 线结构光 The corresponding second weight B is 100%, so the second depth value d 线结构光 That is, the final depth value of the target pixel. Among them, the target pixel is any pixel of the reflected light, and the preset offset can be but is not limited to 0.05 pixel. Those skilled in the art should understand that the preset offset can also be set to other values ​​according to the light spot detection algorithm, and this application does not limit this. In addition, the calibration pixel is pre-calibrated, and the area illuminated by the light spot is calculated during the calibration process, and then the light plane is calibrated to obtain the zero reference point. The number of rows and columns of pixels located in the light plane is zero.

[0124] In some other embodiments, when the emitted light is a single linear modulated structured light or a multi-linear modulated structured light, the weight influencing factor may include the spot width of the target pixel. If the spot width is less than the preset width, the first depth value d may be determined. itof The corresponding first weight A is 0%, and the second depth value d 线结构光 The corresponding second weight B is 100%, so the second depth value d 线结构光 If the spot width is not less than the preset width, the first depth value d can be determined. itof The corresponding first weight A is 100%, and the second depth value d 线结构光 The corresponding second weight B is 0%, so the first depth value d itof That is, the final depth value of the target pixel. Among them, the target pixel is any pixel of the reflected light, and the preset width can be but is not limited to 6 pixels. Those skilled in the art should understand that the preset width can also be set to other values ​​according to the spot detection algorithm, and this application is not limited to this. In addition, the spot width of the target pixel can be obtained by binarizing the image of the reflected light using a preset fixed threshold or dynamic threshold. Among them, the fixed threshold can be set according to the power of the light emitting component 110, the current automatic exposure parameters, and the usage scenario requirements. The dynamic threshold can be obtained by performing data statistics such as histogram statistics or Gaussian curve fitting on the current number of rows or columns. Both the fixed threshold and the dynamic threshold are related to the hardware configuration and usage scenario of the visual system. Considering that in industrial application scenarios, the device is generally fixed exposure, a fixed threshold can be used, while in consumer scenarios, the device is generally automatically exposed, and the width of the spot will change slightly, which is more suitable for using a dynamic threshold.

[0125] The above mainly introduces the depth detection method corresponding to different types of emitted light when determining the first depth value based on the ITOF method. The following introduces the depth detection method corresponding to different types of emitted light when determining the first depth value based on the DTOF method when the weight influencing factor includes the ranging error.

[0126] In some embodiments, Figure 3 As shown, the depth detection method may include: S100, determining a first depth value of each pulse wave peak corresponding to each pixel based on a direct time-of-flight method; S200, determining a second depth value of at least part of the pixels of the reflected light based on a triangulation method; S310, screening out an alternative depth value from the first depth values ​​of each pulse wave peak corresponding to the target pixel based on a ranging error and a fourth depth value of the target pixel, wherein the fourth depth value is determined based at least on the second depth value of at least part of the pixels; S320, screening out a final depth value of the target pixel from the alternative depth values ​​based on the fourth depth value of the target pixel.

[0127] As an example, if the light emitting component 110 emits a single linear modulated structured light of a certain wavelength, such as 850 mm, to the target object, the depth detection method may include: determining the first depth value of each pulse peak corresponding to each pixel based on the DTOF method; since the emitted light is a single linear modulated structured light, the second depth value of each pixel of the reflected light can be directly determined by the triangulation method. Taking any pixel of the reflected light as the target pixel as an example, if the target pixel corresponds to three pulse peaks, the first depth values ​​of the three pulse peaks can be determined based on the DTOF method as d1, d2 and d3, respectively, and the second depth value of the target pixel can be determined based on the triangulation method as d 线结构光 Of course, the target pixel may also correspond to two or more than three pulse peaks, which is not limited in the present application. Next, it is determined whether the three first depth values ​​d1, d2 and d3 are within d′. 线结构光 Within the range of ±a, d′ 线结构光 is the fourth depth value of the target pixel, and the second depth value of the target pixel is its fourth depth value, that is, d′ 线结构光 =d 线结构光 , a is the ranging error. Assume that d1 and d3 are at d′ 线结构光 ±a, then d1 and d3 are candidate depth values. Calculate the second depth difference between each candidate depth value and the fourth depth value of the target pixel; set the candidate depth value corresponding to the second depth difference with the smallest absolute value among the multiple second depth differences as the final depth value of the target pixel. For example, if |d1-d 线结构光 |<|d2-d 线结构光 |, then the candidate depth value d1 is the final depth value of the target pixel. It should be noted that the final depth values ​​of other pixels of reflected light can also be determined through the above steps.

[0128] As can be seen from the above, in the case where the emitted light is a single linear modulated structured light, the final depth value of the pixel of the reflected light after the single linear modulated structured light is reflected by the target object can be determined by, but is not limited to, the above method. In addition, it should be noted that if the light emitting component 110 emits multi-linear modulated structured light to the target object, then the above method can also be used to determine the final depth value of the pixel of the reflected light after each linear modulated structured light in the multi-linear modulated structured light is reflected by the target object. In other words, the depth detection method corresponding to the single linear modulated structured light is also applicable to the multi-linear modulated structured light.

[0129] In some embodiments, if the light emitting component 110 emits a scattered speckle modulated structured light of a certain wavelength, such as 850 mm, to the target object, the depth detection method may include: determining the first depth value of each pulse peak corresponding to each pixel of the reflected light, i.e., the scattered speckle, based on the DTOF method; since the emitted light is scattered speckle modulated structured light, the second depth value of each pixel of the reflected light can be determined by the triangulation method. Taking any pixel of the reflected light, i.e., the target pixel, as an example, if the target pixel corresponds to three pulse peaks, the first depth values ​​of the three pulse peaks can be determined based on the DTOF method, respectively, and the second depth value of the target pixel can be determined based on the triangulation method as d 线结构光 Of course, the target pixel may also correspond to two or more than three pulse peaks, which is not limited in the present application. Next, it is determined whether the three first depth values ​​d1, d2 and d3 are within d′. 线结构光 Within the range of ±a, d′ 线结构光 is the fourth depth value of the target pixel, and the second depth value of the target pixel is its fourth depth value, that is, d′ 线结构光 =d 线结构光 , a is the ranging error. Assume that d1 and d3 are at d′ 线结构光 ±a, then d1 and d3 are candidate depth values. Calculate the second depth value d of each candidate depth value and the target pixel. 线结构光 The candidate depth value corresponding to the second depth difference with the smallest absolute value among the multiple second depth differences is set as the final depth value of the target pixel. 线结构光 |<|d2-d 线结构光|, then the alternative depth value d1 is the final depth value of the target pixel. It should be noted that the final depth values ​​of other pixels of the reflected light can also be determined by the above steps. In addition, the second depth value of each pixel of the reflected light, i.e., the scattered speckle, can be determined by the following method: determining the parallax of the scattered speckle of the reflected light based on the principle of monocular structured light; determining the second depth value based on the parallax, the baseline length between the light emitting component 110 and the light receiving component 120, and the focal length of the light receiving component 120 based on the triangulation method. For example,

[0130]

[0131] Among them, d is the parallax of the scattered speckle, f is the intrinsic focal length of the visual system, b is the baseline length between the light emitting component 110 and the light receiving component 120, z0 is the distance between the reference baseline and the light receiving component 120, and z is the distance between the position where the emitted light irradiates the target object and the light emitting component 110.

[0132] In some embodiments, if the light emitting component 110 emits a single light band structured light of a certain wavelength, such as 850 mm, to the target object, the depth detection method may include: determining the first depth value of each pulse peak corresponding to each pixel based on the DTOF method; since the emitted light is a single light band structured light, only the pixels at the center of the light band of the reflected light can directly use the triangulation method. In other words, the triangulation method can only directly determine the second depth value of the pixels at the center of the light band of the reflected light, and the second depth value of the pixels in other areas of the reflected light cannot be directly determined. Therefore, the pixels of the light band of the reflected light can be divided into two parts: pixels located at the center of the light band and pixels located outside the center of the light band. If any pixel at the center of the light band of the reflected light is taken as the target pixel, assuming that the target pixel corresponds to three pulse peaks, then based on the DTOF method, the first depth values ​​of the three pulse peaks can be determined to be d1, d2 and d3 respectively, and based on the triangulation method, the second depth value of the target pixel can be determined to be d 线结构光 Of course, the target pixel may also correspond to two or more than three pulse peaks, which is not limited in the present application. Next, it is determined whether the three first depth values ​​d1, d2 and d3 are within d′. 线结构光 Within the range of ±a, d′ 线结构光 is the fourth depth value of the target pixel, and the second depth value of the target pixel is its fourth depth value, that is, d′ 线结构光 =d 线结构光 , a is the ranging error. Assume that d1 and d3 are at d′ 线结构光 ±a, then d1 and d3 are candidate depth values. Calculate the second depth value d of each candidate depth value and the target pixel. 线结构光The candidate depth value corresponding to the second depth difference with the smallest absolute value among the multiple second depth differences is set as the final depth value of the target pixel. 线结构光 |<|d2-d 线结构光 |, then the alternative depth value d1 is the final depth value of the target pixel. For pixels in other areas of the light band of reflected light, the first depth value of each pulse wave peak corresponding to each pixel can be determined based on the DTOF method, while the second depth value corresponding to each pixel cannot be directly determined based on the triangulation method. Therefore, it is necessary to use the first depth value of each pulse wave peak of the pixel and the second depth value and first depth value of the pixel in the center of the light band in the same column to estimate the second depth value of the pixel not in the center of the light band. Since the difference between the first depth value and the second depth value of the pixel at the center of the light band and the pixel not at the center of the light band in the same column are basically equal, if any pixel in other areas outside the center of the light band of reflected light is used as the target pixel, the third depth difference between the first depth value of each pulse wave peak of the target pixel and the corresponding first depth value of the auxiliary pixel can be calculated first, and the auxiliary pixel is the pixel at the center of the light band of reflected light and is located in the same column as the target pixel; for example, the target pixel corresponds to three pulse wave peaks, then based on the DTOF method, it can be determined that the first depth values ​​of these three pulse peaks are d1, d2 and d3 respectively, and the third depth difference between the first depth value d1 and the first depth value corresponding to the auxiliary pixel in the same column is d1 diff The third depth difference between the second depth value d2 and the first depth value corresponding to the auxiliary pixel in the same column is d2 diff The third depth difference between the third depth value d3 and the first depth value corresponding to the auxiliary pixel in the same column is d3 diff The target depth difference is determined according to the plurality of third depth differences, for example, the target depth difference d diff Equal to d1 diff , d2 diff and d3 diff The weighted average or arithmetic mean of the target depth difference d diff and the second depth value d of the auxiliary pixel 线结构光 Determine the second depth estimation value d′ of the target pixel 线结构光 Since the auxiliary pixel is located at the center of the light band of the reflected light, its second depth value can be directly determined by triangulation, so the second depth estimation value d′ of the target pixel is 线结构光 It can be equal to the second depth value d of the auxiliary pixel 线结构光 The difference with the third depth d diff The sum of d′ 线结构光 =d 线结构光 +d diff The second depth estimation value of the target pixel is its fourth depth value, that is, the fourth depth value of the target pixel is d′线结构光 Next, determine whether the three first depth values ​​d1, d2 and d3 are within d′. 线结构光 ±a, a is the distance measurement error. Assume that d1 and d3 are within d′ 线结构光 ±a, then d1 and d3 are candidate depth values. Calculate the fourth depth value of each candidate depth value and the target pixel, that is, the second depth estimation value d′ 线结构光 The candidate depth value corresponding to the second depth difference with the smallest absolute value among the multiple second depth differences is set as the final depth value of the target pixel. 线结构光 |<|d2-d′ 线结构光 |, then the alternative depth value d1 is the final depth value of the target pixel.

[0133] As can be seen from the above, in the case where the emitted light is a single light band structured light, the final depth value of the pixel of the reflected light after the single light band structured light is reflected by the target object can be determined by, but is not limited to, the above method. In addition, it should be noted that if the light emitting component 110 emits multiple light band structured lights to the target object, then the above method can also be used to determine the final depth value of the pixel of the reflected light after each light band structured light in the multiple light band structured light is reflected by the target object. In other words, the depth detection method corresponding to the single light band structured light is applicable to the multiple light band structured light.

[0134] In addition, if Figure 4 As shown, an embodiment of the present application also provides a visual system, which includes a light emitting component 110, a light receiving component 120 and a controller 130. Among them, the light emitting component 110 emits a transmitted light to the target object, and the light receiving component 120 receives the reflected light after the transmitted light is reflected by the target object. The controller 130 includes at least one processor and a memory connected to the at least one processor in communication. The memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor so that at least one processor can execute the depth detection method of the above-mentioned visual system. In the embodiment of the present application, the first depth value and the second depth value can be determined by the same processor. In other words, steps S100 and S200 above can be executed by the same processor.

[0135] In some embodiments, the visual system may be, but is not limited to, a depth camera. The light emitting component 110 includes a laser emitter, which may be, but is not limited to, an edge emitting laser, i.e., an EEL, a vertical cavity surface emitting laser (Vertical Cavity Surface Emitting Laser, referred to as VCSEL), a fiber laser, or a semiconductor-pumped solid-state laser. The light receiving component 120 includes a visible light camera or a non-visible light camera, for example, the light receiving component 120 may include an infrared camera.

[0136] Furthermore, an embodiment of the present application also provides a robot, which includes a robot body and the above-mentioned visual system, and the visual system is arranged on the robot body.

[0137] An embodiment of the present application also provides a computer-readable storage medium storing a computer program, which implements the depth detection method of the above-mentioned visual system when the computer program is executed by a processor.

[0138] Figure 5 A schematic block diagram of an example controller, i.e., an electronic device 200, which can be used to implement an embodiment of the present disclosure is shown. An electronic device is intended to represent various forms of digital computers. An electronic device can also represent various forms of mobile devices that can run computing programs. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0139] like Figure 5 As shown, the electronic device 200 includes a computing unit 201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 202 or a computer program loaded from a storage unit 208 into a random access memory (RAM) 203. In the RAM 203, various programs and data required for the operation of the electronic device 200 can also be stored. The computing unit 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0140] A number of components in the electronic device 200 are connected to the I / O interface 205, including: an input unit 206, such as a touch screen, etc.; an output unit 207, such as various types of displays, speakers, etc.; a storage unit 208, such as a disk, etc.; and a communication unit 209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 209 allows the electronic device 200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0141] The computing unit 201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 201 performs the various methods and processes described above, such as the depth detection method of the visual system. For example, in some embodiments, the depth detection method of the visual system may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 208. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 200 via the ROM 202 and / or the communication unit 209. When the computer program is loaded into the RAM 203 and executed by the computing unit 201, one or more steps of the depth detection method of the visual system described above may be performed. Alternatively, in other embodiments, the computing unit 201 may be configured to perform the depth detection method of the visual system in any other appropriate manner (e.g., by means of firmware).

[0142] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow diagram and / or block diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0144] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0146] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A depth detection method for a visual system, characterized in that: The visual system comprises a light emitting component and a light receiving component, wherein the light emitting component is used to emit light to a target object, and the light receiving component is used to receive reflected light after the emitted light is reflected by the target object; Wherein, the depth detection method comprises: Determine a first depth value of each pixel of the reflected light based on a time-of-flight method; Determining a second depth value of at least some pixels of the reflected light based on triangulation; and A final depth value of each pixel is determined according to the first depth value, the second depth value and a weight influence factor, where the weight influence factor is related to an optical parameter of the reflected light and / or a physical parameter of the visual system.

2. The depth detection method of a visual system according to claim 1, wherein: The weight influencing factor includes a ranging error determined based on a physical parameter of the visual system, the time-of-flight method is an indirect time-of-flight method, and the pixel of the reflected light includes a target pixel; Wherein, determining the final depth value of each pixel according to the first depth value, the second depth value and the weight influence factor includes: determining, according to the ranging error, a first weight corresponding to the first depth value of the target pixel and a second weight corresponding to a third depth value of the target pixel, wherein the third depth value is determined based at least on the second depth value of the at least part of the pixels, and the first weight and the second weight are greater than 0 and less than 1; and A weighted sum is performed on the first weight, the first depth value, the second weight, and the third depth value to obtain the final depth value of the target pixel.

3. The depth detection method of a visual system according to claim 2, wherein: Determining, according to the ranging error, a first weight corresponding to the first depth value of the target pixel and a second weight corresponding to the third depth value of the target pixel comprises: In response to the ranging error being not less than a first preset threshold and not greater than a second preset threshold, determining the first weight and the second weight according to the ranging error; Wherein, determining a final depth value of each pixel according to the first depth value, the second depth value and the ranging error further comprises: In response to the ranging error being less than the first preset threshold, setting the second depth value of the target pixel as the final depth value of the target pixel; and In response to the ranging error being greater than the second preset threshold, the first depth value of the target pixel is set as the final depth value of the target pixel.

4. The depth detection method of a visual system according to claim 2, wherein: Performing a weighted summation on the first weight, the first depth value, the second weight, and the third depth value includes: A = 1-1 / a; B = 1 / a; d=A*d itof +B*d 线结构光 ; Wherein, d is the final depth value of the target pixel, A is the first weight, B is the second weight, a is the ranging error, and d itof is the first depth value of the target pixel, d 线结构光 is the third depth value of the target pixel.

5. The depth detection method of a visual system according to claim 1, wherein: The weight influencing factor includes a ranging error determined based on a physical parameter of the visual system, the time-of-flight method is an indirect time-of-flight method, and the pixel of the reflected light includes a target pixel; Wherein, determining the final depth value of each pixel according to the first depth value, the second depth value and the weight influence factor includes: In response to the ranging error being less than a preset threshold, setting a third depth value of the target pixel as the final depth value, the third depth value being determined based at least on the second depth value of the at least part of the pixels; and In response to the ranging error being not less than the preset threshold, setting the first depth value of the target pixel as the final depth value.

6. The depth detection method of a visual system according to any one of claims 2 to 5, wherein: The emitted light is single-linear modulated structured light, multi-linear modulated structured light or scattered-speckle modulated structured light, the target pixel is any pixel of the reflected light, and the third depth value is equal to the second depth value of the target pixel.

7. The depth detection method of a visual system according to claim 6, wherein: The emitted light is the scattered speckle modulated structured light; Wherein, determining the second depth value of at least part of the pixels of the reflected light based on the triangulation method comprises: Determine the parallax of the scattered spots of the reflected light based on the monocular structured light principle; and The second depth value is determined based on triangulation according to the parallax, a baseline length between the light emitting component and the light receiving component, and a focal length of the light receiving component.

8. The depth detection method of a visual system according to any one of claims 2 to 5, wherein: The emitted light is single-light-band structured light or multi-light-band structured light, the target pixel is a pixel at a center of a light band of the reflected light, and the third depth value is equal to the second depth value.

9. The depth detection method of a visual system according to any one of claims 2 to 5, wherein: The emitted light is single-light-band structured light or multi-light-band structured light, the target pixel is a pixel at a non-light-band center of the reflected light, and the at least part of the pixels are pixels at a light-band center of the reflected light; Wherein, the depth detection method of the visual system further includes: calculating a first depth difference between the first depth value of the target pixel and the first depth value of an auxiliary pixel, the auxiliary pixel being a pixel in the at least some pixels that is located in the same column as the target pixel; and Determine a second depth estimation value of the target pixel according to the first depth difference and a second depth value of the auxiliary pixel; The third depth value is equal to the second depth estimation value.

10. The depth detection method of a visual system according to any one of claims 2 to 5, wherein: The physical parameters of the vision system include the actual physical size of the picture element of the light receiving component and the baseline length between the light emitting component and the light receiving component.

Images