Method and electronic device for acquiring depth information

Abstract

[0006] An embodiment of the present application provides a method and an electronic device for acquiring depth information. The method is applicable to the field of image processing technology and can help improve the accuracy of depth information acquisition by electronic devices. A specific solution includes an electronic device including a first camera, a second camera, and an infrared projector. The electronic device receives a first command used to trigger the electronic device to acquire depth information of a target object, transmits infrared light having a light spot by using the infrared projector in response to the first command, collects first image information of the target object by using the first camera, and collects second image information of the target object by using the second camera, the first image information and the second image information including features of the target object and texture features formed when the infrared light having the light spot is irradiated onto the target object, and calculates the depth information of the target object based on the first image information, the second image information, the first length, the lens focal length of the first camera, and the lens focal length of the second camera.

Description

[Technical field]

[0001] TECHNICAL FIELD Embodiments of the present application relate to the field of image processing technology, and in particular to a method and electronic device for obtaining depth information. [Background technology]

[0002] With the development of electronic technology, the functions of electronic devices are becoming more and more diverse. For example, electronic devices may include functions such as face recognition and augmented reality (AR). Taking face recognition as an example, when performing face recognition, the electronic device can not only collect a two-dimensional image of a target object (e.g., a human face), but also obtain depth information of the target object.

[0003] Specifically, the electronic device can use a binocular camera (i.e., two cameras, e.g., camera 1 and camera 2) to separately collect image information of the target object, then identify the same feature in the image information collected by camera 1 and the image information collected by camera 2, calculate the disparity between camera 1 and camera 2 for the same feature, and then calculate the depth of the point of the feature (i.e., the perpendicular distance from the point to the line connecting the two cameras) based on the disparity and the hardware parameters of camera 1 and camera 2. The depths of multiple points of the target object can constitute the depth information of the target object. The depth information of the target object can represent three-dimensional features of the target object.

[0004] It can be understood that the greater the amount of features in the image information collected by camera 1 and camera 2, the greater the amount of the same features identified by electronic device 100 in the image information collected by camera 1 and the image information collected by camera 2. The greater the amount of the same features identified by the electronic device, the more points the electronic device can calculate the depth of, and the more accurate the depth information of the target object.

[0005] The amount of features included in the image information collected by the camera depends on at least the following two parameters: (1) Whether the target object has obvious features. For example, if the target object has clear edges and corners (i.e., the target object has relatively obvious features), the image information of the target object collected by the camera can contain a relatively large amount of features, but if the target object is relatively smooth, i.e., the target object (e.g., a white wall) does not have obvious features, the image information of the target object collected by the camera contains only a relatively small amount of features. (2) Light intensity. For example, if the light is relatively strong, the image information collected by the camera contains a relatively large amount of features, but if the light is relatively poor, the image information collected by the camera contains only a relatively small amount of features.

[0006] In summary, the more features in the image information collected by the binocular camera, the more accurate the depth information of the target object calculated by the electronic device. However, the amount of features in the image information collected by the binocular camera depends on multiple factors. In other words, the accuracy of the depth information is affected by multiple factors. Therefore, it is very difficult for the electronic device to obtain relatively accurate depth information. Summary of the Invention [Means for solving the problem]

[0007] SUMMARY OF THE PRESENT APPLICATION The embodiments of the present application provide a method and an electronic device for acquiring depth information to improve the accuracy of the depth information of a target object acquired by the electronic device.

[0008] According to a first aspect, an embodiment of the present application provides a method for acquiring depth information. The method may be applied to an electronic device. The electronic device includes an infrared projector, a first camera, and a second camera. A distance between the first camera and the second camera is a first length. The method includes: the electronic device receives a first command used to trigger the electronic device to obtain depth information of a target object; emitting infrared light having a light point by using the infrared projector in response to the first command; collecting first image information of the target object by using the first camera; collecting second image information of the target object by using the second camera; and calculating the depth information of the target object based on the first image information, the second image information, the first length, the lens focal length of the first camera, and the lens focal length of the second camera.

[0009] The greater the amount of features and the more obvious the features are in the image information collected by the first camera and the image information collected by the second camera, the greater the amount of the same features identified by the electronic device in the first image information and the second image information. The greater the amount of the same features identified by the electronic device, the greater the amount of points at which the features are located. Since the depth information of the target object includes the depths of multiple points of the target object, the greater the amount of points for which the depth is calculated by the electronic device, the greater the accuracy of the depth information of the target object.

[0010] In this embodiment, the first image information and the second image information include the features of the target object and the texture features formed when the infrared light having the light points is irradiated onto the target object. In other words, the first image information and the second image information include not only the features of the target object, but also the texture features formed when the infrared light having the light points is irradiated onto the target object.

[0011] After the amount of features in the images of the target object collected by the first camera and the second camera is increased, the electronic device can more accurately identify the same features in the image information collected by the first camera and the image information collected by the second camera, and further determine the disparity between the first camera and the second camera for the same features, and calculate the depth of each point to obtain the depth information of the target object, thereby improving the accuracy of the depth information of the target object calculated by the electronic device.

[0012] With respect to the first embodiment, in one possible design approach, the lens focal length of the first camera is the same as the lens focal length of the second camera.

[0013] In relation to the first embodiment, in another possible design manner, a method for calculating depth information of a target object by an electronic device based on first image information, second image information, a first length, a lens focal length of a first camera, and a lens focal length of a second camera may include: the electronic device calculates a disparity between the first camera and the second camera for a plurality of first features in the first image information and the second image information, where the first features are identical features in the first image information and the second image information. For each of the first features, the electronic device calculates a depth Z of a point where the first feature is located based on the disparity between the first camera and the second camera for the first feature, the first length, and the lens focal length by using the following formula (2), thereby obtaining depth information of the target object.

[0014]

number

[0015] where f is the lens focal length, d is the disparity between the first and second cameras for the first feature, and T is the first length.

[0016] It can be understood that the distance between the features of the target object (e.g., the tip of a person's nose and eyes) and the camera can vary. The distance between each feature of the target object and the camera is called the depth of that feature (or the point where the feature is located). The depth of the points of the target object constitutes the depth information of the target object. The depth information of the target object can represent three-dimensional features of the target object.

[0017] In relation to the first embodiment, in another possible design manner, the multiple first features are part of the same feature in the first image information and the second image information. The electronic device 100 can select the multiple first features from the first image information based on a preset feature frequency, then search the second image information for features identical to the multiple first features, and finally calculate the depth of each first feature to obtain depth information of the target object. Alternatively, the electronic device can select some first features from the first image information randomly or at a certain interval.

[0018] picture The feature frequency reflected in the image is a function of the distance between two adjacent first features selected by the electronic device (called the feature distance). Based on The method of selecting a plurality of first features from the first image information based on a preset feature frequency by the electronic device may include the electronic device selecting one first feature from all features in the first image information located at an interval of one feature distance. In other words, the electronic device does not need to calculate the depth of a point where each of the same features in the first image information and the second image information is located, but only needs to select one feature located at an interval of one feature distance and calculate the depth of a point where the selected feature is located.

[0019] In relation to the first embodiment, in another possible design manner, the infrared light having a light point includes a plurality of light points, and the plurality of light points includes a plurality of speckle grating groups, where a speckle grating group includes one or more speckle gratings, and each speckle grating includes a plurality of speckles.

[0020] In relation to the first embodiment, in another possible design manner, at least two of the multiple speckle grating groups are different. Since at least two of the multiple speckle grating groups are different, the repetition of the speckle grating groups in the multiple light points can be reduced, and thus the electronic device can help identify different features.

[0021] In relation to the first embodiment, in another possible design manner, the first speckle grating group is any one of a plurality of speckle grating groups, and at least two of the plurality of speckle gratings included in the first speckle grating group are different. Since at least two of the plurality of speckle gratings are different, the repetition of the speckle gratings in the plurality of light points can be reduced, and thus the electronic device can help identify different features.

[0022] In relation to the first embodiment, in another possible design manner, the first speckle grating is any one of a plurality of speckle gratings. Each of the plurality of speckles in the first speckle grating has the same shape. When all of the plurality of speckles have the same shape, the electronic device can distinguish different speckles based on the position of the speckle in the speckle grating.

[0023] At least two of the speckles in the first speckle grating have different shapes. Because at least two of the speckles have different shapes, repetition of the speckles in the speckle grating can be reduced, thus aiding in identifying different features by the electronic device.

[0024] In relation to the first aspect, in another possible design manner, the electronic device selects a plurality of first features from the first image information and the second image information based on a preset feature frequency, the feature frequency being equal to or greater than a repetition frequency of the speckle grating in the plurality of light spots. RepeatThe repetition frequency is the frequency of the same speckle grating appearing in a predefined area. number It is represented by:

[0025] The feature distance used by the electronic device to select the first feature may be equal to or less than the repetition interval of the speckle grating in the multiple light points. In other words, the feature frequency is equal to or greater than the repetition frequency of the speckle grating in the multiple light points. In this way, it can be guaranteed as much as possible that two adjacent first features selected by the electronic device from the first image information correspond to speckles of different speckle gratings, so that the electronic device can distinguish between the two adjacent first features. This can reduce the possibility of feature matching errors and improve the accuracy of the depth information calculated by the electronic device.

[0026] In another possible design approach related to the first embodiment, both sides of each lens of the first camera and the second camera each include an anti-reflection coating, and each optical filter of the first camera and the second camera each include a cut-off coating. The anti-reflection coating is used to increase the transmittance of infrared light. The cut-off coating is used to remove light other than infrared light and visible light and increase the transmittance of infrared light.

[0027] A typical red green blue (RGB) camera can only sense visible light and cannot sense infrared light. In order to reduce the hardware cost of electronic devices, both sides of each lens of the RGB camera are each provided with an anti-reflection coating, and (one or both sides of) the optical filter of the RGB camera are each provided with a cut-off coating to obtain a dual-path camera (i.e., a first camera and a second camera) in which both sides of each lens each include an anti-reflection coating and the optical filter includes a cut-off coating.

[0028] In another possible design manner related to the first embodiment, the infrared light is infrared light between 890 nanometers (nm) and 990 nm. For example, the infrared light may be, in particular, infrared light at 940 nm. Correspondingly, the anti-reflection coating may be an anti-reflection coating for infrared light between 890 nm and 990 nm, for example, an anti-reflection coating for infrared light at 940 nm.

[0029] In relation to the first embodiment, in another possible design manner, the first length is in the range of 20 millimeters to 30 millimeters.

[0030] In relation to the first aspect, in another possible design manner, the electronic device further includes a third camera, the third camera being an RGB camera. The third camera is configured to collect image information under visible light. The image information collected by the third camera is used for display on a display screen of the electronic device.

[0031] According to a second aspect, an embodiment of the present application provides an electronic device. The electronic device includes one or more processors, a memory, an infrared projector, a first camera, and a second camera. A distance between the first camera and the second camera is a first length. The memory, the infrared projector, the first camera, and the second camera are coupled to the processor. The memory is configured to store information. The processor is configured to receive a first instruction used to trigger the electronic device to obtain depth information of a target object. In response to the first instruction, the processor is further configured to emit infrared light having a light point by using the infrared projector, collect first image information of the target object by using the first camera, and collect second image information of the target object by using the second camera, and the first image information and the second image information include features of the target object and texture features formed when the infrared light having a light point is irradiated onto the target object. The processor is further configured to calculate depth information of the target object based on the first image information collected by the first camera, the second image information collected by the second camera, the first length, the lens focal length of the first camera, and the lens focal length of the second camera.

[0032] With respect to the second embodiment, in one possible design approach, the lens focal length of the first camera is the same as the lens focal length of the second camera.

[0033] In relation to the second aspect, in another possible design manner, the processor being configured to calculate depth information of the target object based on the first image information, the second image information, the first length, the lens focal length of the first camera, and the lens focal length of the second camera includes the processor being configured to calculate a disparity between the first camera and the second camera for a plurality of first features in the first image information and the second image information, where the first features are identical features in the first image information and the second image information, and the processor being configured to calculate, for each of the first features, a depth Z of a point where the first feature is located based on the disparity between the first camera and the second camera for the first feature, the first length, and the lens focal length by using the above equation (2) to obtain depth information of the target object, where f is the lens focal length, d is the disparity between the first camera and the second camera for the first feature, and T is the first length.

[0034] In relation to the second aspect, in one possible design manner, the processor is further configured to select a plurality of first features from the first image information and the second image information prior to calculating the disparity between the first camera and the second camera for a plurality of first features in the first image information and the second image information, the plurality of first features being part of an identical feature in the first image information and the second image information.

[0035] In relation to the second embodiment, in one possible design manner, the infrared light having a light point includes a plurality of light points, and the plurality of light points includes a plurality of speckle grating groups, where a speckle grating group includes one or more speckle gratings, and each speckle grating includes a plurality of speckles.

[0036] It should be noted that in the second embodiment and the possible design manner of the second embodiment, the speckle grating group, the speckle grating, the multiple speckles in the speckle grating, and the wavelength of the infrared light emitted by the infrared projector can refer to the relevant description in the possible design manner of the first embodiment, and the details will not be described again here.

[0037] In relation to the second aspect, in one possible design manner, the processor is configured to select a plurality of first features from the first image information and the second image information based on a predetermined feature frequency, the feature frequency being equal to or greater than a repetition frequency of the speckle grating at the plurality of light points. Repeat The repetition frequency is the frequency of the same speckle grating appearing in a predefined area. number It is represented by:

[0038] In relation to the second embodiment, in one possible design approach, both sides of each lens of the first camera and the second camera each include an anti-reflection coating, and each of the optical filters of the first camera and the second camera each include a cut-off coating. The anti-reflection coating is used to increase the transmittance of infrared light. The cut-off coating is used to remove light other than infrared light and visible light and to increase the transmittance of infrared light.

[0039] In relation to the second aspect, in one possible design manner, the electronic device further includes a third camera and a display screen. The third camera is an RGB camera. The third camera is configured to collect image information under visible light. The image information collected by the third camera is used for display on the display screen.

[0040] According to a third aspect, an embodiment of the present application provides a dual path camera (i.e., a first camera or a second camera). The dual path camera is configured to receive visible light and infrared light. Each of the two sides of each lens of the dual path camera includes an anti-reflective coating, and an optical filter of the dual path camera includes a cut-off coating. The anti-reflective coating is used to increase the transmittance of infrared light. The cut-off coating is used to remove light other than infrared light and visible light and increase the transmittance of infrared light.

[0041] In relation to the third aspect, in one possible design approach, the dual path camera includes an RGB camera, an anti-reflective coating applied to each of both sides of each lens of the RGB camera, and a cut-off coating applied to (each of one or both sides of) an optical filter of the RGB camera.

[0042] According to a fourth aspect, an embodiment of the present application provides a computer storage medium comprising computer instructions which, when operating on an electronic device, enable the electronic device to execute a method for obtaining depth information in a manner of the first aspect and possible designs of the first aspect.

[0043] According to a fifth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer, enables the computer to execute a method for obtaining depth information in a first aspect and a possible design manner of the first aspect.

[0044] It can be understood that the electronic device in the second aspect and the possible design method of the second aspect, the computer storage medium in the third aspect, and the computer program product in the fourth aspect are all configured to execute the corresponding methods shown in the above description. Therefore, for the beneficial effects that can be achieved by the electronic device, the computer storage medium, and the computer program product, please refer to the beneficial effects of the corresponding methods presented in the above description. Details will not be described again here. [Brief description of the drawings]

[0045] [Figure 1A] 1 is a schematic diagram of a portion of an electronic device according to an embodiment of the present application. [Figure 1B] FIG. 1 is a schematic diagram of the calculation principle of depth information according to an embodiment of the present application; [Figure 1C] 1 is a schematic diagram of an example of an identical feature in two image information collected by a binocular camera according to an embodiment of the present application. [Figure 1D] FIG. 2 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application. [Diagram 2] 1 is a flowchart 1 of a method for obtaining depth information according to an embodiment of the present application. [Figure 3(a)] 1 is a schematic diagram of an example of a graphical user interface according to an embodiment of the present application. [Figure 3(b)] 1 is a schematic diagram of an example of a graphical user interface according to an embodiment of the present application. [Figure 4] FIG. 2 is a schematic diagram of an example of a speckle grating according to an embodiment of the present application. [Diagram 5] FIG. 1 is a schematic diagram of an example of a speckle grating group according to an embodiment of the present application. [Figure 6A] FIG. 2 is a schematic diagram of an example of a speckle grating group according to an embodiment of the present application. [Figure 6B] FIG. 3 is a schematic diagram of an example of a speckle grating group according to an embodiment of the present application. [Figure 6C] FIG. 4 is a schematic diagram of an example of a speckle grating group according to an embodiment of the present application. [Figure 7A] FIG. 5 is a schematic diagram of an example of a speckle grating group according to an embodiment of the present application. [Figure 7B] 6 is a schematic diagram of an example of a speckle grating group according to an embodiment of the present application. [Figure 7C] 7 is a schematic diagram of an example of a speckle grating group according to an embodiment of the present application. [Figure 8] FIG. 1 is a schematic structural diagram of a camera module according to an embodiment of the present application; [Figure 9] 2 is a flowchart 2 of a method for obtaining depth information according to an embodiment of the present application. [Figure 10A] FIG. 2 is a schematic diagram of the calculation principle of depth information according to an embodiment of the present application; [Figure 10B] FIG. 2 is a schematic diagram of a parallax calculation principle according to an embodiment of the present application; [Figure 10C] FIG. 2 is a schematic diagram of the relationship between parallax and depth according to an embodiment of the present application; [Figure 11] 1 is a schematic diagram of an example of an infrared light having an object of interest and a light point according to an embodiment of the present application. [Figure 12] FIG. 2 is a schematic diagram of an example of a characteristic frequency according to an embodiment of the present application. [Figure 13] FIG. 2 is a schematic diagram of an example of depth information of a human face according to an embodiment of the present application; [Figure 14] FIG. 2 is a schematic diagram of an example of a graphical user interface according to an embodiment of the present application. [Figure 15] 1 is a schematic diagram of a structural composition of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0046] An embodiment of the present application provides a method for acquiring depth information. The method can be applied to a process in which an electronic device acquires image information of a target object. The image information can include a two-dimensional image of the target object and depth information of the target object.

[0047] Generally, a target object (e.g., a human face) is a three-dimensional object. When a camera of an electronic device captures a target object, the distance between the target object's features (e.g., a person's nose and eyes) and the camera may vary. The distance between each target object's feature and the camera is called the depth of that feature (or the point where that feature is located). The depth of the target object's point constitutes the target object's depth information. The target object's depth information can represent the target object's three-dimensional features.

[0048] In the case of a binocular camera (i.e., two cameras, e.g., a first camera and a second camera), the distance between each feature of the target object and the camera (i.e., the depth of the point) may be the perpendicular distance from the point where each feature of the target object is located to a line connecting the two cameras. For example, as shown in FIG. 1B, P is a feature of the target object, and the depth of feature P is the distance from P to O L O R It is assumed that the vertical distance Z to the L is the position of the first camera, and O R is the position of the second camera.

[0049] The electronic device provided in the embodiment of the present application can include an infrared projector and two dual path cameras (for example, a first camera and a second camera). The distance between the two dual path cameras is a first length. In general, the distance between the centers of the two cameras is called the distance between the two cameras.

[0050] 1A is a schematic diagram of a portion of an electronic device 100 according to an embodiment of the present application. As shown in FIG 1A, the electronic device 100 may include an infrared projector 101, a first camera 102, and a second camera 103. The distance between the first camera 102 and the second camera 103 is a first length T.

[0051] The infrared projector 101 is configured to emit infrared light having a light point. The dual-path camera means that the camera can not only receive visible light, but also infrared light. For example, the dual-path camera can receive visible light and infrared light of 940 nm. 940 nm is the wavelength of infrared light. Of course, the two dual-path cameras mentioned above may be replaced by two all-pass cameras. The all-pass camera means that the camera can receive multiple types of light, including visible light, infrared light, and light of other wavelengths. In contrast, a normal RGB camera can receive visible light but cannot receive infrared light.

[0052] In this embodiment of the present application, the first camera 102 may be a left dual path camera and the second camera 103 may be a right dual path camera, but the first camera 102 may be a right dual path camera and the second camera 103 may be a left dual path camera. In Fig. 1A, the structure of a part of the electronic device 100 is described using an example in which the first camera 102 is a left dual path camera and the second camera 103 is a right dual path camera.

[0053] It should be noted that in this embodiment of the present application, the first camera 102 and the second camera 103 can receive the same type of infrared light. The same type of infrared light is infrared light of the same wavelength. In other words, the first camera 102 and the second camera 103 have the same ability to receive infrared light. For example, the first camera 102 and the second camera 103 can receive infrared light of 940 nm. In addition, in this embodiment of the present application, the infrared light emitted by the infrared projector 101 and the infrared light that can be received by the first camera 102 and the second camera 103 are also the same type of infrared light.

[0054] To obtain depth information of the target object, the electronic device 100 can refer to the hardware parameters of the binocular cameras and calculate the depth of each feature of the target object based on the disparity of the binocular cameras for the same feature by using the principle of triangulation.

[0055] In this embodiment of the present application, a method for calculating depth information by the electronic device 100 based on disparity is explained by means of an example.

[0056] As shown in Figure 1B, L is the position of the first camera 102, and O R is the position of the second camera 103, and O L and O R The distance between is the first length T, i.e., O L O R = T. The lens focal length of the first camera 102 and the lens focal length of the second camera 103 are both f.

[0057] Feature P is a feature of the target object. The perpendicular distance from the point where feature P is located to the line connecting the first camera 102 and the second camera 103 is Z. That is, the depth information of P is Z. The first camera 102 collects an image 1 of the target object, and feature P is located at a point P in image 1. L The second camera 103 collects an image 2 of the object of interest, and the feature P is located at a point PR Point P in image 1 is located at L and the feature corresponding to point P in image 2 R The features corresponding to both are feature P of the target object.

[0058] As shown in FIG. 1B, A L C L =A R C R = x, and A L B L =B L C L =A R B R =B R C R =x / 2. Feature P L and Feature A L The distance between is x L That is, feature P L The distance between and the left edge of image 1 is x L In other words, A L P L =x L Feature P R and Feature A R The distance between is x R That is, feature P R The distance between and the left edge of image 2 is x R In other words, A R P R =x R A L P L and A R P R is the disparity between the first camera 102 and the second camera 103 for the feature P. In other words, the disparity d for the feature P is x L -x R is equal to.

[0059] P L P R O L O R Since it is parallel to , according to the principle of triangles, the following equation (1) can be obtained.

[0060]

number

[0061] In the formula, P L P R = O L O R -B L P L -P R B R , O L O R =T,B L P L =A L P L -A L B L =x L -x / 2, P R B R =x / 2-x R , and P L P R =T-(x L -x / 2)-(x / 2-x R )=T-(x L -x R )=Td.

[0062] In equation (1), P L P R = Td and O L O R By substituting =T, we obtain the following equation:

[0063]

number

[0064] The depth Z of feature P is

[0065]

number

[0066] From the above, it can be seen that the distance T between the two cameras, the lens focal length f of the two cameras, and the parallax d can be calculated.

[0067] From the above description, it can be understood that the more the amount of features and the more obvious the features are in the image information collected by the first camera 102 (i.e., the first image information, e.g., image 1) and the image information collected by the second camera 103 (i.e., the second image information, e.g., image 2), the more the same features are identified in the first image information and the second image information by the electronic device 100. When the electronic device 100 identifies the same features in greater amounts, the electronic device 100 can calculate the depths of more points where the features are located. Because the depth information of the target object includes the depths of multiple points of the target object, the more the amount of points whose depths are calculated by the electronic device 100, the higher the accuracy of the depth information of the target object.

[0068] The same feature in the two image information is information that corresponds to the same feature in the two image information. For example, point A in the first image information L corresponds to part A of the object, and point A in the second image information R If point A also corresponds to part A of the object, then L and point A R are identical features in the two image information. For example, the first image information includes image 1 of a building shown in FIG. 1C, and the second image information includes image 2 of a building shown in FIG. 1C. Point A in image 1 L corresponds to part A of the building, and point A in image 2 R Point B also corresponds to part A of the building. L corresponds to part B of the building, and point B R also corresponds to part B of the building. For part A of the building, the disparity of the binocular camera is x L1 -x R1 For part B of the building, the parallax of the binocular camera is x L2 -x R2 It is.

[0069] In summary, when more features are present in the first image information and the second image information, more of the same features can be more easily obtained from the two image information, and therefore the depth information of the target object obtained by the electronic device 100 will be more accurate.

[0070] In this embodiment of the present application, after the infrared projector 101 irradiates the target object with infrared light having light points, the image information of the target object collected by the first camera 102 and the second camera 103 can not only include the features of the target object, but also include the texture features formed when the infrared light having light points is irradiated to the target object. That is, the amount of features in the image of the target object collected by the first camera 102 and the second camera 103 can be increased. After the amount of features in the image of the target object collected by the first camera 102 and the second camera 103 is increased, the electronic device 100 can more accurately identify the same features in the image information collected by the first camera 102 and the image information collected by the second camera 103, and can further determine the parallax between the first camera 102 and the second camera 103 for the same features, and calculate the depth of each point to obtain the depth information of the target object. This can improve the accuracy of the depth information of the target object calculated by the electronic device 100.

[0071] In this embodiment of the present application, a dual path camera can be obtained by improving the RGB camera, and thus the hardware cost of the electronic device 100 can be reduced. Specifically, anti-reflection coatings can be applied to both sides of each lens of the RGB camera to improve the infrared light sensing ability of the lens so that the lens can receive as much infrared light as possible, and cut-off coatings can be applied to the optical filter of the RGB camera to remove light other than infrared light and visible light and increase the transmittance of infrared light. The RGB camera including anti-reflection coatings and cut-off coatings can not only receive visible light, but also receive infrared light. The RGB camera improved in the above manner can be used as a dual path camera.

[0072] It should be noted that in some embodiments, the schematic diagram of a portion of electronic device 100 shown in FIG. 1A may be a schematic diagram of a portion of the front side of electronic device 100. That is, infrared projector 101, first camera 102, and second camera 103 are disposed on the front side of electronic device 100.

[0073] In addition to the first camera 102 and the second camera 103, the electronic device 100 may further include one or more other cameras. The one or more other cameras may include a third camera 104, where the third camera is an RGB camera. The third camera is configured to collect a two-dimensional image of the target object. The electronic device 100 may display the two-dimensional image collected by the third camera on a display screen. The third camera 104, the first camera 102, the second camera 103, and the infrared projector are located on the same side of the electronic device 100. That is, the third camera 104 may also be located on the front of the electronic device 100. The third camera 104 is a forward-facing camera of the electronic device 100.

[0074] 1A may be a schematic view of a portion of the back surface of the electronic device 100. That is, the infrared projector 101, the first camera 102, the second camera 103, and the RGB camera 104 are disposed on the back surface of the electronic device 100. The RGB camera 104 is a rear-facing camera of the electronic device 100.

[0075] The front of electronic device 100 is the side on which electronic device 100 displays a graphical user interface (e.g., a home screen or desktop of electronic device 100), i.e., the side on which the display panel is located, is typically referred to as the front. The back of electronic device 100 is the side that faces in the opposite direction to the direction that the front faces. Typically, the front of an electronic device is the side that faces a user when the electronic device is normally used by the user, and the side that faces away from the user is referred to as the back.

[0076] Of course, the one or more other cameras can further include another RGB camera or a black and white camera, etc. The another RGB camera or a black and white camera, etc. can be a front-facing camera or a rear-facing camera of the electronic device 100. This is not limited in this embodiment of the present application.

[0077] For example, in this embodiment of the present application, the electronic device may be a portable computer (e.g., a mobile phone) including an infrared projector, an RGB camera (i.e., a third camera), and two dual-path cameras (i.e., a first camera and a second camera), a notebook computer, a wearable electronic device (e.g., a smart watch), a tablet computer, an augmented reality (AR) / virtual reality (VR) device, an in-car device, etc. The specific form of the electronic device is not particularly limited in the following embodiments.

[0078] 1D is a schematic structural diagram of an electronic device 100 according to an embodiment of the present application. The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, a notification device 192, a camera 193, a display screen 194, a subscriber identification module (SIM) card interface 195, and the like.

[0079] The electronic device 100 may further include an infrared projector 196 (for example, the infrared projector 101 shown in FIG. 1A). The infrared projector 196 is configured to emit infrared light having a light spot. For example, the infrared projector 196 may emit infrared light having a wavelength of 940 nm, and the infrared light has a light spot. For the shape and arrangement of the light spot, please refer to the related description later in the embodiment of this application. Details will not be described here.

[0080] The camera 193 may include two dual path cameras (e.g., the first camera 102 and the second camera 103 shown in FIG. 1A) 193B and 193C, as well as 1-N other cameras 193A. The 1-N other cameras 193A may include the third camera 104 shown in FIG. 1A, i.e., an RGB camera, and may further include another camera, e.g., a black and white camera. The dual path cameras 193B and 193C may receive visible light and infrared light. The RGB camera is configured to collect an RGB image, e.g., a human face image (i.e., a two-dimensional person face image).

[0081] The sensor module 180 may include multiple sensors such as a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, an optical proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and a bone conduction sensor 180M.

[0082] It may be understood that the structure shown in this embodiment of the invention does not constitute a particular limitation on the electronic device 100. In some other embodiments of the present application, the electronic device 100 may include more or fewer components than those shown in the figures, or some components may be combined, some components may be divided, or other arrangements of components may exist. The components shown in the figures may be implemented by hardware, software, or a combination of software and hardware.

[0083] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The various processing units may be separate devices or may be integrated into one or more processors.

[0084] The controller may be the nerve center and command center of the electronic device 100. The controller may generate operation control signals based on command operation codes and time series signals to control command acquisition and command execution.

[0085] A memory may further be disposed within the processor 110 for storing instructions and data. In some embodiments, the memory within the processor 110 is a cache. This memory may store instructions or data that have just been used by the processor 110, or instructions or data that are used periodically by the processor 110. When the processor 110 needs to use the instructions or data again, the processor 110 may directly retrieve the instructions or data from this memory. This avoids repeated accesses and reduces the waiting time of the processor 110, thus improving system efficiency.

[0086] In some embodiments, the processor 110 may include one or more interfaces, including an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, a universal serial bus (USB) interface, and the like.

[0087] The I2C interface consists of a serial data line (SDL) and a serial clock line ( serial A bidirectional synchronous serial bus including a serial clock line (SCL) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can be separately coupled to the touch sensor 180K, a charger, a flash, a camera 193, etc. by using different I2C bus interfaces. For example, the processor 110 can be coupled to the touch sensor 180K by using an I2C interface, such that the processor 110 communicates with the touch sensor 180K by using the I2C bus interface to realize the touch function of the electronic device 100.

[0088] The I2S interface may be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 may be coupled to the audio module 170 by using the I2S buses to realize communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 may transmit audio signals to the wireless communication module 160 by using the I2S interface to realize a function of answering a call by using a Bluetooth headset.

[0089] Furthermore, the PCM interface can be used for audio communication to sample, quantize, and encode analog signals. In some embodiments, the audio module 170 can be coupled to the wireless communication module 160 by using a PCM bus interface. In some embodiments, the audio module 170 can also send audio signals to the wireless communication module 160 by using the PCM interface to realize the function of answering a call by using a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0090] The UART interface is a general-purpose serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. This bus converts the transmitted data between serial and parallel communication. In some embodiments, the UART interface is generally configured to connect the processor 110 to the wireless communication module 160. For example, the processor 110 communicates with a Bluetooth module in the wireless communication module 160 by using the UART interface to realize a Bluetooth function. In some embodiments, the audio module 170 can send an audio signal to the wireless communication module 160 by using the UART interface to realize a function of playing music by using a Bluetooth headset.

[0091] The MIPI interface may be configured to connect the processor 110 to peripheral devices such as a display screen 194 or a camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 communicates with the camera 193 by using the CSI interface to realize the photographing function of the electronic device 100, and the processor 110 communicates with the display screen 194 by using the DSI interface to realize the display function of the electronic device 100.

[0092] The GPIO interface may be configured by using software. The GPIO interface may be configured as a control signal or a data signal. In some embodiments, the GPIO interface may be configured to connect the processor 110 to the camera 193, the display screen 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. Alternatively, the GPIO interface may be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0093] The USB interface 130 is an interface conforming to the USB standard specification, and specifically may be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 may be configured to connect to a charger to charge the electronic device 100, may be configured to transmit data between the electronic device 100 and a peripheral device, or may be configured to connect to a headset and play audio files by using the headset. Alternatively, the interface may be configured to connect to another electronic device, such as an AR device.

[0094] It may be understood that the interfacing relationships between modules shown in this embodiment of the present invention are merely illustrative examples and do not constitute limitations on the structure of the electronic device 100. In some other embodiments of the present application, the electronic device 100 may alternatively use different interfacing schemes than those in the above-described embodiments, or may use a combination of interfacing schemes.

[0095] The charging management module 140 is configured to receive a charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive a charging input from a wired charger by using the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive a wireless charging input by using a wireless charging coil of the electronic device 100. When the charging management module 140 charges the battery 142, the power management module 141 may further provide power to the electronic device.

[0096] The power management module 141 is configured to communicate with the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 and provides power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 may be further configured to monitor parameters such as battery capacity, battery cycle count, and battery health (leakage and impedance). In some other embodiments, the power management module 141 may alternatively be located within the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 may alternatively be located within the same device.

[0097] The wireless communication function of the electronic device 100 may be realized by antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, a modem processor, a baseband processor, and so on.

[0098] Antenna 1 and Antenna 2 are configured to transmit and receive electromagnetic signals. Each antenna in electronic device 100 may be configured to cover one or more communication frequency bands. Various antennas may be further multiplexed to improve antenna utilization. For example, Antenna 1 may be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas may be used in combination with tuning switches.

[0099] The mobile communication module 150 can be applied to the electronic device 100 to provide a solution including wireless communication technologies such as 2G, 3G, 4G, and 5G. The mobile communication module 150 can include at least one optical filter, a switch, a power amplifier, a low noise amplifier (LNA), and the like. The mobile communication module 150 can receive electromagnetic waves by using the antenna 1, process the received electromagnetic waves such as filtering and amplifying, and send the processed electromagnetic waves to the modem processor for demodulation. The mobile communication module 150 can further amplify the signal modulated by the modem processor, convert the amplified signal into electromagnetic waves, and radiate the electromagnetic waves by using the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 and at least some of the modules of the processor 110 can be disposed in the same device.

[0100] The modem processor may include a modulator and a demodulator. The modulator is configured to modulate a low-frequency baseband signal to be transmitted into an intermediate or high-frequency signal. The demodulator is configured to demodulate a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the low-frequency baseband signal obtained by demodulation to the baseband processor for processing. The low-frequency baseband signal is processed by the baseband processor, and the processed signal is sent to the application processor. The application processor outputs an acoustic signal by using an audio device (including but not limited to speakers 170A, receiver 170B, etc.) or displays an image or video by using a display screen 194. In some embodiments, the modem processor may be an independent component. In some other embodiments, the modem processor may be independent from the processor 110 and is located in the same device as the mobile communication module 150 or another functional module.

[0101] The wireless communication module 160 can be applied to the electronic device 100 to provide solutions including wireless communication technologies such as wireless local area networks (WLAN) (e.g., Wi-Fi network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), NFC, and infrared (IR) technology. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic wave signals by using the antenna 2, performs frequency modulation and filtering on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can further receive signals to be transmitted from the processor 110, perform frequency modulation and amplification on the signals, convert the processed signals into electromagnetic waves, and radiate the electromagnetic waves by using the antenna 2.

[0102] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150 and antenna 2 is coupled to wireless communication module 160, such that electronic device 100 can communicate with networks and other devices by using wireless communication technologies. Wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, IR technologies, and the like. GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0103] The electronic device 100 realizes display functions by using a GPU, a display screen 194, an application processor, etc. The GPU is a microprocessor for image processing and is connected to the display screen 194 and the application processor. The GPU is configured to perform mathematical and geometric calculations and is used for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0104] The display screen 194 is configured to display images, videos, etc. The display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0105] The electronic device 100 can achieve the photography function by using an ISP, a camera 193, a video codec, a GPU, a display screen 194, an application processor, and the like.

[0106] The ISP is primarily configured to process data fed back by the camera 193. For example, when taking a picture, the shutter is opened, light is transmitted through the lens to the camera's photosensitive elements, the light signal is converted into an electrical signal, and the camera's photosensitive elements send the electrical signal to the ISP for processing to convert the electrical signal into an image that can be seen by the naked eye. The ISP can further optimize the noise, brightness, and color of the image based on algorithms. The ISP can further optimize parameters such as exposure and color temperature for the photography scenario. In some embodiments, the ISP can be located within the camera 193.

[0107] The camera 193 is configured to capture still images or videos. An optical image of an object is generated by using a lens, and the optical image is projected onto a photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then sends the electrical signal to the ISP for converting the electrical signal into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard image signal, such as an RGB format or a YUV format. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0108] The digital signal processor is configured to process digital signals. In addition to digital image signals, the digital signal processor may further process other digital signals. For example, when the electronic device 100 selects a frequency, the digital signal processor is configured to perform a Fourier transform or the like on the frequency energy.

[0109] A video codec is configured to compress or decompress digital video. Electronic device 100 may support one or more types of video codecs. Thus, electronic device 100 may play or record video in multiple encoding formats, such as moving picture experts group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0110] The NPU is a neural-network (NN) computing processor. The NPU can quickly process input information by referring to the structure of biological neural networks, such as the communication mode between human brain neurons, and can also continuously perform self-learning. The NPU can realize applications such as intelligent cognition of the electronic device 100, such as image recognition, face recognition, voice recognition, and text understanding.

[0111] The external memory interface 120 may be configured to connect to an external storage card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external storage card communicates with the processor 110 by using the external memory interface 120 to realize data storage functions. For example, files such as music and videos are stored on the external storage card.

[0112] The internal memory 121 may be configured to store computer-executable program code including instructions. The processor 110 performs various functional applications and data processing of the electronic device 100 by executing the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, an application program required for at least one function (e.g., a sound playback function or an image playback function), and the like. The data storage area may store data generated in the course of use of the electronic device 100 (e.g., audio data and a phone book), and the like. In addition, the internal memory 121 may include a high-speed random access memory, or may include at least one magnetic disk storage device, a flash storage device, and a non-volatile memory such as a universal flash storage (UFS).

[0113] The electronic device 100 can achieve audio functions, such as playing and recording music, by using an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, an application processor, and the like.

[0114] Audio module 170 is configured to convert digital audio information into analog audio signals for output, and is also configured to convert analog audio inputs into digital audio signals. Audio module 170 may be further configured to perform encoding and decoding of audio signals. In some embodiments, audio module 170 may be located within processor 110, or some functional modules of audio module 170 are located within processor 110.

[0115] The speaker 170A, also called a "loudspeaker," is configured to convert audio electrical signals into acoustic signals. The electronic device 100 can be used to listen to music or answer hands-free calls by using the speaker 170A.

[0116] Receiver 170B, also called an "earphone," is configured to convert an audio electrical signal into an acoustic signal. When electronic device 100 is used to answer a call or listen to voice information, receiver 170B can be placed near a human ear to hear the sound.

[0117] The microphone 170C, also referred to as a "mike" or "mic", is configured to convert an acoustic signal into an electrical signal. When making a call or transmitting voice information, a user can put his / her mouth close to the microphone 170C to speak and input a voice signal to the microphone 170C. At least one microphone 170C may be disposed in the electronic device 100. In some other embodiments, two microphones 170C may be disposed in the electronic device 100 to achieve a noise reduction function in addition to collecting acoustic signals. In some other embodiments, three, four, or more microphones 170C may be disposed in the electronic device 100 for collecting acoustic signals and reducing noise. In addition, the microphones can identify sound sources, achieve directional recording functions, and the like.

[0118] Headset interface 170D is configured to connect to a wired headset and may be USB interface 130, or may be a 3.5 mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0119] The pressure sensor 180A is configured to sense a pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed in the display screen 194. There are many types of pressure sensors 180A, such as, for example, resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. The capacitive pressure sensor can include at least two parallel plates having a conductive material. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the strength of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the strength of the touch operation based on the pressure sensor 180A. Furthermore, the electronic device 100 can also calculate the touch position based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation strengths can correspond to different operation commands. For example, if a touch operation with a touch strength less than a first pressure threshold is applied over an icon of an SMS application, instructions are executed to display an SMS message, or if a touch operation with a touch strength equal to or greater than the first pressure threshold is applied over an icon of an SMS application, instructions are executed to create an SMS message.

[0120] The gyroscope sensor 180B may be configured to determine the motion attitude of the electronic device 100. In some embodiments, the gyroscope sensor 180B may be used to determine the angular velocity of the electronic device 100 around three axes (i.e., x-axis, y-axis, and z-axis). The gyroscope sensor 180B may be used for image stabilization during photography. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the shaking of the electronic device 100, and based on this angle, calculates the required compensation distance by the lens module, so as to realize image stabilization by offsetting the shaking of the electronic device 100 by the inverse motion of the lens. The gyroscope sensor 180B may also be further used in navigation and motion sensing game scenarios.

[0121] The air pressure sensor 180C is configured to measure air pressure. In some embodiments, the electronic device 100 calculates altitude by using the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0122] The magnetic sensor 180D includes a Hall effect sensor. The electronic device 100 can detect the opening and closing of the clamshell leather case by using the magnetic sensor 180D. In some embodiments, if the electronic device 100 is a clamshell phone, the electronic device 100 can detect the opening and closing of the clamshell based on the magnetic sensor 180D. Furthermore, functions such as automatic unlocking on flip are set based on the detected opening and closing state of the leather case or the detected opening and closing state of the clamshell.

[0123] The acceleration sensor 180E can detect the magnitude of acceleration of the electronic device 100 in each direction (usually three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. The acceleration sensor 180E may be further configured to identify the orientation of the electronic device, and is applied to applications such as landscape mode / portrait mode switching and pedometer.

[0124] The distance sensor 180F is configured to measure distance. The electronic device 100 can measure distance by using infrared or laser. In some embodiments, in a photography scenario, the electronic device 100 can measure distance by using the distance sensor 180F to perform fast focusing.

[0125] The optical proximity sensor 180G may include a light detector, such as a light emitting diode (LED) and a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light by using the light emitting diode. The electronic device 100 detects infrared reflected light from a nearby object by using the photodiode. If sufficient reflected light is detected, the electronic device 100 can determine that an object is present near the electronic device 100, and if sufficient reflected light is not detected, the electronic device 100 can determine that an object is not present near the electronic device 100. By using the optical proximity sensor 180G, the electronic device 100 can detect that a user is holding the electronic device 100 close to the ear for a call and automatically turn off the screen to save power. Furthermore, the optical proximity sensor 180G can also be used for automatic screen lock or unlock in leather case mode or pocket mode.

[0126] The ambient light sensor 180L is configured to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 based on the sensed brightness of the ambient light. In addition, the ambient light sensor 180L can also be configured to automatically adjust the white balance when taking a picture. In addition, the ambient light sensor 180L can cooperate with the optical proximity sensor 180G to detect whether the electronic device 100 is in a pocket to prevent unintentional touches.

[0127] The fingerprint sensor 180H is configured to collect a fingerprint. The electronic device 100 can use the collected fingerprint characteristics to achieve fingerprint-based unlocking, application lock access, fingerprint-based photo taking, fingerprint-based call answering, and the like.

[0128] The temperature sensor 180J is configured to detect temperature. In some embodiments, the electronic device 100 executes a temperature handling policy by using the temperature detected by the temperature sensor 180J. For example, if the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of the processor near the temperature sensor 180J to reduce power consumption and implement thermal protection. In some other embodiments, if the temperature falls below another threshold, the electronic device 100 warms the battery 142 to avoid an abnormal shutdown of the electronic device 100 due to low temperature. In some other embodiments, if the temperature falls below yet another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid an abnormal shutdown due to low temperature.

[0129] The touch sensor 180K is also referred to as a "touch panel." The touch sensor 180K may be disposed within the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also referred to as a "touch control screen." The touch sensor 180K is configured to detect a touch operation acting on or near the touch sensor. The touch sensor can transmit the detected touch operation to an application processor to determine the type of touch event. A visual output associated with the touch operation can be provided by using the display screen 194. In some other embodiments, the touch sensor 180K may alternatively be disposed on the surface of the electronic device 100, and is located at a position different from the position of the display screen 194.

[0130] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain vibration signals from the vibrating bones of the human vocal part. In addition, the bone conduction sensor 180M may contact the human pulse and receive blood pressure and pulse signals. In some embodiments, the bone conduction sensor 180M may alternatively be disposed in a headset to form a bone conduction headset. The audio module 170 can analyze audio signals based on the vibration signals obtained by the bone conduction sensor 180M from the vibrating bones of the vocal part to realize audio functions. The application processor can analyze heartbeat information based on the blood pressure and pulse signals obtained by the bone conduction sensor 180M to realize heartbeat detection functions.

[0131] The keys 190 include a power key, a volume key, etc. The keys 190 may be mechanical keys or may be touch keys. The electronic device 100 can receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0132] The motor 191 can generate a vibration stimulus. The motor 191 can be configured to generate a vibration stimulus for an incoming call, or can be configured to generate a vibration feedback for a touch. For example, touch operations acting on different applications (e.g., taking pictures and playing audio) may correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, the motor 191 can also correspondingly generate different vibration feedback effects. Different application scenarios (e.g., time reminders, information reception, alarm clocks, and games) may also correspond to different vibration feedback effects. The touch vibration feedback effects may be further customizable.

[0133] Indicator 192 may be an indicator light and may be configured to indicate charging status and changes in battery level, or may be configured to indicate messages, missed calls, notifications, and the like.

[0134] The SIM card interface 195 is configured to connect to a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to achieve contact with or separation from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support a Nano SIM card, a Micro SIM card, a SIM card, etc. Multiple cards may be inserted into one SIM card interface 195. The multiple cards may be of the same type or different types. The SIM card interface 195 can also be compatible with various types of SIM cards. The SIM card interface 195 can also be compatible with an external storage card. The electronic device 100 interacts with a network by using a SIM card to achieve functions such as calling and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card may be integrated into the electronic device 100 and cannot be separated from the electronic device 100.

[0135] An embodiment of the present application provides a method for acquiring depth information. The method is applicable to an electronic device 100. The electronic device 100 includes an infrared projector 101, a first camera 102, and a second camera 103. For example, as shown in FIG. 1A, the lens focal lengths of the first camera 102 and the second camera 103 are both f, and the distance between the first camera 102 and the second camera 103 (i.e., a first length) is T. As shown in FIG. 2, the method for acquiring depth information may include S201 to S203.

[0136] S201. The electronic device 100 receives a first command, which triggers the electronic device 100 to obtain depth information of an object of interest.

[0137] The electronic device 100 can receive a first operation of a user, where the first operation is used to trigger the electronic device 100 to perform a first event. The electronic device 100 needs to use depth information of the target object to perform the first event. Thus, in response to the first operation, the electronic device 100 can obtain a first command, where the first command can trigger the electronic device 100 to obtain depth information of the target object.

[0138] For example, the method in this embodiment of the present application may be applied to multiple scenarios, such as a face-based unlocking scenario, a face-based payment scenario, an AR scenario, a 3D modeling scenario, and a wide-opening scenario.

[0139] In the face-based unlocking scenario, it is assumed that the electronic device 100 is powered on and the screen of the electronic device 100 is off. The first operation may be a hit operation (e.g., a single-hit operation) performed by a user on an associated physical key of the electronic device 100. For example, the associated physical key may be a lock screen key or a Home key. When the electronic device 100 receives the first operation performed by the user on the physical key, it may indicate that the user is about to unlock the electronic device 100.

[0140] Alternatively, the electronic device 100 may include one or more sensors configured to detect the state of the electronic device 100 held by a user. It is assumed that the power of the electronic device 100 is on, the screen of the electronic device 100 is off, or a lock screen interface is displayed on the electronic device 100. The reception of the first operation by the electronic device 100 may specifically be as follows: the sensor may detect that the current state of the electronic device 100 held by the user has undergone a change that satisfies a preset condition. If the state of the electronic device 100 held by the user has undergone a change that satisfies a preset condition (for example, the electronic device 100 is picked up by the user, and the user holds the electronic device 100 so that the included angle between the display screen of the electronic device 100 and a horizontal plane falls within a certain range), it indicates that the user may be about to unlock the electronic device 100.

[0141] In other words, the first operation can be used to trigger the electronic device 100 to perform an unlock (i.e., perform a first event). Prior to unlocking the electronic device 100, a user's identity verification needs to be performed. Facial recognition is one way of verifying the user's identity. To perform facial recognition, the electronic device 100 can obtain depth information of a target object.

[0142] In this embodiment of the present application, one or more sensors can detect whether the electronic device 100 has been rotated, whether the electronic device 100 has moved forward relative to the user, or whether the electronic device 100 has moved upward relative to the horizontal line, thereby determining whether the state of the electronic device 100 held by the user has undergone a change that satisfies a preset condition. Specifically, the electronic device 100 can detect the motion parameters of the electronic device 100, and then determine whether the electronic device 100 has been rotated, whether the electronic device 100 has moved forward relative to the user, and whether the electronic device 100 has moved upward relative to the horizontal line based on the motion parameters, and finally determine whether the state of the electronic device 100 held by the user has undergone a change that satisfies a preset condition based on the determination result.

[0143] For example, in this embodiment, "the current state of electronic device 100 held by the user undergoes a change that satisfies a preset condition" may specifically include the angle between the display screen of electronic device 100 and a horizontal plane falling within a preset range after a sensor detects that electronic device 100 has been rotated and moved upward.

[0144] It should be noted that since different users have different photographing habits, the electronic device 100 can collect statistics on the state change parameters collected by the sensor in the process of most or all users using the electronic device 100 to collect human face images, and determine the change in the state change parameter as a change that satisfies a preset condition. In this way, the electronic device 100 can determine that it is relatively likely that the user wants to unlock the electronic device 100 in response to the change in the state change parameter collected by the sensor in the electronic device 100 satisfying the change in the preset condition. In this case, S202 can be executed.

[0145] For example, in a face-based payment scenario, the first operation may be a touch operation (e.g., a single-touch operation) performed by a user on a payment button on a payment interface. For example, the payment interface 301 shown in FIG. 3(a) includes a "Pay Now" payment button 302, and the first operation may be a single-touch operation performed by a user on the "Pay Now" button 302. Alternatively, the first operation may be a pre-defined gesture input by a user on the payment interface, such as an S-shaped gesture. In response to the first operation, the electronic device 100 may make a payment for the order (i.e., perform a first event). However, before the electronic device 100 makes a payment for the order, a verification of the user's identity needs to be performed. Facial recognition is one way of verifying the user's identity. To perform facial recognition, the electronic device 100 may obtain depth information of a target object. In response to a single-touch operation performed by a user on the "Pay Now" button 302, the electronic device 100 may display a facial recognition interface 303 shown in FIG. 3(b).

[0146] S202. In response to the first command, the electronic device 100 emits infrared light having a light point by using the infrared projector 101, collects first image information of a target object by using the first camera 102, and collects second image information of the target object by using the second camera 103.

[0147] In response to a first command, the electronic device 100 can turn on the infrared projector 101 and turn on the first camera 102 and the second camera 103. After the electronic device 100 turns on the infrared projector 101, the infrared projector 101 can emit infrared light having a light spot. After the electronic device 100 turns on and initializes the first camera 102 and the second camera 103, the first camera 102 and the second camera 103 can collect image information of a target object.

[0148] In this embodiment of the present application, the infrared projector 101 emits infrared light having a light spot to the target object, and the infrared light having a light spot is irradiated to the target object, which can increase the amount of features (i.e., texture features) of the target object, and can improve the identification rate of each feature of the target object by the first camera 102 and the second camera 103.

[0149] In this embodiment of the present application, the infrared light having a light spot emitted by the infrared projector 101 and the principle of the emission of the infrared light having a light spot by the infrared projector 101 are described herein.

[0150] In general, infrared light is not visible to human eyes. However, some infrared light (for example, 850 nm infrared light) has obvious red light characteristics, and a small amount of red light is still visible to human eyes. When infrared projector 101 emits this kind of infrared light to a target object, a user can see the infrared light irradiated to the target object, and thus the user's visual experience may be affected.

[0151] The infrared light emitted by the infrared projector 101 may be completely invisible to the human eye so that the infrared light emitted by the infrared projector 101 to the target object does not affect the visual experience of the user. For example, the infrared light emitted by the infrared projector 101 may be infrared light of 890 nm to 990 nm, and specifically, for example, infrared light of 940 nm.

[0152] The exposure of some cameras (e.g., rolling shutter cameras) is performed line by line. Therefore, when the exposure of the first camera 102 and the second camera 103 is performed line by line, the infrared projector 101 needs to be turned on and emit infrared light having light points during the entire exposure period of the first camera 102 and the second camera 103. Otherwise, the image information collected by the first camera 102 and the second camera 103 during exposure may not have light points in some images. In this case, it is required that the operating power of the infrared projector 101 is not too large, so that the infrared projector 101 generates heat and becomes relatively high temperature, which affects the efficiency of infrared light emission by the infrared projector 101 (i.e., affects the brightness of infrared light) and also affects the effect of texture features formed when the infrared light having light points is irradiated to an object.

[0153] Of course, the operating power of the infrared projector 101 cannot be too low. If the operating power of the infrared projector 101 is too low, it will also affect the radiation efficiency of the infrared light by the infrared projector 101, and even affect the effect of the texture feature formed when the infrared light having a light point is irradiated onto an object.

[0154] In summary, to ensure the effect of the texture feature formed when infrared light is irradiated onto an object, the output of the infrared projector 101 should not be too high or too low. To stabilize the output of the infrared projector 101, the operating current of the infrared projector 101 may be in the range of 100mA to 200mA. For example, the operating current of the infrared projector 101 may be 150mA.

[0155] In general, an infrared projector can include three parts: (1) an infrared light source, (2) a collimating mirror, and (3) a diffractive optical element (DOE).

[0156] The infrared light source may be a laser light source. For example, the laser light source may be a vertical cavity surface emitting laser (VCSEL). A VCSEL is capable of emitting the infrared light described above.

[0157] The collimating mirror may be a 1P lens or a 2P lens. The collimating mirror being a 1P lens means that the collimating mirror includes one lens. The 2P lens means that the collimating mirror includes two lenses. The collimating mirror is configured to convert the non-parallel light into a nearly parallel light source. This conversion can reduce the noise level of the texture of the infrared light emitted by the infrared projector, i.e., reduce the light in the non-textured parts. In other words, this conversion can make the bright dots of infrared light emitted by the infrared projector brighter and the dark dots of infrared light emitted by the infrared projector darker.

[0158] During the daytime, when the visible light is relatively strong, the texture features formed when the infrared light with light spots is irradiated onto the target object are almost invisible in the image information collected by the first camera 102 and the second camera 103. In this case, the electronic device 100 can make a judgment based on the features of the target object when identifying the same features in the two image information. Therefore, the bright dots obtained by converting the light by the collimating mirror do not affect the calculation of the depth.

[0159] In this embodiment of the present application, the collimating mirror may be omitted in the infrared projector 101. Specifically, in this embodiment of the present application, the infrared projector 101 may not include a collimating mirror, but may only include an infrared light source and a DOE. The collimating mirror is configured so that the light emitted by the VCSEL becomes nearly parallel light after passing through the collimating mirror. In this way, when the collimating mirror is present, the diameter of the light spot is small and the contrast is high (i.e., the brightness of the light spot area is high and the brightness of the non-light spot area is low) after the light passes through the DOE. Therefore, when the intensity of visible light is very low or when it is completely dark (e.g., at night), the target object illuminated by the projector will be black or invisible in the area where there is no spot or light spot. The reason is that the light spot and light of the projector with the collimating mirror are more concentrated, and the place where there is no light spot is theoretically no light or has only a little ambient light. If the collimating mirror is removed from the infrared projector, the light before entering the DOE is not parallel light, so the contrast of the light spot is low, and some light still exists in the non-light spot area. A projector without a collimating mirror is a noise in the design of the projector. However, in this embodiment of the present application, the measurement accuracy of the depth information can be improved. In a projector without a collimating mirror, when the visible light is weak or in complete darkness, in the area of ​​the target object where the light spot is irradiated, the target object can be illuminated by the light spot, and in the area not illuminated by the light spot, a relatively weak light is still irradiated to the target object due to this noise. In this case, the target object is visible as a whole. Therefore, the features of the target object can be made clearer, and the measurement accuracy of the depth information can be improved.

[0160] The DOE is configured to convert the parallel light into the required light output form by using Fourier optical principles. The output form can be a point, a line, or a surface. In this embodiment of the present application, the DOE can be arranged to control an infrared projector to emit infrared light with a light spot. The shape, quantity, and arrangement of the light spot can all be realized by arranging the DOE.

[0161] Specifically, the infrared light emitted by the infrared projector 101 can include multiple light points. The multiple light points can include multiple speckle grating groups. A speckle grating group includes one or more speckle gratings, and each speckle grating includes multiple speckles. Figure 4 is a schematic diagram of an example of multiple speckle gratings according to an embodiment of the present application. For example, the speckle grating 401 shown in Figure 4 is a speckle grating including multiple circular speckles.

[0162] In some embodiments, the speckles of a speckle grid can have the same shape. For example, the speckles of speckle grid 401, speckle grid 402, speckle grid 403, speckle grid 404, and speckle grid 408 shown in FIG. 4 are all circular. All speckles of speckle grid 405 shown in FIG. 4 are rectangular.

[0163] In some other embodiments, at least two of the speckles of the speckle grid have different shapes. For example, the speckle grid 406 shown in FIG. 4 includes a circular speckle and a rectangular speckle. The speckle grid 407 shown in FIG. 4 includes a plurality of speckles of different shapes.

[0164] It should be noted that the various speckle gratings may have various shapes. For example, as shown in FIG. 4, speckle grating 401 is a rectangular speckle grating, speckle grating 402 is a triangular speckle grating, speckle grating 403 is an octagonal speckle grating, and speckle grating 404 is another octagonal speckle grating. Since speckle grating 401, speckle grating 402, speckle grating 403, and speckle grating 404 are different shapes, speckle grating 401, speckle grating 402, speckle grating 403, and speckle grating 404 are different speckle gratings.

[0165] Different speckle gratings have different amounts of speckles, shapes of speckles, and shapes of gratings. For example, speckle gratings 401 and 408 shown in FIG. 4 have the same shape, but speckle gratings 401 and 408 contain different amounts of speckles (speckle grating 401 contains one more speckle than speckle grating 408). Therefore, speckle gratings 401 and 408 are different speckle gratings. For example, speckle gratings 401 and 405 shown in FIG. 4 have the same shape, and speckle gratings 401 and 405 contain the same amount of speckles, respectively, but the shape of the speckles in speckle grating 401 (circular) is different from the shape of the speckles in speckle grating 405 (rectangle). Therefore, speckle gratings 401 and 405 are different speckle gratings. 4 have the same shape, and the two speckle gratings each contain the same amount of speckles, but the shape of speckle grating 402 (triangle) is different from the shape of speckle grating 404 (octagon). Thus, speckle grating 402 and speckle grating 404 are different speckle gratings.

[0166] 4 is merely an illustrative schematic diagram of some example speckle gratings. The speckle grating of multiple light points of infrared light includes, but is not limited to, the speckle grating shown in FIG.

[0167] In some embodiments, the multiple light points include multiple identical speckle grating groups. Each speckle grating group includes one speckle grating. For example, it is assumed that the infrared light emitted by the infrared projector 101 has multiple light points 501 shown in FIG. 5. 501 shown in FIG. 5 indicates multiple light points. The multiple light points 501 include multiple identical speckle grating groups 502. The speckle grating group 502 includes one speckle grating 401. As another example, it is assumed that the infrared light emitted by the infrared projector 101 has multiple light points 503 shown in FIG. 5. The multiple light points 503 include multiple identical speckle grating groups 504. The speckle grating group 504 includes one speckle grating 407.

[0168] In some other embodiments, the plurality of light spots includes a plurality of identical speckle gratings, each speckle grating group including a plurality of speckle gratings, at least two of the plurality of speckle gratings being different.

[0169] For example, it is assumed that the infrared light emitted by the infrared projector 101 has a plurality of light points 601 as shown in Fig. 6A. The plurality of light points 601 includes a plurality of identical speckle grating groups 602. The speckle grating group 602 includes the speckle grating 402 and the speckle grating 404. The speckle grating 402 is different from the speckle grating 404.

[0170] As another example, it is assumed that the infrared light emitted by the infrared projector 101 has a plurality of light points 603 shown in FIG. 6B. The plurality of light points 603 includes a plurality of identical speckle grating groups 604. The speckle grating group 604 includes two speckle gratings 401 and two speckle gratings 403. The speckle grating 401 is different from the speckle grating 403.

[0171] As another example, assume that the infrared light emitted by the infrared projector 101 has a plurality of light points 605 shown in Fig. 6C. The plurality of light points 605 includes a plurality of identical speckle grating groups 606. The speckle grating group 606 includes a speckle grating 404, a speckle grating 408, a speckle grating 406, and a speckle grating 407. The speckle grating 404, the speckle grating 408, the speckle grating 406, and the speckle grating 407 are different from each other.

[0172] In some other embodiments, the plurality of light points includes a plurality of speckle grating groups. The plurality of speckle grating groups includes at least two different speckle grating groups. For example, it is assumed that the infrared light emitted by the infrared projector 101 has a plurality of light points 701 shown in FIG. 7A. The plurality of light points 701 includes a plurality of different speckle grating groups, such as a speckle grating group 602, a speckle grating group 604, and a speckle grating group 606.

[0173] In some other embodiments, the light spots include speckles of different shapes. The speckles of different shapes are randomly arranged. For example, as shown in FIG. 7B, the light spots 702 include speckles of different shapes.

[0174] Please note that Figure 5, Figure 6A, Figure 6B, Figure 6C, or Figure 7A is only an example schematic diagram of a plurality of light points in infrared light. The amount of speckle grating in the plurality of light points in infrared light is not limited to the amount of speckle grating shown in Figure 5, Figure 6A, Figure 6B, Figure 6C, or Figure 7A.

[0175] In general, the amount of speckle in the infrared light emitted by the infrared projector 101 is about 3000. Naturally, the greater the amount of speckle in the infrared light emitted by the infrared projector 101, the greater the increase in the amount of features of the target object when the infrared light is irradiated to the target object, and the more convenient it is for the electronic device 100 to calculate the depth information of the target object. For example, the depth information of the target object calculated by the electronic device 100 when the amount of speckle in the infrared light emitted by the infrared projector 101 is about 7000 is more accurate than the depth information of the target object calculated by the electronic device 100 when the amount of speckle is about 3000.

[0176] If the speckle gratings in the infrared light emitted by the infrared projector 101 are periodically repeated, it can be understood that the texture features formed when the infrared light is irradiated onto the target object are also periodically repeated. In this embodiment of the present application, the repetition frequency of the speckle gratings is determined by the number of identical speckle gratings appearing in a predetermined area. number The same speckle grating group appearing in a preset area can be expressed as: number The more the number of the speckle gratings, the higher the repetition frequency of the speckle gratings and the smaller the repetition interval of the speckle gratings. In this case, feature matching errors are likely to occur when the electronic device 100 identifies the same features in the image information collected by the first camera 102 and the image information collected by the second camera 103. In this embodiment of the present application, feature matching means identifying the same features in the image information collected by the first camera 102 and the image information collected by the second camera 103. The same speckle gratings appearing in a preset area are identified by the following formula: number The smaller the repetition frequency of the speckle grating, the lower the repetition interval of the speckle grating.

[0177] The repeat interval of the speckle grating reflected in the image is the shortest distance between two identical speckle gratings. For example, the distance between two speckle gratings may be the distance between the center points of the two speckle gratings. For example, as shown in FIG. 7C, the multiple light points 703 include multiple repeated speckle grating groups, such as speckle grating group 704, speckle grating group 705, and speckle grating group 706. The speckle grating group 704, speckle grating group 705, and speckle grating group 706 are identical. Point K1 is the center point of the speckle grating 707 in the speckle grating group 704. Point K2 is the center point of the speckle grating 708 in the speckle grating group 705. Point K3 is the center point of the speckle grating 709 in the speckle grating group 706. The repeat interval of the speckle grating shown in FIG. 7C can be represented by the shortest distance between the distance between point K1 and point K2 and the distance between point K1 and point K3. Because the distance between points K1 and K2 is shorter than the distance between points K1 and K3, the repeat interval of the speckle grating shown in FIG. 7C can be represented by the distance between points K1 and K2.

[0178] In this embodiment of the present application, the lower the repetition frequency of the speckle grating, the less repetitive the texture features formed when infrared light is irradiated onto the target object, and the more accurate the depth information of the target object calculated by the electronic device 100.

[0179] The electronic device 100 has the formula

[0180]

number

[0181] By using the above, the depth information of the target object is calculated according to the principle of triangulation. Specifically, the depth information of the target object is calculated based on the distance between the first camera 102 and the second camera 103 (i.e., the first length T), the lens focal length f of the first camera 102 and the second camera 103, and the parallax d.

[0182] In order to ensure that the infrared light having a light spot emitted by the infrared projector 101 can be irradiated on the target object, and the target object irradiated with the infrared light having a light spot can be photographed by the first camera 102 and the second camera 103, in this embodiment of the present application, the coverage area of ​​the field of view (FOV) of the infrared projector 101, the coverage area of ​​the FOV of the first camera 102, and the coverage area of ​​the FOV of the second camera 103 partially overlap or completely overlap. The size of the FOV can represent the field of view of an optical device (e.g., a camera).

[0183] The larger the overlapping area of ​​the coverage areas of the FOVs of the infrared projector 101, the first camera 102, and the second camera 103, the greater the amount of texture features collected by the first camera 102 and the second camera 103. In order to make the overlapping area of ​​the coverage areas of the FOVs of the infrared projector 101, the first camera 102, and the second camera 103 relatively large, the infrared projector 101 can be disposed between the first camera 102 and the second camera 103, as shown in FIG. 1A.

[0184] It can be understood that the distance between the first camera 102 and the second camera 103 (i.e., the first length) also affects the size of the overlapping area. For example, the larger the first length, the smaller the overlapping area of ​​the coverage area of ​​the FOV of the first camera 102 and the second camera 103. However, if the first length is too small, the parallax between the first camera 102 and the second camera 103 for each feature of the target object will also be very small and close to 0. Therefore, the first length cannot be too large or too small. If the first length is too large or too small, the accuracy of the depth information calculated by the electronic device 100 will be affected.

[0185] Experiments have shown that the depth information calculated by the electronic device 100 is relatively accurate when the first length T is any length in the range of 20 mm to 30 mm. Therefore, in this embodiment of the present application, the first length T may be any length in the range of 20 mm to 30 mm. For example, the first length T may be 29.5 mm. It should be noted that how to set the distance between the two cameras may be influenced by the camera parameters. Therefore, the first length T being 20 mm to 30 mm is just an example.

[0186] When it is required that the first camera 102 and the second camera 103 can collect image information including the light spots after the infrared projector 101 emits the infrared light having the light spots, the first camera 102 and the second camera 103 are required to receive the infrared light. For example, the first camera 102 and the second camera 103 may be configured to receive infrared light of 890 nm to 990 nm, such as 940 nm infrared light.

[0187] From the above description, it can be understood that in this embodiment of the present application, the first camera 102 and the second camera 103 are required to be able to sense (i.e., receive) infrared light. However, a general RGB camera can only sense visible light, and cannot sense infrared light. Meanwhile, the cost of arranging an infrared camera with an infrared light sensing function in the electronic device 100 is relatively high, and the use of an infrared camera increases the power consumption of the electronic device 100.

[0188] In order to reduce the hardware cost of the electronic device 100 and reduce the power consumption of the electronic device 100, in this embodiment of the present application, a general RGB camera is improved to obtain a first camera 102 and a second camera 103 that can sense infrared light.

[0189] Referring to Fig. 8(a), this figure is a schematic structural diagram of a general RGB camera module according to an embodiment of the present application. As shown in Fig. 8(a), in this embodiment of the present application, a rolling shutter camera is used as an example to explain how to improve a general RGB camera to obtain the first camera 102 and the second camera 103 in this embodiment of the present application.

[0190] As shown in FIG. 8(a), the RGB camera module 800 may include a 3P lens, an optical filter (also called a filter) 804, and a sensor 805. The 3P lens means that the RGB camera includes three lenses: a lens 801, a lens 802, and a lens 803. The sensor 805 may be a 2M sensor. The 2M means that the highest resolution of the RGB camera can reach 2 megapixels.

[0191] In order to enable the RGB camera module 800 shown in FIG. 8(a) to sense both visible light and infrared light, the RGB camera module 800 shown in FIG. 8(a) can be improved as follows: anti-reflection coating is applied to both sides of each lens (e.g., lens 801, lens 802, and lens 803) of the RGB camera module 800 shown in FIG. 8(a) to obtain lens 801, lens 802, and lens 803 shown in FIG. 8(b). A cut-off coating is applied on the light filter 804 to obtain the light filter 804 shown in FIG. 8(b). In this embodiment of the present application, the sensor 805 in the RGB camera module 800 may not be improved. In this embodiment of the present application, a cut-off coating may be applied to both sides of the light filter 804, or a cut-off coating may be applied to one side of the light filter 804.

[0192] It should be noted that the first camera 102 and the second camera 103 need to have the capability of sensing the infrared light emitted by the infrared projector 101 so that the infrared light emitted by the infrared projector 101 can be sensed by the first camera 102 and the second camera 103. Therefore, the anti-reflective coating may be an anti-reflective coating corresponding to the wavelength of the infrared light emitted by the infrared projector 101. For example, the anti-reflective coating may be an anti-reflective coating for infrared light from 890 nm to 990 nm, such as an anti-reflective coating for infrared light of 940 nm.

[0193] The ability of lens 801, lens 802, and lens 803 to sense infrared light emitted by infrared projector 101 can be improved, and an anti-reflective coating is applied to both sides of each lens (e.g., lens 801, lens 802, and lens 803) so that lens 801, lens 802, and lens 803 can receive as much of the infrared light emitted by infrared projector 101 as possible.

[0194] A cut-off coating can be used to remove non-infrared and non-visible light emitted by the infrared projector 101 and increase the transmittance of infrared light.

[0195] For example, a cut-off coating can be used to remove infrared light having a wavelength of 850 nm. It can be understood that infrared light having a wavelength of 850 nm has obvious red light characteristics, so that a severe red exposure phenomenon may occur. Therefore, a cut-off coating can be used to remove infrared light having a wavelength of 850 nm, thereby reducing the possibility of the occurrence of the red exposure phenomenon. The red exposure phenomenon is a problem of unclean light filtering. For example, it may be assumed that only infrared light (i.e., infrared light) is used for illumination, and therefore a filter can be added to the light source to remove light other than infrared light. In this case, a small amount of infrared light may still be visible to the human eye due to unclean light filtering. This phenomenon is called the red exposure phenomenon.

[0196] In this embodiment of the present application, the general RGB camera to be improved includes, but is not limited to, the above-mentioned rolling shutter camera. The reason for improving the rolling shutter camera to obtain the above-mentioned first camera is that the exposure of the rolling shutter camera is performed line by line, and the cost is relatively low. In this embodiment of the present application, the rolling shutter camera is improved to obtain the first camera 102 and the second camera 103, and thus the cost can be further reduced.

[0197] S203. The electronic device 100 calculates depth information of the target object based on the first image information, the second image information, the first length, and the lens focal length.

[0198] The above formula

[0199]

number

[0200] From this, it can be understood that the depth Z of the point where each feature of the target object is located is inversely proportional to the disparity d between the first camera 102 and the second camera 103 for that point, the depth Z of the point where each feature is located is directly proportional to the lens focal length f, and the depth Z of the point where each feature is located is directly proportional to the first length T.

[0201] The first length T and the lens focal length f are hardware parameters of the first camera 102 and the second camera 103. The first length T and the lens focal length f are fixed. In this case, the value of the depth Z of each point of the target object depends on the value of the disparity d between the first camera 102 and the second camera 103 for that point.

[0202] For example, as shown in FIG. 10A(a), the first camera 102 is a left dual path camera and the second camera 1 0 3 is assumed to be the right dual-path camera. The smile, triangle, cylinder, and crescent represent different features of the target object separately. The distances of the smile, triangle, cylinder, and crescent positions to the camera gradually increase. As shown in Fig. 10A(b), O L is the position of the first camera 102, and O R is the position of the second camera 103, and O L and O R The distance between is T (i.e., the first length).

[0203] As shown in FIG. 10A(b), the first camera 102 collects an image A, and the second camera 103 collects an image B. l The coordinate of the crescent moon on the x-axis of the coordinate system (called coordinate system L for short) with the origin at the upper left corner of image A is x L1 The coordinate of the cylinder on the x-axis of the coordinate system L is x L2 The coordinates of the triangle on the x-axis of the coordinate system L are x L3 The coordinate of the smile on the x-axis of the coordinate system L is x L4 It is. r The coordinate of the crescent moon on the x-axis of the coordinate system (called coordinate system R for short) with the origin at the upper left corner of image B is xR1 The coordinate of the cylinder on the x-axis of the coordinate system R is x R2 The coordinates of the triangle on the x-axis of the coordinate system R are x R3 and the coordinate of the smile on the x-axis of the coordinate system R is x R4 It is.

[0204] The parallax between the first camera 102 and the second camera 103 for the crescent moon is d1, where d1=x L1 -x R1 The parallax between the first camera 102 and the second camera 103 about the cylinder is d2, where d2=x L2 -x R2 The parallax between the first camera 102 and the second camera 103 for the triangle is d3, where d3=x L3 -x R3 The disparity between the first camera 102 and the second camera 103 for a smile is d4, where d4=x L4 -x R4 It is.

[0205] In this embodiment of the present application, the smile shown in FIG. 10A(b) is captured by the disparity d4 (d4=x L4 -x R4 ) is used as an example in this specification to illustrate the same.

[0206] As shown in FIG. 10B, the coordinates of the smile captured by the first camera 102 in the coordinate system of image A are (X L4 , y), and the coordinates of the smile captured by the second camera 103 in the coordinate system of image B are (x R4 , y). As shown in FIG. 10B, the coordinates on the x-axis (X L4 ,y) and coordinates (x R4 ,y) is d4, and d4=x L4 -x R4 That is, the disparity d4 between the first camera 102 and the second camera 103 for a smile is x L4 -x R4 is equal to.

[0207] d1 < d2 < d3 < d4. The depth Z of each feature of the target object (i.e., the location where the feature is located) is inversely proportional to the parallax d between the first camera 102 and the second camera 103 for that location. Therefore, a schematic diagram of the relationship between the parallax d and the depth Z can be obtained as shown in FIG. 10C. As shown in FIG. 10C, as the parallax d gradually increases, the depth Z gradually decreases.

[0208] In summary, the electronic device 100 first calculates the parallax d between the first camera 102 and the second camera 103 for the features of the target object, and then calculates the depth Z of the location where the feature is located based on the parallax d of the feature, the first length T, and the lens focal length f. After that, the depth information of the target object can be obtained based on the depths of multiple locations. Specifically, as shown in FIG. 9, S203 shown in FIG. 2 can include S203a and S203b.

[0209] S203a. The electronic device 100 calculates the parallax between the first camera 102 and the second camera 103 for a plurality of first features in the first image information and the second image information, and the first features are the same features in the first image information and the second image information.

[0210] When the infrared light having the light spot emitted by the infrared projector 101 irradiates the target object, the amount of texture features of the target object can be increased. For example, assume that the target object is the human face 1101 shown in FIG. 11(a), and the infrared light 1102 shown in FIG. 11(b) emitted by the infrared projector 101 includes the light spots shown in FIG. 7C. After the infrared light 1102 emitted by the infrared projector 101 irradiates the human face 1101, as shown in FIG. 11(c), the human face 1103 irradiated with the light spots can be obtained. Compared with the human face 1101, the human face 1103 irradiated with the light spots has more feature textures.

[0211] The difference is that the points where the same feature is located in the image information of the human face 1103 collected by the first camera 102 (i.e., the first image information) and the image information of the human face 1103 collected by the second camera 103 (i.e., the second image information) are at different positions on the x-axis of the coordinate system, i.e., the first camera 102 and the second camera 103 have parallax.

[0212] The electronic device 100 can identify the first image information collected by the first camera 102 and the second image information collected by the second camera 103, and determine the same features in the first image information and the second image information. The same features in the first image information and the second image information can include features of the target object itself and texture features formed when infrared light having light points is irradiated onto the target object. Specifically, in this embodiment of the present application, when identifying the same features in the first image information and the second image information, the electronic device 100 can not only identify the same features of the target object in the first image information and the second image information, but also identify the same features from among the texture features formed when infrared light having light points is irradiated onto the target object in the first image information and the second image information. That is, when identifying the same features in the two image information, the electronic device 100 may make a determination based on the features of the target object or the texture features, or may make a determination by combining the two features. For example, if a determination can be made based only on the target object features or only on the texture features, the two features do not need to be combined, but if it is difficult or impossible to determine whether a feature is the same feature based on the target object features, it can be determined whether a feature is the same feature based on both the texture features and the target object features.

[0213] For example, when the visible light is relatively strong during the daytime, the texture features formed when the infrared light having a light spot is irradiated on the target object are almost invisible in the image information collected by the first camera 102 and the second camera 103. However, because the visible light is relatively strong, when the visible light is irradiated on the target object, the features of the target object are relatively obvious. In this case, the electronic device 100 can perform a judgment based on the features of the target object when identifying the same features in the two image information.

[0214] It should be noted that the speckles in the above-mentioned speckle grid may have the same shape (e.g., the speckle grid includes a plurality of dots), but the positions of the speckles in the speckle grid are different. Thus, the electronic device 100 can identify the same features represented by speckles of the same shape in the first image information and the second image information based on the speckle grid and the positions of the speckles in the speckle grid.

[0215] In some embodiments, the plurality of first features includes all identical features in the first image information and the second image information. The electronic device 100 can identify all identical features in the first image information and the second image information, and then perform S203b for each feature to calculate the depth of the feature and obtain depth information of the target object.

[0216] In some other embodiments, the multiple first features are some of the same features in the first image information and the second image information. In this embodiment, the electronic device 100 selects the multiple first features from the first image information based on a preset feature frequency, then searches the second image information for features identical to the multiple first features, and finally performs S203b for each of the first features to calculate the depth of the first feature and obtain the depth information of the target object. Alternatively, the electronic device 100 may select some first features from the first image information randomly or at a certain interval.

[0217] picture The feature frequency reflected in the image is a function of the distance between two adjacent first features selected by the electronic device 100 (called the feature distance). Based on The method for selecting, by the electronic device 100, a plurality of first features from the first image information based on a preset feature frequency may include the electronic device 100 selecting one first feature from all features in the first image information that are spaced apart by one feature distance.

[0218] In other words, the electronic device 100 does not need to calculate the depth of the point where each of the identical features in the first image information and the second image information is located, but only needs to select one feature that is one feature distance apart and calculate the depth of the point where the selected feature is located.

[0219] For example, in this embodiment of the present application, the above-mentioned periodic features are described herein using as examples texture features in the first image information and the second image information formed when infrared light having light spots is irradiated onto a target object.

[0220] As shown in Fig. 12, the feature distance may be the distance between speckle 1201 and speckle 1202, or the distance between speckle 1202 and speckle 1204, or the distance between speckle 1204 and speckle 1203, or the distance between speckle 1201 and speckle 1203. As shown in Fig. 12, in this embodiment of the present application, the manner of marking the speckles in black is used to indicate some first features of the texture features formed when infrared light having a light point is irradiated onto the target object. In other words, the speckles marked in black shown in Fig. 12 are some first features.

[0221] Some speckles in the speckle grid may have the same shape (e.g., the speckle grid includes multiple dots). Although the electronic device 100 can distinguish different speckles based on the positions of the speckles in the speckle grid, it takes a relatively long time for the electronic device 100 to distinguish different speckles based on the positions of the speckles in the speckle grid, which wastes the power consumption of the electronic device 100. The feature distance used by the electronic device 100 in selecting the first feature may be equal to or less than the repetition interval of the speckle grid in the multiple light points. In other words, the feature frequency is equal to or greater than the repetition frequency of the speckle grid in the multiple light points. In this way, it can be guaranteed as much as possible that two adjacent first features selected from the first image information by the electronic device 100 correspond to speckles of different speckle grids, so that the electronic device 100 can distinguish between the two adjacent first features. This can reduce the possibility of feature matching errors and improve the accuracy of the depth information calculated by the electronic device 100.

[0222] S203b. For each first feature, the electronic device 100 uses Equation (2) to calculate the depth of the point where the first feature is located based on the parallax d between the first camera 102 and the second camera 103 for the first feature, the first length T, and the lens focal length f to obtain depth information of the target object.

[0223] Equation (2) is

[0224]

number

[0225] It is.

[0226] For example, it is assumed that the target object is a human face 1301 shown in Fig. 13. The electronic device 100 can calculate the depth of multiple points shown in Fig. 13 to obtain depth information 1302 of the target object 1301.

[0227] An embodiment of the present application provides a method for acquiring depth information. The electronic device 100 can emit infrared light having light points by using an infrared projector 101. In this way, the image of the target object collected by the first camera 102 and the second camera 103 can not only include the features of the target object, but also include texture features formed when the infrared light having light points is irradiated onto the target object. That is, the amount of features in the image of the target object collected by the first camera 102 and the second camera 103 can be increased. After the amount of features in the image of the target object collected by the first camera 102 and the second camera 103 is increased, the electronic device 100 can more accurately identify the same feature in the image information collected by the first camera 102 and the image information collected by the second camera 103, further determine the parallax between the first camera 102 and the second camera 103 for this same feature, and calculate the depth of the point where each feature is located to acquire the depth information of the target object. This can improve the accuracy of the depth information of the target object calculated by the electronic device 100.

[0228] In some embodiments, the electronic device 100 includes a first camera 102 and a second camera 103. The first camera 102 and the second camera 103 can sense visible light. The first camera 102 or the second camera 103 can function as a primary camera. The electronic device 100 can display image information collected by the primary camera under visible light on a display screen of the electronic device 100. Furthermore, the first camera 102 and the second camera 103 can further sense infrared light to cooperate with the infrared projector 101 to enable the electronic device 100 to calculate depth information of a target object.

[0229] However, because the first camera 102 and the second camera 103 can sense infrared light, the image information of the target object collected by the electronic device 100 may be slightly red. In this case, the image information collected by the main camera (the first camera 102 or the second camera 103) and displayed on the display screen of the electronic device 100 may be slightly red, which may affect the user's visual experience.

[0230] In order to avoid the problem that the image information displayed on the display screen of the electronic device 100 is slightly red, in some other embodiments, the electronic device 100 can not only include the first camera 102 and the second camera 103, but also further include a third camera 104. The third camera 104 is an RGB camera (i.e., a general RGB camera). The third camera 104 is configured to collect image information under visible light. The third camera 104 may be disposed between the first camera 102 and the second camera 103, or may be disposed in other positions.

[0231] In this embodiment, the first camera 102 and the second camera 103 can not only sense visible light, but also sense infrared light, that is, the first camera 102 and the second camera 103 can not only collect image information under infrared light, but also be configured to collect image information under visible light. However, the image information collected by the first camera 102 and the second camera 103 is only used to calculate the depth information of the target object. The third camera 104 is configured to collect image information under visible light. The image information collected by the third camera 104 is used for display on the display screen of the electronic device 100. This can avoid the problem that the image information displayed on the display screen of the electronic device 100 is slightly red, and can ensure the user's visual experience when taking an image.

[0232] It should be noted that the above embodiment is described using an example in which the lens focal length of the first camera is the same as the lens focal length of the second camera. However, in another embodiment, the lens focal length of the first camera may alternatively be different from the lens focal length of the second camera. When the lens focal lengths of the two cameras are different, the above formula for calculating depth Z is modified for calculating depth. The specific calculation formula may be a formula in the prior art.

[0233] For example, in a scenario such as a face-based unlocking scenario or a face-based payment scenario, when a two-dimensional human face image used for face recognition is input to the electronic device 100, the image information collected by the third camera 104 can be displayed on the display screen of the electronic device 100. The image information collected by the first camera 102 and the second camera 103 is used to calculate the depth information of the target object when the electronic device 100 performs face recognition. Specifically, the face recognition process of the electronic device 100 can include recognition of two-dimensional human face information and human face depth authentication. The recognition of two-dimensional human face information means that the electronic device 100 determines whether the two-dimensional image of the target object collected by the electronic device 100 matches the two-dimensional human face image stored in the electronic device 100. The human face depth authentication means that the electronic device 100 determines whether the depth information of the target object has the depth features of a real human face. If the depth information of the target object has the depth characteristics of a real human face, it indicates that the target object is a real human face, and if the depth information of the target object does not have the depth characteristics of a real human face, it indicates that the target object is not a real human face. For example, the target object may be a photo including a two-dimensional image that matches a two-dimensional human face image pre-stored in the electronic device. If the depth information of the target object does not have the depth characteristics of a real human face, even if the two-dimensional image of the target object matches the two-dimensional human face image stored in the electronic device 100, the facial recognition of the target object still fails. That is, the facial recognition performed on the electronic device by using the photo fails. This can prevent the leakage of information of the electronic device 100 or the loss of the user's property, protect the security of the information of the electronic device 100, and improve the security performance of the electronic device 100.

[0234] In AR and 3D modeling scenarios, the electronic device 100 can combine depth information of the target object calculated based on image information collected by the first camera 102 and the second camera 103 with image information of the target object collected by the third camera 104 to build a realistic human face model of the target object to improve the realism of the AR and 3D modeling scenarios.

[0235] In a large aperture scenario, the electronic device 100 can perform the above-mentioned method to calculate the depth of each point in the image collected by the electronic device 100, and then perform a blurring process on the points with depths greater than a preset value and emphasize the points with depths less than the preset value to achieve the effect of a large aperture.

[0236] For example, when the electronic device 100 collects an image that includes a human face, the depth of each point on the human face is smaller than the depth of each point on the background (the background is behind the human face and is further away from the camera). Thus, in a large aperture scenario, the human face image can be enhanced and the background image can be blurred.

[0237] For example, in the large aperture scenario, the first operation may include a user's operation of enabling the "camera" application and a user's operation of selecting the "large aperture" mode in the "camera" application. When the electronic device 100 receives the first operation, it indicates that the electronic device 100 needs to take an image by using the large aperture scenario (i.e., execute the first event). In response to the first event, the electronic device 100 executes the above-mentioned method for acquiring depth information to acquire depth information of the target object, and then can realize the effect of the large aperture based on the depth information of the target object. The image collected by the third camera 104 is displayed on the display screen of the electronic device 100. The electronic device 100 can perform large aperture processing on both the displayed captured preview photo and the final captured image based on the acquired depth information, i.e., the nearby target object can be highlighted in both the preview photo and the captured image, and other objects are blurred.

[0238] For example, the user's operation of enabling the "camera" application may be a single touch operation performed by the user on the icon of the "camera" application. In response to the user's operation of enabling the "camera" application, the electronic device 100 can display a photo-taking interface 1401 shown in FIG. 14. The photo-taking interface 1401 includes a viewfinder frame 1402, a camera switching key 1404, a capture key 1403, and an album key 1405. The viewfinder frame 1402 is configured to display a preview image captured by the rear camera or the front camera of the electronic device 100. The camera switching key 1404 is configured to trigger the electronic device 100 to switch between the front camera and the rear camera for capturing images. The capture key 1403 is configured to control the electronic device 100 to save the preview image displayed in the viewfinder frame 1402 captured by the rear camera or the front camera. The album key 1405 is configured to view images stored in the electronic device 100. If the default camera of the electronic device 100 is the rear camera, a preview image captured by the rear camera can be displayed in the viewfinder pane 1402, as shown in Figure 14. The preview image is captured by the third camera 104.

[0239] The photographing interface 1401 may further include a selection of photographing modes of the electronic device 100. The photographing modes of the electronic device 100 may include a rear camera photographing mode, a selfie mode, a video mode, a large aperture mode, a panoramic mode, and the like. For example, as shown in FIG. 14, the photographing interface 1401 may further include a "large aperture" selection 1406, a "photo" selection, a "video" selection, and the like. Of course, in this embodiment of the present application, the photographing modes include, but are not limited to, a rear camera photographing mode, a selfie mode, a panoramic mode, a large aperture mode, and a video mode. Other photographing modes are not described in detail here in this embodiment of the present application.

[0240] The above-mentioned operation of the user selecting the “large aperture” mode in the “camera” application may be a touch operation (e.g., a single-touch operation, a double-touch operation, or a touch-and-hold operation) performed by the user on the “large aperture” option 1406 shown in Fig. 14. Alternatively, the above-mentioned operation of the user selecting the “large aperture” mode in the “camera” application may be a preset gesture (e.g., an L-shaped gesture) input by the user on the capture interface 1401 shown in Fig. 14, which is used to trigger the electronic device 100 to capture an image in the large aperture mode.

[0241] In the large aperture scenario, the method in this embodiment of the present application may further include: in response to the first event, the electronic device 100 collects a two-dimensional image of the target object by using the third camera 104. The electronic device 100 displays the two-dimensional image collected by the third camera 104 (referred to as the RGB image of the target object) on a display screen. The electronic device 100 identifies a human face image (referred to as the RGB image of the human face) in the two-dimensional image collected by the third camera 104. After the electronic device 100 calculates the depth information of the target object (i.e., S203), the method in this embodiment of the present application may further include: the electronic device 100 identifies the depth information of the human face (referred to as the depth image of the human face) from the depth information of the target object (referred to as the depth image of the target object). According to the RGB image of the human face and the depth image of the human face, the electronic device 100 performs mapping on the RGB image of the target object and the depth image of the target object to find a coordinate region in the RGB image of the target object that corresponds to the depth image of the human face in the depth image, i.e., find the coordinate region of the face region in the RGB image, and then blur the part of the RGB image of the target object other than the part of this coordinate region, thereby achieving the effect of blurring the background.

[0242] It can be understood that to realize the above-mentioned functions, the electronic device includes corresponding hardware configurations and / or software modules for performing these functions. In combination with the typical units, algorithms, and steps described in the embodiments disclosed herein, those skilled in the art can easily understand that the embodiments of the present application can be realized by hardware or a combination of hardware and computer software. Whether a certain function is performed by hardware or by hardware driven by computer software depends on the design constraints of a specific application and technical solution. Those skilled in the art can use various methods to realize the above-mentioned functions for each specific application, but the implementation should not be considered beyond the scope of the embodiments of the present application.

[0243] In the embodiment of the present application, the electronic device may be divided into functional modules based on the above-mentioned method example. For example, each functional module may be obtained by division based on the corresponding functions, or two or more functions may be integrated into one processing module. The integrated module may be realized in the form of hardware, or in the form of a software function module. It should be noted that in the embodiment of the present application, the division into modules is merely an example, and is merely a logical division of functions. In actual implementation, other division manners may be used.

[0244] When an integrated unit is used, Fig. 15 is a possible schematic structural diagram of an electronic device 1500 in the above embodiment. The electronic device 1500 may include a processing module 1501, a display module 1502, an infrared emission module 1503, a dual-path collection module 1504, a dual-path collection module 1505, and an RGB collection module 1506. Optionally, the electronic device 1500 may further include a communication module, including a Bluetooth module, a Wi-Fi module, etc.

[0245] The processing module 1501 is configured to control and manage the operation of the electronic device 1500. The RGB collection module 1506 is configured to collect images of the object of interest under visible light. The display module 1502 displays the images generated by the processing module 1501 and the RGB collection module 1506. 6 The electronic device 1500 is configured to display the image collected by the dual path collection module 1504. The infrared emission module 1503 is configured to emit infrared light having a light point. The dual path collection module 1504 and the dual path collection module 1505 are configured to collect an image of the target object under visible light and an image of the target object under infrared light. The communication module is configured to assist the electronic device 1500 in communication with another device. The processing module 1501 is further configured to calculate depth information of the target object based on the image collected by the dual path collection module 1504.

[0246] Specifically, the processing module 1501 may be configured to assist the electronic device 1500 in performing S201, S203, S203a, and S203b in the above-mentioned method embodiments, and / or other processes used in connection with the techniques described herein. The display module 1502 may be configured to assist the electronic device 1500 in performing the operation of "displaying an image collected by the RGB collection module 1506" in the above-mentioned method embodiments, and / or other processes used in connection with the techniques described herein. The infrared emission module 1503 may be configured to assist the electronic device 1500 in the operation of "emitting infrared light having a light point" in S202 in the above-mentioned method embodiments, and / or may be used in other processes in the techniques described herein. The dual-path collection module 1504 may be configured to support the electronic device 1500 in performing the operation of "collecting first image information" in S202 in the above-mentioned method embodiments, and / or other processes used in connection with the techniques described herein. The dual path collection module 1505 may be configured to support the electronic device 1500 in performing the "collect second image information" operation of S202 in the method embodiments described above and / or other processes used in connection with the techniques described herein. The RGB collection module 1506 may be configured to support the electronic device 1500 in collecting image information under visible light and / or may be used in other processes of the techniques described herein.

[0247] Of course, the unit modules in the electronic device 1500 include, but are not limited to, a processing module 1501, a display module 1502, an infrared emission module 1503, a dual path collection module 1504, an RGB collection module 1506, etc. For example, the electronic device 1500 may further include a storage module. The storage module is configured to store program codes and data of the electronic device 1500.

[0248] The processing module 1501 may be a processor or controller, such as a Central Processing Unit (CPU), a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or another programmable logic device, a transistor logic device, a hardware component, or any combination thereof. The processor may include an application processor and a baseband processor. The processor may implement or execute various exemplary logic blocks, modules, and circuits described in connection with the subject matter disclosed in this application. The processor may be a combination of processors that implement computational functions, such as a combination including one or more microprocessors, or a combination of a DSP and a microprocessor. The storage module may be a memory.

[0249] For example, the processing module 1501 is one or more processors (e.g., the processor 110 shown in FIG. 1D ), and the communication module includes a wireless communication module (e.g., the wireless communication module 160 shown in FIG. 1D , where the wireless communication module 160 includes a BT (i.e., a Bluetooth module) and a WLAN (e.g., a Wi-Fi module)). The wireless communication module may be referred to as a communication interface. The storage module may be a memory (e.g., the internal memory 121 shown in FIG. 1D ). The display module 1502 may be a display screen (e.g., the display screen 194 shown in FIG. 1D ). The infrared emission module 1503 may be an infrared projector (e.g., the infrared projector 196 shown in FIG. 1D , i.e., the infrared projector 101 in the above-mentioned embodiment). The two dual-path collection modules 1504 and 1505may be two dual path cameras (e.g., dual path camera 193B (i.e., first camera 102 in the above embodiment) and dual path camera 193C (i.e., second camera 103 in the above embodiment) shown in FIG. 1D). RGB collection module 1506 may be one of 1 to N other cameras 193A shown in FIG. 1D. R The first and second cameras 1504 and 1506 may be a GB camera, i.e., the third camera 104 in the above embodiment. The two dual path collection modules 1504 and the RGB collection module 1506 are located on the same side of the electronic device 100, for example, the front side or the rear side. The electronic device 1500 provided in this embodiment of the present application may be the electronic device 100 shown in FIG. 1D. One or more processors, memories, infrared projectors, the first camera, the second camera, the display screen, the third camera, etc. may be connected, for example, by using a bus.

[0250] An embodiment of the present application further provides a computer storage medium, which stores a computer program code, which, when executed by a processor, causes the electronic device 1500 to perform the relevant method steps in either FIG. 2 or FIG. 9 of the accompanying drawings to realize the method of the above-mentioned embodiment.

[0251] An embodiment of the present application further provides a computer program product, which, when run on a computer, enables the computer to execute the relevant method steps in either Figure 2 or Figure 9 of the accompanying drawings to realize the method of the above-mentioned embodiment.

[0252] The electronic device 1500, the computer storage medium and the computer program product provided in the embodiment of the present application are all configured to execute the corresponding method presented in the above description. Therefore, for the beneficial effects that can be achieved by the electronic device 1500, the computer storage medium and the computer program product, please refer to the beneficial effects of the corresponding method presented in the above description. Details will not be described again here.

[0253] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned division of functional modules is only performed as an example for the purpose of convenient and concise description. In actual application, the above-mentioned functions can be allocated to various functional modules and implemented based on requirements, that is, the internal structure of the device is divided into various functional modules to implement all or part of the above-mentioned functions.

[0254] It should be understood that in some embodiments presented in this application, the disclosed apparatus and method may be implemented in other ways. For example, the above-mentioned apparatus embodiments are merely examples. For example, the division of modules or units is merely a division of logical functions, and may be other divisions in actual implementation. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not implemented. Furthermore, the shown or described mutual couplings or direct couplings or communication connections may be realized by using some interfaces. Indirect couplings or communication connections between apparatuses or units may be realized in electronic, mechanical, or other forms.

[0255] Furthermore, the functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The integrated unit may be realized in the form of hardware or a software functional unit.

[0256] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, the integrated unit may be stored in a readable storage medium. Based on such understanding, the technical solution of the present application in essence, or the contribution to the prior art, or all or part of the technical solution may be realized in the form of a software product. The software product is stored in a storage medium and includes some instructions for instructing a device (which may be a single-chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in the embodiments of the present application. The above-mentioned storage medium includes any medium that can store program code, such as a USB flash drive, a removable hard disk, a ROM, a magnetic disk, or an optical disk.

[0257] The above description is merely a specific embodiment of the present application, and does not limit the scope of protection of the present application. Any variation or replacement within the technical scope disclosed in the present application shall be encompassed by the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims. [Explanation of symbols]

[0258] 100 Electronic Devices 101 Infrared Projector 102 First Camera 103 Second Camera 104 3rd camera, RGB camera 110 Processor 120 External Memory Interface 121 Internal Memory 130 USB interface 140 Charging Management Module 141 Power Management Module 142 Battery 150 Mobile communication module 160 Wireless communication module 170 Audio Module 170A Speaker 170B Receiver 170C Microphone 170D Headset Interface 180 Sensor Module 180A PRESSURE SENSOR 180B Gyroscope Sensor 180C Barometric Pressure Sensor 180D Magnetic Sensor 180E Acceleration Sensor 180F Distance Sensor 180G Optical Proximity Sensor 180H Fingerprint Sensor 180J Temperature Sensor 180K touch sensor 180L Ambient Light Sensor 180M Bone Conduction Sensor 190 keys 191 Motor 192 Notification devices, indicators 193 Camera 193A Other Cameras 193B, 193C Dual Path Camera 194 Display screen 195 SIM card interface 196 Infrared Projector 301 Payment Interface 302 Payment Buttons, "Pay Now" Buttons 303 Face Recognition Interface 401,402,403,404,405,406,407,408,707,708,709 Speckle grating 501,503,601,603,605,701,702,703 light spot 502,504,602,604,606,704,705,706 Speckle grating group 800 RGB Camera Module 801,802,803 Lenses 804 Optical Filter 805 Sensors 1101 Human Face 1102 Infrared light 1103 Human Face 1201,1202,1203,1204 Speckle 1301 Human face, object of interest 1302 Depth information 1401 Shooting interface 1402 viewsFinder frame 1403 Shooting key 1404 Camera switching key 1405 Album Key 1406 "Large opening" option 1500 Electronic Devices 1501 Processing Module 1502 Display module 1503 Infrared Radiation Module 1504,1505 Dual Path Acquisition Module 1506 RGB Acquisition Module K1,K2,K3 points

Claims

1. A method for acquiring depth information, the method being applied to an electronic device, the electronic device comprising an infrared projector, a first camera, and a second camera, a distance between the first camera and the second camera being a first length, the first camera comprising a first optical filter and at least one first lens, both sides of the first lens comprising a first anti-reflection coating, the first optical filter comprising a first cut-off coating, the second camera comprising a second optical filter and at least one second lens, both sides of the second lens comprising a second anti-reflection coating, the second optical filter comprising a second cut-off coating, the first anti-reflection coating and the second anti-reflection coating being used to increase transmittance of infrared light, and the first cut-off coating and the second cut-off coating being used to block light other than infrared light and visible light, The method comprises: receiving a first instruction by the electronic device, receiving a first command, the first command being used to trigger the electronic device to obtain depth information of an object of interest; emitting infrared light having a light spot by using the infrared projector, collecting first image information of the target object by using the first camera, and collecting second image information of the target object by using the second camera, by the electronic device in response to the first command; The first image information and the second image information include features of the object of the target and texture features formed when the infrared light having the light point is irradiated onto the object of the target; The infrared light having the light point includes a plurality of light points, The plurality of light points includes a plurality of speckle grating groups, one speckle grating group of the plurality of speckle grating groups includes two or more speckle gratings; each of the speckle gratings includes a plurality of speckles; a first speckle grating group is any one of the plurality of speckle grating groups, and all of the plurality of speckle gratings included in the first speckle grating group are different from each other in at least one of the number of speckles, the shape of the speckle, and the shape of the speckle grating; the first speckle grating group has the same speckle grating configuration as the other speckle grating groups, and the repeat interval of the speckle gratings is the shortest distance between two identical speckle gratings; emitting infrared light and collecting first image information and second image information; Calculating, by the electronic device, a disparity between the first camera and the second camera for each of a plurality of first features in the first image information and the second image information, the plurality of first features being a plurality of speckles selected within a repeat interval of the speckle grating; the distance between the selected speckles is a preset feature distance; The feature distance is set to be smaller than a repeat interval of the speckle grating in the plurality of light points, and two adjacent first features selected from the first image information correspond to speckles of different speckle gratings. calculating a disparity; acquiring depth information of the target object, obtaining depth information of the target object by calculating, for each of the first features, the depth information of the target object based on the first image information, the second image information, the disparity d between the first camera and the second camera for the first feature, the first length T, a lens focal length f of the first camera, and a lens focal length f of the second camera; A method comprising:

2. The method of claim 1 , wherein the lens focal length of the first camera is the same as the lens focal length of the second camera.

3. The step of calculating, by the electronic device, the depth information of the target object based on the first image information, the second image information, the first length, a lens focal length of the first camera, and a lens focal length of the second camera, further comprises: For each first feature, the electronic device determines a depth Z of the point where the first feature is located using the following formula: [0010] where f is the lens focal length, d is the disparity between the first camera and the second camera for the first feature, and T is the first length. and obtaining the depth information of the target object based on the parallax between the first camera and the second camera for the first feature, the first length, and the lens focal length by using The method of claim 2 , comprising:

4. Prior to the step of calculating, by the electronic device, a disparity between the first camera and the second camera for a plurality of first features in the first image information and the second image information, the method further comprises: Selecting, by the electronic device, a plurality of first features from the first image information and the second image information. Further comprising: The method according to claim 3.

5. a first speckle grating is any one of the plurality of speckle gratings; The method of claim 1 , wherein each of the plurality of speckles in the first speckle grating has the same shape or at least two of the plurality of speckles in the first speckle grating have different shapes.

6. each of the first and second camera lenses includes an anti-reflective coating on both sides; and each of the first and second camera optical filters includes a cut-off coating; The method according to any one of claims 1 to 5, wherein the anti-reflection coating is used to increase the transmittance of infrared light, and the cut-off coating is used to remove light other than infrared light and visible light and to increase the transmittance of infrared light.

7. The method according to any one of claims 1 to 6, wherein the infrared light emitted by the infrared projector is infrared light between 890 nanometers and 990 nanometers.

8. The method according to any one of claims 1 to 7, wherein the infrared light emitted by the infrared projector is 940 nanometer infrared light.

9. The method of any one of claims 1 to 8, wherein the first length is in the range of 20 millimeters to 30 millimeters.

10. The method according to any one of claims 1 to 9, wherein the electronic device further comprises a third camera, the third camera being an RGB camera, the third camera being configured to collect image information under visible light, and the image information collected by the third camera being used for display on a display screen of the electronic device.

11. 1. An electronic device comprising: one or more processors; a memory; an infrared projector; a first camera; and a second camera; a distance between the first camera and the second camera is a first length; the first camera includes a first light filter and at least one first lens, both sides of the first lens include a first anti-reflective coating, the first light filter includes a first cut-off coating; the second camera includes a second light filter and at least one second lens, both sides of the second lens include a second anti-reflective coating, the second light filter includes a second cut-off coating, the first anti-reflective coating and the second anti-reflective coating are used to increase transmittance of infrared light, and the first cut-off coating and the second cut-off coating are used to block light other than infrared light and visible light; the memory, the infrared projector, the first camera, and the second camera are coupled to the processor; and the memory is configured to store information, The processor, configured to receive a first command, the first command being used to trigger the electronic device to obtain depth information of an object of interest; The processor, In response to the first command, emitting infrared light having a light point by using the infrared projector, collecting first image information of the target object by using the first camera, and collecting second image information of the target object by using the second camera; The first image information and the second image information include features of the object of the target and texture features formed when the infrared light having the light point is irradiated onto the object of the target; The infrared light having the light point includes a plurality of light points, The plurality of light points includes a plurality of speckle grating groups, one speckle grating group of the plurality of speckle grating groups includes two or more speckle gratings; each of the speckle gratings includes a plurality of speckles; a first speckle grating group is any one of the plurality of speckle grating groups, and all of the plurality of speckle gratings included in the first speckle grating group are different from each other in at least one of the number of speckles, the shape of the speckle, and the shape of the speckle grating; the first speckle grating group has the same speckle grating configuration as the other speckle grating groups, and the repeat interval of the speckle gratings is the shortest distance between two identical speckle gratings; emitting infrared light and collecting first image information and second image information; calculating, by the electronic device, a disparity between the first camera and the second camera for each of a plurality of first features in the first image information and the second image information; the plurality of first features being a plurality of speckles selected within a repeat interval of the speckle grating; the distance between the selected speckles is a preset feature distance; The feature distance is set to be smaller than a repeat interval of the speckle grating in the plurality of light points, and two adjacent first features selected from the first image information correspond to speckles of different speckle gratings. Calculating the parallax; and Obtaining depth information of the target object, obtaining depth information of the target object by calculating, for each of the first features, the depth information of the target object based on the first image information, the second image information, the disparity d between the first camera and the second camera for the first feature, the first length T, a lens focal length f of the first camera, and a lens focal length f of the second camera; 4. The method of claim 3, further comprising: Electronic devices.

12. The electronic device of claim 11 , wherein the lens focal length of the first camera is the same as the lens focal length of the second camera.

13. The processor is configured to calculate the depth information of the target object based on the first image information, the second image information, the first length, a lens focal length of the first camera, and a lens focal length of the second camera; The processor, For each first feature, the depth Z of the point where the first feature is located is calculated using the following formula: [0025] where f is the lens focal length, d is the disparity between the first camera and the second camera for the first feature, and T is the first length. and calculating based on the parallax between the first camera and the second camera for the first feature, the first length, and the lens focal length, to obtain the depth information of the object of interest by using The electronic device of claim 12 .

14. 14. The electronic device of claim 13, wherein the processor is further configured to select a plurality of first features from the first image information and the second image information before calculating the disparity between the first camera and the second camera for the plurality of first features in the first image information and the second image information.

15. a first speckle grating is any one of the plurality of speckle gratings; 15. The electronic device of claim 14, wherein each of the plurality of speckles in the first speckle grating has the same shape or at least two of the plurality of speckles in the first speckle grating have different shapes.

16. each of the first and second camera lenses includes an anti-reflective coating on both sides; and each of the first and second camera optical filters includes a cut-off coating; The electronic device according to any one of claims 11 to 15, wherein the anti-reflection coating is used to increase the transmittance of infrared light, and the cut-off coating is used to remove light other than infrared light and visible light and to increase the transmittance of infrared light.

17. The electronic device according to any one of claims 11 to 16, wherein the infrared light emitted by the infrared projector is infrared light between 890 nanometers and 990 nanometers.

18. The electronic device according to any one of claims 11 to 17, wherein the infrared light emitted by the infrared projector is 940 nanometer infrared light.

19. The electronic device of any one of claims 11 to 18, wherein the first length is in the range of 20 millimeters to 30 millimeters.

20. a third camera and a display screen; The electronic device of any one of claims 11 to 19, wherein the third camera is a red-green-blue RGB camera, the third camera is configured to collect image information under visible light, and the image information collected by the third camera is used for display on the display screen.

21. A computer storage medium comprising computer instructions, which when executed on an electronic device, enables the electronic device to perform the method for obtaining depth information according to any one of claims 1 to 10.

Images