Occlusion Detection Method for TOF Camera and Terminal Device

Abstract

An embodiment of the present application relates to an occlusion detection method and a terminal device for a TOF camera. The method includes: obtaining an image generated by an image sensor according to the received first reflected light; preliminarily determining whether the TOF camera is occluded according to a first frame image generated by the image sensor when the projection light source is turned on and a second frame image generated by the image sensor when the projection light source is turned off; if it is preliminarily determined that the TOF camera is occluded, obtaining a voltage generated by a photoelectric conversion component according to the received second reflected light; if the voltage is greater than a preset voltage, determining that the TOF camera is occluded. In the embodiment of the present application, the terminal device preliminarily determines whether the TOF camera is occluded according to the first frame image and the second frame image, and then further determines whether the TOF camera is occluded by comparing the voltage generated by the photoelectric conversion component with the preset voltage. This method can accurately detect whether the TOF camera is occluded.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of electronic technologies, and in particular, to an occlusion detection method for a TOF camera and a terminal device. Background Art

[0002] Currently, TOF cameras are widely used in various different fields. A TOF (Time of Flight) camera includes a transmitting end and a receiving end. When the TOF camera captures an image of a target object, the transmitting end emits detection light towards the target object, and the receiving end receives a first reflected light including the reflected light of the detection light by the target object and forms an image.

[0003] When a user uses a TOF camera, the hand often accidentally occludes the TOF camera. When the human skin is irradiated by the detection light emitted by the TOF camera for a long time at a short distance, the detection light may cause certain harm to the human skin. Therefore, when using a TOF camera, the TOF camera needs to detect whether it is occluded. When the TOF camera is occluded, the user can timely move the hand away to avoid harm to the human skin. However, the related technology TOF camera cannot accurately detect whether it is occluded. Summary of the Invention

[0004] Embodiments of the present application provide an occlusion detection method for a TOF camera and a terminal device to solve the problem that it is impossible to accurately detect whether the TOF camera is occluded.

[0005] In a first aspect, an occlusion detection method for a TOF camera is provided. The TOF camera includes a transmitting end and a receiving end. The transmitting end includes a projection light source and a photoelectric conversion component, and the projection light source is used to emit detection light towards a target object; the receiving end includes an image sensor. The method includes: obtaining an image generated by the image sensor according to the received first reflected light, where the image includes a first frame image and a second frame image; where when the TOF camera is not occluded, the first reflected light includes the reflected light of the detection light by the target object, and when the TOF camera is occluded, the first reflected light does not include the reflected light of the detection light by the target object; the first frame image is an image generated by the image sensor when the projection light source is turned on, and the second frame image is an image generated by the image sensor when the projection light source is turned off; preliminarily determining whether the TOF camera is occluded according to the first frame image and the second frame image; if it is preliminarily determined that the TOF camera is occluded, obtaining a voltage generated by the photoelectric conversion component according to the received second reflected light; where when the TOF camera is occluded, the second reflected light includes the reflected light of the detection light by the occluder; when the TOF camera is not occluded, the second reflected light does not include the reflected light of the detection light by the occluder; if the voltage is greater than a preset voltage, determining that the TOF camera is occluded, and the preset voltage is determined according to the voltage generated by the photoelectric conversion component when the preset projection light source is turned off.

[0006] In this way, the terminal device preliminarily determines whether the TOF camera is occluded based on the differences between the first frame of image and the second frame of image determined by the TOF camera itself. After the terminal device preliminarily determines that the TOF camera is occluded, in order to further accurately determine whether the TOF camera is occluded, the voltage generated by the optoelectronic conversion component is compared with a preset voltage. When the terminal device determines that the voltage is greater than the preset voltage, it indicates that the optical signal intensity of the second reflected light received by the optoelectronic conversion component is relatively high. Therefore, the terminal device determines that the TOF camera is occluded.

[0007] In one implementation, the steps of preliminarily determining whether the TOF camera is occluded based on the first frame of image and the second frame of image include: calculating the average value of the gray values of the pixel points in the first frame of image to obtain a first average value; calculating the average value of the gray values of the pixel points in the second frame of image to obtain a second average value; and preliminarily determining whether the TOF camera is occluded based on the first average value and the second average value.

[0008] In this way, the first average value is calculated based on the first frame of image generated by the image sensor when the projection light source is turned on, and the second average value is calculated based on the second frame of image generated by the image sensor when the projection light source is turned off. In this way, the terminal device can use the first average value and the second average value to determine the difference between the first frame of image and the second frame of image, and further preliminarily determine whether the TOF camera is occluded.

[0009] In one implementation, the steps of preliminarily determining whether the TOF camera is occluded based on the first average value and the second average value include: subtracting the first average value from the second average value to obtain a gray value average difference; if the gray value average difference is not greater than the average threshold, it is preliminarily determined that the TOF camera is occluded, and the average threshold is determined based on the second average value of the second frame of image; if the gray value average difference is greater than the average threshold, it is determined that the TOF camera is not occluded.

[0010] In this way, the gray value average difference can accurately describe the difference between the first frame of image and the second frame of image. When the gray value average difference is greater than the average threshold, it indicates that the difference between the first frame of image and the second frame of image is relatively large, and the difference between the optical signal intensity of the first reflected light received by the image sensor when generating the first frame of image and the optical signal intensity of the first reflected light received by the image sensor when generating the second frame of image is relatively large. The detection light of the projection light source is emitted to the target object. At this time, the terminal device can determine that the TOF camera is not occluded; if the gray value average is not greater than the average threshold, it indicates that the difference between the first frame of image and the second frame of image is not large, and the difference between the optical signal intensity of the first reflected light received by the image sensor when generating the first frame of image and the optical signal intensity of the first reflected light received by the image sensor when generating the second frame of image is not large. The detection light of the projection light source is not emitted to the target object. At this time, the terminal device can preliminarily determine that the TOF camera is occluded.

[0011] In one implementation, the steps of preliminarily determining whether the TOF camera is occluded according to the first average value and the second average value include: if the first average value is not greater than the preset average value, it is preliminarily determined that the TOF camera is occluded, and the preset average value is determined according to the second average value; if the first average value is greater than the preset average value, it is determined that the TOF camera is not occluded.

[0012] In this way, the first average value is calculated based on the first frame of image generated by the image sensor when the projection light source is turned on, and the second average value is calculated based on the second frame of image generated by the image sensor when the projection light source is turned off. The preset average value is determined according to the second average value. By comparing the first average value and the preset average value, the difference between the first frame of image and the second frame of image can be accurately determined. If the first average value is not greater than the preset average value, it means that the difference between the first frame of image and the second frame of image is not significant, and the intensity of the optical signal of the first reflected light received by the image sensor when generating the first frame of image is not much different from the intensity of the optical signal of the first reflected light received by the image sensor when generating the second frame of image. At this time, the terminal device can preliminarily determine that the TOF camera is occluded; if the first average value is greater than the preset average value, it means that the difference between the first frame of image and the second frame of image is relatively large, and the intensity of the optical signal of the first reflected light received by the image sensor when generating the first frame of image is quite different from the intensity of the optical signal of the first reflected light received by the image sensor when generating the second frame of image. At this time, the terminal device can determine that the TOF camera is not occluded.

[0013] In one implementation, the steps of preliminarily determining whether the TOF camera is occluded according to the first frame of image and the second frame of image include: determining a set of pixels to be compared according to the gray values of the pixels in the first frame of image and the second frame of image, and the set of pixels to be compared includes target pixels, where the target pixels are the pixels in the first frame of image whose gray values are not equal to the gray values of the pixels corresponding to the positions of the pixels in the first frame of image in the second frame of image; calculating the proportion of the number of pixels in the set of pixels to be compared to the number of all pixels in the first frame of image; if the proportion is not greater than the proportion threshold, it is preliminarily determined that the TOF camera is occluded; if the proportion is greater than the proportion threshold, it is determined that the TOF camera is not occluded.

[0014] In this way, by determining the proportion of the number of target pixels to the number of all pixels in the image, the terminal device can accurately describe the difference between the first frame of image and the second frame of image, and further can accurately preliminarily determine whether the TOF camera is occluded.

[0015] In one implementation, the steps of preliminarily determining whether the TOF camera is blocked based on the first frame image and the second frame image include: determining a comparison image region according to the gray values of the pixel points in the first frame image. The shape of the comparison image region is rectangular, and the comparison image region is formed by the pixel points at the uppermost, lowermost, leftmost, and rightmost positions among all the pixel points whose gray values in the first frame image are greater than the gray values of the corresponding pixel points in the second frame image at the same positions as the pixel points in the first frame image; preliminarily determining whether the TOF camera is blocked according to the gray values of the pixel points in the comparison image region of the first frame image and the gray values of the pixel points in the comparison image region of the second frame image.

[0016] In this way, the gray values of the pixel points in the image are mainly determined by the first reflected light reflected by the target object. When some parts of the target object may be relatively close to the TOF camera and some parts may be relatively far from the TOF camera, the objects that are farther away can only reflect the reflected light with a relatively weak light signal intensity of the detection light. When the image sensor receives the reflected light with a relatively weak light signal intensity, the pixel value of the corresponding pixel point in the image is generated, which may be the same as the gray value of the pixel point in the image generated by the image sensor when the projection light source is turned off. Therefore, in order to reduce the calculation amount, the terminal device determines the comparison image region, tries to exclude the pixel points corresponding to the farther parts of the target object in the first frame image and the second frame image, and then preliminarily determines whether the TOF camera is blocked according to the gray values of the pixel points in the first frame image and the second frame image within the comparison region.

[0017] In one implementation, the method further includes: if the voltage is not greater than the preset voltage, determining that the projection light source fails to emit the detection light normally. In this way, when the voltage is not greater than the preset voltage, it indicates that the light signal intensity of the second reflected light received by the photoelectric conversion component is relatively low. Therefore, the terminal device can accurately determine that the projection light source fails to emit the detection light normally.

[0018] In one implementation, the method further includes: if the TOF camera is blocked, turning off the projection light source or reducing the power of the projection light source. In this way, after the terminal device determines that the TOF camera is blocked, it can turn off the projection light source or reduce the power of the projection light source to reduce the harm of the detection light to the human skin.

[0019] In one implementation, the method further includes: if the TOF camera is blocked, displaying a first prompt window on the display screen, where the first prompt window is used to prompt the user that the TOF camera is blocked. In this way, when the user sees the prompt window on the display screen, they can move the hand that blocks the TOF camera away.

[0020] In one implementation, the method further includes: if the projection light source fails to emit detection light normally, a second prompt window is displayed on the display screen, and the second prompt window is used to prompt the user that the projection light source fails to emit detection light normally. In this way, the user can repair the damaged projection light source in time according to the prompt window.

[0021] In one implementation, the photoelectric conversion component includes a photodiode. In this way, the terminal device can use the photodiode to accurately generate a voltage according to the second reflected light.

[0022] In one implementation, the projection light source includes a vertical cavity surface emitting laser. In this way, the terminal device can emit infrared laser through the vertical cavity surface emitting laser, so that the image sensor generates an image according to the received reflected infrared laser.

[0023] In one implementation, the mean threshold is the second average value of the second frame image that is a first preset multiple, and the first preset multiple is 1.2 times. In this way, by using the second average value of the second frame image that is a first preset multiple as the mean threshold, the terminal device can more accurately preliminarily determine whether the TOF camera is blocked.

[0024] In one implementation, the preset average value is the second average value that is a second preset multiple, and the second preset multiple is 1.2 times. In this way, by using the second average value that is a second preset multiple as the preset average value, the terminal device can more accurately preliminarily determine whether the TOF camera is blocked.

[0025] In one implementation, the preset voltage is the voltage generated by the photoelectric conversion component when the preset projection light source is turned off that is a third preset multiple, and the third preset multiple is 1.2 times. In this way, by using the voltage generated by the photoelectric conversion component when the preset projection light source is turned off that is a third preset multiple as the preset voltage, the terminal device can more accurately determine whether the TOF camera is blocked.

[0026] In a second aspect, an embodiment of the present application provides a terminal device, including: a processor and a memory; the memory stores program instructions, and when the program instructions are executed by the processor, the terminal device executes the methods in the above aspects and their respective implementation manners.

[0027] In a third aspect, an embodiment of the present application further provides a chip system, which includes a processor and a memory, and the memory stores program instructions. When the program instructions are executed by the processor, the chip system executes the methods in the above aspects and their respective implementation manners. For example, generating or processing the information involved in the above methods.

[0028] Fourthly, the embodiments of the present application further provide a computer-readable storage medium. Program instructions are stored in the computer-readable storage medium. When the program instructions run on a computer, the computer is enabled to execute the methods in the above aspects and their respective implementation manners.

[0029] Fifthly, the embodiments of the present application further provide a computer program product. When it runs on a computer, the computer is enabled to execute the methods in the above aspects and their respective implementation manners. Description of the Drawings

[0030] Figure 1 It is a schematic diagram of the structure of a TOF camera provided by the embodiments of the present application and the optical propagation path in Scenario 1.

[0031] Figure 2 It is a schematic diagram of the structure of a TOF camera provided by the embodiments of the present application and the optical propagation path in Scenario 2.

[0032] Figure 3 It is a flowchart of a method for detecting occlusion of a TOF camera provided by the embodiments of the present application.

[0033] Figure 4 It is a schematic diagram of the hardware structure of a terminal device provided by the embodiments of the present application.

[0034] Figure 5 It is a flowchart of another method for detecting occlusion of a TOF camera provided by the embodiments of the present application.

[0035] Figure 6 It is a schematic diagram of the user interface of a terminal device provided by the embodiments of the present application.

[0036] Figure 7 It is a schematic diagram of the user interface of a camera application provided by the embodiments of the present application.

[0037] Figure 8 It is a schematic diagram of an image generated by an image sensor provided by the embodiments of the present application.

[0038] Figure 9 It is a flowchart of yet another method for detecting occlusion of a TOF camera provided by the embodiments of the present application.

[0039] Figure 10 It is a flowchart of yet another method for detecting occlusion of a TOF camera provided by the embodiments of the present application.

[0040] Figure 11 It is a flowchart of yet another method for detecting occlusion of a TOF camera provided by the embodiments of the present application.

[0041] Figure 12Flow chart of another occlusion detection method for the TOF camera provided by the embodiment of the present application;

[0042] Figure 13 Schematic diagram of an image A and an image B provided by the embodiment of the present application;

[0043] Figure 14 Schematic structural diagram of an occlusion detection device for a TOF camera provided by the embodiment of the present application;

[0044] Figure 15 Schematic structural diagram of another occlusion detection device for a TOF camera provided by the embodiment of the present application;

[0045] Figure 16 Schematic structural diagram of a chip system. Detailed implementation manners

[0046] Next, the technical solutions of the embodiments of the present application will be clearly described with reference to the accompanying drawings in the embodiments of the present application.

[0047] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. As used in the specification and the appended claims of the embodiments of the present application, the singular forms "a", "an", "the", "above-mentioned", "said", and "this" are also intended to include the plural forms, unless clearly indicated to the contrary in the context. It should also be understood that although the terms first, second, etc. may be used in the following embodiments to describe a certain type of object, the object should not be limited to these terms. These terms are only used to distinguish specific objects of this type of object. For example, the terms first, second, etc. may be used in the following embodiments to describe display windows, but the display windows should not be limited to these terms. These terms are only used to distinguish different windows displayed on the display screen. The same applies to other types of objects that may be described by the terms first, second, etc. in the following embodiments, and will not be repeated here. In addition, the term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone, where A and B may be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0048] In the embodiments of the present application, the directions of "up", "down", "left", and "right" are subject to the directions of "up", "down", "left", and "right" perceived by the user when the user faces the display screen of the terminal device.

[0049] When users use a TOF camera, it often happens that their hands accidentally block the TOF camera. When the human skin is irradiated by the detection light emitted by the TOF camera at a close distance for a long time, the detection light may cause certain damage to the human skin. Therefore, when using the TOF camera, the TOF camera needs to detect whether it is blocked. When the TOF camera is blocked, the user can move their hand away in time to avoid damaging the human skin. However, in related technologies, the TOF camera cannot accurately detect whether it is blocked.

[0050] To more clearly describe the occlusion detection method of the TOF camera in the embodiments of the present application, the structure of the TOF camera and the light propagation path in different scenarios will be described in detail below.

[0051] Figure 1 FIG. is a schematic diagram of the structure of a TOF camera and the light propagation path in Scenario 1 provided by an embodiment of the present application.

[0052] As Figure 1 shown, Scenario 1 is a scenario where the TOF camera is not blocked by an occluder and the projection light source 11 emits detection light 111.

[0053] The TOF camera includes a transmitting end (TX) 10 and a receiving end (RX) 20. The transmitting end 10 includes a projection light source 11 and a photoelectric conversion component 12; the receiving end 20 includes an image sensor 21.

[0054] The projection light source 11 is used to emit detection light 111 to the target object 30. Exemplarily, the projection light source 11 can be a vertical cavity surface emitting laser (VCSEL), and the detection light 111 emitted by the vertical cavity surface emitting laser can be an infrared laser.

[0055] The photoelectric conversion component 12 is a component that converts an optical signal into an electrical signal through photoelectric conversion. The photoelectric conversion component 12 is used to receive the second reflected light 121 and generate a voltage according to the second reflected light 121.

[0056] The second reflected light may include the light reflected by the target object 30 from the detection light 111. Due to the limitations of the photoelectric conversion component 12 itself, when the photoelectric conversion component 12 is far from the target object 30, the photoelectric conversion component can only receive a small amount of reflected light. The reflected light of the target object 30 reflected from the detection light 111 received by the photoelectric conversion component has little effect on generating a voltage for the photoelectric conversion component and can be ignored in the embodiments of the present application.

[0057] In some embodiments, the photoelectric conversion component 12 can be a photodiode (PD).

[0058] In some embodiments, the transmitting end 10 further includes a diffuser 13, which can provide a uniform surface light source. When the transmitting end 10 further includes the diffuser 13, the second reflected light 121 further includes the reflected light of the diffuser 13 reflecting the detection light 111.

[0059] The image sensor 21 is configured to receive the first reflected light 211 and generate an image based on the first reflected light 211. The first reflected light 211 includes the reflected light of the target object 30 reflecting the detection light 111. In some embodiments, since the target object 30 may also emit optical signals, the first reflected light 211 received by the image sensor 21 further includes the optical signals emitted by the target object 30 itself.

[0060] In some embodiments, the image sensor 21 may be a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor.

[0061] In some embodiments, the receiving end 20 further includes a lens 22, and the first reflected light 211 received by the image sensor 21 is the reflected light passing through the lens.

[0062] In some embodiments, since the target object 30 may also emit optical signals, the first reflected light 211 received by the image sensor 21 may further include the optical signals emitted by the target object 30 itself. The reflected light of the target object 30 reflecting the detection light 111 and the optical signals emitted by the target object 30 itself are superimposed to obtain the first reflected light 211, and the image sensor 21 generates an image based on the first reflected light 211.

[0063] Figure 2 Schematic diagrams of the structure of a TOF camera and the optical propagation path in Scenario 2 provided by the embodiments of the present application.

[0064] As Figure 2 shown, Scenario 2 is a scenario where the TOF camera is blocked by an occluder 40 and the projection light source 11 emits the detection light 111.

[0065] When the TOF camera is blocked by the occluder 40, the detection light 111 emitted by the projection light source 11 cannot reach the target object 30. At this time, the first reflected light 211 does not include the light reflected by the target object 30 from the detection light 111. If the target object 30 also emits optical signals, the first reflected light 211 received by the image sensor 21 further includes the optical signals emitted by the target object 30 itself.

[0066] When the TOF camera is blocked by the blocker 40, the detection light 111 emitted by the projection light source 11 is reflected by the blocker. At this time, the second reflected light 211 includes the reflected light of the blocker 40 reflecting the detection light 111. If the transmitting end 10 further includes a diffusion sheet 13, the second reflected light 211 further includes the reflected light of the diffusion sheet 13 reflecting the detection light 111.

[0067] Figure 3 The flowchart of a method for detecting occlusion of a TOF camera provided by an embodiment of the present application, the method includes steps S11 - S14:

[0068] Step S11, the terminal device responds to the operation of taking a photo, turns on the projection light source 11; and obtains the first voltage generated by the photoelectric conversion component 12.

[0069] In some embodiments, the operation of taking a photo can be that the user clicks the photo - taking button displayed on the terminal device. How the terminal device displays the photo - taking button will be introduced in detail below and will not be elaborated here.

[0070] In the embodiment of the present application, the terminal device responds to the operation of taking a photo, turns on the projection light source 11, so that the projection light source 11 emits the detection light 111. When the projection light source 11 emits the detection light 111, the photoelectric conversion component 12 can receive the second reflected light 121.

[0071] Step S12, compare the first voltage with the first preset voltage.

[0072] In one implementation, the terminal device can define a voltage set, which includes the voltages generated by the photoelectric conversion component when the terminal device takes photos at different exposure times. Exemplarily, the voltage set includes exposure time A, the voltage corresponding to exposure time A; exposure time B, the voltage corresponding to exposure time B; exposure time C, the voltage corresponding to exposure time C. The voltage set can include one or more exposure times and the voltages corresponding to the exposure times.

[0073] The voltage set can be pre - set by the terminal device before leaving the factory, or obtained by the terminal device from the manufacturer side. Of course, it can also be obtained by other means, and the embodiments of the present application do not limit this.

[0074] In an example, the specific determination process of the voltage set is as follows:

[0075] Record the voltages generated by the photoelectric conversion component when different terminal devices take pictures at different exposure times when the TOF camera is not blocked. Combine the exposure times and the corresponding voltages of different terminal devices into total data. For example, the total data includes the voltages generated by the photoelectric conversion component of terminal device A at exposure times such as 100 μs, 200 μs, and 300 μs. The voltages generated by the photoelectric conversion component of terminal device B at exposure times such as 100 μs, 200 μs, and 300 μs. Next, extract a voltage corresponding to a different exposure time from the total data. For example, when the exposure time is 100 μs, if there are voltages a and b corresponding to the exposure time of 100 μs in the total data, at this time, extract one voltage from voltage a and voltage b. Finally, form a voltage set with the voltages corresponding to different exposure times extracted and the exposure times.

[0076] In this implementation manner, the terminal device responds to the operation of taking pictures and records the exposure time when taking pictures. The terminal device filters out the voltage corresponding to the recorded exposure time from the voltage set and uses the filtered voltage as the first preset voltage. Exemplarily, the exposure time when the terminal device records taking pictures is 100 μs. At this time, filter out the voltage corresponding to 100 μs from the voltage set and use this voltage as the first preset voltage.

[0077] In step S13, if the first voltage is greater than the first preset voltage, the terminal device determines that the TOF camera is blocked by an occluder. Exemplarily, the first voltage is 0.2 v and the first preset voltage is 0.1 v. At this time, it is determined that the TOF camera is blocked.

[0078] Refer to again Figure 1 and Figure 2 , if the TOF camera is not blocked by the occluder 40, the second reflected light 121 does not include the reflected light of the occluder 40 reflecting the detection light 111. If the TOF camera is blocked by the occluder 40, the second reflected light 121 includes the reflected light of the occluder 40 reflecting the detection light 111. Since the second reflected light 121 received by the photoelectric conversion component 12 when the TOF camera is blocked by the occluder 40 has a higher optical signal intensity than the second reflected light 121 received by the photoelectric conversion component 12 when the TOF camera is not blocked by the occluder 40, the photoelectric conversion component 12 generates a higher voltage when the TOF camera is blocked by the occluder 40.

[0079] In the embodiments of the present application, the first preset voltage is generated when the TOF camera in the terminal device is not blocked by an object. Therefore, when the first voltage is greater than the first preset voltage, that is, when the optical signal intensity of the second reflected light received by the photoelectric conversion component is higher than the optical signal intensity of the second reflected light received when the TOF camera is blocked, the terminal device determines that the TOF camera is blocked by an object during photographing, and the blocking object reflects more detection light to the photoelectric conversion component.

[0080] In step S14, if the first voltage is not greater than the first preset voltage, the terminal device determines that the TOF camera is not blocked by an object. Exemplarily, the first voltage is 0.05v and the first preset voltage is 0.1v. At this time, it is determined that the TOF camera is not blocked.

[0081] In the embodiments of the present application, the first voltage is not greater than the first preset voltage, that is, the optical signal intensity of the second reflected light received by the photoelectric conversion component is lower or the same as the optical signal intensity of the second reflected light received when the TOF camera is blocked. Therefore, the terminal device determines that the TOF camera is not blocked during photographing.

[0082] In the above method, the terminal device compares the first voltage generated by the photoelectric conversion component in the TOF camera with the first preset voltage to detect whether the TOF camera is blocked.

[0083] However, there are at least the following problems in detecting whether the TOF camera is blocked by the above method: First, the first preset voltage is extracted from the voltage sets determined by different terminal devices, which cannot overcome the performance differences of the photoelectric conversion components in different terminal devices and affects the accuracy of detecting whether the TOF camera is blocked; Second, the photoelectric conversion efficiency of the photoelectric conversion component in the terminal device decreases after long-term use. The usage time of the terminal device when the voltage in the voltage set is generated is not necessarily exactly the same as the usage duration of the photoelectric conversion component in the terminal device for photographing in the embodiments of the present application. Therefore, when receiving the second reflected light with a certain optical signal intensity, the first voltage generated by the photoelectric conversion component in the terminal device for photographing in the embodiments of the present application may be lower than the first preset voltage in the voltage set, which easily determines the blocked situation of the TOF camera as an unblocked situation. It can be seen that the terminal device cannot accurately determine whether the TOF camera is blocked only by using the hardware (photoelectric conversion component) on the TOF camera; Finally, in the process of determining the voltage set, it is necessary to record the voltages generated by the photoelectric conversion components when different terminal devices take pictures at different exposure times when the TOF camera is not blocked, and this process is too cumbersome.

[0084] In summary, in order for the terminal device to more accurately detect whether the TOF camera is blocked, the embodiments of the present application provide another method for detecting the occlusion of the TOF camera.

[0085] The method can be applied to a terminal device. The terminal device includes, but is not limited to, a mobile phone, a tablet computer, a personal computer, a workstation device, a large-screen device (such as a smart screen, a smart TV, etc.), a wearable device (such as a smart bracelet, a smart watch), a handheld game console, a home game console, a virtual reality device, an augmented reality device, a mixed reality device, etc., and an in-vehicle intelligent terminal, etc. The method can also be directly applied to a TOF camera, and the TOF camera directly executes the method of the embodiment of the present application.

[0086] Figure 4 It is a schematic diagram of the hardware structure of the terminal device provided by the embodiment of the present application. As Figure 4 shown, the terminal device 100 may include a processor 110, a memory 120, a universal serial bus (USB) interface 130, a radio frequency circuit 140, a mobile communication module 150, a wireless communication module 160, a camera 170, a display screen 180, a touch sensor 190, a barometric pressure sensor 210, and a key 220, etc.

[0087] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors, such as integrated in a system on a chip (SoC). A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled.

[0088] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0089] The memory 120 may be used to store computer-executable program code, and the executable program code includes instructions. The memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.). The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.). In addition, the memory 120 may include one or more storage units. For example, it may include a volatile memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), etc.; it may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, etc. The processor 110 executes various functional application programs and data processing of the terminal device 100 by running the instructions stored in the memory 120 and / or the instructions stored in the memory provided in the processor.

[0090] It should be supplemented here that the operating systems referred to in the embodiments of the present application include, but are not limited to, Android operating system, IOS operating system, iPad OS, Windows operating system, Linux operating system, MAC OS operating system, embedded systems, etc.

[0091] The wireless communication function of the terminal device 100 can be implemented by a radio frequency circuit 140, a mobile communication module 150, a wireless communication module 160, a modulation and demodulation processor, a baseband processor, etc.

[0092] The radio frequency circuit 140 may include at least one antenna 141 for transmitting and receiving electromagnetic wave signals. Each antenna in the terminal device 100 can be used to cover a single or multiple communication frequency bands. In some embodiments, the antenna can be used in combination with a tuning switch.

[0093] The mobile communication module 150 can provide solutions for wireless communication including 2G / 3G / 4G / 5G, etc. for applications on the terminal device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves by the antenna 141, filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor and convert it into electromagnetic waves through the antenna 141 for radiation. In some embodiments, at least some functional modules of the mobile communication module 150 can be provided in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 can be provided in the same device.

[0094] The modulation and demodulation processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. Subsequently, the demodulator transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs a sound signal through an audio device (including but not limited to a speaker, a receiver, etc.), or displays an image or video through the display screen 180. In some embodiments, the modulation and demodulation processor can be an independent device. In other embodiments, the modulation and demodulation processor can be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

[0095] The wireless communication module 160 may include a wireless fidelity (Wi-Fi) module, a Bluetooth (BT) module, a GNSS module, a near field communication (NFC) module, an infrared (IR) module, etc. The wireless communication module 160 may be one or more devices integrating at least one of the above modules. The wireless communication module 160 receives electromagnetic waves via the antenna 141, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 may also receive the signals to be sent from the processor 110, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through the antenna 141 for radiation.

[0096] In the embodiments of the present application, the wireless communication functions of the terminal device 100 may include, for example, global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), 5th generation mobile networks new radio (5G NR), BT, GNSS, WLAN, NFC, FM, and / or IR, etc. GNSS may include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS).

[0097] The camera 170 is used to capture static images or videos. The camera 170 includes a lens and a photosensitive element. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal and then transmits the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in standard formats such as RGB, YUV, RYYB, etc. In some embodiments, the terminal device 100 may include one or N cameras 170, where N is a positive integer greater than 1.

[0098] In the embodiments of the present application, the camera 170 may be a TOF camera.

[0099] The NPU is a neural-network (NN) computing processor. By drawing on the structure of a biological neural network, such as the transmission pattern between human brain neurons, it can quickly process the input information and can also continuously self-learn. Through the NPU, application programs such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, speech recognition, text understanding, etc.

[0100] The display screen 180 is used to display images, videos, etc. The display screen 180 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include one or N display screens 180, where N is a positive integer greater than 1.

[0101] The touch sensor 190, also known as the "touch control device". The touch sensor 190 can be disposed on the display screen 180. The touch sensor 190 and the display screen 180 form a touch screen, also known as the "touch control screen". The touch sensor 190 is used to detect touch operations acting thereon or nearby. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 180. In some other embodiments, the touch sensor 190 can also be disposed on the surface of the terminal device 100, at a different position from the display screen 180.

[0102] The barometric pressure sensor 210 is used to measure barometric pressure. In some embodiments, the terminal device 100 calculates the altitude based on the barometric pressure value measured by the barometric pressure sensor 210 to assist in positioning and navigation.

[0103] The button 220 includes a power-on button, a volume button, etc. The button 220 can be a mechanical button or a touch button. The terminal device 100 can receive button inputs and generate key signal inputs related to the user settings and function control of the terminal device 100.

[0104] It can be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the terminal device 100. In some other embodiments of the present application, the terminal device may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0105] Figure 5 It is a flowchart of another method for detecting occlusion of a TOF camera provided by the embodiments of the present application. The method includes steps S21 - S28:

[0106] S21, in response to an operation of starting the TOF camera, the terminal device triggers the projection light source to operate according to a preset rule. The preset rule includes turning on the projection light source to emit detection light and turning off the projection light source to stop emitting detection light.

[0107] In the embodiments of the present application, this solution can be applied in fields such as machine vision, unmanned driving, augmented reality (AR) / virtual reality (VR) interaction, and three-dimensional (3D) imaging. The operation of starting the TOF camera can be different operations in different fields.

[0108] In some embodiments, the display screen of the terminal device can be a touch screen or a non-touch screen. In the following with more attached drawings, taking the display screen as a touch screen as an example, the operation process of the terminal device in response to the operation of starting the TOF camera and triggering the projection light source to run according to a preset rule will be described by way of example.

[0109] Figure 6 It is a schematic diagram of a user interface of a terminal device provided by an embodiment of the present application.

[0110] Figure 7 It is a schematic diagram of a user interface of a camera application provided by an embodiment of the present application.

[0111] As Figure 6 and Figure 7 shown, Figure 6 The user interface 61 in shows multiple applications (App), such as file management, email, music, calculator, information, address book, phone, and camera. It should be understood that the user interface 61 may also include other more applications, which are not limited in this application.

[0112] The terminal device can respond to the user's first gesture operation 63 and start the camera application, and display the user interface 71 of the camera application on the display screen. The user interface 71 includes a shooting button 72. Among them, the first gesture operation 63 can be that the user clicks the selection button 62 of the camera application. When the terminal device detects that the user clicks the selection button 62 of the camera application, it can start the camera application.

[0113] The terminal device can respond to the user's second gesture operation 73 and trigger the projection light source to run according to a preset rule. Among them, the second gesture operation 73 can be that the user clicks the shooting button 72. When the terminal device detects that the user clicks the shooting button 72, it triggers the projection light source to run according to a preset rule. In some embodiments, the detection light emitted by the projection light source is a light pulse, and the image sensor generates an image according to the first reflected light of the received target object reflecting a part of the light pulses. Exemplarily, this part of the light pulses can be light pulses of 1 / 4 cycle. The image sensor generates an image according to the first reflected light after the received target object reflects the light pulses of 1 / 4 cycle.

[0114] Figure 8 It is a schematic diagram of an image sensor generating an image provided by an embodiment of the present application.

[0115] As Figure 8 shown, Figure 8The image therein includes six frames of different images, specifically including phase images a - f arranged from top to bottom. Phase image a is an image generated by the image sensor based on the first reflected light of the target object reflecting a 1 / 4 - cycle light pulse. Phase image b is an image generated by the image sensor based on the first reflected light of the target object reflecting a 1 / 4 - cycle light pulse located after the 1 / 4 - cycle light pulse corresponding to phase image a. Phase image c is an image generated by the image sensor based on the first reflected light of the target object reflecting a 1 / 4 - cycle light pulse located after the 1 / 4 - cycle light pulse corresponding to phase image b. Phase image d is an image generated by the image sensor based on the first reflected light of the target object reflecting a 1 / 4 - cycle light pulse located after the 1 / 4 - cycle light pulse corresponding to phase image c. Image e is an image generated by the image sensor when the projection light source does not emit detection light. Phase f is an image generated by the image sensor based on the first reflected light of the target object reflecting a 1 / 4 - cycle light pulse. The start point of the 1 / 4 - cycle light pulse corresponding to phase f is separated from the start point of the 1 / 4 - cycle light pulse corresponding to phase d by 1 / 2 - cycle light pulse.

[0116] Of course, when the terminal device in the embodiment of the present application controls the operation of the projection light source, it can also follow other preset rules, but it is necessary to ensure that the operation of the projection light source includes the processes of turning on the projection light source and turning off the projection light source.

[0117] S22, the terminal device acquires the images generated by the image sensor according to the received first reflected light. The images include a first - frame image and a second - frame image. Among them, the first - frame image is the image generated by the image sensor when the projection light source is turned on, and the second - frame image is the image generated by the image sensor when the projection light source is turned off.

[0118] As Figure 8 shown, the first - frame image can be any one of phase images a - d and phase image f. The second - frame image can be phase image e. In some embodiments, Figure 8 the images therein are stored as raw data (image files). The terminal device can acquire the first - frame image and the second - frame image from the raw data storing phase images a - f.

[0119] In some embodiments, there is noise in the image sensor itself. Exemplarily, it includes fixed - pattern noise (FPN) and dark - current noise, etc. When the image sensor generates an image, the noise of the image sensor itself affects the image. The image sensor receives the light signal emitted by the target object itself and the noise, and generates the second - frame image.

[0120] S23, the terminal device preliminarily determines whether the TOF camera is blocked according to the first - frame image and the second - frame image.

[0121] In the embodiments of the present application, the first frame of image is the image generated by the image sensor when the projection light source is turned on, and the second frame of image is the image generated by the image sensor when the projection light source is turned off. The terminal device preliminarily determines whether the TOF camera is blocked by the difference between the first frame of image and the second frame of image determined by the TOF camera itself. If the difference is large, the terminal device determines that the detection light of the projection light source has been emitted to the target object, and at this time the TOF camera is not blocked. If the difference is small, the terminal device determines that the detection light of the projection light source has not been emitted to the target object, and preliminarily determines that the TOF camera is blocked.

[0122] In fact, if the detection light of the projection light source does not reach the target object, it may be caused by the following two situations: The first situation is that the TOF camera is blocked by an occluder; the second situation is that the TOF camera is not blocked, but the projection light source fails to emit the projection light normally. There may be problems such as damage to the projection light source, resulting in the inability to emit the transmitted light normally.

[0123] In the embodiments of the present application, there are multiple specific implementation manners for step S23.

[0124] Figure 9 It is a flowchart of another method for detecting occlusion of a TOF camera provided by the embodiments of the present application. As Figure 9 shown, in some embodiments, step S23 may be specifically implemented through the following steps:

[0125] S231, the terminal device calculates the average value of the gray values of the pixel points in the first frame of image to obtain the first average value. Exemplarily, the width of the first frame of image is n pixel points, and the height is m pixel points. The first average value is obtained by adding the gray values of all pixel points in the first frame of image and dividing by the number of pixel points (n*m) in the first frame of image, and finally obtaining the first average value.

[0126] S232, the terminal device calculates the average value of the gray values of the pixel points in the second frame of image to obtain the second average value. In the embodiments of the present application, the calculation process of the second average value is the same as that of the first average value, and will not be described in detail here.

[0127] In some embodiments, the number of pixel points in the first frame of image and the second frame of image is the same. Therefore, when calculating the second average value, the terminal device can directly add the gray values of all pixel points in the second frame of image and divide by the number of pixel points in the first frame of image to obtain the second average value.

[0128] In the embodiments of the present application, when the terminal device calculates the first average value and the second average value, there is no fixed execution order. It can calculate the first average value first and then the second average value, or calculate the second average value first and then the first average value. When calculating the second average value first and then the first average value, and the number of pixel points in the first frame image and the second frame image is the same, the terminal device can directly add up the gray values of all pixel points in the first frame image and divide by the number of pixel points in the second frame image to obtain the first average value.

[0129] S233. The terminal device preliminarily determines whether the TOF camera is blocked according to the first average value and the second average value.

[0130] In the embodiments of the present application, the first average value is calculated according to the first frame image generated by the image sensor when the projection light source is turned on, and the second average value is calculated according to the second frame image generated by the image sensor when the projection light source is turned off. In this way, the terminal device can use the first average value and the second average value to determine the difference between the first frame image and the second frame image, and then preliminarily determine whether the TOF camera is blocked.

[0131] Figure 10 It is a flowchart of another method for detecting occlusion of a TOF camera provided by the embodiments of the present application. As Figure 10 shown, in a possible implementation, step S233 can be specifically implemented through the following steps:

[0132] S2331. The terminal device subtracts the second average value from the first average value to obtain the gray mean difference.

[0133] S2332. Compare the gray mean difference with the mean threshold. In the embodiments of the present application, the mean threshold is determined according to the second average value of the second frame image. The second frame image is the image generated by the image sensor when the projection light source is turned off, and the second average value of the second frame image can reflect the situation of the image generated when the projection light source is turned off. The gray mean difference can accurately describe the difference between the first frame image and the second frame image.

[0134] S2333. If the gray mean difference is not greater than the mean threshold, the terminal device preliminarily determines that the TOF camera is blocked.

[0135] S2334. If the gray mean difference is greater than the mean threshold, it is determined that the TOF camera is not blocked.

[0136] When the mean difference of the grayscale values is greater than the mean threshold, it indicates that there is a significant difference between the first frame image and the second frame image. The intensity of the optical signal of the first reflected light received by the image sensor when generating the first frame image is quite different from the intensity of the optical signal of the first reflected light received by the image sensor when generating the second frame image. The detection light of the projection light source is emitted to the target object. At this time, the terminal device can determine that the TOF camera is not blocked. If the grayscale mean is not greater than the mean threshold, it means that there is not much difference between the first frame image and the second frame image. The intensity of the optical signal of the first reflected light received by the image sensor when generating the first frame image is not much different from the intensity of the optical signal of the first reflected light received by the image sensor when generating the second frame image. The detection light of the projection light source is not emitted to the target object. At this time, the terminal device can preliminarily determine that the TOF camera is blocked.

[0137] In some embodiments, the mean threshold is the second average value of the second frame image at a first preset multiple. In order to more accurately detect whether the TOF camera is blocked, the terminal device determines the mean threshold according to the second average value of the second frame image at a first preset multiple, and the first preset multiple is greater than 1. Exemplarily, the first preset multiple is 1.2 times. In the embodiments of the present application, the terminal device uses the second average value of the second frame image at a first preset multiple as the mean threshold, and can more accurately preliminarily determine whether the TOF camera is blocked.

[0138] Figure 11 This is a flowchart of another method for detecting occlusion of a TOF camera provided by the embodiments of the present application. As Figure 11 shown, in one possible implementation, step S233 can be specifically implemented through the following steps:

[0139] S2335, compare the first average value with the preset average value. The preset average value is determined according to the second average value.

[0140] S2336, if the first average value is not greater than the preset average value, the terminal device preliminarily determines that the TOF camera is blocked.

[0141] S2337, if the first average value is greater than the preset average value, the terminal device determines that the TOF camera is not blocked.

[0142] In the embodiments of the present application, the first average value is calculated based on the first frame image generated by the image sensor when the projection light source is turned on, and the second average value is calculated based on the second frame image generated by the image sensor when the projection light source is turned off. The preset average value is determined according to the second average value. By comparing the first average value with the preset average value, the difference between the first frame image and the second frame image can be accurately determined. If the first average value is not greater than the preset average value, it indicates that the difference between the first frame image and the second frame image is not significant, and the intensity of the optical signal of the first reflected light received by the image sensor when generating the first frame image is not much different from the intensity of the optical signal of the first reflected light received by the image sensor when generating the second frame image. At this time, the terminal device can preliminarily determine that the TOF camera is blocked; if the first average value is greater than the preset average value, it indicates that the difference between the first frame image and the second frame image is relatively large, and the intensity of the optical signal of the first reflected light received by the image sensor when generating the first frame image is quite different from the intensity of the optical signal of the first reflected light received by the image sensor when generating the second frame image. At this time, the terminal device can determine that the TOF camera is not blocked.

[0143] In some embodiments, the preset average value is the second average value multiplied by a second preset multiple. In order to more accurately detect whether the TOF camera is blocked, the terminal device determines the preset average value according to the second average value multiplied by the second preset multiple, and the second preset multiple is greater than 1. Exemplarily, the second preset multiple is 1.2 times. In the embodiments of the present application, by using the second average value multiplied by the second preset multiple as the preset average value, the terminal device can more accurately preliminarily determine whether the TOF camera is blocked.

[0144] Figure 12 It is a flowchart of another method for detecting occlusion of a TOF camera provided by the embodiments of the present application. As Figure 12 shown, in some embodiments, step S23 can be specifically implemented through the following steps:

[0145] S234, the terminal device determines a set of pixels to be compared according to the gray values of the pixels in the first frame image and the second frame image. The set of pixels to be compared includes target pixels, and the target pixels are the pixels in the first frame image whose gray values are not equal to the gray values of the pixels corresponding to the positions of the pixels in the first frame image in the second frame image.

[0146] In some embodiments, the position of a pixel can be represented by the coordinates of the pixel in the image. When the coordinates of pixel a in the first frame image in the first frame image are the same as the coordinates of pixel b in the second frame image in the second frame image, pixel a in the first frame image corresponds to pixel b in the second frame image.

[0147] In one example, the coordinates of pixel A in the first frame image are (5, 10), and the grayscale value of pixel A is a. If the grayscale value of pixel B at coordinates (5, 10) in the second frame image is b, and a is not equal to b, then the terminal device determines that pixel A is the target pixel.

[0148] In another example, the coordinates of pixel A in the first frame image are (10, 15), and the grayscale value of pixel A is a. If the grayscale value of pixel B at coordinates (10, 15) in the second frame image is a, then the terminal device determines that pixel A is not the target pixel.

[0149] S235, the terminal device calculates the proportion of the number of pixels in the set of pixels to be compared to the number of all pixels in the first frame image.

[0150] S236, compare the proportion with the proportion threshold. In the embodiments of the present application, the terminal device can define the proportion threshold. The proportion threshold can be preset before the terminal device leaves the factory, or obtained by the terminal device from the manufacturer, and of course, it can also be obtained in other ways. The embodiments of the present application do not limit this.

[0151] S237, if the proportion is not greater than the proportion threshold, then the terminal device preliminarily determines that the TOF camera is blocked.

[0152] S238, if the proportion is greater than the proportion threshold, then the terminal device determines that the TOF camera is not blocked.

[0153] In the embodiments of the present application, by determining the proportion of the number of target pixels to the number of all pixels in the image, the terminal device can accurately describe the difference between the first frame image and the second frame image, and further can accurately preliminarily determine whether the TOF camera is blocked.

[0154] In some embodiments, step S23 can be specifically implemented through the following steps:

[0155] The terminal device determines the image area to be compared according to the grayscale values of the pixels in the first frame image. The shape of the image area to be compared is a rectangle, and the image area to be compared is surrounded by the uppermost, lowermost, leftmost, and rightmost pixels among all the pixels whose grayscale values in the first frame image are greater than the grayscale values of the corresponding pixels in the second frame image at the positions of the pixels in the first frame image.

[0156] Figure 13 This is a schematic diagram of an image A and an image B provided by the embodiments of the present application.

[0157] Image A is the second frame image, and image B is the first frame image. The terminal device searches for the uppermost, lowermost, leftmost, and rightmost pixel points among the pixel points in image B whose grayscale values are greater than the grayscale values of the corresponding pixel points in image A. After comparison, it is determined that the grayscale value of pixel point a in image B is greater than the grayscale value of the pixel point corresponding to the position of pixel point a in image A, and pixel point a is one of the uppermost and leftmost pixel points among all the pixel points in image B whose grayscale values are greater than those in image A (the uppermost pixel point and the leftmost pixel point coincide); and, it is determined that the grayscale value of pixel point b in image B is greater than the grayscale value of the pixel point corresponding to the position of pixel point b in image A, and pixel point b is one of the lowermost and rightmost pixel points among all the pixel points in image B whose grayscale values are greater than those in image A (the lowermost pixel point and the rightmost pixel point coincide). The terminal device determines that the image region 131 to be compared is a rectangular region enclosed by pixel point a and pixel point b.

[0158] In the embodiments of the present application, the terminal device preliminarily determines whether the TOF camera is blocked according to the grayscale values of the pixel points in the image region to be compared in the first frame image and the grayscale values of the pixel points in the image region to be compared in the second frame image.

[0159] In the embodiments of the present application, the grayscale value of the pixel point in the image is mainly determined by the first reflected light reflected by the target object. When a part of the object in the target object may be relatively close to the TOF camera and a part of the object may be relatively far from the TOF camera, the object at a relatively far distance can only reflect the reflected light with a relatively weak signal intensity of the detection light. When the image sensor receives the reflected light with a relatively weak signal intensity, the pixel value of the corresponding pixel point in the image is generated, which may be the same as the grayscale value of the pixel point in the image generated by the image sensor when the projection light source is turned off. Therefore, in order to reduce the calculation amount, the terminal device determines the image region to be compared, tries to exclude the pixel points corresponding to the relatively far part of the target object in the first frame image and the second frame image, and then preliminarily determines whether the TOF camera is blocked according to the grayscale values of the pixel points in the first frame image and the second frame image in the region to be compared.

[0160] In some embodiments, the image region to be compared is represented by the coordinates of the uppermost, lowermost, leftmost, and rightmost pixel points. Exemplarily, the coordinates of the uppermost pixel point are (a, b), the coordinates of the lowermost pixel point are (c, d), the coordinates of the leftmost pixel point are (e, f), and the coordinates of the rightmost pixel point are (g, h). Then the four vertices of the image region to be compared are pixel point (e, b), pixel point (e, d), pixel point (g, b), and pixel point (g, d).

[0161] In a feasible implementation manner, the steps for the terminal device to preliminarily determine whether the TOF camera is blocked according to the gray values of the pixel points in the image area to be compared in the first frame image and the gray values of the pixel points in the image area to be compared in the second frame image include:

[0162] The terminal device calculates the average value of the gray values of the pixel points in the image area to be compared in the first frame image to obtain a third average value.

[0163] The terminal device calculates the average value of the gray values of the pixel points in the image area to be compared in the second frame image to obtain a fourth average value.

[0164] The terminal device preliminarily determines whether the TOF camera is blocked according to the magnitudes of the third average value and the fourth average value.

[0165] In the embodiments of the present application, the process for the terminal device to preliminarily determine whether the TOF camera is blocked according to the magnitudes of the third average value and the fourth average value is the same as the steps for preliminarily determining whether the TOF camera is blocked according to the magnitudes of the first average value and the second average value in the above text.

[0166] S24. If the terminal device determines that the TOF camera is not blocked, it can enable the TOF to continue to work normally.

[0167] S25. If the terminal device preliminarily determines that the TOF camera is blocked, it acquires a second voltage, where the second voltage is the voltage generated by the photoelectric conversion component according to the received second reflected light.

[0168] In some embodiments, in response to an operation of starting the TOF camera, the photoelectric conversion component generates a second voltage, and the terminal device stores the second voltage as matadata (metadata). If the terminal device preliminarily determines that the TOF camera is blocked, it acquires the second voltage from the matadata.

[0169] In the embodiments of the present application, when the terminal device preliminarily determines that the TOF camera is blocked, the TOF camera may be blocked by an object, or the projection light source may fail to emit detection light normally. In order for the terminal device to accurately determine whether the TOF camera is blocked, it acquires the second voltage generated by the photoelectric conversion component according to the received second reflected light, and compares the second voltage with a preset second voltage.

[0170] S26. The terminal device compares the second voltage with a second preset voltage. The second preset voltage is determined according to the voltage generated by the photoelectric conversion component when the preset projection light source is turned off. In the embodiments of the present application, when the projection light source is turned off, the photoelectric conversion component cannot receive the reflected light including the reflected detection light, so the voltage generated by the photoelectric conversion component at this time is relatively low.

[0171] The second preset voltage is a voltage extracted by the terminal device from the voltages generated by the optoelectronic conversion component when taking pictures with the projection light source turned off for multiple different terminal devices. The terminal device can define the second preset voltage, which can be preset before the terminal device leaves the factory, or obtained by the terminal device from the manufacturer side. Of course, it can also be obtained in other ways, and the embodiments of the present application do not limit this.

[0172] S27. When the terminal device determines that the second voltage is greater than the second preset voltage, it indicates that the optical signal intensity of the second reflected light received by the optoelectronic conversion component is higher than the voltage generated by the optoelectronic conversion component when the projection light source is turned off. Therefore, the terminal device determines that the TOF camera is blocked.

[0173] In some embodiments, the second preset voltage is the voltage generated by the optoelectronic conversion component when the preset projection light source is turned off multiplied by a third preset multiple, and the third preset multiple is 1.2 times. In this way, the terminal device uses the voltage generated by the optoelectronic conversion component when the preset projection light source is turned off multiplied by the third preset multiple as the second preset voltage, and can more accurately determine whether the TOF camera is blocked.

[0174] In some embodiments, if the TOF camera is blocked, the terminal device can turn off the projection light source or reduce the power of the projection light source.

[0175] In the embodiments of the present application, after the terminal device determines that the TOF camera is blocked, the terminal device can turn off the projection light source or reduce the power of the projection light source to reduce the harm of the detection light to the human skin.

[0176] In some embodiments, if the TOF camera is blocked, the terminal device can display a first prompt window on the display screen, and the first prompt window is used to prompt the user that the TOF camera is blocked. In some embodiments, the content displayed in the first prompt window can be in the form of text, pictures, or videos, etc. In the embodiments of the present application, when the user sees the first prompt window on the display screen, the user can move the hand that blocks the TOF camera away.

[0177] In some embodiments, S28. If the second voltage is not greater than the second preset voltage, it is determined that the projection light source fails to emit the detection light normally. In the embodiments of the present application, if the projection light source fails to emit the detection light normally, a second prompt window is displayed on the display screen, and the second prompt window is used to prompt the user that the projection light source does not emit the detection light. In some embodiments, the content displayed in the second prompt window can be in the form of text, pictures, or videos, etc. In this way, the terminal device can prompt the user to check the terminal device, and if the projection light source is damaged, the user can repair it in time.

[0178] In the embodiments of the present application, the terminal device preliminarily determines whether the TOF camera is occluded by the difference between the first frame image and the second frame image determined by the TOF camera itself. After the terminal device preliminarily determines that the TOF camera is occluded, in order to further accurately determine whether the TOF camera is occluded, the voltage generated by the optoelectronic conversion component is compared with a preset voltage. When the terminal device determines that the voltage is greater than the preset voltage, it indicates that the optical signal intensity of the second reflected light received by the optoelectronic conversion component is relatively high. Therefore, the terminal device determines that the TOF camera is occluded. The method in the embodiments of the present application adopts a method combining an algorithm (preliminarily determining whether the TOF camera is occluded through the first frame image and the second frame image) and hardware (optoelectronic conversion component), avoiding the problems in the method including steps S11 - S14 mentioned above. While accurately detecting whether the TOF camera is occluded by an occluder, the detection process of the terminal device is simple and fast.

[0179] The above embodiments provided in the present application introduce various solutions of the method for detecting occlusion of the TOF camera provided in the present application. It can be understood that in order for the terminal device to implement the above functions, it includes the corresponding hardware structure and / or software module for executing each function. Those skilled in the art should easily realize that, combining the units and algorithm steps of each example described in the embodiments disclosed in this article, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraint conditions of the technical solution. Professional technicians can use different methods to implement the described function for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0180] Figure 14 It is a schematic structural diagram of an occlusion detection device for a TOF camera provided in an embodiment of the present application. In one embodiment, the terminal device can implement corresponding functions through Figure 14 the hardware device shown. As Figure 14 shown, the detection device may include: a touch screen 1401, a memory 1402, and a processor 1403.

[0181] In one embodiment, the touch screen 1401 may include a display screen and a touch image sensor. The display screen is used to display images, and the touch image sensor can transmit the detected touch operation to the application processor to determine the touch event type and provide visual output related to the touch operation through the display screen. The processor 1403 may include one or more processing units. For example, the processor 1403 may include an application processor, a modem processor, a graphics processor, an image signal processor, a controller, a video codec, a digital signal processor, a baseband processor, and / or a neural network processor, etc. Among them, different processing units may be independent devices or integrated in one or more processors. The memory 1402 is coupled to the processor 1403 and is used to store various software programs and / or multiple sets of instructions. The memory 1402 may include volatile memory and / or non-volatile memory.

[0182] When the software programs and / or multiple sets of instructions in the memory 1402 are executed by the processor 1403, the terminal device implements the following method steps: obtaining an image generated by the image sensor according to the received first reflected light, where the image includes a first frame image and a second frame image; among them, when the TOF camera is not blocked, the first reflected light includes the reflected light of the target object to the detection light, and when the TOF camera is blocked, the first reflected light does not include the reflected light of the target object to the detection light; the first frame image is the image generated by the image sensor when the projection light source is turned on, and the second frame image is the image generated by the image sensor when the projection light source is turned off; according to the first frame image and the second frame image, initially determine whether the TOF camera is blocked; if it is initially determined that the TOF camera is blocked, then obtain the voltage generated by the photoelectric conversion component according to the received second reflected light; among them, when the TOF camera is blocked, the second reflected light includes the reflected light of the occluder to the detection light; when the TOF camera is not blocked, the second reflected light does not include the light reflected by the occluder to the detection light; if the voltage is greater than the preset voltage, then determine that the TOF camera is blocked, and the preset voltage is determined according to the voltage generated by the photoelectric conversion component when the preset projection light source is turned off. In this way, the terminal device initially determines whether the TOF camera is blocked by the difference between the first frame image and the second frame image determined by the TOF camera itself. After the terminal device initially determines that the TOF camera is blocked, in order to further accurately determine whether the TOF camera is blocked, the voltage generated by the photoelectric conversion component is compared with the preset voltage. When the terminal device determines that the voltage is greater than the preset voltage, it means that the light signal intensity of the second reflected light received by the photoelectric conversion component is relatively high. Therefore, the terminal device determines that the TOF camera is blocked.

[0183] In one implementation, when the software program and / or multiple sets of instructions in the memory 1402 are executed by the processor 1403, the terminal device specifically implements the following method steps: The steps for preliminarily determining whether the TOF camera is blocked based on the first frame image and the second frame image include: calculating the average value of the gray values of the pixel points in the first frame image to obtain a first average value; calculating the average value of the gray values of the pixel points in the second frame image to obtain a second average value; and preliminarily determining whether the TOF camera is blocked based on the first average value and the second average value. In this way, the first average value is calculated based on the first frame image generated by the image sensor when the projection light source is turned on, and the second average value is calculated based on the second frame image generated by the image sensor when the projection light source is turned off. In this way, the terminal device can use the first average value and the second average value to determine the difference between the first frame image and the second frame image, and then preliminarily determine whether the TOF camera is blocked.

[0184] In one implementation, when the software program and / or multiple sets of instructions in the memory 1402 are executed by the processor 1403, the terminal device specifically implements the following method steps: subtracting the second average value from the first average value to obtain a gray value average difference; if the gray value average difference is not greater than the average threshold, it is preliminarily determined that the TOF camera is blocked, and the average threshold is determined based on the second average value of the second frame image; if the gray value average difference is greater than the average threshold, it is determined that the TOF camera is not blocked. In this way, the gray value average difference can accurately describe the difference between the first frame image and the second frame image. When the average value of the gray values is greater than the average threshold, it means that the difference between the first frame image and the second frame image is large, and the intensity of the optical signal of the first reflected light received by the image sensor when generating the first frame image is quite different from the intensity of the optical signal of the first reflected light received by the image sensor when generating the second frame image. The detection light of the projection light source is emitted to the target object, and at this time the terminal device can determine that the TOF camera is not blocked; if the average value of the gray values is not greater than the average threshold, it means that the difference between the first frame image and the second frame image is small, and the intensity of the optical signal of the first reflected light received by the image sensor when generating the first frame image is not much different from the intensity of the optical signal of the first reflected light received by the image sensor when generating the second frame image. The detection light of the projection light source is not emitted to the target object, and at this time the terminal device can preliminarily determine that the TOF camera is blocked.

[0185] In one implementation, when the software program and / or multiple sets of instructions in the memory 1402 are executed by the processor 1403, the terminal device specifically implements the following method steps: If the first average value is not greater than the preset average value, it is preliminarily determined that the TOF camera is blocked, and the preset average value is determined according to the second average value; if the first average value is greater than the preset average value, it is determined that the TOF camera is not blocked. In this way, the first average value is calculated based on the first frame of image generated by the image sensor when the projection light source is turned on, and the second average value is calculated based on the second frame of image generated by the image sensor when the projection light source is turned off. The preset average value is determined according to the second average value. By comparing the first average value and the preset average value, the difference between the first frame of image and the second frame of image can be accurately determined. If the first average value is not greater than the preset average value, it indicates that the difference between the first frame of image and the second frame of image is not large, and the light signal intensity of the first reflected light received by the image sensor when generating the first frame of image is not much different from the light signal intensity of the first reflected light received by the image sensor when generating the second frame of image. At this time, the terminal device can preliminarily determine that the TOF camera is blocked; if the first average value is greater than the preset average value, it indicates that the difference between the first frame of image and the second frame of image is large, and the light signal intensity of the first reflected light received by the image sensor when generating the first frame of image is much different from the light signal intensity of the first reflected light received by the image sensor when generating the second frame of image. At this time, the terminal device can determine that the TOF camera is not blocked.

[0186] In one implementation, when the software program and / or multiple sets of instructions in the memory 1402 are executed by the processor 1403, the terminal device specifically implements the following method steps: Determine a set of pixels to be compared according to the gray values of the pixels in the first frame of image and the second frame of image. The set of pixels to be compared includes target pixels, and the target pixels are the pixels in the first frame of image whose gray values are not equal to the gray values of the pixels corresponding to the positions of the pixels in the first frame of image in the second frame of image; Calculate the proportion of the number of pixels in the set of pixels to be compared to the number of all pixels in the first frame of image; If the proportion is not greater than the proportion threshold, it is preliminarily determined that the TOF camera is blocked; if the proportion is greater than the proportion threshold, it is determined that the TOF camera is not blocked. In this way, by determining the proportion of the number of target pixels to the number of all pixels in the image, the terminal device can accurately describe the difference between the first frame of image and the second frame of image, and further can accurately preliminarily determine whether the TOF camera is blocked.

[0187] In one implementation, when the software program and / or multiple sets of instructions in the memory 1402 are executed by the processor 1403, the terminal device specifically implements the following method steps: Determine a to-be-compared image region according to the gray values of the pixel points in the first frame of image. The shape of the to-be-compared image region is rectangular, and the to-be-compared image region is formed by the uppermost, lowermost, leftmost, and rightmost pixel points among all the pixel points whose gray values in the first frame of image are greater than the gray values of the pixel points corresponding to the positions of the pixel points in the first frame of image in the second frame of image; According to the gray values of the pixel points in the to-be-compared image region in the first frame of image and the gray values of the pixel points in the to-be-compared image region in the second frame of image, preliminarily determine whether the TOF camera is blocked. In this way, the gray values of the pixel points in the image are mainly determined by the first reflected light reflected by the target object. When a part of the object in the target object may be relatively close to the TOF camera and a part of the object may be relatively far from the TOF camera, the object that is farther away can only reflect the reflected light with a relatively weak signal intensity of the detection light. When the image sensor receives the reflected light with a relatively weak signal intensity, the pixel value of the corresponding pixel point in the image is generated, which may be the same as the gray value of the pixel point in the image generated when the projection light source is turned off. Therefore, in order to reduce the calculation amount, the terminal device determines the to-be-compared image region, tries to exclude the pixel points corresponding to the farther part of the target object in the first frame of image and the second frame of image, and then preliminarily determines whether the TOF camera is blocked according to the gray values of the pixel points in the to-be-compared region of the first frame of image and the second frame of image.

[0188] In one implementation, when the software program and / or multiple sets of instructions in the memory 1402 are executed by the processor 1403, the terminal device also implements the following method steps: If the second voltage is not greater than the second preset voltage, determine that the projection light source fails to emit the detection light normally. In this way, when the second voltage is not greater than the second preset voltage, it indicates that the signal intensity of the second reflected light received by the photoelectric conversion component is relatively low. Therefore, the terminal device can accurately determine that the projection light source fails to emit the detection light normally.

[0189] In one implementation, when the software program and / or multiple sets of instructions in the memory 1402 are executed by the processor 1403, the terminal device further implements the following method steps: If the TOF camera is blocked, turn off the projection light source or reduce the power of the projection light source. In this way, after the terminal device determines that the TOF camera is blocked, it can turn off the projection light source or reduce the power of the projection light source to reduce the harm of the detection light to the human skin.

[0190] In one implementation, when the software program and / or multiple sets of instructions in the memory 1402 are executed by the processor 1403, the terminal device further implements the following method steps: If the TOF camera is blocked, a first prompt window is displayed on the display screen, and the first prompt window is used to prompt the user that the TOF camera is blocked. In this way, when the user sees the prompt window on the display screen, they can move the hand that blocks the TOF camera.

[0191] In one implementation, when the software program and / or multiple sets of instructions in the memory 1402 are executed by the processor 1403, the terminal device further implements the following method steps: If the projection light source fails to emit detection light normally, a second prompt window is displayed on the display screen, and the second prompt window is used to prompt the user that the projection light source fails to emit detection light normally. In this way, the user can perform timely repairs on the damaged projection light source according to the prompt window.

[0192] In one implementation, if the split-screen score of the target application is greater than the third threshold, the preset split-screen direction of the target application is the first split-screen direction; the second average value of the second frame image with the mean threshold being the first preset multiple, and the first preset multiple is 1.2 times. In this way, the terminal device uses the second average value of the second frame image with the first preset multiple as the mean threshold, and can more accurately initially determine whether the TOF camera is blocked.

[0193] In one implementation, when the software program and / or multiple sets of instructions in the memory 1402 are executed by the processor 1403, the terminal device further implements the following method steps: The preset average value is the second average value with the second preset multiple, and the second preset multiple is 1.2 times. In this way, the terminal device uses the second average value with the second preset multiple as the preset average value, and can more accurately initially determine whether the TOF camera is blocked.

[0194] In one implementation, when the software program and / or multiple sets of instructions in the memory 1402 are executed by the processor 1403, the terminal device further implements the following method steps: The second preset voltage is the voltage generated by the photoelectric conversion component when the preset projection light source is turned off with the third preset multiple, and the third preset multiple is 1.2 times. In this way, the terminal device uses the voltage generated by the photoelectric conversion component when the preset projection light source with the third preset multiple is turned off as the preset voltage, and can more accurately determine whether the TOF camera is blocked.

[0195] Figure 15 It is a schematic structural diagram of another TOF camera occlusion detection device provided by an embodiment of the present application. In one embodiment, the terminal device can Figure 15 implement corresponding functions through the software device shown. As Figure 15As shown in the figure, the detection device includes: a first acquisition module 1501, a first preliminary determination module 1502, a second acquisition module 1503, and a first determination module 1504. Among them, the first acquisition module 1501 is configured to acquire an image generated by an image sensor according to the received first reflected light, and the image includes a first frame image and a second frame image; wherein, when the TOF camera is not blocked, the first reflected light includes the reflected light of the target object to the detection light, and when the TOF camera is blocked, the first reflected light does not include the reflected light of the target object to the detection light; the first frame image is an image generated by the image sensor when the projection light source is turned on, and the second frame image is an image generated by the image sensor when the projection light source is turned off; the first preliminary determination module 1502 is configured to preliminarily determine whether the TOF camera is blocked according to the first frame image and the second frame image; the second acquisition module 1503 is configured to, if it is preliminarily determined that the TOF camera is blocked, acquire the voltage generated by the photoelectric conversion component according to the received second reflected light; wherein, when the TOF camera is blocked, the second reflected light includes the reflected light of the occluder to the detection light; when the TOF camera is not blocked, the second reflected light does not include the reflected light of the occluder to the detection light; the first determination module 1504 is configured to, if the voltage is greater than a preset voltage, determine that the TOF camera is blocked, and the preset voltage is determined according to the voltage generated by the photoelectric conversion component when the preset projection light source is turned off. According to the above device, the terminal device preliminarily determines whether the TOF camera is blocked by the difference between the first frame image and the second frame image determined by the TOF camera itself. After the terminal device preliminarily determines that the TOF camera is blocked, in order to further accurately determine whether the TOF camera is blocked, the voltage generated by the photoelectric conversion component is compared with the preset voltage. When the terminal device determines that the voltage is greater than the preset voltage, it indicates that the light signal intensity of the second reflected light received by the photoelectric conversion component is relatively high. Therefore, the terminal device determines that the TOF camera is blocked.

[0196] In one implementation, the first preliminary determination module includes: a first average value calculation module, a second average value calculation module, and a second preliminary determination module. Among them, the first average value calculation module is configured to calculate the average value of the gray values of the pixel points in the first frame image to obtain a first average value; the second average value calculation module is configured to calculate the average value of the gray values of the pixel points in the second frame image to obtain a second average value; the second preliminary determination module is configured to preliminarily determine whether the TOF camera is blocked according to the first average value and the second average value. According to the above device, the first average value is calculated according to the first frame image generated by the image sensor when the projection light source is turned on, and the second average value is calculated according to the second frame image generated by the image sensor when the projection light source is turned off. In this way, the terminal device can use the first average value and the second average value to determine the difference between the first frame image and the second frame image, and further preliminarily determine whether the TOF camera is blocked.

[0197] In one implementation, the second preliminary determination module includes: a difference module, a third preliminary determination module, and a second determination module. Among them, the difference module is configured to subtract the first average value from the second average value to obtain a grayscale average difference; the third preliminary determination module is configured to preliminarily determine that the TOF camera is blocked if the grayscale average difference is not greater than an average threshold, and the average threshold is determined according to the second average value of the second frame image; the second determination module is configured to determine that the TOF camera is not blocked if the grayscale average difference is greater than the average threshold. According to the above device, the grayscale average difference can accurately describe the difference between the first frame image and the second frame image. When the grayscale average difference is greater than the average threshold, it indicates that the difference between the first frame image and the second frame image is large, and the intensity of the optical signal of the first reflected light received by the image sensor when generating the first frame image is quite different from the intensity of the optical signal of the first reflected light received by the image sensor when generating the second frame image. The detection light of the projection light source is emitted to the target object. At this time, the terminal device can determine that the TOF camera is not blocked; if the grayscale average value is not greater than the average threshold, it indicates that the difference between the first frame image and the second frame image is small, and the intensity of the optical signal of the first reflected light received by the image sensor when generating the first frame image is not much different from the intensity of the optical signal of the first reflected light received by the image sensor when generating the second frame image. The detection light of the projection light source is not emitted to the target object. At this time, the terminal device can preliminarily determine that the TOF camera is blocked.

[0198] In one implementation, the second preliminary determination module includes: a fourth preliminary determination module and a third determination module. Among them, the fourth preliminary determination module is configured to preliminarily determine that the TOF camera is blocked if the first average value is not greater than a preset average value, and the preset average value is determined according to the second average value; the third determination module is configured to determine that the TOF camera is not blocked if the first average value is greater than the preset average value. According to the above device, the first average value is calculated based on the first frame image generated by the image sensor when the projection light source is turned on, and the second average value is calculated based on the second frame image generated by the image sensor when the projection light source is turned off. The preset average value is determined according to the second average value. By comparing the first average value and the preset average value, the difference between the first frame image and the second frame image can be accurately determined. If the first average value is not greater than the preset average value, it indicates that the difference between the first frame image and the second frame image is not significant, and the light signal intensity of the first reflected light received by the image sensor when generating the first frame image is not much different from the light signal intensity of the first reflected light received by the image sensor when generating the second frame image. At this time, the terminal device can preliminarily determine that the TOF camera is blocked; if the first average value is greater than the preset average value, it indicates that the difference between the first frame image and the second frame image is relatively large, and the light signal intensity of the first reflected light received by the image sensor when generating the first frame image is quite different from the light signal intensity of the first reflected light received by the image sensor when generating the second frame image. At this time, the terminal device can determine that the TOF camera is not blocked.

[0199] In one implementation, the first preliminary determination module includes: a set determination module, a calculation module, a fifth preliminary determination module, and a fourth determination module. Among them, the set determination module is configured to determine a set of pixels to be compared according to the gray values of the pixels in the first frame image and the second frame image. The set of pixels to be compared includes target pixels, and the target pixels are the pixels in the first frame image whose gray values are not equal to the gray values of the pixels at the corresponding positions in the second frame image; the calculation module is configured to calculate the proportion of the number of pixels in the set of pixels to be compared to the number of all pixels in the first frame image; the fifth preliminary determination module is configured to preliminarily determine that the TOF camera is blocked if the proportion is not greater than a proportion threshold; the fourth determination module is configured to determine that the TOF camera is not blocked if the proportion is greater than the proportion threshold. According to the above device, by determining the proportion of the number of target pixels to the number of all pixels in the image, the terminal device can accurately describe the difference between the first frame image and the second frame image, and further can accurately preliminarily determine whether the TOF camera is blocked.

[0200] In one implementation, the first preliminary determination module includes: a region determination module and a sixth preliminary determination module. Among them, the region determination module is configured to determine a to-be-compared image region according to the gray values of the pixel points in the first frame image. The shape of the to-be-compared image region is rectangular, and the to-be-compared image region is formed by the pixel points at the uppermost, lowermost, leftmost, and rightmost positions among all the pixel points whose gray values in the first frame image are greater than the gray values of the pixel points corresponding to the positions of the pixel points in the first frame image in the second frame image; the sixth preliminary determination module is configured to preliminarily determine whether the TOF camera is blocked according to the gray values of the pixel points in the to-be-compared image region in the first frame image and the gray values of the pixel points in the to-be-compared image region in the second frame image. According to the above device, the gray value of the pixel point in the image is mainly determined by the first reflected light reflected by the target object. When a part of the object in the target object may be relatively close to the TOF camera and a part of the object may be relatively far from the TOF camera, the object at a relatively far distance can only reflect the reflected light with a relatively weak signal intensity of the detection light. When the image sensor receives the reflected light with a relatively weak signal intensity, the pixel value of the corresponding pixel point in the image is generated, which may be the same as the gray value of the pixel point in the image generated by the image sensor when the projection light source is turned off. Therefore, in order to reduce the calculation amount, the terminal device determines the to-be-compared image region, tries to exclude the pixel points corresponding to the relatively far part of the target object in the first frame image and the second frame image, and then preliminarily determines whether the TOF camera is blocked according to the gray values of the pixel points in the to-be-compared region of the first frame image and the second frame image.

[0201] In one implementation, the detection device further includes: a fifth determination module. The fifth determination module is configured to determine that the projection light source fails to emit the detection light normally if the second voltage is not greater than the second preset voltage. According to the above device, when the second voltage is not greater than the second preset voltage, it indicates that the signal intensity of the second reflected light received by the photoelectric conversion component is relatively low. Therefore, the terminal device can accurately determine that the projection light source fails to emit the detection light normally.

[0202] In one implementation, the detection device further includes: a processing module. The processing module is configured to turn off the projection light source or reduce the power of the projection light source if the TOF camera is blocked. According to the above device, after the terminal device determines that the TOF camera is blocked, it can turn off the projection light source or reduce the power of the projection light source to reduce the harm of the detection light to the human skin.

[0203] In one implementation, the detection device further includes: a first display module. The first display module is configured to display a first prompt window on the display screen if the TOF camera is blocked, and the first prompt window is used to prompt the user that the TOF camera is blocked. According to the above device, when the user sees the prompt window on the display screen, the user can move the hand blocking the TOF camera away.

[0204] In one implementation, the detection device further includes: a second display module. The second display module is configured to display a second prompt window on the display screen if the projection light source fails to emit detection light normally, and the second prompt window is used to prompt the user that the projection light source does not emit detection light. According to the above device, the user can timely repair the damaged projection light source according to the prompt window.

[0205] The embodiments of the present application further provide a computer-readable storage medium, in which instructions are stored. When the instructions run on a computer, the computer is caused to execute the methods in the above aspects.

[0206] The embodiments of the present application further provide a computer program product containing instructions. When the computer program product runs on a computer, the computer is caused to execute the methods in the above aspects.

[0207] The embodiments of the present application further provide a chip system, Figure 16 which is a schematic structural diagram of the chip system. The chip system includes a processor 1601, which is used to support the above device to implement the functions involved in the above aspects. For example, the processor 1601 is used to generate or process the information involved in the above methods. In a possible design, the chip system further includes a memory 1602, which is used to store the necessary computer instructions 1603 and data of the privacy information generation device. The chip system can be composed of chips or can include chips and other discrete devices.

[0208] The above specific implementation manners further elaborate the purpose, technical solutions, and beneficial effects of the embodiments of the present application. It should be understood that the above are only the specific implementation manners of the embodiments of the present application and are not used to limit the protection scope of the embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A method for occlusion detection of a TOF camera, the TOF camera comprising a transmitting end and a receiving end, characterized in that, The transmitting end includes a projection light source and a photoelectric conversion component, and the projection light source is used to emit detection light to a target object; The receiving end includes an image sensor, and the method includes: Obtaining an image generated by the image sensor according to the received first reflected light, where the image includes a first frame image and a second frame image; wherein, when the TOF camera is not blocked, the first reflected light includes the reflected light of the target object to the detection light, and when the TOF camera is blocked, the first reflected light does not include the reflected light of the target object to the detection light; the first frame image is an image generated by the image sensor when the projection light source is turned on, and the second frame image is an image generated by the image sensor when the projection light source is turned off; Based on the first frame image and the second frame image, preliminarily determine whether the TOF camera is blocked; If it is preliminarily determined that the TOF camera is blocked, then obtain the voltage generated by the photoelectric conversion component according to the received second reflected light; wherein, when the TOF camera is blocked, the second reflected light includes the reflected light of the occluder to the detection light; when the TOF camera is not blocked, the second reflected light does not include the reflected light of the occluder to the detection light; if the voltage is greater than a preset voltage, it is determined that the TOF camera is blocked, and the preset voltage is determined according to the voltage generated by the photoelectric conversion component when the preset projection light source is turned off.

2. The method according to claim 1, wherein The step of preliminarily determining whether the TOF camera is blocked based on the first frame image and the second frame image includes: Calculating the average value of the gray values of the pixel points in the first frame image to obtain a first average value; Calculating the average value of the gray values of the pixel points in the second frame image to obtain a second average value; Based on the first average value and the second average value, preliminarily determine whether the TOF camera is blocked.

3. The method according to claim 2, wherein The step of preliminarily determining whether the TOF camera is blocked based on the first average value and the second average value includes: Subtracting the first average value from the second average value to obtain a gray value mean difference; If the gray value mean difference is not greater than a mean threshold value, it is preliminarily determined that the TOF camera is blocked, and the mean threshold value is determined according to the second average value of the second frame image; If the gray value mean difference is greater than the mean threshold value, it is determined that the TOF camera is not blocked.

4. The method according to claim 2, wherein The step of preliminarily determining whether the TOF camera is blocked based on the first average value and the second average value includes: If the first average value is not greater than a preset average value, it is preliminarily determined that the TOF camera is blocked, and the preset average value is determined according to the second average value; If the first average value is greater than the preset average value, it is determined that the TOF camera is not blocked.

5. The method according to claim 1, wherein The step of preliminarily determining whether the TOF camera is blocked based on the first frame image and the second frame image includes: Determine a set of pixels to be compared based on the grayscale values of the pixels in the first frame image and the second frame image. The set of pixels to be compared includes target pixels, where the target pixels are the pixels in the first frame image whose grayscale values are not equal to the grayscale values of the corresponding pixels in the second frame image at the same positions as the pixels in the first frame image. Calculate the proportion of the number of pixels in the set of pixels to be compared to the number of all pixels in the first frame image. If the proportion is not greater than a proportion threshold, preliminarily determine that the TOF camera is blocked. If the proportion is greater than the proportion threshold, determine that the TOF camera is not blocked.

6. The method according to claim 1, wherein The step of preliminarily determining whether the TOF camera is blocked based on the first frame image and the second frame image includes: Determine a region of the image to be compared based on the grayscale values of the pixels in the first frame image. The shape of the region of the image to be compared is rectangular, and the region of the image to be compared is surrounded by the uppermost, lowermost, leftmost, and rightmost pixels among all the pixels in the first frame image whose grayscale values are greater than the grayscale values of the corresponding pixels in the second frame image at the same positions as the pixels in the first frame image. Preliminarily determine whether the TOF camera is blocked based on the grayscale values of the pixels in the region of the image to be compared in the first frame image and the grayscale values of the pixels in the region of the image to be compared in the second frame image.

7. The method according to claim 1, characterized in that Further includes: If the voltage is not greater than a preset voltage, determine that the projection light source fails to emit detection light normally.

8. The method according to claim 1, wherein Further includes: If the TOF camera is blocked, turn off the projection light source or reduce the power of the projection light source.

9. The method according to claim 1, wherein Further includes: If the TOF camera is blocked, display a first prompt window on the display screen, where the first prompt window is used to prompt the user that the TOF camera is blocked.

10. The method according to claim 7, wherein Further includes: If the projection light source fails to emit detection light normally, display a second prompt window on the display screen, where the second prompt window is used to prompt the user that the projection light source does not emit detection light.

11. The method according to claim 1, characterized in that, The photoelectric conversion component includes a photodiode.

12. The method according to claim 1, characterized in that, The projection light source includes a vertical cavity surface emitting laser.

13. The method according to claim 3, characterized in that, The mean threshold is the second average value of the second frame image multiplied by a first preset multiple, and the first preset multiple is 1.2 times.

14. The method according to claim 4, characterized in that, The preset average value is the second average value multiplied by a second preset multiple, and the second preset multiple is 1.2 times.

15. The method according to claim 1, wherein The preset voltage is the voltage generated by the photoelectric conversion component when the preset projection light source is turned off multiplied by a third preset multiple, and the third preset multiple is 1.2 times.

16. A terminal device, characterized in that, Includes: A processor and a memory; the memory stores program instructions, and when the program instructions are executed by the processor, the terminal device executes the method according to any one of claims 1-15.

Images