Three-dimensional recognition device, terminal, image enhancement method, and storage medium

By integrating a three-dimensional recognition device with a dichroic filter to separate light paths within a shared lens, the 'notch' area on mobile terminals is minimized, enhancing display effectiveness and user experience.

JP2024541141AActive Publication Date: 2024-11-07ZTE CORP
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024529441
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-12-07
Filing Date
2022-10-08
Publication Date
2024-11-07
Estimated Expiration
2042-10-08

AI Technical Summary

Technical Problem

The integration of three-dimensional recognition elements and RGB cameras on mobile terminals creates a 'notch' area on the display screen, reducing the effective display area and user experience due to the need for special treatments to enhance light transmission.

Method used

A three-dimensional recognition device is installed inside the display screen, utilizing a lens with an optical path that includes an RGB camera, a three-dimensional recognition unit, and a dichroic filter to separate visible and infrared light paths, allowing both to share a single lens, thus reducing the light transmission enhancement area.

Benefits of technology

This configuration enhances the display effect by minimizing the area affected by light transmission enhancements, improving user experience by reducing the size of the 'notch' area and optimizing screen utilization.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024541141000001_ABST
    Figure 2024541141000001_ABST
Patent Text Reader

Abstract

The present application provides a three-dimensional recognition device, a terminal, an image enhancement method, and a storage medium. The three-dimensional recognition device includes a lens (140) and an optical path (200) installed inside the lens (140), in which an RGB camera (440), a three-dimensional recognition unit, and a first dichroic filter (310) are installed, and the first dichroic filter (310) reflects visible light in the ambient light to the RGB camera (440) and transmits infrared light transmitted and received by the three-dimensional recognition unit.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This application is filed based on a Chinese patent application bearing application number 202111486807.7 and filing date December 7, 2021, and claims priority to the Chinese patent application, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the field of smart terminals, but is not limited thereto, and in particular to a three-dimensional recognition device, a terminal, an image enhancement method, and a storage medium. [Background technology]

[0003] Currently, there are two main schemes for forward 3D recognition of mobile terminals: structured light and time of flight (TOF), and to realize the above two schemes, 3D recognition elements such as infrared cameras and flood illuminators need to be placed on the mobile terminal. In addition, forward RGB (Red Green Blue) cameras are usually juxtaposed between the 3D recognition elements, while the lens areas of these elements are relatively large and are usually placed on the top of the display screen, forming a "notch" area, which results in a smaller effective display area of ​​the screen.

[0004] To solve this problem, more and more smart terminals have begun to adopt under-display photography technology, and place the front RGB camera and three-dimensional recognition element inside the display screen. In order to meet the light requirements of each element, the corresponding screen display area needs to undergo special processing to increase light transmittance, such as reducing the RGB pixels or making the RGB pixels smaller to increase the amount of light transmittance. The display area after special processing has a certain difference in display effect from other display areas, and if each element adopts the current side-by-side layout, the area of ​​the special processing area will be large, which will impair the overall display effect and user experience. Summary of the Invention [Problem to be solved by the invention]

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The embodiments of the present application provide a 3D recognition device, a terminal, an image enhancement method, and a storage medium. [Means for solving the problem]

[0007] In a first aspect, an embodiment of the present application provides a three-dimensional recognition device installed inside a display screen of a terminal, the three-dimensional recognition device including: a lens; and an optical path installed inside the lens, in which an RGB camera, a three-dimensional recognition unit and a first dichroic filter are installed, the first dichroic filter reflects visible light in ambient light to the RGB camera, and further an optical path that transmits infrared light transmitted and received by the three-dimensional recognition unit.

[0008] In a second aspect, an embodiment of the present application provides a terminal comprising a three-dimensional recognition device according to the first aspect and a display screen, wherein the three-dimensional recognition device is installed inside the display screen, and an area of ​​the display screen corresponding to a lens of the three-dimensional recognition device is an enhanced light-transmittance area.

[0009] In a third aspect, an embodiment of the present application provides an image enhancement method applied to a three-dimensional recognition device, the three-dimensional recognition device including a lens and an optical path, the optical path is disposed inside the lens, an RGB camera, an infrared emitter, an infrared dot projector, a first infrared camera and a first dichroic filter are disposed in the optical path, visible light in an ambient light is reflected by the first dichroic filter to the RGB camera, infrared light in the ambient light passes through the first dichroic filter to reach the first infrared camera, and infrared light transmitted by the infrared emitter and the infrared dot projector passes through the first dichroic filter to reach the lens. The image enhancement method includes the steps of: acquiring a three-dimensional point cloud image and a two-dimensional image, where the three-dimensional point cloud image is acquired from a target reflected light by the first infrared camera, and the two-dimensional image is acquired from the target reflected light by the RGB camera, and the target reflected light is a laser speckle transmitted by the infrared dot projector and reflected by a target object; acquiring a three-dimensional structured light point cloud from the three-dimensional point cloud image, and synchronizing the three-dimensional structured light point cloud and the two-dimensional image to obtain a structured light difference value; determining RGB reference data according to the two-dimensional image, and performing an interpolation process on a reference plane according to the RGB reference data and the structured light difference value to increase density of the reference plane; acquiring a three-dimensional point cloud layer surface of the three-dimensional point cloud image, and calibrating the three-dimensional point cloud layer surface and the reference plane; and performing stereo matching between the three-dimensional point cloud layer surface and the reference plane to obtain RGBD depth information.

[0010] In a fourth aspect, an embodiment of the present application provides a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the image enhancement method of the third aspect when it executes the computer program.

[0011] In a fifth aspect, embodiments of the present application provide a computer readable storage medium having stored thereon computer executable instructions for performing the image enhancement method according to the third aspect.

[0012] Other features and advantages of the present application will be set forth in the following specification, and in part will be apparent from the specification, or will be learned by the practice of the present application. The objectives and other advantages of the present application will be achieved and obtained by the structures particularly pointed out in the specification, claims and drawings. The drawings are used to provide a further understanding of the technical solution of the present application, constitute a part of the specification, and are used to interpret the technical solution of the present application together with the examples of the present application, and are not intended to limit the technical solution of the present application. [Brief description of the drawings]

[0013] [Figure 1] 1 is a structural diagram of a three-dimensional recognition device according to a first embodiment of the present application. [Diagram 2] FIG. 11 is a structural diagram of a three-dimensional recognition device according to a second embodiment of the present invention. [Diagram 3] FIG. 11 is a structural diagram of a three-dimensional recognition device according to a third embodiment of the present invention. [Figure 4] FIG. 11 is a structural diagram of a three-dimensional recognition device according to a fourth embodiment of the present invention. [Diagram 5] FIG. 2 is a structural diagram of a terminal according to another embodiment of the present application; [Figure 6] 4 is a flowchart of an image enhancement method according to another embodiment of the present application. [Figure 7] 4 is a flow chart of image calibration according to another embodiment of the present application; [Figure 8] FIG. 2 is a diagram illustrating an apparatus of a terminal according to another embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In order to make the purpose, technical solution and advantages of the present application clearer, the present application will be described in more detail below with reference to the drawings and examples. The specific examples described in this specification are only used to interpret the present application and are not used to limit the present application.

[0015] In addition, although the schematic diagram of the device is divided into functional modules and the flow chart shows a logical order, in some cases, the division into modules within the device may differ, or the steps shown or described may be performed in a different order than that of the flow chart. The terms "first," "second," etc. in the specification, claims, or above drawings are intended to distinguish between similar objects, and are not intended to describe a particular order or priority.

[0016] The present application provides a three-dimensional recognition device, a terminal, an image enhancement method, and a storage medium. The three-dimensional recognition device includes a lens and an optical path installed inside the lens, in which an RGB camera, a three-dimensional recognition unit, and a first dichroic filter are installed in the optical path, and the first dichroic filter reflects visible light in the ambient light to the RGB camera, and transmits infrared light transmitted and received by the three-dimensional recognition unit. According to the technical solution of this embodiment, the RGB lens and the three-dimensional recognition unit can be installed simultaneously in the optical path of the lens, realizing the lens sharing of the front camera and the three-dimensional recognition unit, and reducing the area specially treated to enhance light transmittance to the size of a single lens, effectively enhancing the display effect of the screen, thereby improving the user experience.

[0017] Hereinafter, the embodiments of the present application will be further described with reference to the drawings.

[0018] The present application provides a three-dimensional recognition device installed inside a display screen of a terminal, the three-dimensional recognition device comprising: Lens 140; The optical path 200 is installed inside the lens 140, and an RGB camera 440, a three-dimensional recognition unit and a first dichroic filter 310 are installed in the optical path 200, and the first dichroic filter 310 reflects visible light in the ambient light to the RGB camera 440 and further transmits infrared light transmitted and received by the three-dimensional recognition unit.

[0019] It should be noted that the lens 140 is installed inside the display screen 120 of the terminal and is adjacent to the enhanced light transmission area 130 of the display screen 120. Here, the specific parameters and type of the lens 140 are not particularly limited as long as they can be applied to under-display photography.

[0020] In addition, in order to realize the same lens 140 being shared for photography and 3D recognition, the RGB camera 440 and the 3D recognition unit need to be installed simultaneously in the optical path 200 inside the lens 140, the RGB camera 440 realizes photography, its light source is visible light, while the 3D recognition unit is mainly structured light or TOF unit, and the received or transmitted light is infrared light. Therefore, when the display screen 130 is placed horizontally, the first dichroic filter 310 can be installed in the vertical optical path 200 inside the lens 140, which realizes the transmission of infrared light and the reflection of visible light, and can distinguish the propagation paths of visible light and infrared light, thereby meeting the requirements of the optical propagation paths of the RGB camera 440 and the 3D recognition unit in the same optical path 200.

[0021] In addition, the first dichroic filter 310 may use a visible / infrared light filter film. When light is incident at an angle of 45°, the visible / infrared light filter film has a reflectance of more than 90% for the visible light wavelength band of 0.3 microns to 0.6 microns and a transmittance of more than 90% for near-infrared light of 0.75 microns to 2.5 microns, so that the infrared light can pass through the visible / infrared light filter film, but the visible light is specularly reflected by the visible / infrared light filter film, thereby distinguishing the propagation directions of the infrared light and the visible light. The specific installation angle of the first dichroic filter 310 can be adjusted according to the actual layout manner of the optical path 200, and is not particularly limited in this embodiment.

[0022] In addition, the first dichroic filter 310 realizes different propagation paths of infrared light and visible light, so that the RGB camera 440 and the 3D recognition unit installed in the optical path 200 can operate normally, thereby realizing the 3D recognition unit and the RGB camera 440 to share one lens 140. Compared with the scheme in which the 3D recognition unit and the RGB camera are arranged side by side at the front end of the display screen, the enhanced light-transmitting region 130 only needs to be installed for one lens 140, which greatly reduces the area and effectively improves the user experience.

[0023] Various embodiments of the three-dimensional recognition device will be described below by way of several examples.

[0024] Note that Figures 1 to 4 are structural diagrams of four embodiments of the three-dimensional recognition device according to the present application, and all four of the above diagrams are cut vertically when the terminal is placed horizontally with the front camera facing upwards, and will not be described again later.

[0025] Example 1 As shown in FIG. 1, when the first dichroic filter 310 is installed in the optical path 200, the first dichroic filter 310 divides the optical path 200 to form a first passage 210 and a second passage 220. Based on the above description, when the first dichroic filter 310 is installed at an angle of 45°, the visible light is mirror-reflected by the first dichroic filter 310, so that the second passage 220 is perpendicular to the main body of the optical path 200, and the visible light reflected at 90° can enter the second passage 220 and finally enter the RGB camera 440 installed in the second passage 220. Of course, the angle of the second passage 220 can be adjusted according to the installation angle of the first dichroic filter 310, and is not particularly limited here as long as it can ensure that the visible light can enter the RGB camera 440.

[0026] In this embodiment, the RGB camera 440 includes a second lens group 443, an infrared cut filter 442, and a visible light sensitive substrate 441. The first dichroic filter 310 can transmit infrared light, but it is difficult to achieve 100% transmittance, so the reflected light still contains some infrared light, so the light reflected by the first dichroic filter 310 needs to be filtered out by the infrared cut filter 442 after passing through the second lens group 443, and only the visible light is incident on the visible light sensitive substrate 441, and a photoelectric image signal is generated by a contact image sensor (CIS) chip installed in the visible light sensitive substrate 441, and the photoelectric image signal is sent to the main control chip of the terminal for subsequent imaging processing, which will not be described in detail here.

[0027] In addition, since the process of three-dimensional recognition includes infrared light transmission and infrared light reception, an infrared shielding wall 320 may be installed in the first passage 210, thereby dividing the first passage 210 to form a third passage 230 and a fourth passage 240, and the infrared transmission unit and the infrared camera may be installed in the third passage 230 and the fourth passage 240, respectively. In this embodiment, an example is described in which the infrared transmission unit is installed in the third passage 230 and the first infrared camera 430 is installed in the fourth passage 240. In addition, since the first dichroic filter 310 can transmit infrared light, the infrared shielding wall 320 may be installed in a direction perpendicular to the display screen 120 as shown in FIG. 1, and in order to avoid mutual interference between the transmitted and received infrared light, one end of the infrared shielding wall 320 is closely connected to the first dichroic filter 310, and the other end is closely connected to the bottom side of the optical path 200. The specific position of the infrared blocking wall 320 in the first passage 210 may be adjusted according to the size of the infrared transmitting unit and the first infrared camera 430, as long as a compact layout in the first passage 210 can be realized.

[0028] In addition, in this embodiment, in order to simultaneously realize the three-dimensional recognition of structured light and the three-dimensional recognition of TOF, the embodiment takes as an example that the first emitter is an infrared dot projector 410 and the second emitter is an infrared emitter 420. The infrared dot projector 410 includes a first vertical cavity surface emitting laser (VCSEL) 411, a wafer level optical lens (WLO) 412 and a diffractive optical element (DOE) 413, which are sequentially installed in the direction of infrared light emission. The first VCSEL 411 may use a high-power VCSEL, and the emitted infrared laser is calibrated by the WLO 412 and then modulated and scattered by the DOE 413 to form a speckle pattern. The infrared emitter 420 includes a second VCSEL 421 and a diffuser 422, the second VCSEL 421 may be a low-power VCSEL, and the infrared light is transmitted from the second VCSEL 421 and expanded by the first diffuser 422 to achieve the effect of a low-power flood illuminator. The specific device parameters and specifications of the infrared dot projector 410 and the infrared emitter 420 may be selected according to actual needs.

[0029] In order to avoid mutual interference of the infrared light of the infrared dot projector 410 and the infrared emitter 420, a first reflecting mirror 330 may be installed in the third passage 230 to partition the third passage 320 to form the first extending passage 231 and the second extending passage 232 shown in FIG. 1. In this embodiment, the infrared dot projector 410 is installed in the first extending passage 231, and the infrared emitter 420 is installed in the second extending passage 232. Of course, the infrared dot projector 410 may be installed in the second extending passage 232, and the infrared emitter 420 may be installed in the first extending passage 231, which will not be described in detail here. In this embodiment, the first extending passage 231 and the second extending passage 232 are perpendicular to each other, and the method shown in FIG. 1 may be adopted to obtain the first extending passage 231 by extending vertically from one side of the first passage 210. Of course, the first extending path 231 and the second extending path 232 may have other angles as long as the infrared light can be emitted under the reflection effect of the first reflecting mirror 330.

[0030] The first reflecting mirror 330 may use an infrared total reflection lens or a time-division reflecting mirror. If the first reflecting mirror 330 is an infrared total reflection lens, the infrared dot projector 410 and the infrared emitter 420 can realize simultaneous operation. The angle between the first reflecting mirror 330 and the first extending path 231 is greater than the critical angle of the first reflecting mirror 330, so that the infrared light emitted from the infrared dot projector 410 has an included angle with the mirror surface greater than the critical angle, and is thereby all reflected to the first dichroic filter 310. Conversely, the included angle between the infrared light emitted by the infrared emitter 420 and the mirror surface is smaller than the critical angle, and the infrared light is transmitted to the first dichroic filter 310, so that the infrared light transmitted by the two elements installed in different extending paths can propagate to the first dichroic filter 310. In addition, in the process of transmitting through the lens, due to two slight refractions when entering and exiting, the infrared light remains parallel and does not affect the overall transmission effect. When the first reflecting mirror 330 is a time-division reflecting mirror, the time-division operation of the infrared dot projector 410 and the infrared emitter 420 can be realized by mechanically controlling the deployment state of the first reflecting mirror 330. When the operation of the infrared dot projector 410 is required, the reflecting mirror is mechanically controlled to lower, blocking the infrared light transmitted by the infrared emitter 420 and reflecting the infrared light transmitted by the infrared dot projector 410 to the first dichroic filter 310. When the operation of the infrared emitter 420 is required, the reflecting mirror is mechanically controlled to raise, blocking the infrared light transmitted by the infrared dot projector 410, so that the infrared light transmitted by the infrared emitter 420 can transmit through the first reflecting mirror 330 and propagate to the first dichroic filter 310. In order to avoid mutual interference of infrared light, the infrared dot projector 410 and the infrared emitter 420 need to adopt different infrared frequency bands. In this case, instead of the infrared total reflection lens, an infrared filter that can precisely control the transmission and blocking of different infrared spectrums can be used, and can be selected according to actual needs.

[0031] In the fourth passage 240, the first infrared camera 430 includes a first infrared photosensitive substrate 441, a first infrared low-pass filter 442 and a first lens group 443. The specification parameters of the first lens group 443 may be selected according to actual needs. The first infrared low-pass filter 442 may use an infrared low-pass filter or an infrared narrow-band interference color filter, as long as it only passes near-infrared light of a specific wavelength band. The first infrared photosensitive substrate 441 may be a CIS photosensitive substrate. The first infrared camera 430 receives the infrared light transmitted from the infrared emitter 420 and the infrared dot projector 410 and reflected by the target, and the infrared light is converted into a photoelectric signal in the first infrared photosensitive substrate 441, which can be further processed by the main control chip, and the specific photosensitive principle is not described in detail here. In this embodiment, the second VCSEL is a low-power VCSEL, and the transmitted low-power unstructured infrared light signal is mainly used for the prediction accuracy of the target, for example, the two-dimensional infrared information can be analyzed by an artificial intelligence algorithm to perform target recognition, and after the prediction is successful, an infrared speckle pattern is emitted by the infrared dot projector 410, which is reflected by the target object and received by the first infrared camera 430, thereby obtaining the depth information of the target object. The depth information obtained by the first infrared camera 420 and the two-dimensional information of the target object obtained by the RGB camera 440 can be combined to obtain red, green, and blue depth (RGBD) information, and since the RGBD information is provided, three-dimensional image recognition and three-dimensional image reconstruction can be effectively performed, and the specific image recognition method is not described in detail in this embodiment.

[0032] Example 2 As shown in Fig. 2, in this embodiment, the structure and principle of each member are almost the same as those in the first embodiment, and will not be described again here. The main difference is the layout of the first passage 210. In the first embodiment, as shown in Fig. 1, the first passage 210 is a vertical passage, and can be partitioned by the infrared blocking wall 320 to form the second extension passage 232 and the fourth passage 240 in the vertical direction, and the infrared emitter 420 and the first infrared camera 430 can be installed toward the lens 140, and the infrared light emitted from the infrared emitter 420 can pass through the first dichroic filter 310 and directly exit to the lens 140, and the first infrared camera 430 can receive the infrared light directly incident on the lens 140. Meanwhile, in this embodiment, the layout of the first passage 210 is as follows:

[0033] The third passage 230 has an F-shaped cross section, a first infrared reflection mirror 340 is installed at the corner, the relative vertical relationship between the first extension passage 231 and the second extension passage 232 remains unchanged, the infrared light transmitted by the infrared emitter 420 passes through the first reflection mirror 330 and enters the first infrared reflection mirror 340, the infrared light transmitted by the infrared dot projector 410 is specularly reflected by the first reflection mirror 330 and enters the first infrared reflection mirror 340, and the infrared light incident on the first infrared reflection mirror 340 is specularly reflected, thereby entering the first dichroic filter 310. The other principles are the same as those in the first embodiment, and a duplicated description will be omitted here.

[0034] The fourth passage 240 has an L-shaped cross section, with a second infrared reflecting mirror 350 installed at a corner, and infrared light incident from the lens 140 and transmitted through the first dichroic filter 310 is reflected by the second infrared reflecting mirror 350 to the first infrared camera 430. Other principles are the same as in the first embodiment, so repeated explanations will be omitted here. In addition, the technical principle of this embodiment is similar to that of embodiment 1. Due to the above-mentioned differences, compared with embodiment 1, the depth direction of the optical path 200 is effectively reduced, so that the structural layout of the optical path 200 is more compact. When applied to a terminal, the occupied thickness is reduced, which is favorable for reducing the thickness of the terminal.

[0035] Example 3 As shown in Fig. 3, in this embodiment, the structure and principle of the three-dimensional recognition device are almost the same as those in the second embodiment, and the thickness of the terminal can be effectively reduced. The main difference between this embodiment and the second embodiment is that in this embodiment, the first reflecting mirror 330 and the infrared dot projector 410 are omitted, so that the third passage 230 in this embodiment does not need to separately install the first extension passage 231, and the cross section of the third passage 230 is L-shaped, and the infrared light transmitted from the infrared emitter 420 is directly reflected to the first dichroic filter 310 by the first infrared reflecting mirror 340.

[0036] In addition, since the infrared dot projector 410 is omitted, the second VCSEL 421 of the infrared emitter 420 in this embodiment uses a high-power VCSEL to form a high-power flood illuminator, and the infrared light from the high-power VCSEL is expanded by the diffuser 422 to transmit the TOF surface emission light, so as to realize the imaging of a three-dimensional image in one go, and the specific imaging method is not described in detail here. In addition, since the infrared light emitted from the infrared emitter 420 is different from that in the second embodiment, the internal structure of the first infrared camera 430 may also be adjusted accordingly, as long as a three-dimensional image can be obtained when receiving the infrared light reflected from the target.

[0037] Except for the above differences, the rest of the structure of this embodiment is the same as that of the second embodiment, and for the sake of simplicity, the description will not be repeated here.

[0038] Example 4 In this embodiment, the structure and principle of the three-dimensional recognition device are almost the same as those in the first embodiment, and the main differences are as follows.

[0039] As shown in FIG. 4, the second passage 220 has a T-shaped cross section and is partitioned by the second dichroic filter 360 to form the third extension passage 221 and the fourth extension passage 222, the second infrared camera 450 is installed in the fourth extension passage 222, and the RGB camera 440 is installed in the third extension passage 221. The second dichroic filter 360 may adopt the same material and structure as the first dichroic filter 310, and will not be described here repeatedly. After the light enters the lens 140, most of the light reflected by the first dichroic filter 310 is visible light, but there is still a small amount of infrared light. After entering the second dichroic filter 360, the infrared light is transmitted and enters the second infrared camera 450, and the visible light is reflected to the RGB camera 440. By introducing the second infrared camera 450, a binocular camera for structured light can be formed together with the first infrared camera 430, and the complex calibration of the infrared dot projector 410 can be omitted, and the collective feature information of the object surface can be increased by acquiring the structured light by the second infrared camera 450, and the problem of weak texture and repeated texture areas being difficult to match in binocular stereoscopic vision can be avoided. In addition, in the scene of a monocular camera, the coded spots emitted from the infrared dot projector 410 are easily buried in sunlight, and the binocular camera using structured light can measure depth information using structured light in an indoor environment, and switch to a pure binocular mode when outdoor light causes the failure of the structured light, increasing the collective features of the object surface, and further improving reliability and anti-interference.

[0040] The second infrared camera 450 includes a second infrared-sensitive substrate 451, a second infrared low-pass filter 452 and a third lens group 453, and has a structure similar to that of the first infrared camera 430, and for ease of explanation, will not be repeated here.

[0041] Except for the above differences, the other structure and principles of the three-dimensional recognition device of this embodiment are the same as those of the first embodiment, and for the sake of simplicity, they will not be described here repeatedly.

[0042] It should be noted that in the first to fourth embodiments, the three-dimensional recognition device is installed inside the display screen 120, thereby reducing the area of ​​the enhanced light-transmittance region 130 and improving the user experience, which is applicable to the scene using the under-display front camera. Of course, for a terminal that does not use an under-display front camera, the three-dimensional recognition device of the first to fourth embodiments may be installed in the "notch" region of the display screen 120, thereby realizing the effect of reducing the area of ​​the "notch" region, reducing the occupation of the display screen area by the shooting unit, and improving the user experience.

[0043] The present application further provides a terminal, as shown in FIG. 5 , the terminal 10 includes a three-dimensional perception device described in the above embodiment and a display screen 120, where the three-dimensional perception device is installed inside the display screen 120, and the area of ​​the display screen 120 corresponding to the lens 140 of the three-dimensional perception device is an enhanced light-transmittance area 130.

[0044] In addition, as shown in FIG. 1, a TP cover plate 110 may be further installed on the surface of the display screen 120, so that from the outside to the inside are the TP cover plate 110, the display screen 120, the light-transmitting enhanced area 130 and the lens 140, respectively, and a three-dimensional recognition device described in any one of the above embodiments is installed in the optical path 200 corresponding to the lens 140.

[0045] In addition, in the conventional scheme in which the three-dimensional recognition element and the RGB camera are arranged side by side, the RGB camera and the three-dimensional recognition element are respectively arranged with lenses, so that the area of ​​the light-transmitting reinforcement area 130 is large and has a "notch" shape, which impairs the user experience. According to the three-dimensional recognition device of this embodiment, the three-dimensional recognition unit and the RGB lens 440 are arranged in the same light path 200, and the sharing of one lens 140 is realized by controlling the propagation paths of infrared light and visible light, and only one lens 140 is arranged at the front end of the terminal 10, and the corresponding area is subjected to a special light-transmitting treatment, which greatly reduces the area of ​​the light-transmitting reinforcement area 130, thereby reducing the display area affected by the light-transmitting reinforcement area 130, and improving the full-screen display effect of the terminal, thereby improving the user experience.

[0046] In addition, as shown in FIG. 6, an embodiment of the present application further provides an image enhancement method applied to a three-dimensional recognition device, and the three-dimensional recognition device includes a lens 140 and an optical path 200, as shown in FIG. 1 or FIG. 2, the optical path 200 is installed inside the lens 140, and an RGB camera 440, an infrared emitter 420, an infrared dot projector 410, a first infrared camera 430 and a first dichroic filter 310 are installed in the optical path 200, the visible light in the ambient light is reflected to the RGB camera 440 by the first dichroic filter 310, the infrared light in the ambient light passes through the first dichroic filter 310 to reach the first infrared camera 430, and the infrared light transmitted by the infrared emitter 420 and the infrared dot projector 410 passes through the first dichroic filter 310 to reach the lens 140.

[0047] The image enhancement method includes, but is not limited to, the following steps.

[0048] Step S610: Obtain a three-dimensional point cloud image and a two-dimensional image, where the three-dimensional point cloud image is obtained from the target reflected light by a first infrared camera, and the two-dimensional image is obtained from the target reflected light by an RGB camera, and the target reflected light is the laser speckle transmitted by an infrared dot projector reflected by a target object.

[0049] Step S620: A three-dimensional structured light point cloud is obtained from the three-dimensional point cloud image, and the three-dimensional structured light point cloud is synchronized with the two-dimensional image to obtain a structured light difference value.

[0050] Step S630: Determine RGB reference data according to the two-dimensional image, and perform an interpolation process on the reference surface according to the RGB reference data and the structured light difference value to increase the density of the reference surface.

[0051] Step S640: Obtain a 3D point cloud layer plane of the 3D point cloud image, and calibrate the 3D point cloud layer plane and the reference plane.

[0052] Step S650: Stereo matching is performed between the 3D point cloud layer surface and the reference plane to obtain RGBD depth information.

[0053] It should be noted that when the three-dimensional recognition device adopts the structure of the embodiment shown in Figures 1 and 2, in the single lens, the third passage 230 and the fourth passage 240 are close to each other, and the baseline is almost zero. In this case, the incident angle of the laser speckle emitted by the infrared dot projector 410 is small, which results in a large density error of the calibration reference plane, and increases the noise in the depth calculation process, affecting the accuracy of the measured depth information to a certain extent. In order to realize three-dimensional recognition, the two-dimensional image captured by the RGB camera 440 needs to be synchronized with the three-dimensional point cloud image captured by the first infrared camera 430 to perform the enhancement calculation, thereby increasing the density of the reference plane and ensuring the improvement of the accuracy of stereo matching.

[0054] In addition, since the RGB camera is larger in size than the structured light lens, the amount of incoming light and the imaging radius are correspondingly larger. By synchronizing the three-dimensional structured light point cloud with the two-dimensional image after parameter calibration, the difference between the transmission and reception of structured light can be increased when the baseline is close to zero, thereby obtaining a structured light difference value.

[0055] It should be noted that laser speckle is generated by irradiating coherent light on a random scatterer, and is a random process, and it is necessary to use probability statistics methods to calculate the intensity distribution, contrast, speckle motion law, etc. of the speckle. Therefore, the laser speckle is highly random, and different patterns will occur with different distances, that is, the speckle patterns at any two locations in the same space are different. In this embodiment, the laser speckle transmitted through the infrared dot projector 410 is reflected by the target object, and then the first infrared camera 430 acquires a three-dimensional point cloud image, and according to the synchronization between the three-dimensional structured light point cloud of the three-dimensional point cloud image and the two-dimensional image, a structured light difference value is obtained, and then an interpolation process is performed on the reference plane according to the structured light difference value, thereby realizing the density increase of the reference plane on the two-dimensional plane, and then matching is performed to obtain RGBD information. Obtaining the three-dimensional structured light point cloud and the three-dimensional point cloud layer surface from the three-dimensional point cloud image is a well-known technique in this field, and will not be described in detail here.

[0056] In addition, after obtaining the reference plane after density increase, the 3D point cloud layer plane of each layer of the dot-projected real reflection layer plane can be matched with the reference plane to obtain stereo matching and obtain parallax, or the 3D point cloud layer plane can be merged with the RGB image to perform matching with the reference plane to obtain parallax, thereby further calculating RGBD depth information. The matching process can be realized by conventional matching cost, cost aggregation, parallax calculation and parallax refinement, and is not particularly limited here.

[0057] 6. Also, as shown in FIG. 7, in one embodiment, before performing step S620 in the embodiment shown in FIG. 6, the method further includes, but is not limited to, the following steps:

[0058] Step S710: Obtain an infrared reference heat map and a visible light reference map, and determine calibration parameters according to the infrared reference heat map and the visible light reference map, where the infrared reference heat map is collected by a first infrared camera, and the visible light reference map is collected by an RGB camera.

[0059] Step S720: Calibrate the 3D point cloud image and the 2D image according to the calibration parameters.

[0060] In addition, when obtaining the infrared reference heat map and the visible light reference map, a shape that is easy to see, such as a black and white chessboard pattern or a dot pattern, can be used as a standard plate, infrared light and visible light can be irradiated simultaneously, and images can be collected simultaneously by the first infrared camera and the RGB camera to obtain the infrared reference heat map and the visible light reference map, and a general calibration method can be adopted to realize the calibration of the first infrared camera and the RGB camera to obtain the calibration parameters. For example, the external parameters, internal parameters and distortion coefficients can be output by the Zhang Zhengyou calibration method, and the external parameters can include the previous positional relationship of the two cameras, such as the rotation matrix and the translation vector relative to the world coordinate system, and the internal parameters can be parameters associated with the first infrared camera and the RGB camera, such as the focal length and the principal point.

[0061] It should be noted that the process of calibrating the 3D point cloud image and the 2D image includes translation and rotation operations for each coordinate point. In this process, the 3D coordinates include three coordinate axes, XYZ, while the coordinates of the 2D image only have two coordinate axes, XY. Therefore, to avoid the loss of the Z-axis coordinate, homogeneous coordinates may be introduced into the conversion process, thereby realizing linear change. The specific calibration process will not be described in detail here.

[0062] Moreover, as shown in FIG. 8, an embodiment of the present application further provides a terminal, where the terminal 800 includes a memory 810, a processor 820, and a computer program stored in the memory 810 and executable on the processor 820.

[0063] The processor 820 and the memory 810 may be connected via a bus or in another manner.

[0064] Non-transitory software programs and instructions for implementing the image enhancement methods of the above embodiments are stored in memory 810 and, when executed by processor 820, perform the image enhancement methods of the above embodiments, for example, performing steps S610 to S650 of the method in FIG. 6 and steps S710 to S720 of the method in FIG. 7 described above.

[0065] The above device embodiment is merely schematic, and the units shown as separate components may or may not be physically separated, i.e., they may be located in one place or may be located separately in multiple network units. Some or all of these modules may be selected to achieve the purpose of the embodiment according to actual needs.

[0066] In addition, one embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor or controller, for example, by a processor in the above-mentioned terminal embodiment, the processor is caused to execute the image enhancement method in the above-mentioned embodiment, for example, executing steps S610 to S650 of the method in FIG. 6 described above, and steps S710 to S720 of the method in FIG. 7.

[0067] All or part of the steps in the methods and systems disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processor, digital signal processor, microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer readable medium, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (e.g., computer readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridge, magnetic tape, magnetic disk storage or other magnetic storage device, or any other medium that can be used to store the desired information and that can be accessed by a computer. Additionally, communication media typically includes computer readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transmission mechanism, and may include any information delivery media as known to those skilled in the art.

[0068] The embodiment of the present application includes a lens, and an optical path installed inside the lens, in which an RGB camera, a three-dimensional recognition unit and a first dichroic filter are installed, the first dichroic filter reflects visible light in the ambient light to the RGB camera, and transmits infrared light transmitted and received by the three-dimensional recognition unit. According to the technical solution of the embodiment, the RGB lens and the three-dimensional recognition unit can be installed simultaneously in the optical path of the lens, realizing the lens sharing of the front camera and the three-dimensional recognition unit, reducing the area specially treated to enhance light transmittance to the size of a single lens, effectively improving the display effect of the screen, and thereby improving the user experience.

[0069] Although several implementations of the present application have been specifically described above, the present application is not limited to the above embodiments, and a person skilled in the art may make various equivalent modifications or substitutions without departing from the spirit of the present application, and all of these equivalent modifications or substitutions are intended to be included within the scope defined by the claims of the present application.

Claims

1. A three-dimensional recognition device installed inside a display screen of a terminal, Lenses and An optical path installed inside the lens, in which a red, green, blue (RGB) camera, a three-dimensional recognition unit, and a first dichroic filter are installed, the first dichroic filter reflects visible light in the ambient light to the RGB camera, and further includes an optical path that transmits infrared light transmitted and received by the three-dimensional recognition unit. Three-dimensional recognition device.

2. The first dichroic filter divides the optical path to form a first passage and a second passage, the three-dimensional recognition unit is installed in the first passage, and the RGB camera is installed in the second passage. The three-dimensional recognition device according to claim 1.

3. an infrared ray blocking wall is installed in the first passage, the infrared ray blocking wall is connected to the first dichroic filter, and divides the first passage to form a third passage and a fourth passage; The three-dimensional recognition unit includes an infrared transmitting unit and a first infrared camera, the infrared transmitting unit is installed in the third passage, and the infrared light transmitted by the infrared transmitting unit passes through the first dichroic filter and reaches the lens, and the first infrared camera is installed in the fourth passage and receives the infrared light in the ambient light that passes through the first dichroic filter. The three-dimensional recognition device according to claim 2.

4. the third passage includes a first extension passage and a second extension passage, a first reflection mirror is installed between the first extension passage and the second extension passage, and the first reflection mirror is an infrared total reflection lens or a time division reflection mirror; the infrared transmitting unit includes a first emitter and a second emitter, the first emitter is installed in the first extending passage, the second emitter is installed in the second extending passage, the infrared light transmitted by the first emitter is reflected by the first reflecting mirror to the first dichroic filter, the infrared light transmitted by the second emitter is transmitted through the first reflecting mirror to reach the first dichroic filter, the first emitter is an infrared emitter and the second emitter is an infrared dot projector; Or, the first emitter is an infrared dot projector and the second emitter is an infrared emitter; The infrared dot projector includes a first vertical cavity surface emitting laser (VCSEL), a wafer level optical lens (WLO) and a diffractive optical element (DOE) in order in an infrared light emission direction, and the infrared emitter includes a second VCSEL and a diffusion plate in order in an infrared light emission direction; The three-dimensional recognition device according to claim 3.

5. a first infrared reflecting mirror is further installed in the third passage, and the first infrared reflecting mirror reflects the infrared light that has reached the first reflecting mirror by transmission or reflection to the first dichroic filter; The three-dimensional recognition device according to claim 4.

6. The infrared transmitting unit is an infrared emitter, and the infrared emitter includes a second VCSEL and a diffusion plate in order in an infrared light transmitting direction; The three-dimensional recognition device according to claim 3.

7. a first infrared reflecting mirror is disposed in the third passage, and the first infrared reflecting mirror reflects the infrared light transmitted by the infrared emitter to the first dichroic filter; The three-dimensional recognition device according to claim 6.

8. the first infrared camera includes a first lens group, a first infrared low-pass filter, and a first infrared-sensitive substrate, in that order, in the infrared light incident direction; The three-dimensional recognition device according to any one of claims 3 to 7.

9. a second infrared reflecting mirror is installed in the fourth passage, and the second infrared reflecting mirror reflects infrared light in the ambient light that has passed through the first dichroic filter to the first infrared camera; The three-dimensional recognition device according to claim 8.

10. The RGB camera includes a second lens group, an infrared media filter, and a visible light sensitive substrate in a visible light incident direction, in that order; The three-dimensional recognition device according to any one of claims 3 to 7.

11. the second passage includes a third extension passage and a fourth extension passage, the RGB camera is installed in the third extension passage, and a second infrared camera is installed in the fourth extension passage; a second dichroic filter is installed between the third extension path and the fourth extension path, and the second dichroic filter reflects the visible light reflected by the first dichroic filter to the RGB camera and transmits the infrared light reflected by the first dichroic filter to reach the second infrared camera; the second infrared camera includes a third lens group, a second infrared low-pass filter, and a second infrared-sensitive substrate, in that order, in the infrared light incident direction; The three-dimensional recognition device according to claim 10.

12. A terminal, A three-dimensional recognition device according to any one of claims 1 to 11, a display screen; The three-dimensional recognition device is installed inside the display screen, and an area of ​​the display screen corresponding to the lens of the three-dimensional recognition device is an enhanced light-transmitting area. Terminal.

13. An image enhancement method applied to a three-dimensional recognition device, comprising: The three-dimensional recognition device includes a lens and an optical path, the optical path is installed inside the lens, an RGB camera, an infrared emitter, an infrared dot projector, a first infrared camera and a first dichroic filter are installed in the optical path, visible light in the ambient light is reflected by the first dichroic filter to the RGB camera, infrared light in the ambient light passes through the first dichroic filter to reach the first infrared camera, and infrared light transmitted by the infrared emitter and the infrared dot projector passes through the first dichroic filter to reach the lens; The image enhancement method includes: acquiring a three-dimensional point cloud image and a two-dimensional image, the three-dimensional point cloud image being acquired from a target reflected light by the first infrared camera, the two-dimensional image being acquired from the target reflected light by the RGB camera, the target reflected light being laser speckle light transmitted by the infrared dot projector reflected by a target object; obtaining a three-dimensional structured light point cloud from the three-dimensional point cloud image, and synchronizing the three-dimensional structured light point cloud with the two-dimensional image to obtain a structured light difference value; determining RGB reference data according to the two-dimensional image; and performing an interpolation process on a reference surface according to the RGB reference data and the structured light difference value to increase density of the reference surface; obtaining a 3D point cloud layer surface of the 3D point cloud image, and calibrating the 3D point cloud layer surface and the reference surface; and performing stereo matching between the 3D point cloud layer surface and the reference surface to obtain red-green-blue depth (RGBD) information. Image enhancement methods.

14. Prior to the step of acquiring a three-dimensional structured light point cloud from the three-dimensional point cloud image, obtaining an infrared reference heat map and a visible light reference map, and determining calibration parameters according to the infrared reference heat map and the visible light reference map, wherein the infrared reference heat map is collected by the first infrared camera and the visible light reference map is collected by the RGB camera; calibrating the 3D point cloud image and the 2D image according to the calibration parameters. The method of claim 13.

15. A terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the image enhancement method according to any one of claims 13 to 14 when executing the computer program. Terminal.

16. A computer-readable storage medium having stored thereon computer-executable instructions for carrying out the image enhancement method according to any one of claims 13 to 14. A computer-readable storage medium.

Citation Information

Patent Citations

  • TOF optical sensing device, mobile terminal and image generation method

    CN112118371A

  • Information acquisition device and object detection device

    JP2015132729A

  • Three-dimensional lidar system for autonomous vehicle using dichroic mirror

    JP2019109219A

  • Device

    US20210248770A1