Three-dimensional recognition device and terminal

Abstract

The present application provides a three-dimensional recognition device, a terminal, an image enhancement method, and a storage medium. The three-dimensional recognition device includes a lens (140) and an optical path (200) installed inside the lens (140), in which an RGB camera (440), a three-dimensional recognition unit, and a first dichroic filter (310) are installed, and the first dichroic filter (310) reflects visible light in the ambient light to the RGB camera (440) and transmits infrared light transmitted and received by the three-dimensional recognition unit.

Description

[Technical Field]

[0001] This application is filed based on and claims priority from a Chinese patent application bearing application number 202111486807.7 and filing date December 7, 2021, the entire contents of which are incorporated herein by reference.

[0002] This application relates to the field of smart terminals, but is not limited thereto, and in particular to three-dimensional recognition devices. and end At the end Regarding. [Background technology]

[0003] Currently, there are two main methods for forward 3D recognition in mobile terminals: structured light and time-of-flight (TOF). To realize these two methods, 3D recognition elements such as infrared cameras and flood illuminators must be placed on the mobile terminal. Furthermore, forward RGB (Red, Green, Blue) cameras are usually juxtaposed between the 3D recognition elements. Meanwhile, the lens areas of these elements are relatively large and are usually placed on the top of the display screen, forming a "notch" area, which results in a smaller effective display area of ​​the screen.

[0004] To solve this problem, more and more smart devices are adopting under-display photography technology, placing a front RGB camera and 3D recognition device inside the display screen. To meet the light requirements of each device, the corresponding screen display area needs to be specially processed to increase light transmittance, for example, by reducing the RGB pixels or making the RGB pixels smaller to increase light transmittance. The display effect of the specially processed display area will be different from that of other display areas. If each device is currently arranged side by side, the area of ​​the specially processed area will be large, which will degrade the overall display effect and user experience. Summary of the Invention [Problem to be solved by the invention]

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The embodiment of the present application is a three-dimensional recognition device and end The end provide. [Means for solving the problem]

[0007] In a first aspect, an embodiment of the present application provides a three-dimensional recognition device installed inside a display screen of a terminal, the three-dimensional recognition device including: a lens; and an optical path installed inside the lens, in which an RGB camera, a three-dimensional recognition unit, and a first dichroic filter are installed, the first dichroic filter reflecting visible light in ambient light to the RGB camera and further transmitting infrared light transmitted and received by the three-dimensional recognition unit.

[0008] In a second aspect, an embodiment of the present application provides a terminal including a three-dimensional recognition device according to the first aspect and a display screen, wherein the three-dimensional recognition device is installed inside the display screen, and an area of ​​the display screen corresponding to the lens of the three-dimensional recognition device is an enhanced light-transmittance area.

[0009] In a third aspect, an embodiment of the present application provides an image enhancement method applied to a three-dimensional recognition device, the three-dimensional recognition device including a lens and an optical path, the optical path being installed inside the lens, and an RGB camera, an infrared emitter, an infrared dot projector, a first infrared camera, and a first dichroic filter being installed in the optical path, visible light in ambient light is reflected by the first dichroic filter to the RGB camera, infrared light in the ambient light passes through the first dichroic filter to reach the first infrared camera, and infrared light transmitted by the infrared emitter and the infrared dot projector passes through the first dichroic filter to reach the lens. The image enhancement method includes the steps of: acquiring a three-dimensional point cloud image and a two-dimensional image, wherein the three-dimensional point cloud image is acquired from a target reflected light by the first infrared camera, and the two-dimensional image is acquired from the target reflected light by the RGB camera, and the target reflected light is light of laser speckles transmitted by the infrared dot projector reflected by a target object; acquiring a three-dimensional structured light point cloud from the three-dimensional point cloud image, and synchronizing the three-dimensional structured light point cloud with the two-dimensional image to obtain a structured light difference value; determining RGB reference data according to the two-dimensional image, and performing interpolation on a reference plane according to the RGB reference data and the structured light difference value to increase density of the reference plane; acquiring a three-dimensional point cloud layer surface of the three-dimensional point cloud image, and calibrating the three-dimensional point cloud layer surface and the reference plane; and performing stereo matching between the three-dimensional point cloud layer surface and the reference plane to obtain RGBD depth information.

[0010] In a fourth aspect, an embodiment of the present application provides a terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the image enhancement method of the third aspect when it executes the computer program.

[0011] In a fifth aspect, embodiments of the present application provide a computer-readable storage medium having stored thereon computer-executable instructions for performing the image enhancement method according to the third aspect.

[0012] Other features and advantages of the present application will be set forth in the following specification, and in part will be apparent from the specification, or may be learned by the practice of the present application. The objectives and other advantages of the present application will be achieved and obtained by the structures particularly pointed out in the specification, claims and drawings. The drawings are used to provide a further understanding of the technical solution of the present application, constitute a part of the specification, and are used to interpret the technical solution of the present application together with the examples of the present application, but are not intended to limit the technical solution of the present application. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a structural diagram of a three-dimensional recognition device according to a first embodiment of the present invention. [Figure 2] FIG. 10 is a structural diagram of a three-dimensional recognition device according to a second embodiment of the present invention. [Figure 3] FIG. 10 is a structural diagram of a three-dimensional recognition device according to a third embodiment of the present invention. [Figure 4] FIG. 10 is a structural diagram of a three-dimensional recognition device according to a fourth embodiment of the present invention. [Figure 5] FIG. 10 is a structural diagram of a terminal according to another embodiment of the present application; [Figure 6] 4 is a flowchart of an image enhancement method according to another embodiment of the present application; [Figure 7] 10 is a flowchart of image calibration according to another embodiment of the present application. [Figure 8] FIG. 10 is a diagram illustrating a device of a terminal according to another embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0014] In order to clarify the purpose, technical solution and advantages of the present application, the present application will be described in more detail below with reference to the drawings and examples. The specific examples described in this specification are only used to interpret the present application and are not used to limit the present application.

[0015] Although the schematic diagram of the device is divided into functional modules and the flowchart shows a logical order, in some cases the division into modules within the device may differ, or the steps shown or described may be performed in a different order from that of the flowchart. The terms "first," "second," etc. in the specification, claims, or above drawings are used to distinguish between similar objects, and are not intended to describe a particular order or priority.

[0016] This application provides a 3D recognition device, a terminal, an image enhancement method, and a storage medium. The 3D recognition device includes a lens and an optical path installed inside the lens, where an RGB camera, a 3D recognition unit, and a first dichroic filter are installed in the optical path, and the first dichroic filter reflects visible light in ambient light to the RGB camera and transmits infrared light transmitted and received by the 3D recognition unit. According to the technical solution of this embodiment, an RGB lens and a 3D recognition unit can be installed simultaneously in the optical path of the lens, realizing a shared lens for the front camera and the 3D recognition unit, and reducing the size of a specially treated area for enhanced light transmittance to a single lens, effectively improving the display effect on the screen and thereby improving the user experience.

[0017] Hereinafter, the embodiments of the present application will be further described with reference to the drawings.

[0018] The present application provides a three-dimensional recognition device installed inside a display screen of a terminal, the three-dimensional recognition device comprising: Lens 140 and The optical path 200 is installed inside the lens 140, and an RGB camera 440, a three-dimensional recognition unit, and a first dichroic filter 310 are installed within the optical path 200, and the first dichroic filter 310 reflects visible light in the ambient light to the RGB camera 440, and further includes the optical path 200 that transmits infrared light transmitted and received by the three-dimensional recognition unit.

[0019] The lens 140 is installed inside the display screen 120 of the terminal and is adjacent to the enhanced light-transmitting area 130 of the display screen 120. The specific parameters and type of the lens 140 are not particularly limited as long as they are applicable to under-display photography.

[0020] In order to realize the shared use of the same lens 140 for photography and 3D recognition, an RGB camera 440 and a 3D recognition unit must be installed simultaneously in the optical path 200 inside the lens 140. The RGB camera 440 realizes photography and uses visible light as its light source, while the 3D recognition unit is mainly a structured light or TOF unit and receives or transmits infrared light. Therefore, when the display screen 130 is placed horizontally, a first dichroic filter 310 can be installed in the vertical optical path 200 inside the lens 140 to transmit infrared light and reflect visible light, thereby distinguishing the propagation paths of visible light and infrared light and satisfying the optical propagation path requirements of the RGB camera 440 and the 3D recognition unit in the same optical path 200.

[0021] The first dichroic filter 310 may be a visible / infrared filter film. When light is incident at a 45° angle, the visible / infrared filter film has a reflectance of over 90% for the visible light wavelength range of 0.3 microns to 0.6 microns and a transmittance of over 90% for near-infrared light from 0.75 microns to 2.5 microns. This allows infrared light to pass through the visible / infrared filter film, but causes visible light to be specularly reflected by the visible / infrared filter film, thereby separating the propagation directions of infrared light and visible light. The specific installation angle of the first dichroic filter 310 can be adjusted according to the actual layout of the optical path 200 and is not particularly limited in this embodiment.

[0022] In addition, the first dichroic filter 310 realizes different propagation paths for infrared light and visible light, allowing the RGB camera 440 and the 3D recognition unit installed in the optical path 200 to operate normally, thereby realizing the 3D recognition unit and the RGB camera 440 sharing one lens 140. Compared with a scheme in which the 3D recognition unit and the RGB camera are arranged side by side at the front edge of the display screen, the enhanced light-transmitting area 130 only needs to be installed for one lens 140, which significantly reduces the area and effectively improves the user experience.

[0023] Various embodiments of the three-dimensional recognition device will be described below by way of several examples.

[0024] 1 to 4 are structural diagrams of four embodiments of the three-dimensional recognition device according to the present application. All four diagrams are cut vertically when the terminal is placed horizontally with the front camera facing upwards, and will not be described again later.

[0025] Example 1 1 , when a first dichroic filter 310 is installed in the optical path 200, the first dichroic filter 310 divides the optical path 200 into a first passage 210 and a second passage 220. Based on the above description, when the first dichroic filter 310 is installed at a 45° angle, visible light is specularly reflected by the first dichroic filter 310, and the second passage 220 is perpendicular to the main body of the optical path 200. The visible light reflected at 90° can enter the second passage 220 and finally enter the RGB camera 440 installed in the second passage 220. Of course, the angle of the second passage 220 can be adjusted according to the installation angle of the first dichroic filter 310, and is not particularly limited herein as long as it ensures that visible light can enter the RGB camera 440.

[0026] In this embodiment, the RGB camera 440 includes a second lens group 443, an infrared cutoff filter 442, and a visible light sensitive substrate 441. Although the first dichroic filter 310 can transmit infrared light, it is difficult to achieve 100% transmittance, so the reflected light still contains some infrared light. Therefore, the light reflected by the first dichroic filter 310 passes through the second lens group 443 and then needs to be filtered out by the infrared cutoff filter 442, so that only visible light is incident on the visible light sensitive substrate 441, and a photoelectric image signal is generated by a contact image sensor (CIS) chip installed in the visible light sensitive substrate 441, and the photoelectric image signal is sent to the main control chip of the terminal for subsequent imaging processing, which will not be described in detail here.

[0027] Furthermore, since the 3D recognition process includes infrared light transmission and infrared light reception, an infrared shielding wall 320 may be installed in the first passage 210 to divide the first passage 210 into a third passage 230 and a fourth passage 240, and the infrared transmitting unit and the infrared camera may be installed in the third passage 230 and the fourth passage 240, respectively. In this embodiment, an example is described in which the infrared transmitting unit is installed in the third passage 230 and the first infrared camera 430 is installed in the fourth passage 240. Note that, since the first dichroic filter 310 can transmit infrared light, the infrared shielding wall 320 may be installed perpendicular to the display screen 120 as shown in FIG. 1 . To avoid mutual interference between the transmitted and received infrared light, one end of the infrared shielding wall 320 is closely connected to the first dichroic filter 310 and the other end is closely connected to the bottom side of the optical path 200. The specific position of the infrared blocking wall 320 in the first passage 210 may be adjusted according to the size of the infrared transmitting unit and the first infrared camera 430, as long as a compact layout can be achieved in the first passage 210.

[0028] In addition, in this embodiment, in order to simultaneously realize three-dimensional recognition using structured light and three-dimensional recognition using TOF, this embodiment takes as an example that the first emitter is an infrared dot projector 410 and the second emitter is an infrared emitter 420. The infrared dot projector 410 includes a first vertical cavity surface emitting laser (VCSEL) 411, a wafer level optical lens (WLO) 412, and a diffractive optical element (DOE) 413, which are sequentially arranged in the direction of infrared light emission. The first VCSEL 411 may be a high-power VCSEL, and the emitted infrared laser is calibrated by the WLO 412 and then modulated and scattered by the DOE 413 to form a speckle pattern. The infrared emitter 420 includes a second VCSEL 421 and a diffuser 422. The second VCSEL 421 may be a low-power VCSEL, and the infrared light emitted from the second VCSEL 421 is expanded by the first diffuser 422 to achieve the effect of a low-power flood illuminator. The specific device parameters and specifications of the infrared dot projector 410 and the infrared emitter 420 may be selected according to actual needs.

[0029] To avoid mutual interference between the infrared light from the infrared dot projector 410 and the infrared emitter 420, a first reflecting mirror 330 may be installed in the third passage 230 to divide the third passage 320 into the first extending passage 231 and the second extending passage 232 as shown in FIG. 1 . In this embodiment, the infrared dot projector 410 is installed in the first extending passage 231, and the infrared emitter 420 is installed in the second extending passage 232. Of course, the infrared dot projector 410 may be installed in the second extending passage 232, and the infrared emitter 420 may be installed in the first extending passage 231, which will not be described in detail here. In this embodiment, the first extending passage 231 and the second extending passage 232 are perpendicular to each other, and the first extending passage 231 may be obtained by vertically extending from one side of the first passage 210 as shown in FIG. 1 . Of course, the first extending path 231 and the second extending path 232 may have other angles as long as the infrared light can be emitted under the reflection effect of the first reflecting mirror 330.

[0030] The first reflecting mirror 330 may be an infrared total reflection lens or a time-division reflecting mirror. If the first reflecting mirror 330 is an infrared total reflection lens, the infrared dot projector 410 and the infrared emitter 420 can operate simultaneously. The angle between the first reflecting mirror 330 and the first extending path 231 is greater than the critical angle of the first reflecting mirror 330, so that the included angle between the infrared dot projector 410 and the mirror surface is greater than the critical angle, and all of the infrared light is reflected by the first dichroic filter 310. Conversely, the included angle between the infrared light emitted by the infrared emitter 420 and the mirror surface is smaller than the critical angle, so that the infrared light is transmitted through the first dichroic filter 310, allowing the infrared light transmitted by the two elements installed in different extending paths to propagate to the first dichroic filter 310. Furthermore, during the process of transmitting through the lens, the infrared light remains parallel due to two slight refractions upon entering and exiting, and does not affect the overall transmission effect. When the first reflecting mirror 330 is a time-sequential reflecting mirror, the deployment state of the first reflecting mirror 330 can be mechanically controlled to realize time-sequential operation of the infrared dot projector 410 and the infrared emitter 420. When the infrared dot projector 410 needs to operate, the reflecting mirror is mechanically controlled to lower, blocking the infrared light transmitted by the infrared emitter 420 and reflecting the infrared light transmitted by the infrared dot projector 410 to the first dichroic filter 310. When the infrared emitter 420 needs to operate, the reflecting mirror is mechanically controlled to raise, blocking the infrared light transmitted by the infrared dot projector 410, allowing the infrared light transmitted by the infrared emitter 420 to transmit through the first reflecting mirror 330 and propagate to the first dichroic filter 310. In order to avoid mutual interference of infrared light, the infrared dot projector 410 and the infrared emitter 420 need to adopt different infrared frequency bands. In this case, instead of the infrared total reflection lens, an infrared filter can be used that can precisely control the transmission and blocking of different infrared spectrums, and can be selected according to actual needs.

[0031] In the fourth path 240, the first infrared camera 430 includes a first infrared-sensitive substrate 441, a first infrared low-pass filter 442, and a first lens group 443. The specifications of the first lens group 443 can be selected according to actual needs. The first infrared low-pass filter 442 can be an infrared low-pass filter or an infrared narrow-band interference color filter, as long as it only passes near-infrared light of a specific wavelength band. The first infrared-sensitive substrate 441 can be a CIS photosensitive substrate. The first infrared camera 430 receives the infrared light transmitted from the infrared emitter 420 and the infrared dot projector 410 and reflected by the target. The infrared light is converted into a photoelectric signal in the first infrared-sensitive substrate 441, which can be further processed by the main control chip. The specific photoelectric principle is not described in detail here. In this embodiment, the second VCSEL is a low-power VCSEL, and the transmitted low-power unstructured infrared optical signal is mainly used for target prediction accuracy. For example, an artificial intelligence algorithm can analyze the two-dimensional infrared information to perform target recognition. After successful prediction, an infrared speckle pattern is emitted by the infrared dot projector 410, which is reflected by the target object and received by the first infrared camera 430, thereby obtaining depth information of the target object. The depth information obtained by the first infrared camera 420 can be combined with the two-dimensional information of the target object obtained by the RGB camera 440 to obtain red, green, and blue depth (RGBD) information. With the RGBD information, three-dimensional image recognition and three-dimensional image reconstruction can be effectively performed. The specific image recognition method will not be described in detail in this embodiment.

[0032] Example 2 As shown in FIG. 2, the structure and principles of each component in this embodiment are substantially the same as those in the first embodiment, and will not be described again here. The main difference is the layout of the first passage 210. In the first embodiment, as shown in FIG. 1, the first passage 210 is a vertical passage, and can be partitioned by an infrared-shielding wall 320 to form a second elongated passage 232 and a fourth passage 240 in the vertical direction. The infrared emitter 420 and the first infrared camera 430 can be installed facing the lens 140. The infrared light emitted from the infrared emitter 420 can pass through the first dichroic filter 310 and directly exit to the lens 140, and the first infrared camera 430 can receive the infrared light directly incident on the lens 140. Meanwhile, in this embodiment, the layout of the first passage 210 is as follows:

[0033] The third passage 230 has an F-shaped cross section, and a first infrared reflecting mirror 340 is installed at a corner thereof, the relative vertical relationship between the first extension passage 231 and the second extension passage 232 remains unchanged, the infrared light transmitted by the infrared emitter 420 passes through the first reflecting mirror 330 and enters the first infrared reflecting mirror 340, the infrared light transmitted by the infrared dot projector 410 is specularly reflected by the first reflecting mirror 330 and enters the first infrared reflecting mirror 340, and the infrared light incident on the first infrared reflecting mirror 340 is specularly reflected and thereby enters the first dichroic filter 310. The other principles are the same as those in the first embodiment, and a duplicated description will be omitted here.

[0034] The fourth passage 240 has an L-shaped cross section, and a second infrared reflecting mirror 350 is installed at a corner thereof, and infrared light incident from the lens 140 and transmitted through the first dichroic filter 310 is reflected by the second infrared reflecting mirror 350 to the first infrared camera 430. Other principles are the same as those in the first embodiment, and therefore a repeated explanation will be omitted here. It should be noted that the technical principle of this embodiment is similar to that of embodiment 1, and due to the above-mentioned differences, the depth direction of the optical path 200 is effectively reduced compared to embodiment 1, thereby making the structural layout of the optical path 200 more compact. When applied to a terminal, the occupied thickness is reduced, which is advantageous for reducing the thickness of the terminal.

[0035] Example 3 As shown in Figure 3, the structure and principle of the 3D recognition device in this embodiment are similar to those in the second embodiment, and the thickness of the terminal can be effectively reduced. The main difference between this embodiment and the second embodiment is that the first reflecting mirror 330 and the infrared dot projector 410 are omitted in this embodiment, so that the third passage 230 in this embodiment does not require the first extending passage 231 to be separately installed. The cross section of the third passage 230 is L-shaped, and the infrared light transmitted from the infrared emitter 420 is directly reflected by the first infrared reflecting mirror 340 to the first dichroic filter 310.

[0036] In addition, since the infrared dot projector 410 is omitted, in order to realize the application of a single lens in the TOF scheme, the second VCSEL 421 of the infrared emitter 420 in this embodiment uses a high-power VCSEL, thereby forming a high-power flood illuminator, and the infrared light from the high-power VCSEL is expanded by a diffuser 422 to transmit the TOF surface-emission light, thereby realizing one-time imaging of a 3D image; the specific imaging method will not be described in detail here. In addition, since the infrared light emitted from the infrared emitter 420 is different from that in the second embodiment, the internal structure of the first infrared camera 430 may be adjusted accordingly, as long as it can obtain a 3D image when receiving the infrared light reflected from the target.

[0037] Except for the above differences, the other structures of this embodiment are the same as those of the second embodiment, and for the sake of simplicity, they will not be described again here.

[0038] Example 4 In this embodiment, the structure and principle of the three-dimensional recognition device are almost the same as those in the first embodiment, and the main differences are as follows.

[0039] 4, the second passage 220 has a T-shaped cross section and is divided by the second dichroic filter 360 to form the third and fourth extension passages 221 and 222. The second infrared camera 450 is installed in the fourth extension passage 222, and the RGB camera 440 is installed in the third extension passage 221. The second dichroic filter 360 may use the same material and structure as the first dichroic filter 310, and will not be described again here. After light enters the lens 140, most of the light reflected by the first dichroic filter 310 is visible light, but there is still a small amount of infrared light. After entering the second dichroic filter 360, the infrared light is transmitted to the second infrared camera 450, and the visible light is reflected to the RGB camera 440. By introducing the second infrared camera 450, a binocular camera for structured light can be formed together with the first infrared camera 430, eliminating the need for complex calibration of the infrared dot projector 410. The second infrared camera 450 can capture structured light to increase the collective feature information of the object surface, thereby avoiding the difficulty of matching weak textures or repeated texture areas in binocular stereoscopic vision. Furthermore, in monocular camera scenes, the coded spots emitted by the infrared dot projector 410 are easily obscured by sunlight. A binocular camera using structured light can measure depth information using structured light in indoor environments, and can switch to pure binocular mode when outdoor light causes the structured light to malfunction, increasing the collective feature information of the object surface and further improving reliability and interference resistance.

[0040] The second infrared camera 450 includes a second infrared-sensitive substrate 451, a second infrared low-pass filter 452 and a third lens group 453, and has a structure similar to that of the first infrared camera 430, and for ease of explanation, it will not be described again here.

[0041] Except for the above differences, the other structure and principles of the three-dimensional recognition device of this embodiment are the same as those of the first embodiment, and for the sake of simplicity, they will not be described again here.

[0042] It should be noted that in Examples 1 to 4, the 3D recognition device is installed inside the display screen 120, thereby reducing the area of ​​the enhanced light-transmittance region 130 and improving the user experience, which is applicable to scenes using an under-display front camera. Of course, for terminals that do not use an under-display front camera, the 3D recognition device of Examples 1 to 4 can be installed in the "notch" region of the display screen 120, thereby realizing the effect of reducing the area of ​​the "notch" region, reducing the occupation of the display screen area by the shooting unit, and improving the user experience.

[0043] The present application also provides a terminal, as shown in FIG. 5, which includes a three-dimensional recognition device described in the above embodiment and a display screen 120, wherein the three-dimensional recognition device is installed inside the display screen 120, and the area of ​​the display screen 120 corresponding to the lens 140 of the three-dimensional recognition device is an enhanced light-transmitting area 130.

[0044] In addition, as shown in Figure 1, a TP cover plate 110 may be further installed on the surface of the display screen 120, so that from the outside to the inside are the TP cover plate 110, the display screen 120, the light-transmitting reinforced area 130 and the lens 140, respectively, and a three-dimensional recognition device described in any one of the above embodiments is installed in the optical path 200 corresponding to the lens 140.

[0045] In addition, in the conventional scheme in which the 3D recognition element and the RGB camera are arranged side by side, lenses are arranged respectively for the RGB camera and the 3D recognition element, resulting in a large area of ​​the enhanced light-transmitting area 130 and a "notch" shape, which impairs the user experience. In the 3D recognition device of this embodiment, the 3D recognition unit and the RGB lens 440 are arranged in the same optical path 200, and one lens 140 is shared by controlling the propagation paths of infrared light and visible light. By simply placing only one lens 140 at the front end of the terminal 10 and performing special light-transmitting treatment on the corresponding area, the area of ​​the enhanced light-transmitting area 130 is significantly reduced, thereby reducing the display area affected by the enhanced light-transmitting area 130 and improving the full-screen display effect of the terminal, thereby improving the user experience.

[0046] 6, an embodiment of the present application further provides an image enhancement method applied to a 3D recognition device, the 3D recognition device includes a lens 140 and an optical path 200, as shown in FIG. 1 or 2, the optical path 200 is installed inside the lens 140, and an RGB camera 440, an infrared emitter 420, an infrared dot projector 410, a first infrared camera 430 and a first dichroic filter 310 are installed in the optical path 200, the visible light in the ambient light is reflected by the first dichroic filter 310 to the RGB camera 440, the infrared light in the ambient light passes through the first dichroic filter 310 to reach the first infrared camera 430, and the infrared light transmitted by the infrared emitter 420 and the infrared dot projector 410 passes through the first dichroic filter 310 to reach the lens 140.

[0047] The image enhancement method includes, but is not limited to, the following steps.

[0048] Step S610: Obtain a three-dimensional point cloud image and a two-dimensional image. The three-dimensional point cloud image is obtained from the target reflected light by a first infrared camera, and the two-dimensional image is obtained from the target reflected light by an RGB camera. The target reflected light is the laser speckle light transmitted by the infrared dot projector reflected by the target object.

[0049] Step S620: Obtain a three-dimensional structured light point cloud from the three-dimensional point cloud image, and synchronize the three-dimensional structured light point cloud with the two-dimensional image to obtain a structured light difference value.

[0050] Step S630: Determine RGB reference data according to the two-dimensional image, and perform interpolation processing on the reference surface according to the RGB reference data and the structured light difference value to increase the density of the reference surface.

[0051] Step S640: Obtain the 3D point cloud layer surface of the 3D point cloud image, and calibrate the 3D point cloud layer surface and the reference plane.

[0052] Step S650: Stereo matching is performed on the 3D point cloud layer surface and the reference surface to obtain RGBD depth information.

[0053] 1 and 2, the single lens configuration of the 3D recognition device allows the third and fourth channels 230 and 240 to be close to each other, resulting in a near-zero baseline. This reduces the incident angle of the laser speckles emitted by the infrared dot projector 410, resulting in a large density error in the calibration reference plane, increasing noise in the depth calculation process and affecting the accuracy of the measured depth information. To achieve 3D recognition, the 2D image captured by the RGB camera 440 must be synchronized with the 3D point cloud image captured by the first infrared camera 430 for enhancement calculation, thereby increasing the density of the reference plane and improving the accuracy of stereo matching.

[0054] In addition, since the RGB camera is larger in size than the structured light lens, the amount of incoming light and the imaging radius are correspondingly larger. By synchronizing the three-dimensional structured light point cloud with the two-dimensional image after parameter calibration, the difference between the transmission and reception of structured light can be increased when the baseline is close to zero, thereby obtaining a structured light difference value.

[0055] Laser speckle is generated by irradiating a random scatterer with coherent light. It is a random process, and requires the use of probability and statistical methods to calculate the speckle intensity distribution, contrast, and speckle motion law. Therefore, laser speckle is highly random, resulting in different patterns at different distances. That is, the speckle patterns at any two locations in the same space are different. In this embodiment, the laser speckle transmitted through the infrared dot projector 410 is reflected by the target object and then captured by the first infrared camera 430. The structured light difference value is obtained by synchronizing the three-dimensional structured light point cloud of the three-dimensional point cloud image with the two-dimensional image. The structured light difference value is then used to perform interpolation on the reference plane, thereby increasing the density of the reference plane on the two-dimensional plane, and subsequently matching to obtain RGBD information. Obtaining the three-dimensional structured light point cloud and the three-dimensional point cloud layer plane from the three-dimensional point cloud image is well known in the art and will not be described in detail here.

[0056] After obtaining the density-increased reference plane, the 3D point cloud layer plane of each layer of the dot-projected real-reflection layer plane can be matched with the reference plane to achieve stereo matching and obtain disparity. Alternatively, the 3D point cloud layer plane can be fused with the RGB image and then matched with the reference plane to obtain disparity, thereby further calculating RGBD depth information. The matching process can be achieved by conventional matching cost, cost aggregation, disparity calculation, and disparity refinement, and is not particularly limited here.

[0057] 7, in one embodiment, before performing step S620 in the embodiment shown in FIG. 6, the following steps are further included, but not limited to:

[0058] Step S710: Obtain an infrared reference heat map and a visible light reference map, and determine calibration parameters according to the infrared reference heat map and the visible light reference map, where the infrared reference heat map is collected by a first infrared camera and the visible light reference map is collected by an RGB camera.

[0059] Step S720: Calibrate the 3D point cloud image and the 2D image according to the calibration parameters.

[0060] The infrared reference heat map and the visible light reference map can be obtained by using a standard board with an easily recognizable shape, such as a black and white chessboard pattern or a dot pattern, irradiating it with infrared and visible light simultaneously and collecting images simultaneously with the first infrared camera and the RGB camera. A common calibration method can then be used to calibrate the first infrared camera and the RGB camera and obtain calibration parameters. For example, the Zhang Zhengyou calibration method can be used to output extrinsic parameters, intrinsic parameters, and distortion coefficients. The extrinsic parameters can include the previous positional relationship between the two cameras, such as a rotation matrix and a translation vector relative to the world coordinate system. The intrinsic parameters can be parameters associated with the first infrared camera and the RGB camera, such as the focal length and principal point.

[0061] It should be noted that the process of calibrating the 3D point cloud image and the 2D image involves performing translation and rotation operations on each coordinate point. In this process, the 3D coordinates include three coordinate axes, XYZ, while the 2D image coordinates only have two coordinate axes, XY. Therefore, to avoid the loss of the Z-axis coordinate, a homogeneous coordinate may be introduced into the conversion process, thereby realizing a linear change. The specific calibration process will not be described in detail here.

[0062] Also, as shown in FIG. 8, an embodiment of the present application further provides a terminal, where the terminal 800 includes a memory 810, a processor 820, and a computer program stored in the memory 810 and executable on the processor 820.

[0063] The processor 820 and the memory 810 may be connected via a bus or in other ways.

[0064] Non-transitory software programs and instructions for implementing the image enhancement methods of the above embodiments are stored in memory 810, and when executed by processor 820, perform the image enhancement methods of the above embodiments, for example, steps S610 to S650 of the method in Figure 6 and steps S710 to S720 of the method in Figure 7 described above.

[0065] The above device embodiments are merely schematic, and the units shown as separate components may or may not be physically separated, i.e., they may be located in one place or may be located separately in multiple network units. Some or all of these modules may be selected to achieve the purpose of the embodiment according to actual needs.

[0066] In addition, one embodiment of the present application further provides a computer-readable storage medium, in which computer-executable instructions are stored, and the computer-executable instructions are executed by a processor or controller, for example, by a processor in the above-mentioned terminal embodiment, to cause the processor to execute the image enhancement method in the above-mentioned embodiment, for example, to execute steps S610 to S650 of the method in Figure 6 described above, and steps S710 to S720 of the method in Figure 7 described above.

[0067] All or part of the steps in the methods and systems disclosed above may be implemented as software, firmware, hardware, and any suitable combination thereof. Some or all of the physical components may be implemented as software executed by a processor, such as a central processor, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (e.g., computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cartridge, magnetic tape, magnetic disk storage or other magnetic storage device, or any other medium that can be used to store desired information and that can be accessed by a computer. Additionally, communication media typically includes computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and may include any information delivery media, as known to those skilled in the art.

[0068] An embodiment of the present application includes a lens and an optical path installed inside the lens, where an RGB camera, a 3D recognition unit, and a first dichroic filter are installed in the optical path, and the first dichroic filter reflects visible light in ambient light to the RGB camera and transmits infrared light transmitted and received by the 3D recognition unit. According to the technical solution of this embodiment, the RGB lens and the 3D recognition unit can be installed simultaneously in the optical path of the lens, realizing the front camera and the 3D recognition unit to share a lens, and reducing the area specially treated to enhance light transmittance to the size of a single lens, effectively improving the display effect on the screen and thereby improving the user experience.

[0069] Although the above describes several implementations of the present application in detail, the present application is not limited to the above embodiments, and a person skilled in the art may make various equivalent modifications or substitutions without departing from the spirit of the present application, and all of these equivalent modifications or substitutions are intended to be included in the scope defined by the claims of the present application.

Claims

1. A three-dimensional recognition device installed inside a display screen of a terminal, Lenses and an optical path installed inside the lens, in which an RGB camera, a three-dimensional recognition unit, and a first dichroic filter are installed, the first dichroic filter reflecting visible light in ambient light to the RGB camera and transmitting infrared light transmitted and received by the three-dimensional recognition unit; the first dichroic filter divides the optical path to form a first passage and a second passage, the three-dimensional recognition unit is installed in the first passage, and the RGB camera is installed in the second passage; an infrared blocking wall is installed in the first passage, the infrared blocking wall is connected to the first dichroic filter, and divides the first passage to form a third passage and a fourth passage; the three-dimensional recognition unit includes an infrared transmitting unit and a first infrared camera, the infrared transmitting unit is installed in the third passage, and the infrared light transmitted by the infrared transmitting unit passes through the first dichroic filter and reaches the lens; the first infrared camera is installed in the fourth passage and receives the infrared light in the ambient light that passes through the first dichroic filter; a first infrared reflecting mirror is installed in the third passage, and the first infrared reflecting mirror reflects the infrared light transmitted by the infrared transmitting unit to the first dichroic filter; a second infrared reflecting mirror is installed in the fourth passage, and the second infrared reflecting mirror reflects infrared light in the ambient light that has passed through the first dichroic filter to the first infrared camera; Three-dimensional recognition device.

2. the third passage includes a first extending passage and a second extending passage, a first reflecting mirror is installed between the first extending passage and the second extending passage, and the first reflecting mirror is an infrared total reflection lens or a time-division reflecting mirror; the infrared transmitting unit includes a first emitter and a second emitter, the first emitter is installed in the first extending passage, the second emitter is installed in the second extending passage, the infrared light transmitted by the first emitter is reflected by the first reflecting mirror to the first dichroic filter, the infrared light transmitted by the second emitter passes through the first reflecting mirror to reach the first dichroic filter, the first emitter is an infrared emitter and the second emitter is an infrared dot projector; Or, the first emitter is an infrared dot projector and the second emitter is an infrared emitter; the first infrared reflecting mirror is also used to reflect, to the first dichroic filter, infrared light that has reached the first reflecting mirror by transmission or reflection; The three-dimensional recognition device according to claim 1 .

3. The infrared dot projector includes a first vertical cavity surface emitting laser, a wafer level optical lens and a diffractive optical element in that order in the direction of infrared light emission, and the infrared emitter includes a second vertical cavity surface emitting laser and a diffuser in that order in the direction of infrared light emission. The three-dimensional recognition device according to claim 2 .

4. The infrared transmitting unit is an infrared emitter, and the infrared emitter includes a second vertical cavity surface emitting laser and a diffuser plate in order in an infrared light transmitting direction; The three-dimensional recognition device according to claim 1 .

5. the first infrared camera includes a first lens group, a first infrared low-pass filter, and a first infrared-sensitive substrate arranged in this order in an infrared light incident direction; The three-dimensional recognition device according to claim 1 .

6. the RGB camera includes a second lens group, an infrared media filter, and a visible light sensitive substrate in the visible light incident direction, in this order; The three-dimensional recognition device according to claim 1 .

7. A terminal, The three-dimensional recognition device according to any one of claims 1 to 6, a display screen; The three-dimensional recognition device is installed inside the display screen, and an area of ​​the display screen corresponding to the lens of the three-dimensional recognition device is an enhanced light-transmitting area. Terminal.

Images