Three-dimensional recognition device, terminal, image enhancement method, storage medium

Abstract

The application provides a three-dimensional identification device, a terminal, an image enhancement method and a storage medium. The three-dimensional identification device comprises a lens, a light channel arranged on the inner side of the lens, an RGB camera, a three-dimensional identification component and a first color separation filter arranged in the light channel. The first color separation filter is used for reflecting visible light of ambient light to the RGB camera, and the first color separation filter is also used for transmitting infrared light received and transmitted by the three-dimensional identification component. According to the technical scheme of the embodiment, the RGB camera and the three-dimensional identification component can be arranged in the light channel of the lens, the lens multiplexing of the front camera and the three-dimensional identification component is realized, the area of the region which is specially processed for improving the light transmission is reduced to the size of a single lens, the display effect of the screen is effectively improved, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to, but is not limited to, the field of smart terminals, and particularly to a three-dimensional recognition device, a terminal, an image enhancement method, and a storage medium. Background Technology

[0002] Currently, forward-facing 3D recognition on mobile terminals mainly employs two methods: structured light and time-of-flight (TOF). To implement these methods, 3D recognition devices, such as infrared cameras and flood illuminators, need to be deployed on the mobile terminal. Furthermore, forward-facing red-green-blue (RGB) cameras are typically arranged side-by-side between the 3D recognition devices. Since the lens areas of these devices are relatively large, they are usually positioned at the top of the display screen, forming a "notch" area, which reduces the effective display area of ​​the screen.

[0003] To address this issue, an increasing number of smart devices are adopting under-display camera technology, placing the front-facing RGB camera and 3D recognition devices inside the display screen. To meet the lighting requirements of these devices, special processing is needed to enhance the light transmittance of the corresponding screen display areas, such as reducing or shrinking RGB pixels to increase light transmission. However, this specially processed display area differs somewhat in display effect from other display areas. If the devices are arranged in a parallel layout as before, the specially processed area will be too large, affecting the overall display effect and user experience. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] This invention provides a three-dimensional recognition device, terminal, image enhancement method, and storage medium, which can reduce the area of ​​special processing required to improve light transmittance and enhance user experience.

[0006] In a first aspect, embodiments of the present invention provide a three-dimensional recognition device disposed inside the display screen of a terminal, the three-dimensional recognition device comprising:

[0007] Lens;

[0008] An optical channel is provided inside the lens. The optical channel includes an RGB camera, a 3D recognition component, and a first color filter. The first color filter is used to reflect visible light from the ambient light to the RGB camera and also to transmit infrared light transmitted and received by the 3D recognition component.

[0009] Secondly, embodiments of the present invention provide a terminal, including:

[0010] The three-dimensional recognition device as described in the first aspect;

[0011] The display screen has the three-dimensional recognition device disposed inside it, and the area corresponding to the lens of the display screen and the three-dimensional recognition device is a light-transmitting enhancement area.

[0012] Thirdly, embodiments of the present invention provide an image enhancement method applied to a three-dimensional recognition device. The three-dimensional recognition device includes a lens and an optical channel. The optical channel is disposed inside the lens and includes an RGB camera, an infrared emitter, an infrared dot matrix projector, a first infrared camera, and a first color separation filter. Visible light of ambient light is reflected to the RGB camera through the first color separation filter, and infrared light of ambient light is transmitted to the first infrared camera through the first color separation filter. Infrared light emitted by the infrared emitter and the infrared dot matrix projector is transmitted to the lens through the first color separation filter.

[0013] The image enhancement method includes:

[0014] Acquire a 3D point cloud image and a 2D image, wherein the 3D point cloud image is acquired by the first infrared camera from the target reflected light, and the 2D image is acquired by the RGB camera from the target reflected light, wherein the target reflected light is the light reflected by the target object after the laser speckle emitted by the infrared dot matrix projector;

[0015] A three-dimensional structured light point cloud is obtained from the three-dimensional point cloud image, and the structured light difference value is obtained by synchronizing the three-dimensional structured light point cloud and the two-dimensional image.

[0016] RGB reference data is determined based on the two-dimensional image, and interpolation is performed on the reference surface based on the RGB reference data and the structured light difference value to increase the density of the reference surface.

[0017] Obtain the three-dimensional point cloud map layer of the three-dimensional point cloud image, and calibrate the three-dimensional point cloud map layer and the reference surface;

[0018] Stereo matching is performed between the three-dimensional point cloud map layer and the reference surface to obtain RGBD depth information.

[0019] Fourthly, embodiments of the present invention provide a terminal, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image enhancement method as described in the third aspect.

[0020] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing computer-executable instructions for performing the image enhancement method as described in the third aspect.

[0021] This invention includes: a lens; and an optical channel disposed inside the lens. The optical channel houses an RGB camera, a 3D recognition component, and a first color filter. The first color filter reflects visible light from the ambient light to the RGB camera and also transmits infrared light transmitted and received by the 3D recognition component. According to this embodiment, an RGB lens and a 3D recognition component can be simultaneously disposed within the optical channel of the lens, achieving lens reuse for both the front-facing camera and the 3D recognition component. This reduces the area requiring special processing to improve light transmittance to the size of a single lens, effectively improving the screen's display effect and thus enhancing the user experience.

[0022] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0023] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0024] Figure 1 This is a structural diagram of the three-dimensional recognition device provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is a structural diagram of the three-dimensional recognition device provided in Embodiment 2 of the present invention;

[0026] Figure 3 This is a structural diagram of the three-dimensional recognition device provided in Embodiment 3 of the present invention;

[0027] Figure 4 This is a structural diagram of the three-dimensional recognition device provided in Embodiment 4 of the present invention;

[0028] Figure 5 This is a structural diagram of a terminal provided in another embodiment of the present invention;

[0029] Figure 6 This is a flowchart of an image enhancement method provided in another embodiment of the present invention;

[0030] Figure 7 This is a flowchart of a calibration image provided in another embodiment of the present invention;

[0031] Figure 8 This is a device diagram of a terminal provided in another embodiment of the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0033] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0034] This invention provides a 3D recognition device, a terminal, an image enhancement method, and a storage medium. The 3D recognition device includes: a lens; and an optical channel disposed inside the lens. The optical channel houses an RGB camera, a 3D recognition component, and a first color filter. The first color filter reflects visible light from the ambient light to the RGB camera and also transmits infrared light transmitted and received by the 3D recognition component. According to this embodiment, an RGB lens and a 3D recognition component can be simultaneously disposed within the optical channel of the lens, achieving lens reuse for both the front-facing camera and the 3D recognition component. This reduces the area requiring special processing to improve light transmittance to the size of a single lens, effectively improving the screen display effect and thus enhancing the user experience.

[0035] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0036] This invention provides a three-dimensional recognition device disposed inside the display screen of a terminal. The three-dimensional recognition device includes:

[0037] 140mm lens;

[0038] The light channel 200 is located inside the lens 140. The light channel 200 includes an RGB camera 440, a 3D recognition component, and a first color filter 310. The first color filter 310 is used to reflect the visible light of the ambient light to the RGB camera 440 and also to transmit the infrared light transmitted and received by the 3D recognition component.

[0039] It should be noted that the lens 140 is located inside the display screen 120 of the terminal and close to the light transmission enhancement area 130 of the display screen 120. The specific parameters and type of the lens 140 are not limited here, as long as it can be used for under-display photography.

[0040] It is worth noting that in order to reuse the same lens 140 for both shooting and 3D recognition, an RGB camera 440 and a 3D recognition component need to be set up simultaneously in the light channel 200 inside the lens 140. The RGB camera 440 is used for shooting, and its light source is visible light, while the 3D recognition component is mainly a structured light or TOF component, and the light received or emitted is infrared light. Therefore, when the display screen 130 is placed horizontally, a first color filter 310 can be set up in the vertical light channel 200 inside the lens 140 to achieve the transmission of infrared light and the reflection of visible light. By distinguishing the propagation paths of visible light and infrared light, the light propagation path requirements of the RGB camera 440 and the 3D recognition component in the same light channel 200 are met.

[0041] It should be noted that the first dichroic filter 310 can be a visible / infrared filter film. When light is incident at a 45-degree angle, the reflectivity of the visible / infrared filter film is greater than 90% for the visible light band of 0.3 micrometers to 0.6 micrometers, and the transmittance of the near-infrared light band of 0.75 micrometers to 2.5 micrometers is greater than 90%. Therefore, infrared light can penetrate the visible / infrared filter film, while visible light will be reflected by the mirror surface of the visible / infrared filter film, thereby distinguishing the propagation directions of infrared light and visible light. The specific setting angle of the first dichroic filter 310 can be adjusted according to the actual layout of the optical channel 200, and this embodiment does not impose any limitations on this.

[0042] It is worth noting that the first color filter 310 enables infrared light and visible light to travel along different paths, allowing the RGB camera 440 and the 3D recognition component set in the light channel 200 to operate normally. This allows the 3D recognition component and the RGB camera 440 to reuse a single lens 140. Compared to the solution where the 3D recognition component and the RGB camera are placed side by side at the front of the display screen, the light transmittance enhancement area 130 only needs to be set for a single lens 140, and its area is greatly reduced, effectively improving the user experience.

[0043] The following describes various embodiments of the 3D recognition device through several examples.

[0044] It should be noted that, Figures 1 to 4 The above four figures are structural diagrams of four embodiments of the three-dimensional recognition device provided by the present invention. All four figures are obtained by taking vertical cropping when the terminal is placed horizontally and the front camera is facing upwards. They will not be described again in detail below.

[0045] Example 1:

[0046] Reference Figure 1 When a first color filter 310 is set in the light channel 200, the first color filter 310 divides the light channel 200 into a first channel 210 and a second channel 220. Based on the above description, when the first color filter 310 is set at a 45-degree angle, visible light will be specularly reflected at the first color filter 310. Therefore, the second channel 220 can be perpendicular to the main body of the light channel 200, so that visible light reflected at a 90-degree angle can enter the second channel 220 and finally be incident on the RGB camera 440 set in the second channel 220. Of course, the angle of the second channel 220 can be adjusted according to the setting angle of the first color filter 310, as long as it can ensure that visible light can be incident on the RGB camera 440. No further limitations are made here.

[0047] In this embodiment, the RGB camera 440 includes a second lens group 443, an infrared cut-off filter 442, and a visible light photosensitive substrate 441. Although the first color filter 310 can transmit infrared light, its transmittance is difficult to achieve 100%. Therefore, some infrared light is still carried in the reflected light. Thus, after the light reflected by the first color filter 310 passes through the second lens group 443, the infrared light needs to be filtered out by the infrared cut-off filter 442, leaving only visible light to enter the visible light photosensitive substrate 441. The photoelectric image signal is generated by the contact image sensor (CIS) chip set in the visible light photosensitive substrate 441, and the photoelectric image signal is sent to the main control chip of the terminal for subsequent imaging processing. This will not be elaborated further here.

[0048] Furthermore, the 3D recognition process includes infrared light emission and infrared light reception. Therefore, an infrared isolation wall 320 can be set in the first channel 210 to divide the first channel 210 into a third channel 230 and a fourth channel 240. The infrared emitting component and the infrared camera can be respectively set in the third channel 230 and the fourth channel 240. This embodiment uses the example of the infrared emitting component being set in the third channel 230 and the first infrared camera 430 being set in the fourth channel 240 for illustration. It should be noted that since the first color filter 310 transmits infrared light, the infrared isolation wall 320 can be as follows: Figure 1 As shown, the infrared isolation wall 320 is arranged in a direction perpendicular to the display screen 120. In order to avoid mutual interference between the transmitted and received infrared light, one end of the infrared isolation wall 320 is tightly connected to the first color filter 310, and the other end is tightly connected to the bottom side of the light channel 200. Furthermore, the specific position of the infrared isolation wall 320 in the first channel 210 can be adjusted according to the size of the infrared emitting component and the first infrared camera 430, so that a compact layout can be achieved in the first channel 210.

[0049] In addition, in this embodiment, to simultaneously achieve structured light 3D recognition and TOF 3D recognition, this embodiment takes an infrared dot projector 410 as the first emitter and an infrared emitter 420 as the second emitter as an example. The infrared dot projector 410 includes a first vertical cavity surface emitting laser (VCSEL) 411, a wafer level optical lens (WLO) 412, and a diffractive optical element (DOE) 413 arranged sequentially in the infrared light emission direction. The first VCSEL 411 can be a high-power VCSEL. The emitted infrared laser is calibrated by the WLO 412, modulated by the DOE 413, and scattered to form a speckle pattern. The infrared emitter 420 includes a second VCSEL 421 and a diffuser 422. The second VCSEL 421 can be a low-power VCSEL. The infrared light emitted by the second VCSEL 421 is expanded by the first diffuser 422, acting as a low-power floodlight illuminator. The specific component parameters and specifications of the infrared dot matrix projector 410 and the infrared emitter 420 can be selected according to actual needs.

[0050] To prevent interference between the infrared light from the infrared dot projector 410 and the infrared emitter 420, a first reflector 330 can be installed in the third channel 230 to divide the third channel 320 into two parts. Figure 1 The first extension channel 231 and the second extension channel 232 shown are, in this embodiment, an infrared dot matrix projector 410 is disposed in the first extension channel 231 and an infrared emitter 420 is disposed in the second extension channel 232. Alternatively, the infrared dot matrix projector 410 can be disposed in the second extension channel 232 and the infrared emitter 420 in the first extension channel 231; this will not be elaborated further here. In this embodiment, the first extension channel 231 and the second extension channel 232 are perpendicular to each other, and can be adopted... Figure 1 As shown, the first extension channel 231 is obtained by extending vertically from one side of the first channel 210. Of course, the first extension channel 231 and the second extension channel 232 can also be at other angles, as long as infrared light can be emitted under the reflection of the first reflector 330.

[0051] The first reflecting mirror 330 can be an infrared total internal reflection lens or a time-division reflecting mirror. When the first reflecting mirror 330 is an infrared total internal reflection lens, the infrared dot matrix projector 410 and the infrared emitter 420 can work simultaneously. The angle between the first reflecting mirror 330 and the first extension channel 231 is greater than the critical angle of the first reflecting mirror 330, so that the angle between the infrared light emitted by the infrared dot matrix projector 410 and the mirror surface is greater than the critical angle, and thus it is completely reflected towards the first dichroic filter 310. Conversely, the angle between the infrared light emitted by the infrared emitter 420 and the mirror surface is less than the critical angle, and the infrared light is transmitted to the first dichroic filter 310. This allows the infrared light emitted by the two devices located in different extension channels to propagate to the first dichroic filter 310. At the same time, during the transmission process through the lens, the slight refraction at the entry and exit points keeps the infrared light parallel and does not affect the overall transmission effect. When the first reflector 330 is a time-division reflector, its movement can be mechanically controlled to allow the infrared dot projector 410 and the infrared emitter 420 to operate in a time-division manner. When the infrared dot projector 410 needs to operate, the reflector is lowered via mechanical control, blocking the infrared light emitted by the infrared emitter 420 and reflecting it onto the first dichroic filter 310. When the infrared emitter 420 needs to operate, the reflector is raised via mechanical control, blocking the infrared light emitted by the infrared dot projector 410 and allowing it to penetrate the first reflector 330 and reach the first dichroic filter 310. To avoid mutual interference between infrared lights, the infrared dot projector 410 and the infrared emitter 420 need to use different infrared frequency bands. In this case, the infrared total internal reflection lens can also be replaced by an infrared filter that precisely controls the penetration and blocking of different infrared spectra, selected according to actual needs.

[0052] In the fourth channel 240, the first infrared camera 430 includes a first infrared light-sensing substrate 441, a first infrared low-pass filter 442, and a first lens group 443. The specifications of the first lens group 443 can be selected according to actual needs. The first infrared low-pass filter 442 can also be an infrared low-pass filter or an infrared narrowband interference filter, which can only pass near-infrared light of a specific wavelength. The first infrared light-sensing substrate 441 can be a CIS light-sensing substrate. The first infrared camera 430 is used to receive infrared light emitted by the infrared emitter 420 and the infrared dot matrix projector 410 and reflected by the target object. The infrared light is converted into an infrared photoelectric signal in the first infrared light-sensing substrate 441, which can be further processed by the main control chip. The specific light-sensing principle will not be elaborated here. In this embodiment, the second VCSEL is a low-power VCSEL. The low-power unstructured infrared light signal emitted by it is mainly used to predict the accuracy of the target object. For example, target recognition can be performed by analyzing two-dimensional infrared information through artificial intelligence algorithms. After the prediction is successful, an infrared speckle pattern is projected by an infrared dot projector 410, reflected by the target object, and received by the first infrared camera 430 to obtain the depth information of the target object. The depth information obtained by the first infrared camera 420 and the two-dimensional information of the target object obtained by the RGB camera 440 can be combined to obtain Red Green Blue Depth (RGBD) information. Based on the RGBD information, three-dimensional image recognition and three-dimensional image reconstruction can be effectively performed. This embodiment will not elaborate on the specific image recognition method.

[0053] Example 2:

[0054] refer to Figure 2 In this embodiment, the structure and principle of each component are basically the same as in Embodiment 1, and will not be repeated here. The main difference lies in the layout of the first channel 210. In Embodiment 1, as shown in the figure... Figure 1 As shown, the first channel 210 is a vertical channel. After being separated by the infrared isolation wall 320, a second extended channel 232 and a fourth channel 240 can be formed vertically, allowing the infrared emitter 420 and the first infrared camera 430 to be positioned towards the lens 140. The infrared light emitted by the infrared emitter 420 can pass through the first dichroic filter 310 and directly exit to the lens 140, while the first infrared camera 430 can receive the infrared light directly incident on the lens 140. In this embodiment, the layout of the first channel 210 is as follows:

[0055] The cross-section of the third channel 230 is F-shaped, and a first infrared reflector 340 is set at the corner. The relative vertical relationship between the first extended channel 231 and the second extended channel 232 remains unchanged. The infrared light emitted by the infrared emitter 420 passes through the first reflector 330 and then enters the first infrared reflector 340. The infrared light emitted by the infrared dot matrix projector 410 is reflected by the mirror surface of the first reflector 330 and then enters the first infrared reflector 340. The infrared light entering the first infrared reflector 340 undergoes mirror reflection and thus enters the first color filter 310. The rest of the principle is the same as in Embodiment 1, and will not be repeated here.

[0056] The cross-section of the fourth channel 240 is L-shaped, and a second infrared reflector 350 is set at the corner. Infrared light entering and exiting from the lens 140 and passing through the first color filter 310 is reflected by the second infrared reflector 350 to the first infrared camera 430. The rest of the principle is the same as in Embodiment 1, and will not be repeated here.

[0057] It is worth noting that this embodiment is similar to the first embodiment in terms of technical principle. Through the above differences, the depth of the optical channel 200 is effectively reduced compared to the first embodiment, so that the structure of the optical channel 200 is more compact and occupies less thickness after being applied to the terminal, which is conducive to reducing the thickness of the terminal.

[0058] Example 3:

[0059] refer to Figure 3 In this embodiment, the structure and principle of the three-dimensional recognition device are basically the same as those in Embodiment 2, which can effectively reduce the thickness of the terminal. The main difference between this embodiment and Embodiment 2 is that this embodiment omits the first reflector 330 and the infrared dot matrix projector 410. As a result, the third channel 230 in this embodiment does not need to be additionally set with the first extension channel 231. The cross-section of the third channel 230 is L-shaped, and the infrared light emitted by the infrared emitter 420 is directly reflected to the first color filter 310 through the first infrared reflector 340.

[0060] It is worth noting that, since the infrared dot projector 410 is omitted, in order to achieve single-lens application in the TOF scheme, the second VCSEL 421 of the infrared emitter 420 in this embodiment adopts a high-power VCSEL, thereby forming a high-power floodlight illuminator. The infrared light from the high-power VCSEL is expanded by the diffuser 422 and then emitted as TOF surface-emitting light, which can realize a three-dimensional image in a single imaging step. The specific imaging method will not be described in detail here. At the same time, since the infrared light emitted by the infrared emitter 420 is different from that in Embodiment 2, the internal structure of the first infrared camera 430 can also be adjusted accordingly, so that a three-dimensional image can be obtained when the infrared light reflected from the target object is received.

[0061] Apart from the differences mentioned above, the other structures in this embodiment are the same as those in Embodiment 2, and will not be repeated here for the sake of simplicity.

[0062] Example 4:

[0063] In this embodiment, the structure and principle of the 3D recognition device are basically the same as those in Embodiment 1, with the following main differences:

[0064] refer to Figure 4 The cross-section of the second channel 220 is T-shaped, and it is divided into a third extension channel 221 and a fourth extension channel 222 by the second color filter 360. The fourth extension channel 222 is equipped with a second infrared camera 450, and the third extension channel 221 is equipped with an RGB camera 440. The second color filter 360 can use the same material and structure as the first color filter 310, which will not be repeated here. After light enters the lens 140, most of the light reflected by the first dichroic filter 310 is visible light, but a small amount of infrared light remains. After passing through the second dichroic filter 360, the infrared light is transmitted into the second infrared camera 450, and the visible light is reflected to the RGB camera 440. By introducing the second infrared camera 450, a binocular camera targeting structured light can be formed with the first infrared camera 430. This eliminates the need for complex calibration of the infrared dot projector 410. At the same time, the structured light obtained by the second infrared camera 450 can increase the aggregate feature information of the object surface, avoiding the problem of difficulty in matching weak or repetitive texture areas in binocular stereo vision. In addition, in monocular camera scenarios, the coded light spot emitted by the infrared dot projector 410 is easily overwhelmed by sunlight. The binocular camera using structured light can use structured light to measure depth information in indoor environments, and switch to pure binocular mode when outdoor lighting causes the structured light to become ineffective, increasing the aggregate feature of the object surface and further improving reliability and anti-interference.

[0065] The second infrared camera 450 includes a second infrared light-sensing substrate 451, a second infrared low-pass filter 452, and a third lens group 453. Its structure is similar to that of the first infrared camera 430, and will not be repeated here for the sake of simplicity.

[0066] Apart from the differences mentioned above, the remaining structure and principle of the three-dimensional recognition device in this embodiment are the same as those in Embodiment 1, and will not be repeated here for the sake of simplicity.

[0067] It is worth noting that in Embodiments 1 to 4, the 3D recognition device is located inside the display screen 120, thereby reducing the area of ​​the light transmittance enhancement area 130 and improving the user experience. This is for the use scenario of an under-display front camera. Of course, for terminals that do not use an under-display front camera, the 3D recognition device in Embodiments 1 to 4 can also be located in the "notch" area of ​​the display screen 120, thereby reducing the area of ​​the "notch" area, reducing the occupation of the display screen area by the camera component, and improving the user experience.

[0068] In addition, the present invention also provides a terminal, as shown in the following figure. Figure 5 The terminal 10 includes:

[0069] The three-dimensional detection device as described in the above embodiment;

[0070] The display screen 120 has a 3D recognition device located inside it. The area corresponding to the lens 140 of the 3D recognition device on the display screen 120 is a light-transmitting area 130.

[0071] It should be noted that the reference Figure 1 A TP cover plate 110 can also be provided on the surface of the display screen 120, so that from the outside to the inside are the TP cover plate 110, the display screen 120, the light transmittance enhancement area 130 and the lens 140, and the three-dimensional recognition device described in any of the above embodiments is provided in the light channel 200 corresponding to the lens 140.

[0072] It is worth noting that in conventional solutions where 3D recognition devices and RGB cameras are placed side by side, each device has its own lens. As a result, the area of ​​the light-enhancing region 130 is relatively large and resembles a "notch," which affects the user experience. In the 3D recognition device of this embodiment, the 3D recognition component and the RGB lens 440 are placed in the same light channel 200. By controlling the propagation paths of infrared and visible light, a single lens 140 can be reused. The front end of the terminal 10 only needs to be equipped with one lens 140, and special light-transmitting processing is performed in the corresponding area. This significantly reduces the area of ​​the light-enhancing region 130, thereby reducing the display area affected by the light-enhancing region 130, improving the overall display effect of the terminal, and thus enhancing the user experience.

[0073] Additionally, refer to Figure 6 This invention also provides an image enhancement method applied to a 3D recognition device, such as... Figure 1 or Figure 2As shown, the system includes a lens 140 and an optical channel 200. The optical channel 200 is disposed inside the lens 140. The optical channel 200 is equipped with an RGB camera 440, an infrared emitter 420, an infrared dot matrix projector 410, a first infrared camera 430, and a first color filter 310. Visible light from the ambient light is reflected to the RGB camera 440 through the first color filter 310, and infrared light from the ambient light is transmitted to the first infrared camera 430 through the first color filter 310. Infrared light emitted by the infrared emitter 420 and the infrared dot matrix projector 410 is transmitted to the lens 140 through the first color filter 310.

[0074] Image enhancement methods include, but are not limited to, the following steps:

[0075] Step S610: Acquire a three-dimensional point cloud image and a two-dimensional image. The three-dimensional point cloud image is acquired by the first infrared camera from the target reflected light, and the two-dimensional image is acquired by the RGB camera from the target reflected light. The target reflected light is the light reflected by the laser speckle emitted by the infrared dot matrix projector after passing through the target object.

[0076] Step S620: Obtain a three-dimensional structured light point cloud from the three-dimensional point cloud image, and obtain the structured light difference value by synchronizing the three-dimensional structured light point cloud and the two-dimensional image;

[0077] Step S630: Determine RGB reference data based on the two-dimensional image, and perform interpolation processing on the reference surface based on the RGB reference data and the structured light difference value to increase the density of the reference surface;

[0078] Step S640: Obtain the three-dimensional point cloud map layer of the three-dimensional point cloud image, and calibrate the three-dimensional point cloud map layer and the reference surface.

[0079] Step S650: Perform stereo matching between the 3D point cloud map layer and the reference surface to obtain RGBD depth information.

[0080] It should be noted that when the 3D recognition device uses Figure 1 and Figure 2 In the structure of the illustrated embodiment, under single-lens conditions, the third channel 230 and the fourth channel 240 are close to each other, and the baseline is almost zero. In this case, the incident angle of the laser speckle projected by the infrared dot matrix projector 410 is small, which leads to an increase in the density error of the calibration reference surface, increases the noise in the depth calculation process, and affects the accuracy of the measured depth information to a certain extent. In order to achieve three-dimensional recognition, it is necessary to use the two-dimensional image obtained by the RGB camera 440 to synchronize the three-dimensional point cloud image obtained by the first infrared camera 430 for enhancement calculation, thereby improving the density of the reference surface and ensuring more accurate stereo matching.

[0081] It should be noted that RGB cameras are larger in size than structured light lenses, so the amount of light entering and the imaging radius are also correspondingly larger. After parameter calibration, the synchronization of 3D structured light point cloud and 2D image can increase the difference between transmitted and received structured light when the baseline is close to zero, thereby obtaining the structured light difference value.

[0082] It should be noted that laser speckle is generated by random scattering bodies being irradiated by coherent light, which is a random process. Probabilistic statistical methods are needed to calculate the intensity distribution, contrast, and motion patterns of the speckle. Therefore, laser speckle has a high degree of randomness, and different patterns appear with different distances; that is, the speckle patterns at any two locations in the same space are not the same. In this embodiment, the laser speckle transmitted through the infrared dot matrix projector 410 is reflected by the target object, and a three-dimensional point cloud image is acquired by the first infrared camera 430. Based on the synchronization of the three-dimensional structured light point cloud and the two-dimensional image, a structured light difference value is obtained. Then, interpolation processing is performed on the reference plane based on the structured light difference value, thereby realizing the density enhancement of the reference plane on the two-dimensional plane, so as to facilitate subsequent matching to obtain RGBD information. It is understood that obtaining the three-dimensional structured light point cloud and the three-dimensional point cloud image from the three-dimensional point cloud image is a well-known technique in the art, and will not be elaborated upon here.

[0083] It should be noted that after obtaining the reference surface with increased density, each layer of the 3D point cloud map of the dot matrix projection object reflection map can be matched and calculated with the reference surface to achieve stereo matching and obtain disparity; alternatively, the 3D point cloud map layers and the RGB image can be fused and then matched and calculated with the reference surface to obtain disparity, thereby further calculating the RGBD depth information. The matching process can be implemented using conventional matching cost, cost aggregation, disparity calculation, and disparity refinement, which will not be limited here.

[0084] Additionally, refer to Figure 7 In one embodiment, during execution Figure 6 Before step S620 in the illustrated embodiment, the following steps may also be included, but are not limited to:

[0085] Step S710: Obtain an infrared reference thermal image and a visible light reference image, and determine calibration parameters based on the infrared reference thermal image and the visible light reference image. The infrared reference thermal image is acquired by a first infrared camera, and the visible light reference image is acquired by an RGB camera.

[0086] Step S720: Calibrate the 3D point cloud image and the 2D image according to the calibration parameters.

[0087] It should be noted that when acquiring the infrared reference thermal image and the visible light reference image, a clearly contrasting shape such as a black and white checkerboard or a dot pattern can be used as a reference plate. The plate is simultaneously illuminated with infrared and visible light, and images are acquired by the first infrared camera and the RGB camera at the same time to obtain the infrared reference thermal image and the visible light reference image. Then, the first infrared camera and the RGB camera are calibrated using common calibration methods to obtain calibration parameters. For example, the extrinsic parameters, intrinsic parameters, and distortion coefficients can be output using the Zhang Zhengyou calibration method. The extrinsic parameters can include the positional relationship between the two cameras, such as the rotation matrix and translation vector relative to the world coordinate system. The intrinsic parameters can be parameters associated with the first infrared camera and the RGB camera, such as the focal length and the principal point.

[0088] It is worth noting that the calibration process for 3D point cloud images and 2D images includes translation and rotation operations for each coordinate point. During this process, since 3D coordinates include three coordinate axes (X, Y, and Z), while 2D image coordinates only have two coordinate axes (X and Y), homogeneous coordinates can be introduced during the transformation process to avoid the loss of the Z-axis coordinate, thereby achieving linear transformation. The specific calibration process will not be elaborated here.

[0089] Additionally, refer to Figure 8 An embodiment of the present invention also provides a terminal 800, which includes a memory 810, a processor 820, and a computer program stored in the memory 810 and executable on the processor 820.

[0090] The processor 820 and memory 810 can be connected via a bus or other means.

[0091] The non-transient software program and instructions required to implement the statistical data acquisition method of the above embodiments are stored in memory 810. When executed by processor 820, the image enhancement method of the above embodiments is executed, for example, the method described above is executed. Figure 6 Method steps S610 to S650 Figure 7 Method steps S710 to S720.

[0092] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0093] Furthermore, one embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described terminal embodiment, causing the processor to perform the image enhancement method described above, for example, performing the above-described... Figure 6 Method steps S610 to S650 Figure 7 The method steps S710 to S720 are described above. Those skilled in the art will understand that all or some of the steps in the methods disclosed above, and the system, can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be stored on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0094] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A three-dimensional recognition device, characterized in that, The three-dimensional recognition device, located inside the display screen of the terminal, includes: Lens; An optical channel is provided inside the lens. An RGB camera, a 3D recognition component, and a first color separation filter are provided in the optical channel. The first color separation filter is used to reflect visible light from the ambient light to the RGB camera. The first color separation filter is also used to transmit infrared light transmitted and received by the 3D recognition component. The first color filter divides the light channel into a first channel and a second channel, the three-dimensional recognition component is disposed in the first channel, and the RGB camera is disposed in the second channel; An infrared isolation wall is provided in the first channel, and the infrared isolation wall is connected to the first color filter. The infrared isolation wall divides the first channel into a third channel and a fourth channel. The three-dimensional recognition component includes an infrared emitting component and a first infrared camera. The infrared emitting component is disposed in the third channel. The infrared light emitted by the infrared emitting component is transmitted to the lens through the first color filter. The first infrared camera is disposed in the fourth channel and is used to receive the infrared light of the ambient light transmitted from the first color filter. The third channel includes a first extension channel and a second extension channel. A first reflector is disposed between the first extension channel and the second extension channel. The first reflector is an infrared total reflection lens or a time-division reflector. The infrared emitting assembly includes a first emitter and a second emitter. The first emitter is disposed in the first extension channel, and the second emitter is disposed in the second extension channel. Infrared light emitted by the first emitter is reflected by the first reflector to the first dichroic filter, and infrared light emitted by the second emitter is transmitted through the first reflector to the first dichroic filter. The first transmitter is an infrared transmitter, and the second transmitter is an infrared dot matrix projector; or, The first transmitter is an infrared dot matrix projector, and the second transmitter is an infrared transmitter; The infrared dot matrix projector includes a first VCSEL, a WLO, and a DOE in sequence along the infrared light emission direction, and the infrared emitter includes a second VCSEL and a diffuser in sequence along the infrared light emission direction.

2. The three-dimensional recognition device according to claim 1, characterized in that: The third channel is also provided with a first infrared reflector, which is used to reflect infrared light transmitted or reflected by the first reflector to the first color filter.

3. The three-dimensional recognition device according to any one of claims 1 to 2, characterized in that: The first infrared camera includes, in sequence along the infrared light incident direction, a first lens group, a first infrared low-pass filter, and a first infrared photosensitive substrate.

4. The three-dimensional recognition device according to claim 3, characterized in that: The fourth channel is equipped with a second infrared reflector, which is used to reflect the infrared light of the ambient light transmitted from the first color filter to the first infrared camera.

5. The three-dimensional recognition device according to any one of claims 1 to 2, characterized in that: The RGB camera includes, in sequence along the direction of visible light incidence, a second lens group, an infrared medium filter, and a visible light photosensitive substrate.

6. The three-dimensional recognition device according to claim 5, characterized in that: The second channel includes a third extension channel and a fourth extension channel. The RGB camera is disposed in the third extension channel, and a second infrared camera is disposed in the fourth extension channel. A second color filter is provided between the third extension channel and the fourth extension channel. The second color filter is used to reflect the visible light reflected by the first color filter to the RGB camera, and to transmit a small amount of infrared light in the light reflected by the first color filter to the second infrared camera. The second infrared camera includes, in sequence along the infrared light incident direction, a third lens group, a second infrared low-pass filter, and a second infrared photosensitive substrate.

7. A terminal, characterized in that, include: The three-dimensional recognition device as described in any one of claims 1 to 6; The display screen has the three-dimensional recognition device disposed inside it, and the area corresponding to the lens of the display screen and the three-dimensional recognition device is a light-transmitting enhancement area.

8. An image enhancement method, applied to the three-dimensional recognition device as described in claim 1; The image enhancement method includes: Acquire a 3D point cloud image and a 2D image, wherein the 3D point cloud image is acquired by the first infrared camera of the 3D recognition device from the target reflected light, and the 2D image is acquired by the RGB camera of the 3D recognition device from the target reflected light, wherein the target reflected light is the light reflected by the target object after the laser speckle emitted by the infrared dot matrix projector of the 3D recognition device. A three-dimensional structured light point cloud is obtained from the three-dimensional point cloud image, and the structured light difference value is obtained by synchronizing the three-dimensional structured light point cloud and the two-dimensional image. RGB reference data is determined based on the two-dimensional image, and interpolation is performed on the reference surface based on the RGB reference data and the structured light difference value to increase the density of the reference surface. Obtain the three-dimensional point cloud map layer of the three-dimensional point cloud image, and calibrate the three-dimensional point cloud map layer and the reference surface; Stereo matching is performed between the three-dimensional point cloud map layer and the reference surface to obtain RGBD depth information.

9. The method according to claim 8, characterized in that, Prior to acquiring the three-dimensional structured light point cloud from the three-dimensional point cloud image, the method further includes: An infrared reference thermal image and a visible light reference image are acquired, and calibration parameters are determined based on the infrared reference thermal image and the visible light reference image, wherein the infrared reference thermal image is acquired by the first infrared camera, and the visible light reference image is acquired by the RGB camera; The three-dimensional point cloud image and the two-dimensional image are calibrated according to the calibration parameters.

10. A terminal, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the image enhancement method as described in any one of claims 8 to 9.

11. A computer-readable storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are used to perform the image enhancement method as described in any one of claims 8 to 9.

Images