3D recognition device, terminal

Abstract

The present application provides a three-dimensional recognition device, a terminal, a calibration method, and a storage medium, and the three-dimensional recognition device includes an RGB unit including an RGB camera (210) and a first lens (212), where a first optical path (211) is connected between the first lens (212) and the RGB camera (210), an infrared transmitting unit including an infrared floodlight illuminator (310) and a second lens (312) adjacent to the first lens (212), where the second optical path (311) is connected between the infrared floodlight illuminator (310) and the second lens (312), and an infrared receiving unit including a first infrared camera (410) and a third lens (412) adjacent to the first lens (212), where the third optical path (411) is connected between the first infrared camera (410) and the third lens (412).

Description

[Technical field]

[0001] This application is filed based on a Chinese patent application having application number 202111516186.2 and filing date December 7, 2021, and claims priority to that Chinese patent application, the entire contents of which are incorporated herein by reference.

[0002] The present application relates to the technical field of smart terminals, but is not limited thereto, and in particular to a three-dimensional recognition device, a terminal, a calibration method and a storage medium. [Background technology]

[0003] With the development of display screen technology, under-display screen cameras have been applied to various smart terminals and are popular with many consumers. Because the lens is installed inside the display screen, the area of ​​the display screen corresponding to the lens needs to be specially treated to increase light transmittance, for example, by reducing the number of red, green and blue (RGB) pixels or by making the RGB pixels smaller to increase the amount of light transmitted.

[0004] With the development of 3D recognition technology, smart terminals not only need to install a front RGB camera on the display screen, but also need to install various 3D recognition elements. Meanwhile, the number of each 3D recognition element is large, and currently, the parallel arrangement method shown in Figure 1 is mainly adopted, so the area of ​​the display screen that is specially treated to improve light transmittance is relatively large. Due to the difference in light transmittance, there will be a certain difference in the display effect between the area that is specially treated and the other areas of the display screen, which will affect the user experience of the full display. Summary of the Invention [Problem to be solved by the invention]

[0005] The following is a brief summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The embodiments of the present application provide a three-dimensional recognition device, a terminal, a calibration method, and a storage medium. [Means for solving the problem]

[0007] In a first aspect, an embodiment of the present application provides a three-dimensional recognition device provided inside a display screen of a terminal, the three-dimensional recognition device including: an RGB unit including an RGB camera and a first lens, where a first optical path is connected between the first lens and the RGB camera; an infrared transmitting unit including an infrared floodlight illuminator and a second lens adjacent to the first lens, where a second optical path is connected between the infrared floodlight illuminator and the second lens; and an infrared receiving unit including a first infrared camera and a third lens adjacent to the first lens, where a third optical path is connected between the first infrared camera and the third lens.

[0008] In a second aspect, an embodiment of the present application provides a terminal, the terminal including a three-dimensional recognition device according to the first aspect, and a display screen, the three-dimensional recognition device being arranged inside the display screen, and an area of ​​the display screen corresponding to a lens of the three-dimensional recognition device being an enhanced light-transmittance area.

[0009] In a third aspect, an embodiment of the present application provides a calibration method applied to a three-dimensional recognition device, the three-dimensional recognition device including: an RGB unit including an RGB camera and a first lens, the RGB unit being connected with a first optical path between the first lens and the RGB camera; an infrared transmission unit including an infrared floodlight illuminator and a second lens adjacent to the first lens, the infrared transmission unit being connected with a second optical path between the infrared floodlight illuminator and the second lens; and an infrared receiving unit including a first infrared camera and a third lens adjacent to the first lens, the infrared receiving unit being connected with a third optical path between the first infrared camera and the third lens, the calibration method including: when the infrared floodlight illuminator is in an operating state, a first predetermined timing sequence is generated. taking a first image set using the first infrared camera and a second image set using the RGB camera according to a sequence, wherein different target objects are photographed by the first infrared camera at different times, different target objects are photographed by the RGB camera at different times, and the same target object is photographed by the first infrared camera and the RGB camera at the same time; performing a calibration between the first infrared camera and the infrared floodlight illuminator based on the first image set; fusing images in the first image set and the second image set, each taken at the same time, to obtain a first fused image set, and performing a calibration between the RGB camera and the first infrared camera based on the first fused image set.

[0010] In a fourth aspect, an embodiment of the present application provides a terminal, the terminal including a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor, when executing the computer program, realizing the calibration method described in the third aspect.

[0011] In a fifth aspect, embodiments of the present application provide a computer-readable storage medium storing computer-executable instructions, the computer-executable instructions being used to perform the calibration method according to the third aspect.

[0012] Other features and advantages of the present application will be set forth in the following specification, and in part will be obvious from the specification, or may be learned by the practice of the present application. The objectives and other advantages of the present application may be realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0013] The accompanying drawings are intended to provide a further understanding of the technical solution of the present application, constitute a part of the specification, and are used to interpret the technical solution of the present application together with the embodiments of the present application, and are not intended to constitute limitations on the technical solution of the present application. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing an arrangement of a full-display 3D recognition device and an RGB camera in the prior art. [Diagram 2] FIG. 2 is a schematic diagram showing a three-dimensional recognition device installed on a terminal according to an embodiment of the present application; [Diagram 3] 1 is a schematic cross-sectional view of a three-dimensional recognition device provided by the present application. [Figure 4] FIG. 1 is a front view of a three-dimensional recognition device provided by the present application. [Diagram 5] FIG. 1 is a schematic diagram showing the layout of a first embodiment of the present application. [Figure 6] FIG. 13 is a schematic diagram showing the layout of a second embodiment of the present application. [Figure 7] FIG. 13 is a schematic diagram showing the layout of a third embodiment of the present application. [Figure 8] 4 is a flowchart of a calibration method for a three-dimensional recognition device applied to a first embodiment provided by the present application. [Figure 9] 11 is a flowchart of a calibration method for a three-dimensional recognition device applied to a second embodiment provided by the present application. [Figure 10] 13 is a flowchart of a calibration method for a three-dimensional recognition device applied to a third embodiment provided by the present application. [Figure 11] 4 is a flow chart for filtering out laser speckle information provided by another embodiment of the present application; [Figure 12] 4 is a flowchart of a global calibration provided by another embodiment of the present application; [Figure 13] FIG. 2 illustrates a target object provided by the present application. [Figure 14] FIG. 2 is an apparatus diagram of a terminal provided according to another embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be described in more detail below in conjunction with the accompanying drawings and examples, in which the specific examples described herein are only used to interpret the present application, and are not used to limit the present application.

[0016] In addition, although the schematic diagram of the device shows a functional module division and the flowchart shows a logical order, in some cases, the division into modules in the device may be different, or the steps shown or described may be performed in a different order from that in the flowchart. The terms "first", "second", etc. in the specification, claims, or above drawings are intended to distinguish similar objects, and are not necessarily intended to describe a specific order or a chronological order.

[0017] The present application provides a three-dimensional recognition device, a terminal, a calibration method, and a storage medium, and the three-dimensional recognition device includes an RGB unit including an RGB camera and a first lens, where a first optical path is connected between the first lens and the RGB camera, an infrared transmission unit including an infrared floodlight irradiator and a second lens adjacent to the first lens, where a second optical path is connected between the infrared floodlight irradiator and the second lens, and an infrared receiving unit including a first infrared camera and a third lens adjacent to the first lens, where a third optical path is connected between the first infrared camera and the third lens. According to the technical solution of this embodiment, the RGB camera, the infrared floodlight irradiator, and the first infrared camera can be arranged separately from the lens, and the propagation of light rays can be realized through the optical path, and the arrangement of multiple lenses under the display screen can be made more compact, so as to effectively reduce the area of ​​the area where special processing is performed to enhance light transmittance, effectively improve the display effect of the full-screen display, and improve the user experience.

[0018] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0019] An embodiment of the present application provides a three-dimensional recognition device disposed inside a display screen of a terminal, the three-dimensional recognition device comprising: an RGB unit including an RGB camera 210 and a first lens 212, the first lens 212 and the RGB camera 210 being connected by a first optical path 211; an infrared transmission unit including an infrared floodlight irradiator 310 and a second lens 312 adjacent to the first lens 211, with a second optical path 311 being connected between the infrared floodlight irradiator 310 and the second lens 312; The infrared receiving unit includes a first infrared camera 410 and a third lens 412 adjacent to the first lens 212, and a third optical path 411 is connected between the first infrared camera 410 and the third lens 412.

[0020] In addition, in the layout method of a general three-dimensional recognition device, as shown in FIG. 1, the three-dimensional recognition elements are arranged in parallel on the display screen 110 of the terminal 10, and the body size of the three-dimensional recognition element is large and there is a certain distance between each lens, so the area of ​​the light-transmitting reinforced area 120 is large, which affects the display experience of the full-screen display. By using the three-dimensional recognition device of this embodiment, it is possible to realize the separation of the body of the three-dimensional recognition element and the lens. In the manner shown in FIG. 2, the lens can be arranged compactly by considering only the size of the lens. Since the body of the three-dimensional recognition element is distributed inside the terminal 10 and the light path realizes the propagation of the light ray, the area of ​​the light-transmitting reinforced area 120 can be effectively reduced and the user experience can be improved.

[0021] In addition, since the first lens 212 of the RGB camera 210 is large in size, and the lenses of the infrared transmitting unit and the infrared receiving unit are small in size and usually small-diameter lenses, as shown in FIG. 2, the first lens 212 can be surrounded by multiple small-diameter lenses, making the arrangement more compact.

[0022] In this embodiment, the second lens 312 and the third lens 412 are small-diameter lenses, the amount of transmitted light required for the RGB camera 210 is large, and the diameters of the second lens 312 and the third lens 412 are relatively small compared to the diameter of the first lens 212, so even if the second lens 312 and the third lens 412 are disposed adjacent to the first lens 212, they will not interfere with the main body of the RGB camera 210. The diameters of the second lens 312 and the third lens 412 may be adjusted according to the actual requirements for the amount of incident light, and the present specification does not particularly limit the specific sizes.

[0023] Note that, referring to FIG. 3, as long as it is possible to ensure that light can enter or exit, the first optical path 211, the second optical path 311 and the third optical path 411 may have any shape, and are not particularly limited in this specification.

[0024] Various embodiments of the three-dimensional recognition device will be described below using several examples.

[0025] First Example: Referring to FIG. 3, FIG. 3 is a cross-sectional view of the terminal cut in the vertical direction in a horizontally placed state. When the RGB lens 210 is connected to the first lens 212 through the first optical path 211, there is a certain distance between the RGB lens 210 and the first lens 212. In order to realize a compact arrangement, the second lens 312 and the third lens 412 may be provided adjacent to the first lens 212, i.e., surrounding the periphery of the first lens 212, as shown in FIG. 5. As long as there is no mutual interference, the specific positions may be adjusted according to actual needs, and this embodiment does not particularly limit this.

[0026] In this embodiment, when the body size of the RGB camera 210, the infrared floodlight irradiator 310, and the first infrared camera 410 is larger than the corresponding lens, and the optical paths are all linear paths, in order to realize the layout of the rays as shown in FIG. 5, it is necessary to provide a relatively long first optical path 211 so that the infrared floodlight irradiator 310, the first infrared camera 410, and the RGB camera 210 do not interfere with each other, which makes the terminal thicker. Therefore, in this embodiment, the second optical path 311 and the third optical path 411 are both made into a right-angle structure as shown in FIG. 3, and a first reflecting mirror 313 is provided at the corner of the second optical path 311, and a second reflecting mirror 413 is provided at the corner of the third optical path 411. By changing the propagation direction of the infrared light using the reflecting mirror, the infrared floodlight irradiator 310 and the first infrared camera 410 can be shifted laterally from the RGB camera 210. The resulting internal layout may be referred to the front view shown in FIG. 4. This effectively reduces the vertical space required for the three-dimensional recognition device, and effectively reduces the thickness of the three-dimensional recognition device.

[0027] In addition, by using the configuration of this embodiment, three-dimensional recognition using time of flight (TOF) becomes possible. Based on this, a high-power vertical cavity surface emitting laser (VCSEL) is adopted for the infrared flood light irradiator 310, and as long as the TOF can be collected accurately, the inside of the first infrared camera 410 can be adjusted according to actual needs, and the internal configuration of the element is not particularly limited in this embodiment.

[0028] Second Example: Based on the first embodiment, the infrared transmission unit of this embodiment further includes an infrared dot projector 320 and a fourth lens 322 adjacent to the first lens 212, and a fourth optical path 321 is connected between the infrared dot projector 320 and the fourth lens 322. In addition, the fourth optical path 321 is further provided with a third reflecting mirror (not shown) used to reflect the infrared light emitted from the infrared dot projector 320 to the fourth lens. The principles of the setting of the infrared dot projector 320, the fourth lens 322, the fourth optical path 321 and the third reflecting mirror can be referred to the infrared flood light irradiator 310, the second lens 312, the second optical path 311 and the first reflecting mirror 313, and the redundant description will be omitted here for brevity.

[0029] 4 and 6, the fourth lens 322, the second lens 312 and the third lens 412 are arranged to surround the first lens 311, and their specific positions can be adjusted according to actual needs, and are not particularly limited in this embodiment.

[0030] In the first embodiment, a high-output VCSEL was used for the infrared floodlight irradiator 310 to realize three-dimensional recognition by TOF, but since the configuration of this embodiment is a monocular structured light system, a relatively low-output VCSEL may be used for the infrared floodlight irradiator 310, and a high-output VCSEL may be used for the infrared dot projector 320 to project laser speckles. Also, since the configuration of the first infrared camera 410 can be simplified compared to the first embodiment, the internal configuration of the element will not be particularly described in this specification.

[0031] Third Example: Based on the second embodiment, the infrared receiving unit of this embodiment includes a second infrared camera 420 and a fifth lens 422 adjacent to the first lens 212, and a fifth optical path 421 is connected between the second infrared camera 420 and the fifth lens 422. In addition, the fifth optical path 421 is further provided with a fourth reflecting mirror (not shown) used to reflect the light incident through the fifth lens 422 to the second infrared camera 420. The principles of setting the second infrared camera 420, the fifth lens 422, the fifth optical path 421, and the fourth reflecting mirror can be referred to the first infrared camera 410, the third lens 412, the third optical path 411, and the second reflecting mirror, and for convenience and brevity, duplicated explanations will be omitted here.

[0032] 4 and 7, the fifth lens 422, the fourth lens 322, the second lens 312 and the third lens 412 are arranged to surround the first lens 311, and their specific positions can be adjusted according to actual needs, and are not particularly limited in this embodiment.

[0033] In addition, while the second embodiment can realize three-dimensional recognition by monocular structured light, the present embodiment adds a second infrared camera 420, so that three-dimensional recognition by binocular structured light can be realized. The specific element specifications can be selected according to actual needs, and are not particularly limited here.

[0034] According to the three embodiments of the three-dimensional recognition device described above, the main body of the recognition element can be arranged by utilizing the internal space of the terminal, and the area of ​​the light-transmitting enhanced area 120 can be effectively reduced, thereby reducing the display area affected by the light-transmitting enhanced area 120 and improving the full-surface display effect of the terminal, thereby improving the user experience.

[0035] Further, referring to FIG. 2, an embodiment of the present application further provides a terminal, the terminal 10 comprising: The three-dimensional recognition device according to the above embodiment; The display screen includes a display screen 110, the three-dimensional recognition device is disposed inside the display screen 110, and an area of ​​the display screen 110 corresponding to the lens of the three-dimensional recognition device is an enhanced light-transmitting area 120.

[0036] In addition, referring to the description of the first to third embodiments, the number of lenses of the three-dimensional recognition device depends on the solution of the three-dimensional recognition device. As shown in FIG. 4, which is a schematic diagram of the layout of the three-dimensional recognition device corresponding to the third embodiment, when it is necessary to realize three-dimensional recognition by TOF using the configuration of the first embodiment, the infrared dot projector 320, the second infrared camera 420, and the corresponding optical path and reflecting mirror may be reduced based on FIG. 4. In this case, the area of ​​the light-transmitting reinforced region 120 can be further reduced based on FIG. 4. The specific area of ​​the light-transmitting reinforced region 120 may be adjusted according to the number of lenses corresponding to the configuration required for three-dimensional recognition.

[0037] Further, referring to FIG. 8, the present application further provides a calibration method applied to the three-dimensional recognition device described in the first embodiment, the calibration method comprising: S810, when the infrared floodlight illuminator is in an operational state, taking a first set of images using a first infrared camera and taking a second set of images using an RGB camera according to a first predefined timing sequence, where target objects taken by the first infrared camera at different times are different, target objects taken by the RGB camera at different times are different, and target objects taken by the first infrared camera and the RGB camera at the same time are the same; A step S820 of calibrating between a first infrared camera and an infrared floodlight illuminator based on the first set of images; The method includes a step S830 of fusing images taken at the same time in the first image set and the second image set to obtain a first fused image set, and performing calibration between the RGB camera and the first infrared camera based on the first fused image set.

[0038] In addition, in the conventional layout method shown in FIG. 1, since the baselines 130 between the lenses overlap, the multiple photosensitive elements can be directly calibrated, but as shown in FIG. 5, the second lens 312 and the third lens 412 are adjacent to the first lens 212, and it is not possible to guarantee that the first baseline 510 between the second lens 312 and the first lens 212 and the second baseline 520 between the third lens 412 and the first lens 212 are on the same line. Therefore, based on the configuration of the three-dimensional recognition device of the embodiment of the present application, in order to ensure that the multiple optical sensing elements that are not distributed in parallel can be properly calibrated and can be used in combination, it is necessary to calibrate the optical sensing elements two by two, that is, to calibrate the first infrared camera 410 and the RGB camera 210, and to calibrate the infrared flood light irradiator 310 and the RGB camera 210.

[0039] In addition, in the calibration process of the photosensitive element, the target object may be a calibration object with clear contrast, such as a chessboard pattern or a dot pattern, and the same pattern may be photographed from different angles using the photosensitive element. For example, in this embodiment, the calibration object shown in FIG. 13 may be used as the target object, and this target object has four sides, each side has the same chessboard pattern, and the angles of the patterns of the chessboard pattern on each side are different, so that the image contents obtained by photographing different sides are different. Furthermore, the target object shown in FIG. 13 may be provided with a mechanical device so that the image contents can be photographed from more various angles by realizing the forward or backward tilt of each side under the control of a command. In addition, when adopting a target object such as that shown in FIG. 13, a background infrared light source or a background visible light source may be added according to actual needs to ensure sufficient light, but the description is omitted here.

[0040] The three-dimensional recognition device may be fixed and installed so that the RGB camera 210 views the target object head-on. Since the first lens 212 is adjacent to the second lens 312 and the third lens 412 and has a small aperture, it can be considered to view the calibration object head-on.

[0041] It should be noted that the first predefined timing sequence is the interval time between two adjacent shots, and may be set to be the same as the rotation period of the target object to realize automatic shooting. For example, when taking one shot per side, if it takes 2 seconds to rotate the target object from one side to another, the first predefined timing sequence may be 0 seconds, 2 seconds, 4 seconds, etc. Those skilled in the art may set the corresponding parameters so that the two devices can work together to take pictures.

[0042] In order to determine the calibration parameters, it is necessary to take a plurality of images using the first infrared camera 410 and the RGB camera 210, and the specific number may be determined based on the number of homography matrices corresponding to the number of solutions of the internal parameters among the calibration parameters. For example, if the number of solutions of the internal parameters is n, and n (n is an even number) or n+1 (n is an odd number) equations that can be solved by the least squares method are required to be solved, and the number of homography matrices in this case is n / 2 (n is an even number) or (n+1) / 2 (n is an odd number), the number of times of taking images of a target object with a different inclination in the next revolution can be determined as n / 2 (n is an even number) or (n+1) / 2 (n is an odd number). Of course, if calibration can be realized, the number of images can be increased according to actual needs, and is not particularly limited in this specification.

[0043] It should be noted that the reflected image of the infrared floodlight irradiator 310 is a two-dimensional image, and the two-dimensional image can be used to estimate the plane equation of the target object's plane in the camera coordinate system, and then associate the laser point cloud of the infrared floodlight irradiator 310 with the plane, and use the mapping relationship from the laser coordinate system to the camera coordinate system to transform the laser point cloud into the camera coordinate system to establish the minimum distance from each point in the laser point cloud to the plane, and use the least squares method to obtain the value of this minimum distance to form the calibration. The above calibration method is only an example, and the calibration between the first infrared camera 410 and the RGB camera 210 can also be realized by other methods after providing the first image set, and is not particularly limited in this specification.

[0044] It should be noted that the images in the first and second image sets are taken at the same time using different devices, so that the images taken at the same time can be fused. With the first fused image set, those skilled in the art are familiar with how to calibrate the RGB camera 210, and can obtain the respective intrinsic parameters and distortion coefficients by, for example, Zhang Zhengyou's method, OpenCV, Matlab, etc., but the description is omitted here.

[0045] Also, referring to FIG. 9, the calibration method of the present application can be applied to the three-dimensional recognition device described in the second embodiment described above, S910, when the infrared dot projector is in an operational state, taking a third set of images using the first infrared camera and taking a fourth set of images using the RGB camera according to a second predefined timing sequence, where the target objects taken by the first infrared camera at different times are different, the target objects taken by the RGB camera at different times are different, and the target objects taken by the first infrared camera and the RGB camera at the same time are the same; A step S920 of calibrating between the first infrared camera and the infrared dot projector based on the third set of images; The method further includes, but is not limited to, a step S930 of fusing images taken at the same time in the third image set and the fourth image set to obtain a second fused image set, and performing calibration between the RGB camera and the first infrared camera based on the second fused image set.

[0046] In addition, when two infrared emitting elements are provided, the first image set may be taken before the third image set is taken. Of course, in order to improve efficiency, the infrared flood light irradiator 310 and the infrared dot projector 320 may be alternately operated to take pictures, that is, the first image set and the third image set may be alternately taken. A specific operation method may be selected according to timing needs, and is not particularly limited in this specification.

[0047] The photographing method and principle of the fourth image set may refer to the photographing method and principle of the second image set in the embodiment shown in FIG. 8, and a duplicated description will be omitted here.

[0048] In addition, by alternately operating the infrared dot projector 320 and the infrared flood light illuminator 310, it is possible to prevent the laser speckles of the infrared dot projector 320 from interfering with the infrared light emitted from the infrared flood light illuminator 310.

[0049] Note that the method for acquiring the second fusion image set and the method for calibrating the RGB camera 210 and the first infrared camera 410 may refer to the description of the embodiment shown in FIG. 8, and duplicated description will be omitted here.

[0050] It should be noted that in this embodiment, the infrared light emitted by the infrared dot projector 320 is a laser speckle, and the first infrared camera 410 receives a number of three-dimensional laser speckle images irradiated and reflected by the target object by the infrared dot projector 320, and then converts the three-dimensional laser speckle images to correspond to the two-dimensional images captured by the first infrared camera 410. In this conversion process, homogeneous coordinates are introduced to realize the conversion from three-dimensional space points to two-dimensional images, thereby realizing the translation and rotation operations for each point, and further obtaining plane equations in the camera coordinates of multiple sets of target objects, optimizing the distance error between the points and the plane to realize calibration.

[0051] In addition, referring to FIG. 10, the calibration method of the present application can be applied to the three-dimensional recognition device described in the third embodiment described above, S1010, when the infrared dot projector is in an operational state, taking a third set of images using a second infrared camera and taking a fourth set of images using an RGB camera according to a second predefined timing sequence, where target objects taken by the second infrared camera at different times are different, target objects taken by the RGB camera at different times are different, and target objects taken by the first infrared camera and the RGB camera at the same time are the same; S1020 performing a calibration between the second infrared camera and the infrared dot projector based on the third set of images; The method further includes, but is not limited to, a step S1030 of fusing images taken at the same time in the third image set and the fourth image set to obtain a second fused image set, and performing calibration between the RGB camera and the second infrared camera based on the second fused image set.

[0052] In addition, the technical principle of this embodiment may refer to the principle of the embodiment described in Figure 9, and the difference is that the third image set is obtained by photographing with the second infrared camera 420, that is, the first infrared camera 410 operates in accordance with the infrared flood light illuminator 310, and the second infrared camera 420 operates in accordance with the infrared dot projector 320, of course, the relationship of the operations in accordance with each other may be reversed, and is not particularly limited in this specification.

[0053] In addition to the above differences, the first image set may also be captured once by the first infrared camera 410 and the second infrared camera 420 when the infrared floodlight illuminator 310 is in operation, to obtain two available images for cross-referencing, which can effectively improve the efficiency of capture and calibration. A similar operation may also be performed for the third image set, and the description will be omitted here.

[0054] In addition, after the first fusion image set is used to calibrate the first infrared camera 410 and the RGB camera 210, the second fusion image set must be used to calibrate the second infrared camera 420 and the RGB camera 210. Because the baselines of the photosensitive elements are not on the same line, two calibrations must be performed at a time to ensure the normal operation of the three-dimensional recognition device.

[0055] In addition, when there are two infrared cameras and two infrared emitters, and it is necessary to alternately operate the infrared dot projector 320 and the infrared floodlight irradiator 310, according to the description of the above embodiment, the target object is rotated according to a predetermined timing sequence, i.e., the first and second image sets are taken for the first side, and the third and fourth image sets are taken for the second side. Even if the tilt angle of the side of the target object to be photographed can be adjusted by controlling the mechanical device, it is difficult to ensure that the images taken by the target object in one rotation are sufficient. In this case, after the first rotation of the target object is completed, the first rotation in the second rotation may be increased by 90 degrees. That is, by taking the first and second image sets of the second side, it is ensured that the images taken in each image set are different from the images taken in the first rotation.

[0056] Of course, if the number of images captured in the second rotation is still insufficient, the target object may be rotated a third rotation at a non-uniform angle or a small rotation angle, and the first infrared camera 410, the second infrared camera 420, and the RGB camera 210 may be used to synchronously capture the sides of the target object. Rotating the calibration object at a non-uniform angle or a small rotation angle may result in the camera capturing images across two sides or only capturing a local portion of a single side, which may result in a large difference compared to the previous two rotations. Therefore, this one rotation is merely a supplementary capture of the previous two rotations, which on the one hand only supplements a small amount of images for correction against interference caused by laser speckles, and on the other hand is also used for effective supplementation or supplementation when the images of the previous two rotations of the target object are insufficient. The above-mentioned adjustment method of the target object is only one example of the present embodiment, and may be adjusted according to timing needs or multiple different target objects may be arranged, but is not particularly limited in this specification.

[0057] In addition, in one embodiment, referring to FIG. 11, after completing execution of step S910 shown in FIG. 9 or completing execution of step S1010 shown in FIG. 10, the method further includes, but is not limited to, the following steps:

[0058] In step S1110, laser speckle information in the images of the third image set is filtered out.

[0059] In addition, when the infrared dot projector 320 is operating, the image captured by the second infrared camera 420 contains laser speckle information, which must be filtered and removed by noise removal to avoid interference with the calibration. For example, to eliminate multiplicative distortion due to speckle, noise may be converted to an additive model using natural logarithm during iteration, and the image may be converted from RGB space to Hue Lightness Saturation (HLS) space, the red spatial range may be extracted, inversely converted to RGB space, and further converted to grayscale space, histogram equalization and filtering may be performed, corner points may be calculated to generate a chessboard image, and then calibration may be performed.

[0060] In addition, in one embodiment, referring to FIG. 12, after completing execution of step S1030 shown in FIG. 10, the method further includes, but is not limited to, the following steps:

[0061] A step S1210 of obtaining calibration parameters including a first calibration parameter of a first infrared camera, a second calibration parameter of a second infrared camera, and a third calibration parameter of an RGB camera; The method further includes, but is not limited to, a step S1220 of performing a global calibration among the first infrared camera, the second infrared camera, and the RGB camera based on the calibration parameters.

[0062] After completing the calibration of the first infrared camera 410 and the RGB camera 210, and the calibration of the second infrared camera 420 and the RGB camera 210, it is necessary to perform global calibration for three more cameras to realize coordinated operation. Based on this, it is necessary to calculate the respective calibration parameters, where the calibration parameters usually include intrinsic parameters, distortion coefficients, extrinsic parameters and image scale factors.

[0063] In addition, the internal parameters and distortion coefficients may be obtained by calculation by fusing photos of the target object taken by each of the three cameras after one or more rotations. Those skilled in the art will be familiar with how to calculate the corresponding parameters, so the explanation will be omitted here.

[0064] In addition, the calculation of the external parameters requires that the three cameras simultaneously perform one shooting, that is, the three cameras simultaneously shoot the same stationary target object, and then perform feature calculation for each image. The calculation of the external parameters is a technique well known to those skilled in the art, so the description will be omitted here.

[0065] The image scale factor represents the difference in the formation of two types of images of a spatial object due to the misalignment of the optical centers of the infrared camera and the RGB camera and the difference in the focal length between infrared and visible light. This allows the image scale factor to be obtained by comparing the pixel difference between infrared and RGB in the two-dimensional calibration image of the side of the captured target object, thereby realizing the unification of the size of the spatial object across the two types of images, infrared and RGB.

[0066] Furthermore, even if the sizes of the two types of images, infrared and RGB, are unified, there will still be a deviation when moving from the infrared image to the RGB image. However, the corresponding pixel difference can be calculated based on the coordinate positions of the chessboard or dots in the calibration diagram on the side of the calibration object and the pixel coordinate positions of the infrared and RGB images, so that the infrared and RGB pixels can be aligned.

[0067] Further, referring to FIG. 14, an embodiment of the present application further provides a terminal, the terminal 1400 including a memory 1410 , a processor 1420 and a computer program stored in the memory 1410 and executable on the processor 1420 .

[0068] The processor 1420 and the memory 1410 may be connected by a bus or in another manner.

[0069] Non-transitory software programs and instructions necessary to realize the calibration methods of the above-mentioned embodiments are stored in memory 1410 and, when executed by processor 1420, perform the calibration methods of the above-mentioned embodiments, such as method steps S810 to S830 in FIG. 8, method steps S910 to S930 in FIG. 9, method steps S1010 to S1030 in FIG. 10, method step S1110 in FIG. 11, and method steps S1210 to S1220 in FIG. 12.

[0070] The above described device embodiments are merely exemplary, and the units described as separate parts may or may not be physically separated, i.e., located in one place or distributed among multiple network units. Some or all of these modules can be selected according to actual needs to achieve the objectives of the aspects of the present embodiment.

[0071] An embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor or controller, such as a processor in the above-mentioned terminal embodiment, the processor can execute the calibration method in the above-mentioned embodiment, such as the above-mentioned method steps S810 to S830 in FIG. 8, method steps S910 to S930 in FIG. 9, method steps S1010 to S1030 in FIG. 10, method step S1110 in FIG. 11, and method steps S1210 to S1220 in FIG. 12. Those skilled in the art can understand that all or part of the steps of the above-disclosed method and system can be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all of the physical units may be implemented as software executed by a processor such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit such as an application specific integrated circuit. Such software may be distributed on computer readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). Those skilled in the art will recognize that the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (computer readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cartridges, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer.Additionally, communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and may include any information delivery media, as known to those skilled in the art.

[0072] The embodiment of the present application includes an RGB unit including an RGB camera and a first lens, the RGB unit having a first optical path connected between the first lens and the RGB camera, an infrared transmission unit including an infrared floodlight irradiator and a second lens adjacent to the first lens, the infrared transmission unit having a second optical path connected between the infrared floodlight irradiator and the second lens, and an infrared receiving unit including a first infrared camera and a third lens adjacent to the first lens, the infrared receiving unit having a third optical path connected between the first infrared camera and the third lens. According to the technical solution of this embodiment, the RGB camera, the infrared floodlight irradiator and the first infrared camera can be arranged separately from the lens, the propagation of light rays can be realized through the optical path, and the arrangement of multiple lenses under the display screen can be made more compact, so as to effectively reduce the area of ​​the area where special processing is performed to enhance light transmittance, effectively improve the display effect of the full-screen display, and improve the user experience.

[0073] Although some examples of the present application have been specifically described above, the present application is not limited to the above embodiments. Those skilled in the art may make various equivalent modifications or substitutions within the scope of the claims of the present application without departing from the essence of the present application.

Claims

1. A three-dimensional recognition device provided inside a display screen of a terminal, a red-green-blue RGB unit including an RGB camera and a first lens, the red-green-blue RGB unit being connected with a first optical path between the first lens and the RGB camera; an infrared transmission unit including an infrared floodlight illuminator and a second lens adjacent to the first lens, a second optical path being connected between the infrared floodlight illuminator and the second lens; an infrared receiving unit including a first infrared camera and a third lens adjacent to the first lens, the infrared receiving unit having a third optical path connected between the first infrared camera and the third lens; Including, The infrared transmitting unit further includes an infrared dot projector and a fourth lens adjacent to the first lens, and a fourth optical path is connected between the infrared dot projector and the fourth lens; A three-dimensional recognition device, wherein the fourth lens, the second lens, and the third lens are arranged to surround the first lens, and the first lens has a larger aperture than the fourth lens, the second lens, and the third lens.

2. The second optical path further includes a first reflecting mirror for reflecting the infrared light emitted from the infrared floodlight irradiator to the second lens; The third optical path further includes a second reflecting mirror that is used to reflect the light incident through the third lens to the first infrared camera. The three-dimensional recognition device according to claim 1.

3. The fourth optical path further includes a third reflecting mirror used to reflect the infrared light emitted from the infrared dot projector to the fourth lens. The three-dimensional recognition device according to claim 2.

4. the infrared receiving unit further includes a second infrared camera and a fifth lens adjacent to the first lens, and a fifth optical path is connected between the second infrared camera and the fifth lens; The fifth optical path further includes a fourth reflecting mirror that is used to reflect the light incident through the fifth lens to the second infrared camera. The three-dimensional recognition device according to claim 3.

5. A three-dimensional recognition device according to any one of claims 1 to 4; A display screen, the three-dimensional recognition device being disposed inside the display screen, and an area of ​​the display screen corresponding to a lens of the three-dimensional recognition device being an enhanced light-transmitting area; The device includes:

Images