Depth camera 3D real-time surface capturing system

Abstract

The utility model provides a 3D real-time surface capturing system of a depth camera, and relates to the technical field of computer vision and graphics. Fixing seats are fixed to the left side and the right side of the rear end plate, sponge pads are fixed to the two fixing seats in an adhesive mode, a head fixing plate is fixed to the upper ends of the two fixing seats, and a first movable arm plate and a second movable arm plate which can rotate are installed on the fixing seats close to the left end side and the right end side of the head fixing plate respectively; according to the utility model, the Orbian medium light Astra Pro depth camera is selected, the frame rate is 60fps, rapid actions of a human face can be completely captured, image jamming and smear are avoided, most areas of the human face are covered by a 70-degree * 60-degree field angle, shooting dead angles are reduced, data processing is efficient and accurate, algorithm optimization is excellent, data transmission is stable and rapid, and output format compatibility is strong; through cooperative work of a plurality of motors, multi-dimensional flexible rotation of the camera is realized, face information can be captured in all directions, and the accuracy and comprehensiveness of 3D real-time face capture are greatly improved.

Description

Technical Field

[0001] This utility model relates to the field of computer vision and graphics technology, and in particular to a 3D real-time face capture system for a depth camera. Background Technology

[0002] With the rapid development of the digital entertainment industry, such as film, games, virtual reality (VR), and augmented reality (AR), the demand for high-precision, real-time 3D facial capture technology is increasing. In film and television production, to create realistic virtual characters and special effects scenes, it is necessary to accurately capture actors' facial expressions and translate them into virtual character movements, thereby enhancing the film's visual effects and immersion. In game development, especially VR games, real-time and accurate facial capture allows players' expressions to be reflected on the game character in real time, significantly improving the game's interactivity and realism. Therefore, a depth camera 3D real-time facial capture system is designed.

[0003] Existing depth cameras have limitations in frame rate, field of view, and measurement accuracy. During the data processing stage, the algorithm is not effective, making it difficult to improve the quality of the processed data and providing a good foundation for subsequent analysis. Utility Model Content

[0004] This disclosure relates to a 3D real-time face capture system using a depth camera to address the problems mentioned in the background section.

[0005] In a first aspect, this disclosure provides a 3D real-time face capture system for a depth camera, specifically including: a back-end board;

[0006] The rear end plate is fixed with mounting seats on both the left and right sides. Sponge pads are glued to both sets of mounting seats. A head mounting plate is fixed to the upper end of the two sets of mounting seats. Rotatable movable arm plate No. 1 and movable arm plate No. 2 are respectively installed on the mounting seats near the left and right ends of the head mounting plate. Locking knobs are installed at the rear ends of movable arm plate No. 1 and movable arm plate No. 2. The two sets of locking knobs are screwed onto the two sets of mounting seats. A left end plate is fixed to movable arm plate No. 1 and a right end plate is fixed to movable arm plate No. 2. A middle rod is fixed between the left end plate and the right end plate. A No. 1 motor is installed at the lower end of the left end plate by bolts. A pulley is fixed to the output shaft of the No. 1 motor. A mounting plate is installed at the right end of the middle rod by bolts. A pulley is installed on the mounting plate by bearings. A connecting belt connects the two sets of pulleys.

[0007] In at least some embodiments,

[0008] A movable seat is installed on the intermediate rod, and three sets of contact wheels are installed on the movable seat. All three sets of contact wheels are in contact with the intermediate rod, and the movable seat is fixedly connected to the connecting belt by bolts.

[0009] In at least some embodiments,

[0010] A pad is installed at the lower end of the movable seat, and a U-shaped frame is installed above the movable seat. A fixing column is fixed at the lower end of the U-shaped frame, and a locking bolt is threaded between the fixing column and the pad.

[0011] In at least some embodiments,

[0012] A housing is bolted onto the U-shaped frame. Locking plates are fixed at both ends of the housing, and a connecting plate is fixed at the rear end of the housing. The connecting plate and the two sets of locking plates are all connected to the U-shaped frame by bolts.

[0013] In at least some embodiments,

[0014] A ring frame is installed at the rear end of the housing, and a transparent cover is fixed at the rear end of the ring frame. Locking rings are connected between the ring frame and the two sets of locking plates.

[0015] In at least some embodiments,

[0016] The housing contains a second motor, whose output shaft is fixedly connected to the housing. A support plate is bolted to the second motor, a base plate is bolted to the support plate, and a third motor is bolted to the base plate. A locking seat is bolted to the output shaft of the third motor, and a fourth motor is bolted to the locking seat. A frame is fixed to the output shaft of the fourth motor, and a camera is bolted to the frame.

[0017] In at least some embodiments,

[0018] The camera has a resolution of 1280×720, can simultaneously capture two-dimensional color images of faces and depth information, has a frame rate of 60fps, a field of view of 70°×60°, is equipped with a USB 3.2 Gen2 interface and a 10 Gigabit Ethernet card, and has an internal data processing module.

[0019] This invention provides a 3D real-time face capture system for a depth camera, which has the following advantages:

[0020] In this invention, the second motor can drive the entire camera assembly to rotate, the third motor can adjust the camera's angle in the horizontal direction, and the fourth motor can adjust the camera's angle in the vertical direction. Through the coordinated work of multiple motors, the camera can rotate flexibly in multiple dimensions, enabling it to capture facial information from all directions and greatly improving the accuracy and comprehensiveness of 3D real-time face capture.

[0021] Furthermore, this invention utilizes the Orbbec Astra Pro depth camera, which, with a frame rate of 60fps, can fully capture fast facial movements, avoiding image stuttering and ghosting. Its 70°×60° field of view covers most of the face area, reducing blind spots. The camera also features efficient and accurate data processing, excellent algorithm optimization, stable and fast data transmission, and strong output format compatibility. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this utility model, the accompanying drawings of the embodiments will be briefly described below.

[0023] The accompanying drawings described below are only related to some embodiments of the present invention and are not intended to limit the scope of the present invention.

[0024] In the attached diagram:

[0025] Figure 1 The system flowchart of this application is shown;

[0026] Figure 2 A schematic diagram of the overall structure of this application is shown;

[0027] Figure 3 A structural schematic diagram of the intermediate rod portion of this application is shown;

[0028] Figure 4 This application shows Figure 3 A magnified structural diagram of part A in the middle;

[0029] Figure 5 A structural schematic diagram of the U-shaped frame portion of this application is shown;

[0030] Figure 6 This application shows Figure 5 A magnified structural diagram of part B in the middle section;

[0031] Figure 7 A schematic diagram of the camera portion of this application is shown.

[0032] List of reference numerals

[0033] 1. Rear end plate; 11. Fixing seat; 111. Sponge pad; 12. Head fixing plate; 13. No. 1 movable arm plate; 131. Left end plate; 1311. No. 1 motor; 1312. Connecting belt; 132. Locking knob; 14. No. 2 movable arm plate; 141. Right end plate; 1411. Mounting plate; 15. Intermediate rod; 16. Movable seat; 161. Contact wheel; 162. Pad; 163. Locking bolt;

[0034] 2. U-shaped frame; 21. Fixed column; 22. Housing base; 221. Locking plate; 222. Connecting plate; 23. Ring frame; 231. Transparent cover; 232. Locking ring; 24. Motor No. 2; 241. Support plate; 242. Seat plate; 25. Motor No. 3; 251. Locking seat; 26. Motor No. 4; 27. Frame; 28. Camera. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this utility model clearer, the technical solutions of the embodiments of this utility model will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this utility model. Based on the described embodiments of this utility model, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this utility model.

[0036] Example 1: Please refer to Figures 1 to 7 :

[0037] This utility model proposes a 3D real-time face capture system for a depth camera, including: a back-end board 1;

[0038] Both sides of the rear end plate 1 are fixed with mounting bases 11. Sponge pads 111 are glued to both sets of mounting bases 11. A head mounting plate 12 is fixed to the upper end of both sets of mounting bases 11. Rotatable movable arm plates 13 and 14 are respectively installed on the mounting bases 11 near the left and right ends of the head mounting plate 12. Locking knobs 132 are installed at the rear ends of both movable arm plates 13 and 14. The two sets of locking knobs 132 are screwed onto the two sets of mounting bases 11. A left end plate 131 is fixed on the movable arm plate 13, and a right end plate 141 is fixed on the second movable arm plate 14. A middle rod 15 is fixed between the left end plate 131 and the right end plate 141. A first motor 1311 is installed at the lower end of the left end plate 131 by bolts. A pulley is fixed on the output shaft of the first motor 1311. A mounting plate 1411 is installed at the right end of the middle rod 15 by bolts. A pulley is installed on the mounting plate 1411 by bearings. A connecting belt 1312 connects the two sets of pulleys.

[0039] A movable seat 16 is installed on the intermediate rod 15. Three sets of contact wheels 161 are installed on the movable seat 16. All three sets of contact wheels 161 are in contact with the intermediate rod 15. The movable seat 16 is fixedly connected to the connecting belt 1312 by bolts.

[0040] In this embodiment of the disclosure,

[0041] A pad 162 is installed at the lower end of the movable base 16, and a U-shaped frame 2 is installed above the movable base 16. A fixing post 21 is fixed at the lower end of the U-shaped frame 2. A locking bolt 163 is threaded between the fixing post 21 and the pad 162. A housing 22 is installed on the U-shaped frame 2 by bolts. Locking plates 221 are fixed at both ends of the housing 22. A connecting plate 222 is fixed at the rear end of the housing 22. The connecting plate 222 and the two sets of locking plates 221 are all connected to the U-shaped frame 2 by bolts. A ring frame 23 is installed at the rear end of the housing 22. A transparent cover 231 is fixed at the rear end of the ring frame 23. Locking rings 232 are connected between the ring frame 23 and the two sets of locking plates 221. The function of the ring frame 23 and the transparent cover 231 installed at the rear end of the housing 22 is to provide protection for the internal camera 28 and other key equipment, effectively blocking dust, water vapor and other pollutants from entering and extending the service life of the equipment.

[0042] Example 2, based on Example 1,

[0043] The housing 22 houses a second motor 24, whose output shaft is fixedly connected to the housing 22. A support plate 241 is bolted to the second motor 24, and a base plate 242 is bolted to the support plate 241. A third motor 25 is bolted to the base plate 242, and a locking seat 251 is bolted to the output shaft of the third motor 25. A fourth motor 26 is bolted to the locking seat 251, and a frame 27 is fixed to the output shaft of the fourth motor 26. A camera 28 is bolted to the frame 27. The functions of the second motor 24 are as follows: the second motor 24 drives the entire assembly mounted on it to rotate relative to the housing 22; the third motor 25 can adjust the horizontal angle of the locking seat 251 and subsequent assemblies; and the fourth motor 26 can adjust the vertical angle of the frame 27 and the camera 28. Through the coordinated work of these three motors, the camera 28 can achieve flexible rotation in multiple dimensions, thereby capturing facial information from all directions and greatly improving the accuracy and comprehensiveness of 3D real-time face capture.

[0044] Example 3, based on Examples 1 and 2,

[0045] Camera 28 has a resolution of 1280×720 and can simultaneously capture 2D color images and depth information of faces. It operates at a frame rate of 60fps, has a field of view of 70°×60°, and is equipped with a USB 3.2 Gen 2 interface and a 10 Gigabit Ethernet card. Internally, camera 28 incorporates a data processing module. Using the Orbbec Astra Pr model, the module removes image noise using Gaussian and bilateral filtering, corrects image distortion based on the Zhang Zhengyou calibration method, and improves data quality. It employs the MobileNetV3 network model, designed for mobile and embedded devices, for feature extraction. After training on a large-scale face dataset, it accurately extracts 68 key feature points. Utilizing NVIDIA RTX 3090 GPU parallel computing, the speedup is more than 10 times that of traditional CPU computing, significantly improving feature extraction speed. In the 3D reconstruction stage, based on the Poisson reconstruction algorithm and Kalman filter real-time tracking algorithm, the model is continuously updated at a frequency of 60 times per second, achieving accurate real-time face capture. It uses a USB 3.2 Gen 2 interface. With two interfaces and a transmission rate of up to 10Gbps, the system enables rapid transmission of data collected by the 28 cameras to computing devices, reducing data transmission latency. It also features a 10 Gigabit Ethernet card, further enhancing data transmission stability and speed through the Ethernet interface. The generated 3D facial models are output in FBX format, a format widely used in game development and film production. It can be directly integrated with the Unity development engine for VR / AR immersive interactive experience development and can also be imported into Autodesk Maya film production software to give animated characters realistic facial expressions, expanding the system's application scenarios.

[0046] The working principle of this embodiment is as follows: The device is worn on the user's head. The fixed seats 11 and sponge pads 111 on both sides of the rear plate 1 ensure comfortable and stable wearing. The first movable arm plate 13 and the second movable arm plate 14 can rotate and their positions can be adjusted by locking knob 132 to adapt to different user head shapes. The first motor 1311 is run so that the connecting belt 1312 drives the movable seat 16 to move flexibly on the middle rod 15, thereby driving the camera 28 mounted on the U-shaped frame 2 to adjust its position. The second motor 24 in the housing 22 can drive the entire camera 28 assembly to rotate. The third motor 25 can adjust the angle of the camera 28 in the horizontal direction, and the fourth motor 26 can adjust the angle of the camera 28 in the vertical direction, realizing the flexible rotation of the camera 28 in multiple dimensions and ensuring that facial information can be captured from all directions.

[0047] Camera 28 operates based on the principle of structured light. It projects a Gray code pattern onto the face and calculates the depth using triangulation based on the deformation of the pattern at different angles. Simultaneously, it acquires a two-dimensional color image of the face and depth information. In the data processing module, the acquired data is first preprocessed. Gaussian filtering with a standard deviation of 1.5 is used to remove image noise, and bilateral filtering preserves image edge details while removing noise. Then, camera parameters obtained based on Zhang Zhengyou calibration are used to correct image distortion and improve data quality. Next, a MobileNetV3 network model based on convolutional neural networks (CNN) is used for feature extraction. Parallel computing is performed using an NVIDIA RTX 3090 GPU to accelerate feature extraction. Finally, based on the extracted feature points and depth information, an initial 3D facial model is constructed using a Poisson reconstruction algorithm. Then, a real-time tracking algorithm based on Kalman filtering continuously updates the model at a frequency of 60 times per second according to facial dynamics. The generated 3D facial model is finally output in FBX format, which can be directly integrated into the Unity development engine or imported into Autodesk Maya video production software.

[0048] The following points should be noted in this article:

[0049] 1. The accompanying drawings of the embodiments disclosed herein only relate to the structures involved in the embodiments disclosed herein; other structures can be referred to in general design.

[0050] 2. Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0051] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A 3D real-time face capture system using a depth camera, comprising: Back-end board (1); characterized in that, The rear end plate (1) is fixed with mounting bases (11) on both the left and right sides. Sponge pads (111) are glued to both mounting bases (11). A head mounting plate (12) is fixed to the upper end of both mounting bases (11). A first movable arm plate (13) and a second movable arm plate (14) are respectively installed on the mounting bases (11) near the left and right ends of the head mounting plate (12). Locking knobs (132) are installed at the rear ends of the first movable arm plate (13) and the second movable arm plate (14). The two sets of locking knobs (132) are screwed onto the two mounting bases (11). A left end plate (131) is fixed on the first movable arm plate (13), and a right end plate (141) is fixed on the second movable arm plate (14). A middle rod (15) is fixed between the left end plate (131) and the right end plate (141). A first motor (1311) is installed on the lower end of the left end plate (131) by bolts. A pulley is fixed on the output shaft of the first motor (1311). A mounting plate (1411) is installed on the right end of the middle rod (15) by bolts. A pulley is installed on the mounting plate (1411) by bearings. A connecting belt (1312) connects the two sets of pulleys.

2. The 3D real-time face capture system for a depth camera according to claim 1, characterized in that, The intermediate rod (15) is equipped with a movable seat (16), and the movable seat (16) is equipped with three sets of contact wheels (161). All three sets of contact wheels (161) are in contact with the intermediate rod (15), and the movable seat (16) is fixedly connected to the connecting belt (1312) by bolts.

3. The 3D real-time surface capture system for a depth camera according to claim 2, characterized in that, A pad (162) is installed at the lower end of the movable seat (16), and a U-shaped frame (2) is installed above the movable seat (16). A fixing column (21) is fixed at the lower end of the U-shaped frame (2), and a locking bolt (163) is threaded between the fixing column (21) and the pad (162).

4. A 3D real-time face capture system for a depth camera according to claim 3, characterized in that, A housing (22) is bolted onto the U-shaped frame (2). Locking plates (221) are fixed at both ends of the housing (22). A connecting plate (222) is fixed at the rear end of the housing (22). The connecting plate (222) and the two sets of locking plates (221) are all connected to the U-shaped frame (2) by bolts.

5. A 3D real-time face capture system for a depth camera according to claim 4, characterized in that, The rear end of the housing (22) is equipped with a ring frame (23), and the rear end of the ring frame (23) is fixed with a transparent cover (231). The ring frame (23) and the two sets of locking plates (221) are connected with locking rings (232).

6. A 3D real-time face capture system for a depth camera according to claim 5, characterized in that, The housing (22) houses a second motor (24), the output shaft of which is fixedly connected to the housing (22). A support plate (241) is bolted to the second motor (24), a base plate (242) is bolted to the support plate (241), a third motor (25) is bolted to the base plate (242), a locking seat (251) is bolted to the output shaft of the third motor (25), a fourth motor (26) is bolted to the locking seat (251), a frame (27) is fixed to the output shaft of the fourth motor (26), and a camera (28) is bolted inside the frame (27).

7. A 3D real-time face capture system for a depth camera according to claim 6, characterized in that, The camera (28) has a resolution of 1280×720, can simultaneously capture two-dimensional color images and depth information of a face, has a frame rate of 60fps, a field of view of 70°×60°, is equipped with a USB 3.2 Gen2 interface and a 10 Gigabit Ethernet card, and has a data processing module inside the camera (28).

Images