Method and device for realizing 3D virtual mask based on monocular camera

Abstract

The invention relates to a method and a device for realizing a 3D virtual mask based on a monocular camera. The method comprises the following steps: acquiring a scene video image by using the monocular camera; acquiring a face image of each person in each frame of video image in the scene video image; extracting geometric feature information in the face image of each person; obtaining a 3D mask of each person in each frame of video image by using the extracted geometric feature information of each person; mapping the 3D mask of each person in each frame of video image to a corresponding face in each frame of video image; smoother and real-time interactive experience is provided, high synchronization with real expressions is ensured, and the immersion of the user is improved.

Description

Technical Field

[0001] The present application relates to the technical field of 3D virtual masks, and in particular to a method and device for realizing a 3D virtual mask based on a monocular camera. Background Art

[0002] With the development of augmented reality (AR) and virtual reality (VR) technologies, 3D virtual masks have become an emerging interactive experience. Existing technologies can capture the user's facial expressions through a camera and generate a 3D virtual mask that is synchronized with the user's facial expressions through image processing technology. However, these technologies are mostly limited to single-person use, and when used in multi-person environments such as scenic spots, there are challenges in real-time, accuracy, and interactivity. Although the 3D virtual mask technology currently on the market has made some progress in providing user interactive experience, it is still mainly limited to mobile devices for single use and has not yet fully met the needs of multi-person interactive experience in scenic spots.

[0003] Among them, the application publication number is CN118318252A, and the application name is a facial driving method, device, equipment and storage medium for a virtual model, which belongs to the fields of Internet and computers. The method includes: detecting and obtaining two-dimensional facial key points from a two-dimensional facial image; iteratively updating two-dimensional rigid transformation parameters and expression transformation parameters based on the two-dimensional facial key points and the three-dimensional facial key points in the three-dimensional facial model to obtain the final two-dimensional rigid transformation parameters and the final expression transformation parameters; based on the final two-dimensional rigid transformation parameters and the final expression transformation parameters, determining the target rigid transformation parameters and the target expression transformation parameters. In the technical solution provided in the embodiment of the present application, the flexibility of acquiring two-dimensional facial images is improved, so that the driving scheme of the three-dimensional virtual model has strong versatility, and the accuracy of the target rigid transformation parameters and the target expression transformation parameters is improved.

[0004] Application publication number: CN118314197A, application name: A method, device, equipment and medium for locating human eyes. According to an embodiment of the present invention, by detecting human face key points on an image to be processed including a human face, the human face key points in the image to be processed are obtained; based on the human face key points in the image to be processed, the head posture information used to indicate the deflection angle between the human head and the preset direction is determined; and based on the human eye pupil distance in the image to be processed, the camera parameters of the camera used to shoot the image to be processed, the head posture information, and the pixel coordinates of the left eye and the pixel coordinates of the right eye in the image to be processed, the three-dimensional coordinates of the human eye are determined, so as to achieve the purpose of correcting the human eye coordinates according to the head posture information, so that the position of the human eye in the image can be determined more accurately.

[0005] The problems with the above patents are:

[0006] Image processing lacks real-time interaction and does not support interactive scenes involving multiple people at the same time, which limits its potential for application in multi-person environments such as scenic spots. Most 3D virtual mask solutions rely on mobile devices, which may limit their processing power, camera quality and display effects, constraining the provision of high-quality 3D interactive experience. Summary of the invention

[0007] In view of the above problems, the present invention provides a method and device for realizing 3D virtual mask based on a monocular camera, which can be applicable to scenic environment and can realize high-precision face tracking and recognition, as well as generation and real-time mapping of 3D virtual mask.

[0008] The present application is implemented through the following technical solutions, a method for realizing a 3D virtual mask based on a monocular camera, comprising:

[0009] Use a monocular camera to obtain scene video images;

[0010] Obtaining a face image of each person in each frame of the scene video image;

[0011] Extract geometric feature information from each person's face image;

[0012] The 3D mask of each person in each frame of video image is obtained by using the extracted geometric feature information of each person;

[0013] The 3D mask of each person in each frame of video image is mapped to the corresponding face in each frame of video image.

[0014] Preferably, the method for extracting geometric feature information from each person's face image includes:

[0015] Perform face detection on each person's face;

[0016] The 2D key points of the face are extracted, and the parameters of the 3D model are obtained using the extracted 2D key points, and the 3D model parameters are used as geometric feature information.

[0017] Preferably, the 2D key points of the face include eyes, nose, mouth and eyebrows.

[0018] A device for realizing a 3D virtual mask based on a monocular camera, comprising:

[0019] An image acquisition unit collects video images in the scene using a monocular camera;

[0020] An image processing unit, used for receiving the video image collected by the image acquisition unit, and extracting the face image of each person in the received video image;

[0021] A feature extraction unit, used to extract geometric feature information from a face image;

[0022] The 3D modeling unit generates a 3D mask of each person in each frame image using several feature information extracted by the feature extraction unit;

[0023] A mapping unit maps the 3D mask generated by the 3D modeling unit onto the corresponding human face;

[0024] The display unit displays the mapped face image.

[0025] Preferably, it also includes a scene modeling unit, which generates a static scene virtual image using the scene image acquired by the image acquisition unit, and the scene virtual image includes one or more of the landmark buildings and cultural symbols in the scene.

[0026] Preferably, it also includes a data communication unit, which is used to encode and compress data during transmission.

[0027] Preferably, it also includes:

[0028] The rendering unit is used to render the 3D mask obtained in each frame image, and complete the rendering of the animation formed by the 3D mask in the continuous frame images.

[0029] An electronic device, comprising:

[0030] one or more processors;

[0031] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a method for realizing a 3D virtual mask based on a monocular camera as described above.

[0032] A computer-readable medium stores a computer program, and when the computer program is executed by a processor, a method for realizing a 3D virtual mask based on a monocular camera is implemented as described above.

[0033] The beneficial effects of this application are:

[0034] 1. Enhanced real-time performance: Through optimized data processing and multi-threaded rendering technology, it provides a smoother and more real-time interactive experience, and can maintain high frame rate and low latency even in complex scenes such as scenic spots.

[0035] 2. Improved multi-person interaction support: Simultaneously processing facial expression data from multiple users enables more natural and accurate 3D virtual mask mapping, significantly improving the application potential in multi-person environments such as scenic spots.

[0036] 3. Efficient data transmission: The use of a separate data transmission unit ensures the stability and reliability of the data flow, while improving the efficiency of data parsing and reducing compatibility issues in cross-platform applications.

[0037] 4.3D rendering capability: 3D virtual mask rendering is realized to ensure high synchronization with real expressions, enhancing the user's sense of immersion. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 This is a flow chart of a method for realizing a 3D virtual mask based on a monocular camera in an embodiment of the present invention;

[0040] Figure 2 is a block diagram of a device for realizing a 3D virtual mask based on a monocular camera in an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of the structure of the computer system of the electronic device in this embodiment;

[0042] Figure 4 Schematic diagram of a virtual mask in this embodiment. DETAILED DESCRIPTION

[0043] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0044] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, systems, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0045] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0046] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0047] like Figure 1 The embodiment shown provides a method for realizing a 3D virtual mask based on a monocular camera, comprising:

[0048] include:

[0049] A monocular camera is used to acquire scene video images. The monocular camera in this embodiment uses a high-resolution and high-frame-rate camera to capture high-quality real-time video images and provide raw data for facial feature capture. When acquiring scene video images, the captured images may be multi-person scene video images. The scene images in this embodiment may be multi-person scene video images of up to 5 people. At the same time, the scenes provided by this embodiment may be complex scenes such as tourist attractions. The use of a monocular camera can integrate advanced image sensors and optical systems, and can achieve high-precision face tracking and recognition.

[0050] Obtain the facial image of each person in each frame of the scene video image; this embodiment uses an image segmentation method to extract the facial image in the video image, or uses a deep learning-based face detection and recognition algorithm to detect and recognize the facial images of 5 people in the video image in real time. The method for extracting facial images used in this embodiment belongs to the conventional technology in this technical field and will not be described in detail here.

[0051] Extract geometric feature information from each person's face image; after completing the extraction of the face image, it is necessary to extract the geometric feature information of each face to achieve high-precision face tracking and recognition. When extracting facial geometric feature information, this embodiment is developed using the Python programming language, taking advantage of its powerful data processing capabilities and rich library function support; on this basis, the open source Mediapipe framework, an advanced computer vision framework, is integrated for real-time face detection and feature point tracking.

[0052] By applying machine learning models, the Mediapipe framework can stably identify and track facial key points in various complex environments, including eyebrows, eyes, nose, mouth and other key areas. In addition, the module can also obtain expression fusion information blendshape, which is a parametric representation method defined by the Facial Action Coding System (FACS) that can describe the changes in facial expressions in detail and provide rich details for the dynamic modeling of 3D virtual masks;

[0053] In order to further improve the capture accuracy, image processing technologies are also implemented, including but not limited to noise reduction, sharpening and contrast enhancement, to ensure that the user's facial features can be accurately captured even in poor lighting conditions or complex backgrounds. Through the combination of these technologies, the facial feature capture module can provide stable and accurate facial expression data for the generation of 3D virtual masks, meeting the high requirements for real-time and accuracy in multi-person interactive environments such as scenic spots.

[0054] Specifically, when extracting geometric features, a face recognition algorithm is used for implementation. This algorithm is based on the Mediapipe library and is responsible for capturing and analyzing facial images from video streams in real time. First, face detection is performed to locate the face and track its dynamic changes. Then, Mediapipe is used to extract 2D key points of the face and further deduce 3D landmarks, which provides a basis for accurate expression mapping. During the calculation process, subtle changes in facial expressions are analyzed and quantified into blendshapes values, which are key parameters for expression deformation of 3D models as geometric feature information.

[0055] The 3D mask of each person in each frame of the video image is obtained by using the extracted geometric feature information of each person; this embodiment uses the Unity platform to implement the 3D mask generation. The Unity engine not only supports advanced lighting models, shadow processing and post-processing effects, but also can efficiently manage complex 3D scenes and character animations, which provides strong technical support for the realization of realistic 3D virtual masks; in the specific implementation, the Unity engine receives facial expression data transmitted by Protobuf, which includes accurate position information and dynamic expression parameters. Using Unity's powerful scripting capabilities, we have developed a set of algorithms to map these data to the 3D model in real time, achieving high-precision reproduction of facial expressions. Expression fusion is achieved by controlling Unity's SkinnedMeshRenderer component, so that the 3D virtual mask can be synchronized with the user's real expression, and even the most subtle facial movements can be captured and accurately displayed.

[0056] In one embodiment, a C# script is written to receive and process Protobuf data sent from the Python backend. The core function of the script is to parse the serialized data and extract key facial feature information, including facial key point coordinates, expression parameters, etc. The script then applies this data to the 3D model in Unity and uses the SkinnedMeshRenderer component to dynamically adjust the weight of the model to match the user's real-time facial expression.

[0057] The 3D mask of each person in each frame of video image is mapped to the corresponding face in each frame of video image. In order to achieve realistic expression mapping of the mask, the script of this embodiment will control the SkinnedMeshRenderer component to perform detailed expression deformation on the 3D mask according to the extracted expression parameters, including but not limited to the raising of eyebrows, the opening and closing of eyes, the raising of the corners of the mouth, etc., to ensure that these mappings are smooth and natural, and synchronized with the user's facial movements; it can simultaneously receive, analyze and synchronously process facial expression data from multiple users, to ensure that each user's 3D virtual mask can independently and in real time reflect their facial movements.

[0058] Each frame of the image after the 3D mask is mapped is rendered to obtain a rendered video image, and the obtained video image is displayed through a display device.

[0059] In one embodiment, Figure 2 The invention provides a device for realizing 3D virtual mask based on a monocular camera, including: an image acquisition unit, an image processing unit, a feature extraction unit, a 3D modeling unit, a mapping unit and a display unit. All the units can be integrated into an all-in-one machine for realization; a highly integrated system can realize efficient deployment and stable operation in complex environments such as scenic spots.

[0060] The image acquisition unit uses a monocular camera to collect video images in the scene; the monocular camera is arranged in the middle of the top of the all-in-one machine for collecting scene images, the display unit is a touch screen display, which is arranged at the front of the all-in-one machine, and the other units are located inside the all-in-one machine.

[0061] The image processing unit is used to receive the video image collected by the image acquisition unit and extract the face image of each person in the received video image; specifically, the face detection and recognition algorithm based on deep learning is used to detect and recognize up to 5 faces in the video image in real time; in this embodiment, when collecting face images, Python programming language is used for development, taking advantage of its powerful data processing capabilities and rich library function support. On this basis, the open source Mediapipe framework, an advanced computer vision framework, is integrated for real-time face detection and feature point tracking.

[0062] By applying machine learning models, the Mediapipe framework can stably identify and track facial key points in various complex environments, including eyebrows, eyes, nose, mouth and other key areas. In addition, the module can also calculate the expression fusion information blendshape, which is a parametric representation method defined by the Facial Action Coding System (FACS) that can describe the changes in facial expressions in detail and provide rich details for the dynamic modeling of 3D virtual masks.

[0063] In order to further improve the capture accuracy, this module also implements image processing technology, including but not limited to noise reduction, sharpening and contrast enhancement, to ensure that the user's facial features can be accurately captured even in poor lighting conditions or complex backgrounds; through the combination of these technologies, the facial feature capture module can provide stable and accurate facial expression data for the generation of 3D virtual masks, meeting the high requirements for real-time and accuracy in multi-person interactive environments such as scenic spots.

[0064] Before image processing, portrait enhancement needs to be performed. The enhancement method used in this embodiment includes:

[0065] Sharpening technology: It uses spatial frequency filters and edge enhancement algorithms to carefully highlight the contours of facial features and optimize the key feature extraction process of the 3D model to ensure that the subtle differences in expression changes are realistically presented.

[0066] Noise reduction processing: Through three-dimensional spatial filtering and time domain noise reduction technology, image noise can be effectively removed, especially in low-light or high-sensitivity shooting conditions, to improve the image signal-to-noise ratio and ensure data transmission quality.

[0067] Grayscale adjustment: By automatically analyzing the brightness and contrast of the image, the grayscale level of the image is dynamically adjusted to ensure the clarity and visibility of facial features under different lighting conditions, thereby improving the accuracy of facial recognition.

[0068] The feature extraction unit is used to extract geometric feature information from the face image; in this embodiment, the method used for geometric feature extraction is to first perform face detection based on the Mediapipe library, locate the face and track its dynamic changes, then use Mediapipe to extract the 2D key points of the face, and further calculate the 3D landmarks, which provides a basis for accurate expression mapping, analyzes the subtle changes in facial expressions, and quantifies them into blendshapes values, which are key parameters for 3D model expression deformation. The parameters are used as geometric features.

[0069] In order to ensure that the 3D mask matches the facial size of different users, the algorithm also includes dynamic calculation of the facial scaling factor. Through this step, the system can adaptively adjust the mask size to ensure a high fit between users. In addition, the algorithm is designed with real-time requirements in mind and optimizes the data processing flow to ensure that the response time meets the needs of real-time interaction while maintaining high accuracy. Through these comprehensive measures, the face recognition algorithm provides solid and accurate data support for the generation and real-time rendering of 3D virtual masks.

[0070] The image acquisition unit, image processing unit and feature extraction unit in this embodiment use high-performance computing chips and integrate independent graphics cards. This configuration provides strong support for analysis algorithms and three-dimensional rendering. It includes face detection, feature point tracking technology and image enhancement functions, ensuring that facial expressions can be accurately captured and analyzed even under various complex lighting conditions and backgrounds.

[0071] The 3D modeling unit generates a 3D mask for each person in each frame of the image using the feature information extracted by the feature extraction unit; specifically, based on the feature information of the face, a 3D virtual mask matching the shape and expression of each face is generated to achieve accurate fitting of the mask and the face and dynamic expression; when performing 3D modeling, i.e., 3D rendering, this embodiment integrates a 3D graphics engine to convert the processed data into a 3D virtual mask in real time and perform high-precision expression synchronization;

[0072] This embodiment uses the Unity platform, which is an industry-leading real-time 3D rendering engine known for its excellent graphics processing capabilities and highly optimized rendering pipeline. The Unity engine not only supports advanced lighting models, shadow processing and post-processing effects, but also can efficiently manage complex 3D scenes and character animations, which provides strong technical support for the realization of realistic 3D virtual masks.

[0073] In this solution, the Unity engine receives facial expression data transmitted by Protobuf, which includes precise position information and dynamic expression parameters. Using Unity's powerful scripting capabilities, we developed an algorithm to map this data to the 3D model in real time, achieving high-precision reproduction of facial expressions. Expression fusion is achieved by controlling Unity's SkinnedMeshRenderer component, so that the 3D virtual mask can be synchronized with the user's real expression, and even the most subtle facial movements can be captured and accurately displayed.

[0074] In addition, multi-threaded processing ensures smoothness and responsiveness when processing large amounts of data and complex rendering tasks. We also used Unity's particle system to add richer dynamic visual effects to the 3D virtual mask, further enhancing the realism of the user experience.

[0075] The mapping unit maps the 3D mask generated by the 3D modeling unit onto the corresponding human face; the mapping unit also includes a facial expression capture system while performing mapping, so as to achieve dynamic synchronization between the user's facial expression and the 3D virtual mask, and to achieve real-time and accurate reproduction of the user's facial expression, whether subtle changes in expression or complex movements; the facial expression capture system adopts the Mediapipe framework, combined with optimized facial key point detection and expression parameterization technology, to achieve accurate capture and reproduction of subtle facial expressions.

[0076] The display unit displays the mapped face image. The display unit in this embodiment is a display that uses high resolution and high dynamic range (HDR) display technology to display the generated face image. Figure 4 The effect of the 3D virtual mask in a multi-person scene is shown, providing high-definition and color-accurate image display; providing users with clear and realistic 3D virtual mask visual effects and enhancing immersion.

[0077] In one embodiment, it also includes a scene modeling unit, which generates a static scene virtual image using the scene image acquired by the image acquisition unit, and the scene virtual image includes one or more of the iconic buildings and cultural symbols in the scene. The scene modeling unit is used to display static patterns with scenic spot characteristics, such as landmark buildings or cultural symbols; 3D masks customized according to the scenic spot's festival activities or specific theme activities, such as Spring Festival, Christmas, Halloween and other festival masks, or theme masks related to the scenic spot's characteristics; 3D masks of historical figures or characters related to the history or culture of the scenic spot are provided to enhance tourists' sense of immersion.

[0078] It also includes a data communication unit, which is used to encode and compress data during transmission.

[0079] This embodiment uses Protobuf (Protocol Buffers) as the data transmission protocol, which is a lightweight and efficient structured data storage format developed by Google and is widely used for data serialization.

[0080] Protobuf's efficiency comes from its compact binary format, which optimizes storage space and reduces parsing time, thereby speeding up data transmission. The design of the serialization mechanism allows data to be seamlessly converted between different system architectures without worrying about data integrity and accuracy, which is especially important for cross-platform applications.

[0081] In this solution, the use of Protobuf ensures that the data flow from the facial feature capture module to the 3D rendering engine is highly stable and reliable, and defines a precise Protobuf message structure for facial expression data, including the coordinates of facial key points, expression fusion information blendshape, and other related metadata. This structured message design not only improves the efficiency of data parsing, but also enhances the readability and maintainability of the data.

[0082] In addition, Protobuf supports backward compatibility, which means that even if the data structure is expanded or modified in the future, the old system can recognize and process the new data format, thus ensuring the long-term scalability and compatibility of the system. By realizing the automated serialization and deserialization process, this solution significantly reduces the programming burden of developers in data transmission, and also reduces the risk of errors caused by manual parsing.

[0083] Also includes:

[0084] The rendering unit is used to render the 3D mask obtained in each frame image and complete the rendering of the animation formed by the 3D mask in the continuous frame images; the rendering unit has the following characteristics:

[0085] Multi-view consistency: By integrating view transformation and geometric correction algorithms, the 3D virtual mask can maintain consistency and coherence at different viewing angles, regardless of the change of view angle;

[0086] Dynamic animation support: Supports complex 3D animation effects, including but not limited to facial expression animation, head movement tracking, and synchronization of mask and user facial movements, to achieve a highly natural and smooth dynamic interactive experience;

[0087] Optimized rendering process: Multi-threaded rendering technology and graphics pipeline optimization are used to shorten rendering time and ensure high frame rate output

[0088] In this embodiment, since multi-person images are used, multi-person synchronous 3D modeling and mapping can be performed, and facial expression data from multiple users can be received, analyzed and synchronously processed at the same time, ensuring that each user's 3D virtual mask can independently and in real time reflect their facial movements.

[0089] This multi-person synchronization algorithm adopts an efficient data distribution and processing architecture, and uses multi-threading and asynchronous programming techniques to optimize data flow management, reduce processing delays, and improve system response speed. Each user's expression data is processed independently and then mapped to the corresponding 3D model through a precise algorithm, achieving personalized expression synchronization.

[0090] The device provided in this embodiment adopts system integration and integrated design. The image processing unit in the system adopts a face detection and recognition algorithm based on deep learning, which can accurately process high-resolution video streams in real time. The algorithm is optimized to reduce computing resource consumption while maintaining high-precision face tracking and recognition, and can run stably even in complex lighting or dynamic scenes.

[0091] The 3D mask generation unit uses advanced 3D modeling technology and extracts facial geometric feature information to generate a 3D virtual mask that is highly consistent with the user's facial expression. The optimization focus of this unit is to improve the speed and quality of mask generation and ensure synchronization with the user's expression.

[0092] The performance optimization of the tracking and mapping unit is reflected in the feature point capture, tracking recognition and machine learning technologies it uses, which work together to accurately track faces in complex dynamic scenes. Through algorithm optimization, the unit can quickly adapt to the facial features and expression changes of different users, and maintain high accuracy even in multi-person interaction scenarios.

[0093] The rendering output unit is equipped with a high-performance real-time graphics rendering engine. Through multi-view coherence algorithms and dynamic animation support, it ensures that the 3D virtual mask can maintain consistency and coherence at different viewing angles. The optimized rendering process, including multi-threaded rendering technology and graphics pipeline optimization, effectively shortens the rendering time, ensures high frame rate output, and provides users with smooth visual effects.

[0094] In addition, special consideration was given to the compatibility and scalability of the all-in-one machine with other external hardware devices. The all-in-one machine is designed with standardized interfaces and communication protocols, enabling it to seamlessly connect to various external hardware, further enhancing the functionality of the device and the user experience.

[0095] By integrating with the printer, the all-in-one machine can not only provide a virtual 3D mask experience, but also take photos of the user's real-time image or 3D mask effect in a multi-person scene and print it out. This function provides users with the option of a physical souvenir, increasing the value and appeal of the interactive experience.

[0096] When implementing it, proceed as follows:

[0097] 1. Installing the Mediapipe library is the basis for building the facial feature capture module. In the backend development environment, first confirm the compatibility of the Python version with Mediapipe, and then use the pip tool to install Mediapipe and its dependencies. This process requires careful inspection and resolution of possible dependency conflicts to ensure that OpenCV and other necessary machine learning libraries are correctly installed and updated to the recommended versions. After the installation is complete, perform comprehensive version compatibility and functional testing to ensure that the Mediapipe library runs smoothly on the current system and verify the integration of its API with the existing system.

[0098] 2. Using the Protobuf compiler (protoc) is a key step in achieving cross-platform data exchange. First, according to the characteristics of facial expression data, carefully design the .proto file and clearly define the data type and structural relationship of each field to ensure the accuracy and extensibility of the data model. Then, run the protoc compiler to convert the interface description language in the .proto file into code in a specific programming language. This system corresponds to Python and C# languages ​​for the Unity engine. During the compilation process, the version of the proto file is strictly controlled to ensure that it matches the Protobuf plug-in version of Unity to avoid compatibility issues. In addition, the generated code will undergo rigorous unit testing to verify the accuracy and efficiency of the serialization and deserialization process, and ensure the integrity and reliability of information during data transmission. Through this process, a solid foundation is laid for the system's data exchange and processing.

[0099] 3. Detection The face recognition algorithm is the core link to implement this embodiment. The algorithm is based on the Mediapipe library and is responsible for capturing and analyzing face images from the video stream in real time. First, face detection is performed to locate the face and track its dynamic changes. Subsequently, Mediapipe is used to extract the 2D key points of the face and further deduce the 3D landmarks, which provides the basis for accurate expression mapping. During the calculation process, the subtle changes in facial expressions will be analyzed and quantified into blendshapes values, which are the key parameters for the deformation of 3D model expressions.

[0100] In order to ensure that the 3D mask matches the facial size of different users, it also includes dynamic calculation of the facial scaling factor. Through this step, the system can adaptively adjust the mask size to ensure a high fit between users. In addition, the algorithm is designed with real-time requirements in mind and optimizes the data processing flow to ensure that the response time meets the needs of real-time interaction while maintaining high accuracy. Through these comprehensive measures, the face recognition algorithm provides solid and accurate data support for the generation and real-time rendering of 3D virtual masks.

[0101] 4. The data communication layer is a key technical link to ensure accurate and efficient transmission of facial feature data to the Unity client. This layer is built based on network programming technology and uses Protobuf as the data serialization format to ensure the consistency of data structure and the reliability of transmission. During data transmission, the communication layer implements efficient encoding and compression of data, and uses compression algorithms such as gzip to reduce the size of data packets, thereby reducing network transmission delays and improving transmission efficiency.

[0102] In addition, this layer also implements data transmission reliability mechanisms, such as timeout retransmission and data packet confirmation, to ensure that data can reach the receiving end accurately. After the data reaches the Unity client, the communication layer is also responsible for decoding and deserializing the data to ensure that Unity can correctly parse and use the transmitted data.

[0103] 5. In the Unity platform, developers need to write a C# script to receive and process the Protobuf data sent from the Python backend. The core function of the script is to parse the serialized data and extract key facial feature information, including the coordinates of facial key points, expression parameters, etc. The script then applies this data to the 3D model in Unity and uses the SkinnedMeshRenderer component to dynamically adjust the weight of the model to match the user's real-time facial expression.

[0104] In order to achieve realistic expression mapping of the mask, the script will control the SkinnedMeshRenderer component to perform detailed expression deformation on the 3D mask according to the extracted expression parameters. This includes but is not limited to the raising of eyebrows, the opening and closing of eyes, the raising of the corners of the mouth, etc. Ensure that these mappings are smooth and natural, and synchronized with the user's facial movements.

[0105] 6. The realization of multi-person synchronization mechanism is the key innovation in this technical solution. By developing an efficient parallel processing algorithm, the system can simultaneously receive, analyze and synchronously process facial expression data from multiple users. This mechanism uses advanced multi-threading technology and reasonably allocates computing resources to ensure that each user's 3D mask can independently reflect their facial movements in real time and achieve high synchronization.

[0106] In order to optimize the management of data flow, the system has designed an efficient data distribution and collection architecture, using message queues and event-driven methods to reduce data processing delays.

[0107] 7. During the system integration and testing phase, we first performed comprehensive integration testing in a controlled development environment. This process covered all key aspects of the system, including the accuracy of face detection, the efficiency of data serialization, data transmission, the real-time performance of 3D mask generation, and the accuracy of expression synchronization.

[0108] After completing the testing of the development environment, we will move on to the next key step of system integration, which is to deploy the system to a dedicated all-in-one device. On the all-in-one device, we will focus on testing the system's startup speed, response time, and long-term operational stability to ensure that the system can provide consistent and uninterrupted services under various operating conditions.

[0109] In one embodiment, Figure 3 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0110] It should be noted that the computer system 400 of the electronic device shown in the figure is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0111] The computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 402 or the program loaded from the storage part 408 to the random access memory (RAM) 403, such as executing the method for realizing a 3D virtual mask based on a monocular camera in the above-mentioned embodiment. Various programs and data required for system operation are also stored in the RAM 403. The CPU 401, the ROM 402 and the RAM 403 are connected to each other through the bus 404. The input / output (I / O) interface 405 is also connected to the bus 404.

[0112] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed so that a computer program read therefrom is installed into the storage section 408 as needed.

[0113] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 409, and / or installed from a removable medium 411. When the computer program is executed by a central processing unit (CPU) 401, various functions defined in the system of the present application are executed.

[0114] It should be noted that the computer-readable medium shown in the embodiment of the present application is used to implement the method for realizing a 3D virtual mask based on a monocular camera; the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0115] The technical solution provided by this embodiment has the following effects:

[0116] The technical implementation has at least the following beneficial effects:

[0117] 1. Enhanced real-time performance: Through optimized data processing and multi-threaded rendering technology, the solution of this application can provide a smoother and real-time interactive experience, maintaining high frame rate and low latency even in complex scenes with multiple participants.

[0118] 2. Improved multi-person interaction support: The specially designed multi-person synchronization mechanism allows the system to process facial expression data from multiple users simultaneously, achieving more natural and accurate 3D virtual mask mapping, significantly improving the application potential in multi-person environments such as scenic spots.

[0119] 3. Improved image quality: The integrated portrait enhancement algorithm can provide clear and accurate facial image capture under various lighting and background conditions, and ensures high-quality visual effects of 3D virtual masks through image processing technologies such as sharpening, noise reduction and grayscale adjustment.

[0120] 4. Efficient data transmission: Protobuf is used as the data transmission protocol to ensure the stability and reliability of the data flow, while improving the efficiency of data parsing and reducing compatibility issues in cross-platform applications.

[0121] 5.3D rendering capability: Utilize the advanced graphics processing capabilities of the Unity engine to achieve 3D virtual mask rendering. The weights of the SkinnedMeshRenderer components are fused through algorithms to ensure high synchronization with real expressions, thereby enhancing the user's sense of immersion.

[0122] 6. Scalability and compatibility: The design of standardized interfaces and communication protocols allows the all-in-one machine to be seamlessly integrated with a variety of external devices such as printers, providing a richer user experience and value-added services.

[0123] 7. System Integration: System integration ensures the compatibility and integration of cameras, display devices and other peripherals with the software platform. This includes testing of hardware drivers and interfaces, as well as ensuring the reliability and durability of the hardware under different environmental conditions.

[0124] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0126] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0127] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0128] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0129] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0130] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0131] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for realizing a 3D virtual mask based on a monocular camera, characterized in that: include: Use a monocular camera to obtain scene video images; Obtaining a face image of each person in each frame of the scene video image; Extract geometric feature information from each person's face image; The 3D mask of each person in each frame of video image is obtained by using the extracted geometric feature information of each person; The 3D mask of each person in each frame of video image is mapped to the corresponding face in each frame of video image.

2. The method for realizing a 3D virtual mask based on a monocular camera according to claim 1, characterized in that: The method of extracting geometric feature information from each person's face image includes: Perform face detection on each person's face; The 2D key points of the face are extracted, and the parameters of the 3D model are obtained using the extracted 2D key points, and the 3D model parameters are used as geometric feature information.

3. The method for realizing a 3D virtual mask based on a monocular camera according to claim 2, characterized in that: The 2D key points of the face include eyes, nose, mouth and eyebrows.

4. A device for realizing a 3D virtual mask based on a monocular camera, characterized in that: include: An image acquisition unit collects video images in the scene using a monocular camera; An image processing unit, used for receiving the video image collected by the image acquisition unit, and extracting the face image of each person in the received video image; A feature extraction unit, used to extract geometric feature information from a face image; The 3D modeling unit generates a 3D mask of each person in each frame image using several feature information extracted by the feature extraction unit; A mapping unit maps the 3D mask generated by the 3D modeling unit onto the corresponding human face; The display unit displays the mapped face image.

5. The device for realizing a 3D virtual mask based on a monocular camera according to claim 4, characterized in that: It also includes a scene modeling unit, which generates a static scene virtual image using the scene image acquired by the image acquisition unit, and the scene virtual image includes one or more of the landmark buildings and cultural symbols in the scene.

6. The method for realizing a 3D virtual mask based on a monocular camera according to claim 4, characterized in that: It also includes a data communication unit, which is used to encode and compress data during transmission.

7. The device for realizing a 3D virtual mask based on a monocular camera according to claim 4, characterized in that: Also includes: The rendering unit is used to render the 3D mask obtained in each frame image, and complete the rendering of the animation formed by the 3D mask in the continuous frame images.

8. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement a method for realizing a 3D virtual mask based on a monocular camera as described in any one of claims 1 to 6.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, a method for realizing a 3D virtual mask based on a monocular camera as claimed in any one of claims 1 to 6 is implemented.

Images