Acceleration optimization and stand-alone deployment system of real-time live broadcast face changing technology

Abstract

The invention discloses an accelerated optimization and stand-alone deployment system of a real-time live broadcast face changing technology. The accelerated optimization and stand-alone deployment system comprises a hardware system and a software system. The hardware system mainly comprises a face acquisition unit, a face processing unit, a man-machine interaction unit and other modules. The software system mainly comprises a face acquisition module, a face detection module, a face alignment module, a three-dimensional reconstruction module, a face change generation module, a GUI module, a data storage module, a portrait selection module, a playing module and the like. The multi-thread technology is adopted, real-time collection, detection, alignment, three-dimensional reconstruction, face changing generation and display can be carried out on multiple face images, independent GPU operation acceleration is carried out on face detection, three-dimensional reconstruction and face changing generation at the same time, the speed of the whole process is increased, and the real-time face changing display effect is achieved; and a modular design mode is adopted, and each module is highly independent, so that adaptation for deployment of different devices is facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical fields of image processing, software engineering, computer technology, real-time human-computer interaction technology, etc. Specifically, it is an acceleration optimization and single-machine deployment system for real-time live face swapping technology, integrating various key technologies such as face detection, alignment, 3D reconstruction, and face swapping generation, and completing the accelerated inference of relevant models, which can be applied to demonstration and technical application description scenarios that require real-time and rapid face swapping. Background Art

[0002] With the continuous development of deep learning and computer vision technologies, image processing technologies based on artificial intelligence have become an important support in multiple fields. Among them, face swapping technology is a technology that swaps identities according to the facial feature information of people, with characteristics such as covert operation, non-contact acquisition, strong interactivity, and low cost. It can be widely used in film and television production, virtual anchors, social media special effects, etc. With the progress and development of technology, face swapping technology has also been applied by criminals in aspects such as online fraud and public opinion guidance. The social focus and concerns have pushed the application of face swapping technology to a new climax. Currently, most face swapping methods are mainly based on offline image processing, using pictures and videos to complete the modification of the human face in the video. Although the face swapping effect can be achieved, the process is time-consuming and difficult to meet the needs of real-time interaction scenarios.

[0003] Since the current face swapping cannot achieve real-time interaction and cannot conduct real-time and intuitive comparison between the actual acquisition and the face after face swapping, people cannot have a comprehensive understanding of the technical security, which is not conducive to the promotion and use of the technology. At the same time, many current technologies are based on the server side, posing a great hidden danger to the privacy and security of users. Summary of the Invention

[0004] The objective of the present invention is to provide an acceleration optimization and single-machine deployment system for real-time live face swapping technology, solve the engineering application problems of the face swapping system for local deployment, and use model acceleration technology to achieve rapid detection, alignment, 3D reconstruction, and face swapping generation of multiple face images. The overall process time is compressed to the millisecond level, and through high-performance computing and parallel processing technologies, it is ensured that the face swapping generation is synchronized with the camera acquisition.

[0005] The technical solution of the present invention is as follows:

[0006] An acceleration optimization and single-machine deployment system for real-time live face swapping technology, including a hardware system and a software system;

[0007] The hardware system includes a face acquisition unit, a face processing unit, and a human-computer interaction unit. The face acquisition unit is used to collect face images and transmit them to the face processing unit; the face processing unit is used to analyze and process the collected images; the face interaction unit is used to perform human-computer interaction operations;

[0008] The software system includes a face collection module, a face detection module, a face alignment module, a face three-dimensional reconstruction module, a face swapping generation module, a target face addition module, a target face selection module, a GUI module, a camera control module, a face swapping control module, and an image display module. Among them, the face collection module, the detection module, the face alignment module, the face three-dimensional reconstruction module, the face swapping generation module, and the GUI module are executed in different threads respectively.

[0009] The advantages of the present invention are as follows:

[0010] (1) The model acceleration technology is adopted to improve the processing speed of face detection, alignment, three-dimensional reconstruction, and face swapping, making the overall processing flow reach the millisecond level, ensuring real-time display of the captured images and face swapping.

[0011] (2) The multi-face tracking technology is adopted to achieve the detection, tracking, and face swapping of multiple faces.

[0012] (3) The multi-thread optimization technology is adopted to achieve independent processing of target face selection and replacement, video collection, and face swapping generation, improving the operation and running efficiency of the system.

[0013] (4) The frame-by-frame face swapping method is adopted to avoid edge jitter and flickering after face swapping, improving the face swapping effect.

[0014] (5) Combining with the three-dimensional reconstruction technology, it can achieve face swapping with large-angle poses, accurately match details such as face skin color and expressions, and improve the authenticity of face swapping.

[0015] (6) The modular algorithm integration, compilation, and deployment can be independently adapted and compiled, and can be adapted to different high-performance edge computing devices.

[0016] (7) The model pruning and optimization technology is adopted to reduce the consumption of computing resources of the model, lower the standard requirements of basic hardware conditions, and facilitate the migration and deployment of the model.

[0017] (8) The convenient user interaction settings support the quick addition, selection of target faces, and comparison preview of generation effects. The online target face selection provides great flexibility and experience for users.

[0018] (9) Comprehensive science popularization education demonstration, through occlusion, large-angle rotation, facial wear, etc., intuitively demonstrates the problems existing in the technology and identification means, facilitating science popularization and publicity demonstrations.

[0019] (10) The present invention adopts multi-thread technology, which can perform real-time acquisition, detection, alignment, 3D reconstruction, face swapping generation and display on multiple face images. At the same time, it performs separate GPU operation acceleration on face detection, 3D reconstruction and face swapping generation, improving the speed of the entire process and achieving a real-time face swapping display effect; adopting a modular design method, each module is highly independent, facilitating adaptation for different device deployments; the system can use the built-in camera of a mobile workstation or an external camera to achieve real-time acquisition and display of a large range of multiple faces; adopting a process-based processing and GPU operation acceleration technology, it completes face detection, tracking, positioning, 3D reconstruction and generation frame by frame, maximizing the avoidance of display latency and edge flickering before and after face swapping; the target face supports personalized customized upload, and adopts a unified normalization processing method, which can adapt to faces of different pixels and images of different sizes; the selection of the target face can be switched online or selected offline, facilitating adaptation to different target face swapping effects and being convenient for demonstrating to visitors; in addition, online switching does not require turning off the camera and face swapping operations, realizing seamless switching and connection of different target face swapping generations; providing a fast and convenient interaction window, facilitating user operation and effect display. Description of the Drawings

[0020] Figure 1 is the hardware structure diagram of the present invention;

[0021] Figure 2 is the software structure diagram of the present invention. Detailed Embodiments

[0022] In order to make the purpose, technical solutions and advantages of the present invention clearer, the following further elaborates on the present invention in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above object, the present invention adopts the following technical solutions.

[0023] The following will further elaborate on the present invention in combination with the drawings and embodiments.

[0024] The acceleration optimization and single-machine deployment system of the real-time live face swapping technology of the present invention is jointly composed of a hardware system and a software system. Among them, the hardware system is responsible for providing basic data acquisition, processing and interaction support, and the software system realizes seamless cooperation of each functional module through multi-thread and modular design.

[0025] Such as Figure 1As shown in the figure, the hardware system of the present invention includes three modules: a face acquisition unit, a face processing unit, and a human-computer interaction unit. The face acquisition unit is the device's built-in camera or its external camera, which is used to collect face images in real time; the face processing unit is mainly a device with high computing performance (such as a mobile workstation or a host), which is used to perform face analysis and face swapping processing; the human-computer interaction unit includes a display screen, a mouse, etc. The display screen is used to display the user operation interface, the camera capture screen, and the face swapping effect screen, and the mouse provides an interaction function.

[0026] As Figure 2 shown in the figure, the software system of the present invention includes a face acquisition module, a face detection module, a face alignment module, a face three-dimensional reconstruction module, a face swapping generation module, a target face addition module, a target face selection module, a GUI module, a camera control module, a face swapping control module, and an image display module. Among them, the face acquisition module, the face processing part (face detection module, face alignment module, face three-dimensional reconstruction module), the face swapping generation module, and the GUI module are executed in different threads respectively. This implementation method of combining software and hardware and modular design not only ensures real-time performance and high efficiency but also provides a good user interaction experience, meeting the requirements of real-time face swapping in multiple scenarios.

[0027] The face acquisition module is used to mobilize the camera driver interface in the hardware system to capture and collect images in real time, and transfer them to the face detection module for detection and display on the GUI module. The face acquisition module discovers and scans the camera hardware device by calling a unified image acquisition interface for different camera drivers. For different camera input sources, only the corresponding camera signal sequence needs to be replaced, and the driver interface of the camera is scheduled to achieve real-time capture and collection of images. This module provides data output for the processing of subsequent modules.

[0028] The face detection module first performs face detection on the collected image or the selected target face image, locates the face, locates key points according to the face position, and outputs the pixel coordinates of the face position and key point information. In order to improve the rapid detection of faces, the image is first normalized, and then the face detection algorithm is accelerated by GPU calculation for resolution adaptation and quantization acceleration , where is the resolution adaptation factor for acceleration, is the height of the input image, is the height of the target image, is the width of the input image, is the width of the target image, is the integer value of the image width, is the floating-point value of the image width, is the integer value of the image height, is the floating-point value of the image height, Let [[ID=]] be the scaling factor and Z be the zero offset. To quickly locate the face position in the image, the key point information of 106 faces is extracted. Multiple faces within the image can be detected and tracked simultaneously. The detected faces are counted, the position of each face is marked, and its position is located and key points are extracted.

[0029] The face alignment module corrects and performs affine geometric transformation on the face according to the detected face position and key points. First, a dynamic adaptive expansion of the key points of the detected face is performed , where is the scaling factor, is the face image blur factor; then, the detected face is adjusted using the facial feature positions of the five sense organs to perform high-precision planar correction, aligning the face to a standardized coordinate system. Face alignment is achieved by aligning the key point positions of the three organs, namely the eyes, nose, and mouth, to a fixed position. The affine transformation matrix is obtained using the following formula , , where is the affine transformation matrix, controls rotation, scaling, and shearing, controls translation, are the key point positions after alignment. Then, the coordinate values of the transformed key points are obtained . Due to face images at different angles and expressions, there may be different rotation, scaling, and tilting problems. Through the affine geometric transformation algorithm, the face image can be accurately adjusted to the standard position. First, through shape constraint compensation, rigid transformation constraints are imposed on key regions such as the eyes and the tip of the nose through local geometric constraint compensation to avoid interference from occlusion or noise points , where, is the constraint factor for the eyes, are the original key points, are the target reference points, is the regularization quantization operation; then, non-linear distortion compensation is performed , where , , are the distortion parameters, is the number of radial basis functions (usually 4), is the distance between the current point and the center point, is the center point position of the image, are the characteristic positions of the distortion (usually the four corner coordinates); then, through the rotation angle , where is the height difference before and after image rotation, is the width difference before and after image rotation, and the rotation matrix ​​​​​​​​​​​​​​​​​​​, the scaling factor is obtained by using the distance between the two eyes in the target image and the two eyes in the original image. , where is the distance between the two eyes of the target, is between the two eyes of the original image, and then the scaling matrix is obtained; finally, the constraint compensation parameter is estimated by using the residual of the remaining key points, and the rotation matrix and the scaling matrix are used to perform symmetric constraint and prior constraint on the affine transformation matrix , reducing the computational complexity while ensuring the robustness of rotation correction and tilt compensation, so as to ensure that subsequent operations such as 3D reconstruction and face swapping generation can be performed on the face of a unified face.

[0030] The 3D face reconstruction module analyzes elements such as depth information, lighting, and shadows of the face in the 2D image to reconstruct a highly accurate 3D face model. First, a 3D feature model of the face is constructed by combining the modeling coefficients of shape, expression, and pose, and its formula expression is , where are the shape, expression, and pose coefficients after 3D reconstruction, is the average face shape, are the shape basis coefficients, are the expression basis coefficients, are the pose basis coefficients; then the 3D face reconstruction algorithm is optimized, and the results of the 2D and 3D models are fused by dynamically adjusting the weights, and its formula expression is , is the dynamically learned weight, are the 2D feature parameters of the face, are the 3D feature parameters of the face. The 3D face reconstruction process not only considers factors such as facial expressions and skin details, but also can adjust according to the rotation and tilt of the head and other postures, and outputs a three-dimensional and realistic face model. This module uses the 3D face feature model to map the non-integrable gradient field to the frequency domain through Fourier transform for the 3D face reconstruction algorithm , where is the reconstruction frequency domain weight, is the reconstruction surface gradient, is the measured gradient, and the solution in the frequency domain can be obtained through Fourier transform as , is the value in the frequency domain. By compressing the sparse matrix, the memory occupancy is reduced, and GPU acceleration is performed using kernel auto-tuning, which can efficiently and quickly process face images from different angles and different expressions, output a 3D model with strong realism and high precision, provide necessary 3D structural support for the face swapping generation module, enhance the naturalness and credibility of the face swapping effect, and ensure the seamless connection of information such as posture and expression between the target face and the captured face.

[0031] The face-swapping generation module synthesizes the target face and the video-captured face by using any face-swapping algorithm based on a generative adversarial network (which consists of a generator and a discriminator. The generator maps a random noise vector into a realistic data sample, and the discriminator distinguishes between real data and generated data. Through the adversarial training of the generator and the discriminator, high-quality data generation is achieved. The objective function is a minimax game, and the training process is realized through alternating optimization). This module will, according to the face data processed by the face detection module and the face alignment module, achieve the perfect fusion of the features of the target face and the captured face, and then, combined with the three-dimensional model of the face modeled by the three-dimensional reconstruction module, adjust the skin color, lighting, facial expressions, etc., so that the generated face-swapping image not only remains highly natural visually but also ensures synchronization in terms of facial expressions, postures, etc.

[0032] Specifically, the implementation process first adopts an asymmetric method to map the source face ( ) and the target face ( ) into two independent high-dimensional spaces respectively:

[0033] ;

[0034] Among them, is the high-dimensional space of identity features, is the high-dimensional space of attribute features, is the feature region, is the identity feature, is the attribute feature;

[0035] Then, alignment is performed through the Wasserstein distance constraint in the optimal transport theory. The specific formula is:

[0036] ;

[0037] Among them, is the Wasserstein distance constraint, refers to the infimum of the set, is the distribution function from the source domain to the target domain, represents the source domain (the attribute features of the known source face), is the target domain (the attribute features of the target face to be migrated), is the expectation of the sample distance, is the sample of the source domain, target domain sample, is the cost function, and the function expression is:

[0038] ;

[0039] Among them, is the identity feature distribution of the face, is the balance hyperparameter, is the divergence of the attribute feature representation, is the attribute feature distribution of the face.

[0040] Joint optimization is performed using deformable convolutional kernels and optical flow. The specific formula is:

[0041] ;

[0042] Among them, is the output of the variable convolution, is the number of convolutional kernels, is the th convolutional kernel weight, represents the sampling position of the fixed convolutional kernel, is the th convolutional sampling position, is the predicted value of the optical flow network, which is jointly optimized with the 3D model parameters of the face to achieve dynamic geometric alignment and solve the edge artifacts caused by traditional fixed receptive fields; finally, through hybrid adversarial generation, a block mixing discrimination mechanism is introduced in the pixel space, and the output image is randomly cut into mesh blocks. The true / false image blocks are mixed according to the Poisson distribution and then input into the discriminator. Through the constraint, the constraint formula is:

[0043] ;

[0044] Among them, is the adversarial loss function, is the average value, is the number of network blocks, is the true / false similarity of each image block, is the th segmentation mask, is the output image of the generator, is the real image, is the block-by-block mixing operation;

[0045] Combined with the frequency domain consistency constraint, the constraint formula is:

[0046] ;

[0047] Among them, is the frequency domain loss function, is the phase component of the Fourier transform, is the output image of the generator, is the real image. The amplitude spectrum of the generated image is retained to maintain details and solve the problem of frequency domain discontinuity in traditional methods.

[0048] The face-swapping algorithm performs computational graph pruning, with the calculation formula being:

[0049] ;

[0050] Among them, is the pruning objective function (the optimization objective for measuring the redundancy of the computational graph), is the indicator function (the value of non-critical path nodes is 1, otherwise 0), is a node, is a set of nodes, is the critical path, is the node weight. At the same time, an operator fusion strategy is adopted to fuse each layer of the model. The specific formula is:

[0051] ;

[0052] Among them, is the fusion weight, is the convolutional kernel weight, is the scaling parameter, is the fusion offset, is the bias of the convolution operation, is the offset parameter, is the mean and variance of the batch normalization layer, is the numerical stability constant;

[0053] By regularizing to prune non-critical nodes for quantization calibration and dynamic video memory optimization for GPU acceleration, efficient generation of face feature replacement can be achieved. The design of this module ensures high-precision face fusion. The output face-swapping effect is almost indistinguishable from real faces, while also ensuring pose and expression information. Through the acceleration of the algorithm, real-time synchronous face-swapping generation is guaranteed.

[0054] The target face addition module allows users to upload, store, and build an index for it. The original image is saved in the directory path of the target face. At the same time, an index number is added during program operation to quickly load the image, realizing fast and seamless replacement of the user-defined face image and the existing target face image. Through this module, users can flexibly add different target faces, providing diverse choices for effect display.

[0055] The target face selection module provides users with an intuitive target face selection interface for selecting the target face to be used in the face swap demonstration. Users can view all uploaded target face images through this module and select a suitable face for the face swap operation. This module also supports real-time preview of the target face. Users can adjust the display effect of the target face to ensure that the selected face meets the actual requirements, providing a high degree of interactivity and visualization functions, which is convenient for users to make flexible selections during the face swap process.

[0056] The GUI (Graphical User Interface) module is mainly divided into two parts: the upper real-time display area and the lower operation selection area. Through the operation options of the GUI, the acquisition and face swap effects can be displayed in real time. Users can easily perform operations such as target face selection, image preview, and face swap effect adjustment through the GUI module. The GUI module supports real-time update and display. On the left side of the upper display area, the real-time captured image of the camera is shown, and on the right side, the real-time face swap generated image is shown. Below, there are respectively the camera control button and the face swap generation control button. Below the control buttons is the target face selection area, and at the very bottom is the information prompt for operation instructions. To ensure a smooth user experience, the interface design of the GUI module is simple and intuitive, adapting to support a variety of high-performance mobile terminal operation devices.

[0057] The camera control module is responsible for turning on and off the camera connected to the system, supporting various different types of camera hardware, and being able to switch different camera input sources according to requirements. This module processes the real-time captured images in a procedural manner, and at the same time controls the display switching of the camera control button. According to the change of the button state, the captured image is displayed in the camera capture input screen area, and at the same time, the frame rate captured by the current camera is obtained.

[0058] The face swap control module is responsible for managing and scheduling each sub-module of the entire face swap process. According to the user's operations, it sequentially starts and coordinates the operations of modules such as face detection, alignment, 3D reconstruction, and face swap generation for the target face and the face captured by the camera, ensuring the smooth progress of the entire face swap process. This module also controls the display switching of the face swap generation button. According to the change of the button state, the generated image is displayed in the real-time facial generation output screen.

[0059] The image display module is responsible for real-time displaying the camera captured screen and presenting the image effect after face swap. By combining with the GUI module, the image display module can display the result of the fusion of the face captured by the camera and the target face to the user in real time, providing instant feedback, and at the same time displaying the capture frame rate in a specific area.

Claims

1. An acceleration optimization and single - machine deployment system for real - time live face - swapping technology, characterized in that, It includes a hardware system and a software system; The hardware system includes a face acquisition unit, a face processing unit, and a human-computer interaction unit. The face acquisition unit is used to acquire face images and transmit them to the face processing unit; The face processing unit is used to analyze and process the acquired images; the face interaction unit is used for human-computer interaction operations; The software system includes a face acquisition module, a face detection module, a face alignment module, a face three-dimensional reconstruction module, a face swapping generation module, a target face addition module, a target face selection module, a GUI module, a camera control module, a face swapping control module, and an image display module. Among them, the face acquisition module, the detection module, the face alignment module, the face three-dimensional reconstruction module, the face swapping generation module, and the GUI module are executed in different threads respectively.

2. The accelerated optimization and single-machine deployment system for real-time live face-swapping technology according to claim 1, characterized in that The face acquisition module is used to mobilize the camera driver interface in the hardware system to capture acquisition images in real time, and transmit them to the face detection module for detection and display on the GUI module; the face acquisition module discovers and scans camera hardware devices by calling the unified image acquisition interface for different camera drivers, replaces the corresponding camera signal sequences for different camera input sources, and schedules the driver interface of the camera to achieve real-time capture of images.

3. The acceleration optimization and single-machine deployment system of the real-time live face-swapping technology according to claim 1, wherein The face detection module first performs face detection on the acquired images or the selected target face images, locates the face, locates key points according to the face position, and outputs the pixel coordinates of the face position and key point information.

4. The acceleration optimization and single-machine deployment system for real-time live face-swapping technology according to claim 3, wherein The face alignment module corrects and performs affine geometric transformation on the face according to the detected face position and key points; then adjusts the detected face using the positions of the facial features of the face, performs plane correction, and aligns the face to a standardized coordinate system; adjusts the face image to the standard position through the affine geometric transformation algorithm.

5. The accelerated optimization and single-machine deployment system for real-time live face-swapping technology according to claim 1, characterized in that, The face three-dimensional reconstruction module analyzes the depth information, illumination, and shadow elements of the face in the two-dimensional image, considers facial expressions, skin textures, head rotations, and tilts, and reconstructs a three-dimensional face model. This module uses face three-dimensional image reconstruction technology to perform GPU acceleration on the three-dimensional reconstruction algorithm, processes face images from different angles and different expressions, and outputs the face three-dimensional model of the face.

6. The acceleration optimization and single-machine deployment system of the real-time live face-swapping technology according to claim 1, characterized in that, The face swapping generation module generates feature fusion of the target face and the video-acquired face by using an arbitrary face swapping algorithm based on the generative adversarial network; this module will achieve perfect fusion of the features of the target face and the acquired face according to the face data processed by the face detection and face alignment modules, and then combine the three-dimensional model of the face modeled by the three-dimensional reconstruction module to adjust skin color, illumination, and facial expressions to make the generated face swapping images visually natural while keeping facial expressions and postures synchronized; the face swapping algorithm performs GPU acceleration to achieve face feature replacement.

7. The acceleration optimization and single-machine deployment system for real-time live face-swapping technology according to claim 1, wherein The target face addition module allows users to upload, construct its storage and indexing, save the original image under the directory path of the target face, and add an index number during program operation to quickly load the image, realizing the replacement of the user-defined face image and the existing target face image.

8. The acceleration optimization and single-machine deployment system for real-time live face-swapping technology according to claim 1, characterized in that, The target face selection module is used to select the target face to be used in the face swap demonstration; through this module, the user can view all the uploaded target face images and select a suitable face for the face swap operation.

9. The acceleration optimization and single-machine deployment system for real-time live face-swapping technology according to claim 1, wherein The camera control module is used to handle the opening and closing operations of the camera connected to the system. It supports various types of camera hardware, can switch different camera input sources according to requirements, processes the real-time captured images in a streamlined manner, and simultaneously controls the display switching of the camera control buttons. According to the change of the button state, it displays the captured image in the camera capture input screen area and obtains the frame rate of the current camera capture.

10. The accelerated optimization and single-machine deployment system of the real-time live face-swapping technology according to claim 1, characterized in that, The face swap control module is used to manage and schedule each module in the entire face swap process. According to the user's operation, it sequentially starts and coordinates the face detection, alignment, 3D reconstruction, and face swap generation processing processes for the target face and the face captured by the camera. This module also controls the display switching of the face swap generation button and, according to the change of the button state, displays the generated image in the facial real-time generation output screen.

11. The acceleration optimization and single-machine deployment system of the real-time live face-swapping technology according to claim 1, characterized in that, The image display module is used to display the camera capture screen in real time and show the image effect after face swap.

Images