Real-time face changing method based on relative face feature similarity

Abstract

The invention discloses a real-time face changing method based on relative facial feature similarity, and relates to the technical field of artificial intelligence, and the method comprises the following steps: S1, verifying the relative relationship of a user through facial geometric feature extraction and similarity calculation; s2, realizing real-time rendering within 200ms by adopting framing asynchronous processing and dynamic code rate adaptation; and S3, face changing content and user identity information are bound through a digital watermarking technology, a relative similarity threshold value is set to be 0.78, when cosine similarity is larger than or equal to 0.78, a relative relationship is judged, dynamic code rate adaptation comprises automatic degradation to 720P resolution according to network delay, and a face detection step is further included before face geometric feature extraction. According to the real-time face changing method based on the relative facial feature similarity, the relative relationship of the user is verified through facial geometric feature extraction and similarity calculation, the problem of potential safety hazards existing in an existing face changing technology is solved, and safety is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a real-time face-swapping method based on similarity of facial features of relatives. Background Art

[0002] Currently, there are many applications on the market that can generate face-swapped videos, such as "ZAO." In these applications, users only need to upload a photo to replace the face of an actor in the video with their own. However, this technology also has certain risks. Existing face-swapped technologies (such as Deepfake) often lack authentication mechanisms, posing the risk of abuse. At the same time, traditional methods have shortcomings in real-time performance (such as latency > 500ms) and rendering quality (such as insufficient resolution adaptation). Summary of the Invention

[0003] The purpose of the present invention is to provide a real-time face-swapping method based on the similarity of facial features of relatives to solve the problems raised in the above background technology.

[0004] To achieve the above-mentioned object, the present invention provides the following technical solution: a real-time face-swapping method based on the similarity of facial features of relatives, comprising the following steps:

[0005] S1. Verify user kinship by extracting facial geometric features and calculating similarity;

[0006] S2. Uses asynchronous frame processing and dynamic bitrate adaptation to achieve real-time rendering within 200ms;

[0007] S3. Bind the face-changing content and user identity information through digital watermark technology.

[0008] Furthermore, the kinship similarity threshold is set to 0.78, and when the cosine similarity is ≥ 0.78, it is determined to be a kinship relationship.

[0009] Furthermore, dynamic bitrate adaptation includes automatic downgrading to 720P resolution based on network latency.

[0010] Furthermore, a face detection step is also included before extracting facial geometric features.

[0011] Furthermore, it also includes: if it is determined that the person is not a relative, an error prompt is returned.

[0012] Furthermore, the method further includes: if it is determined that the person is a relative, performing face detection and key point positioning.

[0013] Furthermore, the method further includes: utilizing facial key points to perform face alignment and cropping, and adjusting the image to a uniform size.

[0014] Furthermore, it also includes: using a pre-trained model to perform face fusion and output the result.

[0015] The present invention provides a real-time face-changing method based on the similarity of relatives' facial features, which has the following beneficial effects: the present invention verifies the user's kinship relationship through facial geometric feature extraction and similarity calculation, solves the security risks of existing face-changing technology, and improves security. DETAILED DESCRIPTION

[0016] The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0017] The embodiment of the present application provides a real-time AI face-changing method based on kinship feature verification, comprising the following steps:

[0018] S1. Verify user kinship by extracting facial geometric features and calculating similarity;

[0019] S2. Uses asynchronous frame processing and dynamic bitrate adaptation to achieve real-time rendering within 200ms;

[0020] S3. Bind the face-changing content and user identity information through digital watermark technology.

[0021] Specifically, the real-time AI face-changing method based on kinship feature verification provided in the embodiment of the present application is applied to face recognition scenarios. It can verify the user's kinship relationship through facial geometric feature extraction and similarity calculation, solves the security risks of existing face-changing technology, and improves security.

[0022] Furthermore, the kinship similarity threshold is set to 0.78, and when the cosine similarity is ≥ 0.78, it is determined to be a kinship relationship.

[0023] In practical applications, since different faces have different characteristics, an algorithm can be used to extract some feature vectors from facial images for comparison. If the distance between two feature vectors is less than a certain threshold, it is considered that there is a high probability that the two faces belong to the same person. In this embodiment of the present application, whether two faces are related can be determined based on a preset kinship similarity threshold. For example, the kinship similarity threshold is set to 0.78, and a cosine similarity ≥ 0.78 determines that the two faces are related.

[0024] Furthermore, dynamic bitrate adaptation includes automatic downgrading to 720P resolution based on network latency.

[0025] In actual application, due to different network environments, the data transmission rate may change, so it is necessary to select an appropriate bit rate for encoding based on the current network conditions. In the embodiment of the present application, the bit rate can be automatically adjusted according to the current network delay. For example, when the network delay is high, a lower bit rate (such as 720P) can be selected for encoding to ensure video smoothness.

[0026] Furthermore, it also includes: if it is determined that the person is not a relative, an error prompt is returned.

[0027] In actual application, if the system recognizes that the input face does not belong to a relative, it will return an error message so that the user can re-upload the correct face picture.

[0028] Furthermore, the method further includes: if it is determined that the person is a relative, performing face detection and key point positioning.

[0029] Furthermore, the method further includes: utilizing facial key points to perform face alignment and cropping, and adjusting the image to a uniform size.

[0030] Furthermore, it also includes: using a pre-trained model to perform face fusion and output the result.

[0031] In summary, the specific steps are as follows:

[0032] Face detection and preprocessing: Use MTCNN (Multi-Task Convolutional Neural Network) to detect face areas in the input video stream and return the face bounding box coordinates; if no face is detected, an error message "No valid face detected" is returned;

[0033] Kinship verification: Facial geometric feature extraction: The ResNet-50 model is used to extract facial feature vectors and calculate the cosine similarity of two faces;

[0034] Threshold determination: The kinship similarity threshold is set to 0.78 (based on training of a large-scale kinship dataset). If the similarity is ≥ 0.78, it is determined to be a kinship relationship; otherwise, an error message "Not a kinship relationship, operation terminated" is returned;

[0035] Keypoint positioning and alignment: Use the Dlib library or HRNet to detect 68 key points of the face (including eyes, nose, mouth corners, etc.); align the face to a standard pose through affine transformation and crop it to a uniform size of 256×256 pixels;

[0036] Frame asynchronous processing and real-time face swapping: Frame processing: The video stream is framed at 30FPS, and each frame is independently fed into the face swapping model to avoid delay accumulation caused by inter-frame dependencies;

[0037] Model inference: Use the pre-trained FaceShifter model to perform face fusion and generate the face-swapped frame image;

[0038] Dynamic bitrate adaptation: Real-time monitoring of network delay. If the delay is greater than 150ms, the output resolution will be automatically downgraded from 1080P to 720P and the bitrate will be reduced to 2Mbps.

[0039] Digital watermark embedding: Use the LSB (least significant bit) algorithm to embed the user ID information into the video frame after face swapping. The watermark is invisible and compression-resistant. The watermark information includes: user registration timestamp and kinship verification result hash value;

[0040] Real-time rendering and output: Frame rendering is completed through GPU acceleration (CUDA core), ensuring end-to-end latency < 200ms; output streaming protocols support WebRTC or RTMP, and are compatible with major video call platforms.

[0041] The embodiments of the present invention are presented for purposes of illustration and description and are not intended to be exhaustive or to limit the invention to the disclosed forms. Many modifications and variations will be apparent to those skilled in the art. The embodiments are chosen and described in order to better illustrate the principles of the invention and its practical application and to enable those skilled in the art to understand the invention and design various embodiments with various modifications as suited for specific applications.

Claims

1. A real-time face swapping method based on similarity of facial features of relatives, characterized by: The following steps are involved: S1. Verify user kinship by extracting facial geometric features and calculating similarity; S2. Uses asynchronous frame processing and dynamic bitrate adaptation to achieve real-time rendering within 200ms; S3. Bind the face-changing content and user identity information through digital watermark technology.

2. A real-time face swapping method based on similarity of facial features of relatives according to claim 1, characterized in that: The kinship similarity threshold was set at 0.78, and kinship was determined when the cosine similarity was ≥ 0.

78.

3. The real-time face swapping method based on similarity of facial features of relatives according to claim 2, characterized in that: Dynamic bitrate adaptation includes automatic downgrading to 720P resolution based on network latency.

4. The real-time face swapping method based on similarity of facial features of relatives according to claim 3, characterized in that: Before extracting facial geometric features, a face detection step is also included.

5. The real-time face swapping method based on similarity of facial features of relatives according to claim 4, characterized in that: Also includes: If it is determined that they are not relatives, an error message will be returned.

6. The real-time face swapping method based on similarity of facial features of relatives according to claim 5, characterized in that: Also includes: If it is determined that the person is a relative, face detection and key point positioning will be performed.

7. The real-time face swapping method based on similarity of facial features of relatives according to claim 6, characterized in that: Also includes: Use facial key points to perform face alignment and cropping, and resize the images to a uniform size.

8. The real-time face swapping method based on similarity of facial features of relatives according to claim 7, characterized in that: Also includes: Use the pre-trained model to perform face fusion and output the result.