A tracking face-changing method, system, device and medium based on multi-person scene

Abstract

The present application discloses a tracking face-changing method, system, device and medium based on a multi-person scene, the method comprising: obtaining a video to be processed; performing voice detection and lip recognition processing on the video to be processed, performing similarity calculation based on the detection result and the recognition result, and determining the target object; performing task selection processing on each frame image of the video to be processed to obtain the selection task of the image to be processed; when the selected task is a target tracking task, performing tracking and matching processing on the target object through a target tracking model to obtain a target position frame; when the selected task is a target re-identification task, performing positioning and matching processing on the target object through a re-identification model to obtain a target position frame; performing face-changing processing on the video to be processed based on the target position frame to obtain a face-changing video. The embodiments of the present application can accurately track the target object, improve the efficiency of face-changing tracking, and can be widely used in the field of computer vision technology.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a method, system, device and medium for tracking and swapping faces based on a multi-person scenario. Background Art

[0002] The application scenarios of face-changing technology are mainly static image face-changing and dynamic video face-changing. With the rise and explosion of short videos, video ringtones and other product forms, dynamic video face-changing has gradually become one of the main application scenarios of portrait face-changing. However, in actual applications, it is found that in multi-person scenes of dynamic videos, face tracking of related technologies may lose the tracking target or offset the tracking target, resulting in the inability to achieve real-time and accurate results when tracking face-changing, affecting the efficiency of tracking face-changing.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a tracking face-changing method, system, device and medium based on a multi-person scenario, which can accurately locate the tracking target and improve the efficiency of tracking face-changing.

[0005] To achieve the above purpose, an embodiment of the present application proposes a face-changing tracking method based on a multi-person scene, the method comprising:

[0006] Get the video to be processed;

[0007] Performing speech detection and lip reading recognition processing on the video to be processed, performing similarity calculation based on the detection results and the recognition results, and determining the target object;

[0008] Performing task selection processing on each frame image of the video to be processed to obtain a selection task for the image to be processed;

[0009] When the selected task is a target tracking task, the target object is tracked and matched by a target tracking model to obtain a target position frame;

[0010] When the selected task is a target re-identification task, positioning and matching the target object is performed through a re-identification model to obtain a target position frame;

[0011] The video to be processed is subjected to face-changing processing according to the target position frame to obtain a face-changing video.

[0012] In some embodiments, performing speech detection and lip reading recognition processing on the video to be processed, performing similarity calculation based on the detection result and the recognition result, and determining the target object includes:

[0013] Performing content detection processing on the voice or subtitle of the video to be processed to obtain a detection result;

[0014] Performing lip reading recognition processing on each face image in the video to be processed through a lip reading recognition network to obtain a recognition result;

[0015] Calculating the edit distance between the detection result and the recognition result, and determining the face image with the smallest edit distance as the initial target;

[0016] Within a preset time, the face image that is used as the initial target most times in the video to be processed is determined as the target object.

[0017] In some embodiments, the lip reading recognition network is used to perform lip reading recognition processing on each face image in the video to be processed to obtain a recognition result, including:

[0018] Inputting the face image into the lip reading recognition network, wherein the lip reading recognition network includes a convolutional neural network, a long short-term memory recursive neural network, a multi-layer perceptron and a classifier;

[0019] Performing feature extraction processing on the face image through the convolutional neural network to obtain a first feature;

[0020] Performing time domain information extraction processing on the first feature through the long short-term memory recurrent neural network to obtain a second feature;

[0021] Performing text prediction processing on the second feature by the multi-layer perceptron to obtain predicted text;

[0022] The predicted text is subjected to label generation processing by the classifier to obtain a recognition result.

[0023] In some embodiments, performing task selection processing on each frame image of the video to be processed to obtain the selection task of the image to be processed includes:

[0024] Performing frame processing on the video to be processed to obtain an image to be processed and adjacent images;

[0025] Inputting the image to be processed and the adjacent image into a two-way neural network for feature vector extraction processing to obtain a first feature vector and a second feature vector;

[0026] Performing similarity calculation processing on the first feature vector and the second feature vector to obtain feature vector similarity;

[0027] The feature vector similarities are compared according to a preset similarity threshold to obtain a selection task.

[0028] In some embodiments, the step of inputting the image to be processed and the adjacent image into a dual-path neural network for feature vector extraction to obtain a first feature vector and a second feature vector includes:

[0029] Inputting the image to be processed and the adjacent image into a two-way neural network, wherein the two-way neural network includes two feature extraction networks with the same network scale and shared weights, and the feature extraction network includes multiple convolutional layers, pooling layers, and fully connected layers;

[0030] Performing hierarchical extraction processing on the image to be processed and the adjacent images through the multi-layer convolutional layers to obtain multi-granularity features;

[0031] Performing residual connection processing on the multi-granularity features through the pooling layer to obtain intermediate features;

[0032] The intermediate features are mapped through the fully connected layer to obtain global image features, where the global image features include a first feature vector and a second feature vector.

[0033] In some embodiments, the tracking and matching process of the target object by using a target tracking model to obtain a target position frame includes:

[0034] Performing feature extraction processing on the target object to obtain a feature map;

[0035] Performing pyramid pooling processing on the feature map to obtain multi-dimensional features;

[0036] Generate a candidate frame according to the multi-dimensional features, calculate the confidence of the candidate frame, and determine the position information;

[0037] According to the position information, a matching process is performed on the face target in the image to be processed to obtain a target position frame.

[0038] In some embodiments, performing positioning and matching processing on the target object by using a re-identification model to obtain a target position frame includes:

[0039] Performing feature extraction processing on the target object to obtain an extracted feature vector;

[0040] Performing high-dimensional space mapping and metric learning processing on the extracted feature vector to obtain a high-dimensional feature vector;

[0041] Feature matching processing is performed on the face target in the image to be processed according to the high-dimensional feature vector to obtain a target position frame.

[0042] To achieve the above purpose, another aspect of the embodiment of the present application proposes a face-changing tracking system based on a multi-person scene, the system comprising:

[0043] The first module is used to obtain the video to be processed;

[0044] The second module is used to perform speech detection and lip reading recognition on the video to be processed, calculate similarity based on the detection results and recognition results, and determine the target object;

[0045] The third module is used to perform task selection processing on each frame image of the video to be processed to obtain a selection task for the image to be processed;

[0046] The fourth module is used for performing tracking and matching processing on the target object through a target tracking model to obtain a target position frame when the selected task is a target tracking task;

[0047] A fifth module is used for performing positioning and matching processing on the target object through a re-identification model to obtain a target position frame when the selected task is a target re-identification task;

[0048] The sixth module is used to perform face-changing processing on the video to be processed according to the target position frame to obtain a face-changing video.

[0049] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

[0050] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0051] The embodiments of the present application include at least the following beneficial effects: The present application provides a method, system, device and medium for tracking and changing faces based on a multi-person scenario. The scheme performs voice detection and lip reading recognition processing on the video to be processed, calculates the similarity based on the detection results and the recognition results, determines the target object, and can accurately locate the target object that needs to change faces by analyzing multimodal information; then, task selection processing is performed on each frame of the video to be processed to obtain the selection task of the image to be processed, and the target object can be continuously tracked or re-identified and positioned according to different selection tasks, and the target object is subjected to face-changing processing according to the target position frame, thereby achieving real-time and accurate tracking when tracking face-changing, and improving the efficiency of tracking face-changing. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of a tracking face-changing method based on a multi-person scene provided in an embodiment of the present application;

[0053] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0054] Figure 3 yes Figure 2 Flow chart of step S202 in FIG.

[0055] Figure 4 is a schematic diagram of a lip reading recognition network provided in an embodiment of the present application;

[0056] Figure 5 yes Figure 1 Flow chart of step S103 in FIG.

[0057] Figure 6 yes Figure 5 Flow chart of step S502 in FIG.

[0058] Figure 7 This is a flow chart of a customized face-changing method provided in an embodiment of the present application;

[0059] Figure 8 This is a flowchart of a personalized face-changing method provided in an embodiment of the present application;

[0060] Fig. 9 It is a structural diagram of a tracking face-changing system based on a multi-person scenario provided in an embodiment of the present application;

[0061] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of systems and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0063] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0064] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0067] Face-changing technology, also known as face replacement or face manipulation, is an advanced digital technology that uses artificial intelligence (AI) and machine learning (ML) technology. This technology can transfer an image or video frame of a face to another image or video frame through an algorithm, thereby replacing or changing facial features. In recent years, with the rapid development of deep learning, face-changing technology has made significant progress and has been widely used in filmmaking, video editing, virtual reality (VR), augmented reality (AR) and other fields.

[0068] Currently, the application scenarios of face-changing are mainly divided into static picture face-changing and dynamic video face-changing. With the rise and explosion of product forms such as short videos and video ringtones, dynamic video face-changing has gradually become one of the main application scenarios of portrait face-changing.

[0069] In the related art, there is a video containing multiple people, and the face of the speaker in the video needs to be replaced. However, how to accurately locate the face replacement target in the multi-person scene and continuously track the target has become a difficulty. In actual applications, it is found that in the multi-person scene of dynamic video, the face tracking of the related technology will lose the tracking target or the tracking target will be offset, resulting in the inability to achieve real-time and accurate results when tracking face replacement, affecting the efficiency of tracking face replacement.

[0070] For example, in a video with multiple people, there may be multiple scenes. In one scene, the host keeps moving continuously, but between multiple scenes, the camera position changes, and the host's body shape and position in the previous frame and the next frame jump for a moment, making it difficult to track the host in real time. In dynamic videos with multiple people, the tracking target is often lost or offset, resulting in poor efficiency of face-changing tracking, which affects the user experience.

[0071] In view of this, an embodiment of the present application provides a tracking and face-changing method, system, device and medium based on a multi-person scenario. The scheme performs voice detection and lip reading recognition processing on the video to be processed, calculates the similarity based on the detection results and the recognition results, and determines the target object. It can accurately locate the target object that needs to be changed by analyzing the multimodal information; then, task selection processing is performed on each frame image of the video to be processed to obtain the selection task of the image to be processed, and the target object can be continuously tracked or re-identified and positioned according to different selection tasks, and the target object is changed according to the target position frame, thereby achieving real-time and accurate effects when tracking face changing, and improving the efficiency of tracking face changing.

[0072] The embodiment of the present application provides a method for tracking and changing faces based on a multi-person scene, which relates to the field of computer vision technology. The embodiment of the present application provides a method for tracking and changing faces based on a multi-person scene, which can be applied to a terminal, a server, or software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a method for tracking and changing faces based on a multi-person scene, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0074] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0075] Figure 1 is an optional flowchart of a face-swapping tracking method based on a multi-person scenario provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0076] Step S101, obtaining a video to be processed;

[0077] Step S102, performing speech detection and lip reading recognition processing on the video to be processed, performing similarity calculation based on the detection results and the recognition results, and determining the target object;

[0078] Step S103, performing task selection processing on each frame image of the video to be processed to obtain a selection task for the image to be processed;

[0079] Step S104, when the selected task is a target tracking task, tracking and matching processing is performed on the target object through a target tracking model to obtain a target position frame;

[0080] Step S105, when the selected task is a target re-identification task, positioning and matching the target object is performed through a re-identification model to obtain a target position frame;

[0081] Step S106, performing face-changing processing on the video to be processed according to the target position frame to obtain a face-changing video.

[0082] In the steps S101 to S106 shown in the embodiment of the present application, a video to be processed is obtained, which is a video to be processed for face-changing processing, and then voice detection and lip reading recognition are performed on the video to be processed, and the voice content of the video to be processed is detected and lip reading recognition is performed on each face image in the video to be processed, so as to determine the target object by similarity calculation based on the detection results and the recognition results, and the target object is the target to be processed for face-changing processing. Since multiple scenes may be switched in a dynamic video of a multi-person scene, it is easy to lose the tracking target or offset the tracking target. Therefore, after locating the target object in the video through multimodal information, it is necessary to perform task selection processing on each frame of the video to be processed to obtain a selection task for the image to be processed. The selection task includes a target tracking task and a target re-identification task, so as to determine to continuously track or re-identify and track each frame of the image, and obtain a target position frame of the target object. According to the target position frame, the video to be processed can be subjected to face-changing processing to obtain a face-changing video.

[0083] In step S101 of some embodiments, the video to be processed may be uploaded by a user or obtained by other means such as from a video library, but is not limited thereto.

[0084] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S204:

[0085] Step S201, performing content detection processing on the voice or subtitle of the video to be processed to obtain a detection result;

[0086] Step S202, performing lip reading recognition processing on each face image in the video to be processed through a lip reading recognition network to obtain a recognition result;

[0087] Step S203, calculating the edit distance between the texts of the detection result and the recognition result, and determining the face image with the smallest edit distance as the initial target;

[0088] Step S204: within a preset time, determine the face image that is used as the initial target most times in the video to be processed as the target object.

[0089] In an embodiment of the present application, semantic content can be obtained by recognizing the speech and audio in the video to be processed, or the subtitle content can be obtained by performing text recognition processing on the subtitles in the video to be processed, thereby obtaining the detection content. Then, the lip reading recognition network is used to perform lip reading recognition processing on each face image in the video to be processed to obtain a recognition result. Among them, the embodiment of the present application uses an end-to-end lip reading recognition network with a long short-term memory recursive neural network (LSTM) as the core. Then, the edit distance between the texts of the detection results obtained by the video content detection and the recognition results obtained by the lip reading recognition is calculated, and the face image with the smallest edit distance is determined as the initial target. For example, let the character string obtained by the video speech content recognition be A, the character string obtained by the face lip reading recognition be B, and the length of the character string A be L A , the length of string B is L B The similarity between strings is expressed by the edit distance. The edit distance (ED) refers to the minimum number of operations required to convert one string into another. The operations include the following: adding a character, deleting a character, and modifying a character. The specific edit distance is calculated as follows:

[0090] If L A or L B is equal to 0, then ED=max(L A ,L B ), in general, it is calculated recursively through the state transfer equation, and the expression of the state transfer equation is as follows:

[0091]

[0092] In the formula, the value of i is 1…L A , j is 1…L B . Represents the edit distance between string A and string B.

[0093] The embodiment of the present application calculates the edit distance between the detection results and the recognition results, thereby determining the face image corresponding to the recognition result with the smallest edit distance as the initial target. And within a preset time, the face image that is the initial target the most times in the video to be processed is determined as the target object. The preset time can be set according to actual conditions. In the embodiment of the present application, it can be set to within 20 seconds. The embodiment of the present application can improve the accuracy of positioning by combining multimodal information, such as voice information, image information, lip reading recognition information, etc. to locate the target object.

[0094] See also Figure 3 In some embodiments, step S202 may include but is not limited to steps S301 to S305:

[0095] Step S301, inputting the face image into the lip reading recognition network, wherein the lip reading recognition network includes a convolutional neural network, a long short-term memory recurrent neural network, a multi-layer perceptron and a classifier;

[0096] Step S302, performing feature extraction processing on the face image through the convolutional neural network to obtain a first feature;

[0097] Step S303, performing time domain information extraction processing on the first feature through the long short-term memory recurrent neural network to obtain a second feature;

[0098] Step S304, performing text prediction processing on the second feature through the multi-layer perceptron to obtain a predicted text;

[0099] Step S305: performing label generation processing on the predicted text through the classifier to obtain a recognition result.

[0100] In the embodiment of the present application, each face image in the video to be processed is input into the lip reading recognition network for lip reading recognition, thereby obtaining a recognition result. Figure 4 , Figure 4 The process of extracting lip sequence features from a continuous lip image sequence is shown in Figure 1. The image input below is a continuous lip action image of the same person, and the output above is y i It is the text feature predicted by the lip movement corresponding to each frame. The lip reading recognition network includes a convolutional neural network, a long short-term memory recursive neural network (LSTM), a multi-layer perceptron (MLP) and a classifier (not shown). The embodiment of the present application constructs a lip reading recognition network with a long short-term memory recursive neural network as the core. At the input end of the network, the face image must first be processed by the previous convolutional neural network for feature extraction to obtain the first feature. Then the first feature information obtained by the last layer of the convolutional neural network is passed as input to the LSTM layer, and the time domain information of the first feature is extracted, so that the time domain information of the sequence can be obtained. The last second feature vector obtained by the last layer of the LSTM layer is passed through a softmax classifier after the text prediction by the multi-layer perceptron, so that the corresponding label information can be generated to obtain the final recognition result. The embodiment of the present application performs lip reading recognition on each face image in the processed video through the lip reading recognition network, so as to track the target according to the recognition result, determine the target object that needs to be processed for face replacement, and improve the accuracy of target tracking and the efficiency of video face replacement.

[0101] See also Figure 5 In some embodiments, step S103 may include but is not limited to steps S501 to S504:

[0102] Step S501, performing frame processing on the video to be processed to obtain an image to be processed and adjacent images;

[0103] Step S502, inputting the image to be processed and the adjacent image into a two-way neural network for feature vector extraction processing to obtain a first feature vector and a second feature vector;

[0104] Step S503, performing similarity calculation processing on the first feature vector and the second feature vector to obtain feature vector similarity;

[0105] Step S504: performing a comparison process on the similarity of the feature vectors according to a preset similarity threshold to obtain a selection task.

[0106] In an embodiment of the present application, each frame image of the video to be processed is obtained by performing frame processing on the video to be processed, and the image to be processed and the adjacent image are determined therefrom, wherein the image to be processed is the image that currently needs to be processed for task selection, and the adjacent image is the previous frame image or the next frame image of the image to be processed. The embodiment of the present application performs similarity comparison processing on the image to be processed and the adjacent image, thereby determining whether a scene change occurs in the video to be processed, such as a scene change such as camera switching or jumping in the video. Specifically, the image to be processed and the adjacent image are input into a two-way neural network for feature vector extraction processing to obtain a first feature vector and a second feature vector, the first feature vector being a feature vector obtained by feature extraction of the image to be processed, and the second feature vector being a feature vector obtained by feature extraction of the adjacent image. Then, the first feature vector and the second feature vector are subjected to similarity calculation processing, and the feature vector similarity can be obtained by calculating the cosine similarity or the like. Finally, the feature vector similarity is compared and processed according to the preset similarity threshold to obtain the selection task, so as to perform the subsequent tasks according to the selection task. When the similarity is lower than the preset similarity threshold, the image or video re-identification task is performed, and when the similarity is higher than the threshold, the target tracking task in the image or video is performed. Among them, the similarity threshold can be set according to the actual situation, and can be set to 0.5 in the embodiment of the present application. The embodiment of the present application selects tasks for each frame of the video to be processed, thereby distinguishing whether the video has a scene change and performing corresponding task processing, such as target tracking tasks and re-identification tasks, thereby reducing the loss of tracking targets or tracking target offsets during face tracking, and improving the accuracy and efficiency of video face swapping.

[0107] See also Figure 6 In some embodiments, step S502 may include but is not limited to steps S601 to S604:

[0108] Step S601, inputting the image to be processed and the adjacent image into a two-way neural network, wherein the two-way neural network includes two feature extraction networks with the same network scale and shared weights, and the feature extraction network includes multiple convolutional layers, pooling layers, and fully connected layers;

[0109] Step S602, performing hierarchical extraction processing on the image to be processed and the adjacent images through the multi-layer convolution layer to obtain multi-granularity features;

[0110] Step S603, performing residual connection processing on the multi-granularity features through the pooling layer to obtain intermediate features;

[0111] Step S604: Mapping the intermediate features through the fully connected layer to obtain global image features, where the global image features include a first feature vector and a second feature vector.

[0112] In an embodiment of the present application, a dual-path neural network includes a feature extraction network with the same network scale and shared weights, wherein the feature extraction network includes multiple layers of convolutional layers, pooling layers, and fully connected layers. By inputting the image to be processed and the adjacent image into the two feature extraction networks in the dual-path neural network respectively, a first feature vector and a second feature vector are extracted. The feature extraction network uses a multi-layer convolutional neural network as the core, and performs hierarchical extraction processing on the image to be processed and the adjacent image through multiple layers of convolutional layers to obtain multi-granularity features, thereby extracting features of different granularities in layers and stages, and then performing residual connection on the multi-granularity features through the pooling layer, which can be assisted by residual connection to summarize stable intermediate features, and finally obtain the final global image features through the fully connected layer. In an embodiment of the present application, the image to be processed and the adjacent image are input into the dual-path neural network for feature vector extraction processing, which provides a data basis for subsequent feature vector similarity comparison to distinguish scene changes.

[0113] In step S104 of some embodiments, when the selected task is a target tracking task, tracking and matching processing is performed on the target object through a target tracking model to obtain a target position frame, including:

[0114] Performing feature extraction processing on the target object to obtain a feature map;

[0115] Performing pyramid pooling processing on the feature map to obtain multi-dimensional features;

[0116] Generate a candidate frame according to the multi-dimensional features, calculate the confidence of the candidate frame, and determine the position information;

[0117] According to the position information, a matching process is performed on the face target in the image to be processed to obtain a target position frame.

[0118] In the embodiment of the present application, tracking matching is performed by the target tracking model, and a feature extraction network is first constructed. A classical neural network can be used, including but not limited to using VGG, ResNet, DenseNet, transformer, etc. When these classical skeleton networks are used, the parameters trained on a large-scale data set can be directly used to initialize the model, and the convergence speed of the network on a small-scale data set (such as a medical image data set) can be accelerated. The feature extraction network obtained by constructing the target object is subjected to feature extraction processing to obtain a feature map. Then the features in different dimensions in the feature map are obtained by pyramid pooling to obtain multidimensional features. A candidate frame is generated according to the multidimensional features, and the confidence of the candidate frame is calculated by the sotfmax function, and finally the candidate frame with high confidence is output as a result to obtain position information, and finally the face target in the processed image is matched by a matching algorithm, and it is calculated whether different targets in the processed image are target objects. The embodiment of the present application tracks and matches the target object by the target tracking model, calculates the position information according to the confidence and matches the target, and can accurately match the target object, thereby improving the target tracking accuracy in the face tracking process.

[0119] In step S105 of some embodiments, when the selected task is a target re-identification task, positioning and matching the target object is performed by a re-identification model to obtain a target position frame, including:

[0120] Performing feature extraction processing on the target object to obtain an extracted feature vector;

[0121] Performing high-dimensional space mapping and metric learning processing on the extracted feature vector to obtain a high-dimensional feature vector;

[0122] Feature matching processing is performed on the face target in the image to be processed according to the high-dimensional feature vector to obtain a target position frame.

[0123] In an embodiment of the present application, when it is determined that the image to be processed needs to perform a target re-identification task, the re-identification model is used to position and match the target object, so that the target object can be relocated under the scene change conditions such as the new camera angle and conditions. The embodiment of the present application first uses residual structure, pyramid pooling, and attention mechanism to build an efficient feature extraction network. The skeleton network of the feature extraction network can use the skeleton network of the classical neural network, including but not limited to using VGG, ResNet, DenseNet, transformer, etc. When using these classic skeleton networks, the parameters trained on large-scale data sets can be directly used to initialize the model to accelerate the convergence speed of the network on small-scale data sets (such as medical image data sets). Then the extracted feature vector is mapped to a high-dimensional space, and metric learning is performed to obtain a high-dimensional feature vector, so that the same target is closer in the high-dimensional space, and different target distances are farther. Finally, the targets of different pictures or video frames are matched by feature matching, so as to determine the position frame of the target object in each frame image of the video to be processed. The embodiment of the present application performs positioning and matching processing on the target object through the re-identification model, so as to quickly locate the target object when the scene changes, improve the accuracy of the face tracking process, and reduce the occurrence of tracking target loss or tracking target deviation.

[0124] In step S105 of some embodiments, the video to be processed is subjected to face-changing processing according to the target position frame to obtain a face-changing video. The position frame area of ​​the target object in each frame image of the video to be processed can be face-changed, and other areas, regardless of whether they contain human bodies, can be regarded as backgrounds and subtracted, so as to finally obtain a face-changing video containing only the face-changing results of the target object.

[0125] The following is a detailed description of the embodiments of the present invention with reference to specific application examples:

[0126] The embodiments of the present application can be applied to the field of computer vision technology, especially in application scenarios such as film production and video editing. For example, in a video editing scenario, a user inputs a street shot scene video of multiple people, in which the host is introducing the scenery, the camera keeps switching between the host and the scenery, and other passers-by come and go. According to a tracking and face-changing method based on a multi-person scene in an embodiment of the present application, voice detection and lip reading recognition are performed on the street shot scene video, and the similarity is calculated between the detected voice content and the recognized lip reading content, so as to locate the host's face image as the target object for tracking and face-changing. The speaker can be quickly located by combining multimodal information (voice, image, video subtitles, etc.), and the target object can be locked, thereby improving the accuracy of target tracking. Then, we select tasks for each frame of the street scene video, and identify the scene change by calculating the similarity of the feature vectors of each frame and the adjacent images, so as to determine the target tracking task or target re-identification task for each frame. We can track the target of the image with a small scene change to obtain the target position frame, and re-identify the target of the image with a large scene change to obtain the target position frame. When the video is in the same scene, we can track the host, and when the camera switches or jumps in the video, we can re-identify and locate the host, and then continue to track the target. The final output only includes the face-changing video of the host, and removes other passers-by. Please refer to Figure 7 , Figure 7 A flowchart of customized face-swapping is provided, in which the face-swapping model library contains multiple customized templates. The user uploads a video to be tracked and swapped, and the video is input into the customized face model for processing based on a multi-person scene tracking face-swapping method, thereby outputting a customized template face-swapping result video. Figure 8 , Figure 8 A flowchart of personalized face-changing is provided, wherein a user inputs a selfie video, and a user-personalized face model is customized and trained through the tracking face-changing method based on a multi-person scenario of this solution to obtain a user-personalized face-changing model library, thereby inputting the video to be tracked and face-changed into the user-personalized face-changing model library, and outputting a customized template face-changing result video.

[0127] See also Fig. 9 The embodiment of the present application further provides a tracking face-changing system based on a multi-person scene, which can implement the above-mentioned tracking face-changing method based on a multi-person scene, and the system includes:

[0128] The first module 901 is used to obtain a video to be processed;

[0129] The second module 902 is used to perform speech detection and lip reading recognition processing on the video to be processed, and perform similarity calculation based on the detection result and the recognition result to determine the target object;

[0130] The third module 903 is used to perform task selection processing on each frame image of the video to be processed to obtain a selection task for the image to be processed;

[0131] The fourth module 904 is used for performing tracking and matching processing on the target object through a target tracking model to obtain a target position frame when the selected task is a target tracking task;

[0132] The fifth module 905 is used for performing positioning and matching processing on the target object through the re-identification model to obtain a target position frame when the selected task is a target re-identification task;

[0133] The sixth module 906 is used to perform face-changing processing on the video to be processed according to the target position frame to obtain a face-changing video.

[0134] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0135] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned tracking face-changing method based on a multi-person scene when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0136] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0137] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0138] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0139] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes a tracking face-changing method based on a multi-person scene in the embodiment of this application;

[0140] Input / output interface 1003, used to implement information input and output;

[0141] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0142] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0143] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0144] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned tracking and face-changing based on a multi-person scene.

[0145] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0146] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The embodiments of the present application provide a tracking face-changing system, device and medium based on a multi-person scenario, which performs voice detection and lip reading recognition processing on the video to be processed, calculates the similarity based on the detection results and the recognition results, determines the target object, and can accurately locate the target object that needs to be replaced by analyzing the multimodal information; then performs task selection processing on each frame image of the video to be processed to obtain the selection task of the image to be processed, and can continuously track or re-identify and locate the target object according to different selection tasks, and performs face-changing processing on the target object according to the target position frame, thereby achieving real-time and accurate effects when tracking face-changing, and improving the efficiency of tracking face-changing.

[0148] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0149] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0150] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0151] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0152] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0153] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0155] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0158] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A tracking face-changing method based on multi-person scenes, It is characterized in that The method comprises: Get the video to be processed; Performing speech detection and lip reading recognition processing on the video to be processed, performing similarity calculation based on the detection results and the recognition results, and determining the target object; Performing task selection processing on each frame image of the video to be processed to obtain a selection task for the image to be processed; When the selected task is a target tracking task, the target object is tracked and matched by a target tracking model to obtain a target position frame; When the selected task is a target re-identification task, positioning and matching the target object is performed through a re-identification model to obtain a target position frame; Performing face-changing processing on the video to be processed according to the target position frame to obtain a face-changing video; The step of performing task selection processing on each frame of the video to be processed to obtain a selection task for the image to be processed includes: Performing frame processing on the video to be processed to obtain an image to be processed and adjacent images; Inputting the image to be processed and the adjacent image into a two-way neural network for feature vector extraction processing to obtain a first feature vector and a second feature vector; Performing similarity calculation processing on the first feature vector and the second feature vector to obtain feature vector similarity; Comparing the similarities of the feature vectors according to a preset similarity threshold to obtain a selection task; The step of inputting the image to be processed and the adjacent image into a two-way neural network for feature vector extraction processing to obtain a first feature vector and a second feature vector comprises: Inputting the image to be processed and the adjacent image into a two-way neural network, wherein the two-way neural network includes two feature extraction networks with the same network scale and shared weights, and the feature extraction network includes multiple convolutional layers, pooling layers, and fully connected layers; Performing hierarchical extraction processing on the image to be processed and the adjacent images through the multi-layer convolutional layers to obtain multi-granularity features; Performing residual connection processing on the multi-granularity features through the pooling layer to obtain intermediate features; The intermediate features are mapped through the fully connected layer to obtain global image features, where the global image features include a first feature vector and a second feature vector.

2. The method according to claim 1, It is characterized in that The performing speech detection and lip reading recognition processing on the video to be processed, performing similarity calculation according to the detection result and the recognition result, and determining the target object includes: Performing content detection processing on the voice or subtitle of the video to be processed to obtain a detection result; Performing lip reading recognition processing on each face image in the video to be processed through a lip reading recognition network to obtain a recognition result; Calculating the edit distance between the texts of the detection result and the recognition result, and determining the face image with the smallest edit distance as the initial target; Within a preset time, the face image that is used as the initial target most times in the video to be processed is determined as the target object.

3. The method according to claim 2, It is characterized in that The lip reading recognition network is used to perform lip reading recognition processing on each face image in the video to be processed to obtain a recognition result, including: Inputting the face image into the lip reading recognition network, wherein the lip reading recognition network includes a convolutional neural network, a long short-term memory recursive neural network, a multi-layer perceptron and a classifier; Performing feature extraction processing on the face image through the convolutional neural network to obtain a first feature; Performing time domain information extraction processing on the first feature through the long short-term memory recurrent neural network to obtain a second feature; Performing text prediction processing on the second feature by the multi-layer perceptron to obtain predicted text; The predicted text is subjected to label generation processing by the classifier to obtain a recognition result.

4. The method according to claim 1, It is characterized in that The tracking and matching process of the target object is performed by the target tracking model to obtain a target position frame, including: Performing feature extraction processing on the target object to obtain a feature map; Performing pyramid pooling processing on the feature map to obtain multi-dimensional features; Generate a candidate frame according to the multi-dimensional features, calculate the confidence of the candidate frame, and determine the position information; According to the position information, a matching process is performed on the face target in the image to be processed to obtain a target position frame.

5. The method according to claim 1, It is characterized in that The performing positioning and matching processing on the target object by the re-identification model to obtain a target position frame includes: Performing feature extraction processing on the target object to obtain an extracted feature vector; Performing high-dimensional space mapping and metric learning processing on the extracted feature vector to obtain a high-dimensional feature vector; Feature matching processing is performed on the face target in the image to be processed according to the high-dimensional feature vector to obtain a target position frame.

6. A tracking face-changing system based on multi-person scenes, It is characterized in that The system comprises: The first module is used to obtain the video to be processed; The second module is used to perform speech detection and lip reading recognition on the video to be processed, calculate similarity based on the detection results and recognition results, and determine the target object; The third module is used to perform task selection processing on each frame image of the video to be processed to obtain a selection task for the image to be processed; The fourth module is used for performing tracking and matching processing on the target object through a target tracking model to obtain a target position frame when the selected task is a target tracking task; A fifth module is used for performing positioning and matching processing on the target object through a re-identification model to obtain a target position frame when the selected task is a target re-identification task; A sixth module is used to perform face-changing processing on the video to be processed according to the target position frame to obtain a face-changing video; The third module is used to perform task selection processing on each frame image of the video to be processed to obtain the selection task of the image to be processed, including: Performing frame processing on the video to be processed to obtain an image to be processed and adjacent images; Inputting the image to be processed and the adjacent image into a two-way neural network for feature vector extraction processing to obtain a first feature vector and a second feature vector; Performing similarity calculation processing on the first feature vector and the second feature vector to obtain feature vector similarity; Comparing the similarities of the feature vectors according to a preset similarity threshold to obtain a selection task; The step of inputting the image to be processed and the adjacent image into a two-way neural network for feature vector extraction processing to obtain a first feature vector and a second feature vector comprises: Inputting the image to be processed and the adjacent image into a two-way neural network, wherein the two-way neural network includes two feature extraction networks with the same network scale and shared weights, and the feature extraction network includes multiple convolutional layers, pooling layers, and fully connected layers; Performing hierarchical extraction processing on the image to be processed and the adjacent images through the multi-layer convolutional layers to obtain multi-granularity features; Performing residual connection processing on the multi-granularity features through the pooling layer to obtain intermediate features; The intermediate features are mapped through the fully connected layer to obtain global image features, where the global image features include a first feature vector and a second feature vector.

7. An electronic device, It is characterized in that The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images