Method, device and equipment for synchronizing mouth shapes of multiple people in a video, and storage medium

Abstract

Embodiments of the present application provide a method, device and equipment for character lip synchronization of multi-person video, and a storage medium. The method comprises: obtaining an original video, lip synchronization audio and character description information of a target character, determining a target face region of the target character in the original video; performing mask processing on a face region other than the target face region in the original video to generate a target character reserved video, adding silent audio at a starting position of the lip synchronization audio to obtain extended driving audio; generating a new lip sequence for the target character in the target character reserved video based on the extended driving audio and the target face region to obtain a lip video, and cutting off a silent segment corresponding to the silent audio in the lip video to obtain a lip synchronization video of the target character. The present scheme realizes accurate and stable lip synchronization of only the target character.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device, and storage medium for synchronizing lip movements in multi-person video. Background Technology

[0002] In recent years, with the development of artificial intelligence and deep learning technologies, video lip-syncing technology has been widely used in scenarios such as virtual digital humans, film and television dubbing, game characters, and advertising video production. This type of technology typically generates lip movements that match the speech content and rhythm based on the speech features in the input audio, thereby improving the realism of the spoken characters and the efficiency of video production.

[0003] Current lip-syncing technologies are primarily designed for single-person video scenarios. In multi-person video scenarios, the system struggles to accurately identify the person requiring lip-syncing, easily misselecting other individuals or switching between them, resulting in abnormal lip movements from non-target individuals. Furthermore, facial movements from multiple faces can interfere with each other, causing a mismatch between the target person's lip movements and the audio. Additionally, the lack of a stable initial alignment between audio and video at the beginning of the video can lead to lip-syncing deviations, affecting the accuracy and stability of multi-person video lip-syncing. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for lip-syncing in multi-person videos. It can locate the target person in a multi-person video by using the person's description information and mask non-target faces. At the same time, it adds silent audio before the lip-syncing audio to establish a closed initial state. After generating a new lip shape sequence, it cuts out the corresponding silent segment, so as to achieve accurate and stable lip-syncing only for the target person. It effectively solves the problems of difficulty in locking the target person, easy mis-modification of non-target persons, and unstable initial lip shape in multi-person scenes.

[0005] In a first aspect, embodiments of this application provide a method for lip-syncing in a multi-person video, the method comprising: Acquire the original video, lip-synced audio, and character description information of the target person, and determine the target face region of the target person in the original video; Masking is performed on the face regions outside the target face region in the original video to generate a video that retains the target person, and silent audio is added at the beginning of the lip-sync audio to obtain extended driving audio; Based on the extended driving audio and the target face region, a new lip shape sequence is generated for the target person in the video, resulting in a lip-sync video. The silent segments corresponding to the silent audio are then removed from the lip-sync video to obtain a lip-sync synchronized video of the target person.

[0006] In one embodiment, adding silent audio at the beginning of the lip-sync audio to obtain extended drive audio includes: The silence duration is determined based on the frame rate of the original video and the preset number of silence frames; Generate all-zero audio data corresponding to the silence duration according to the sampling parameters of the lip-sync audio, and obtain the silence audio; The silent audio is spliced ​​before the lip-sync audio, and the splicing time of the silent audio and the lip-sync audio is recorded to obtain the extended driving audio.

[0007] In one embodiment, generating a new lip shape sequence for the target person in the video based on the extended driving audio and the target face region to obtain a lip-sync video includes: According to the frame rate of the video of the target person, the extended driving audio is segmented to obtain silent audio segments and multiple speech audio segments, and each audio segment is matched with each frame of the video of the target person in time sequence. The silent audio segment is determined as the initial audio segment, and based on the initial audio segment and the target face region in the corresponding frame image, the initial closed-mouth state of the target person is determined. Based on each speech audio segment, the initial closed state, and the target face region in the corresponding frame image, a new lip shape sequence of the target person is generated, and the new lip shape sequence is fused into the retained video of the target person to obtain the lip shape video of the target person.

[0008] In one embodiment, generating a new lip shape sequence for the target person based on each speech audio segment, the initial closed-mouth state, and the target face region in the corresponding frame image includes: Extract the speech-driven features corresponding to each speech audio segment; Using the lip key point position corresponding to the initial closed state as the starting constraint, and based on each voice-driven feature, the position of the lip key point in the target face region in the corresponding frame image is adjusted to obtain the target lip key point corresponding to each frame image. Based on the temporal sequence of each frame image, a new lip shape is generated corresponding to each target lip key point, thus obtaining a new lip shape sequence for the target person.

[0009] In one embodiment, the step of trimming the silent segment corresponding to the silent audio from the lip-sync video to obtain the lip-sync video of the target person includes: Based on the time interval of the silent audio in the extended driving audio, determine the corresponding silent segment in the lip-sync video; The video and audio content of the silent segment are simultaneously cut out from the lip-sync video, and the start time of the retained video and audio content is reset to zero to obtain the lip-sync video of the target person.

[0010] In one embodiment, determining the target face region of the target person in the original video includes: Target frame images for character localization are extracted from the original video, and multi-scale facial feature extraction is performed on the target frame images to obtain the face bounding box, facial key points and detection confidence for each character. Filter out face bounding boxes whose detection confidence is lower than a preset confidence threshold to obtain multiple face regions. Sort the multiple face regions according to their horizontal positions in the target frame image and configure a person index identifier for each face region. Based on the person description information, a person location instruction is generated, and multimodal semantic analysis is performed on the person location instruction and the target frame image to determine the target person index identifier that matches the person description information. Based on the target person index identifier, the corresponding face bounding box and face key points are extracted from the multiple face regions to obtain the target face region of the target person.

[0011] In one embodiment, the step of masking the face region outside the target face region in the original video to generate a video retaining the target person includes: Using the target face region as the initial tracking region, the face region corresponding to the target person in each frame of the original video is determined frame by frame, and the remaining face regions in the same frame are determined as a set of non-target face regions. For each frame of the original video, blurring and masking are performed on each face region in the corresponding set of non-target face regions, while keeping the face region corresponding to the target person in the corresponding frame unchanged, to obtain the corresponding masked image. The target person's video is obtained by combining the masked images according to their temporal order in the original video.

[0012] Secondly, embodiments of this application also provide a lip-syncing device for multi-person video, comprising: The determination module is configured to acquire the original video, lip-sync audio, and character description information of the target person, and determine the target face region of the target person in the original video; The extension module is configured to mask the face region outside the target face region in the original video, generate a video retaining the target person, and add silent audio at the beginning of the lip-sync audio to obtain extended driving audio; The lip-sync module is configured to generate a new lip shape sequence for the target person in the video based on the extended driving audio and the target face region, to obtain a lip-sync video, and to remove the silent segments in the lip-sync video that correspond to the silent audio, to obtain a lip-sync video of the target person.

[0013] Thirdly, embodiments of this application also provide a lip-syncing device for multi-person videos, the device comprising: One or more processors; Storage device, configured to store one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method for lip-syncing in multi-person video as described in the embodiments of this application.

[0014] Fourthly, embodiments of this application also provide a non-volatile storage medium for storing computer-executable instructions, which, when executed by a computer processor, are configured to perform the lip-syncing method for multi-person video described in embodiments of this application.

[0015] Fifthly, embodiments of this application also provide a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads from the computer-readable storage medium and executes the computer program, causing the device to perform the lip-syncing method for multi-person video described in embodiments of this application.

[0016] In this embodiment, face detection is performed on multi-person videos, and the target face region is determined by combining the target person's description information. Non-target face regions are masked, so that the subsequent lip-sync generation process focuses on the target person. Simultaneously, silent audio is added at the beginning of the original audio to provide a stable closed-mouth initialization state for the target person's lip-sync generation, and the corresponding silent segment is removed after generation. Therefore, new lip-sync sequences for a specified target person are generated and audio-visual timing is synchronized in multi-person scenarios, avoiding mis-driving of other people and improving the accuracy of target person localization and lip-sync synchronization. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a method for synchronizing lip movements in a multi-person video, as provided in an embodiment of this application; Figure 2A flowchart illustrating a specific implementation process for determining the target face region of a target person in an original video, as provided in this application embodiment; Figure 3 A flowchart illustrating a specific implementation process for masking non-target face regions provided in this application embodiment; Figure 4 A flowchart illustrating a specific implementation process for adding silent audio at the beginning position of lip-sync audio, as provided in this application embodiment; Figure 5 A flowchart illustrating a specific implementation process for generating a new lip shape sequence for a target person and obtaining a lip-sync video, provided in this application embodiment; Figure 6 A flowchart illustrating a specific implementation process for generating a new lip shape sequence for a target person, provided in this application embodiment; Figure 7 A flowchart illustrating a specific implementation process for trimming silent segments from a lip-syncing video, provided in this application embodiment; Figure 8 A structural block diagram of a multi-person video lip-syncing device provided in this application embodiment; Figure 9 This is a schematic diagram of the structure of a multi-person video lip-syncing device provided in an embodiment of this application. Detailed Implementation

[0018] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the scope of the embodiments. Furthermore, it should be noted that, for ease of description, only the parts relevant to the embodiments of this application are shown in the accompanying drawings, not the entire structure.

[0019] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or more. The term "several" means one or more, and multiple means two or more. Furthermore, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0020] The method for lip-syncing in multi-person video provided in this application can be applied to scenarios such as film and television dubbing, short video production, virtual digital humans, advertising video generation, online conference video processing, and game character animation production. The aforementioned application scenarios are merely exemplary and illustrative. In practical applications, this method for lip-syncing in multi-person video can also be used in other scenarios, and this application does not limit these applications.

[0021] The method for synchronizing lip movements in multi-person video provided in this application embodiment can be executed by a computer device. The computer device refers to any electronic device with data computing, processing and storage capabilities, such as a server or other terminal device. This application embodiment does not limit this.

[0022] Figure 1 This is a flowchart illustrating a method for synchronizing lip movements in a multi-person video, as provided in an embodiment of this application. This method can be applied to video processing devices, such as… Figure 1 As shown, the method for synchronizing lip movements in a multi-person video includes the following steps: Step S101: Obtain the original video, lip-sync audio, and description information of the target person, and determine the target face region of the target person in the original video.

[0023] The original video can be a video file containing two or more people; the lip-sync audio can be speech audio used to drive the target person to produce corresponding lip movements; the target person can be the person in the original video who needs to be lip-synced; the person description information can be text information used to describe the appearance characteristics of the target person, such as the person's location, clothing color, hairstyle, gender representation, or accessories worn.

[0024] Optionally, after acquiring the original video, video adaptation processing can be performed to ensure that the target person's face is continuously visible in the video and that the target person's lip area remains intact and unobstructed. The lip-sync audio can be pre-processed with standardized sampling parameters, noise filtering, and volume normalization to facilitate the subsequent extraction of stable speech-driven features.

[0025] The target face region can be the image area occupied by the target person's face in the frame image of the original video. It can be represented by face bounding box coordinates and can be associated with and recorded facial key points such as eyes, nose, mouth, and facial contours. The person description information can be entered by the user or read from pre-configured person information.

[0026] Optionally, the method for determining the target face region of the target person in the original video can be as follows: extract the target frame image for person localization from the original video, perform face detection on each person in the target frame image to obtain multiple face regions; configure person index identifiers according to the position of each face region in the target frame image; input the target frame image and person description information into a multimodal processing module with image and text joint analysis capabilities, determine the person index identifier that matches the person description information, and determine the face region corresponding to the person index identifier as the target face region.

[0027] For example, if the target frame image contains three people from left to right, and the description of each person is "the person in the middle of the image wearing a red shirt", then the index of the person in the middle can be determined based on the image content and the description of the person, and the face bounding box and facial key points of that person can be obtained.

[0028] In one embodiment, a specific implementation process for identifying the target facial region of a target person in an original video is described. Please refer to [reference needed]. Figure 2 , Figure 2 This is a flowchart illustrating a specific implementation process for determining the target face region of a target person in an original video, as provided in an embodiment of this application. Figure 2 As shown, the specific steps for determining the target face region of the target person in the original video are as follows: Step S201: Extract target frame images for character localization from the original video, and perform multi-scale facial feature extraction on the target frame images to obtain the face bounding boxes, facial key points and detection confidence scores for each character.

[0029] The target frame image can be a frame image selected from the original video that displays the appearance and facial regions of each person. Optionally, the target frame image can be the first frame image of the original video, or a clear frame image selected within the beginning time segment of the original video. Multi-scale facial feature extraction can identify faces of different sizes in feature maps at different resolutions. The face bounding box can be a rectangular box used to define the position of a person's face; facial key points can include corner points of the eyes, tip points of the nose, corner points of the mouth, upper lip points, lower lip points, and facial contour points; detection confidence can represent the degree to which the detected region belongs to the face region.

[0030] Optionally, the method for extracting target frame images for person localization from the original video and performing multi-scale facial feature extraction on the target frame images to obtain the face bounding boxes, facial key points, and detection confidence scores for each person can be as follows: scaling the target frame images to form multiple scale images to be detected; extracting facial features from the images to be detected at each scale; predicting candidate face bounding boxes, facial key points, and corresponding detection confidence scores based on the facial features; and mapping the detection results at different scales to the original coordinate space of the target frame images.

[0031] Step S202: Filter out face bounding boxes whose detection confidence is lower than the preset confidence threshold to obtain multiple face regions. Sort the multiple face regions according to their horizontal positions in the target frame image and configure person index labels for each face region.

[0032] The preset confidence threshold can be a threshold used to filter low-confidence candidate face boxes. The lateral position can be determined by the x-coordinate of the left boundary of the face box, the x-coordinate of the center point of the face box, or other coordinates that can characterize the left and right position of the face. The person index identifier can be a sequence number used to distinguish different people in the target frame image.

[0033] Optionally, filtering out face bounding boxes with detection confidence scores below a preset confidence threshold to obtain multiple face regions can be achieved by comparing the detection confidence score of each candidate face bounding box with the preset confidence threshold, and deleting face bounding boxes with detection confidence scores below the preset confidence threshold. For example, the preset confidence threshold can be 0.7.

[0034] Optionally, multiple face regions can be sorted according to their horizontal positions in the target frame image, and person index identifiers can be configured for each face region as follows: overlap detection is performed on the retained face bounding boxes, and duplicate face bounding boxes representing the same person are removed; the retained face bounding boxes are sorted in ascending order of their center point horizontal coordinates, and person index identifiers are configured sequentially. For example, when three face regions are detected in the target frame image, person index identifiers 0, 1, and 2 can be configured respectively from left to right.

[0035] Step S203: Generate a person location instruction based on the person description information, and perform multimodal semantic analysis on the person location instruction and the target frame image to determine the target person index identifier that matches the person description information.

[0036] The character location instruction can be a text instruction formed by combining character description information, the number of candidate characters, and output format requirements. Multimodal semantic analysis can jointly understand the visual content in the target frame image and the text content in the character location instruction, and determine the matching relationship between the two.

[0037] Optionally, the method for generating a person location instruction based on the person description information can be as follows: read the person description information and the number of face regions in the target frame image; write the person description information into a preset instruction template, and limit the selection of a unique matching person from multiple persons and output the corresponding person index identifier in the preset instruction template.

[0038] Optionally, the method for performing multimodal semantic analysis on the person location command and the target frame image to determine the target person index identifier that matches the person description information can be as follows: input the person location command and the target frame image into the multimodal processing module; parse the index result output by the multimodal processing module to obtain the target person index identifier.

[0039] For example, if the target frame image contains three people and the description information of the people is "the person wearing glasses on the right side of the image", the multimodal processing module can combine the spatial position and wearing characteristics of the people to output the index identifier of the person on the right.

[0040] Step S204: Based on the target person index identifier, extract the corresponding face bounding boxes and face key points from multiple face regions to obtain the target face region of the target person.

[0041] The target person index identifier can be matched one-to-one with the sorted face regions. The target face region can include not only the face bounding box of the target person, but also the facial landmarks within the face bounding box and the target person's index information.

[0042] Optionally, the method to extract the corresponding face bounding box and facial key points from multiple face regions based on the target person's index identifier to obtain the target person's target face region can be as follows: query the target person's index identifier in the correspondence between the person's index identifier and the face region; read the coordinates of the face bounding box and the coordinates of the facial key points of the corresponding face region; associate and encapsulate the face bounding box coordinates, the coordinates of the facial key points and the target person's index identifier to obtain the target person's target face region.

[0043] As can be seen from the above, by horizontally sorting the face regions and configuring person index identifiers, the correspondence between each person in the image and the index can be established; through multimodal semantic analysis of person description information and target frame images, the target person matching the person description information can be determined from a multi-person scene.

[0044] Step S102: Mask the face area outside the target face area in the original video to generate a video that retains the target person, and add silent audio at the beginning of the lip-sync audio to obtain extended driving audio.

[0045] The target person retention video can be a video formed by retaining the facial image of the target person and blurring / masking the facial images of other people. Silent audio can be audio data with all audio sample values ​​being zero or below a preset volume threshold. Extended-drive audio can be audio formed by splicing silent audio and lip-synced audio in chronological order.

[0046] Optionally, masking is performed on the face regions outside the target face region in the original video to generate a video that preserves the target person. This can be done by: using the target face region as the initial tracking region, determining the face region corresponding to the target person frame by frame in the original video, and identifying other face regions in the same frame as non-target face regions; blurring and masking each non-target face region, and then combining the processed frame images according to the original time sequence to obtain the video that preserves the target person.

[0047] Optionally, adding silent audio at the beginning of the lip-sync audio to obtain extended driving audio can be achieved by generating all-zero audio data with the same sampling parameters as the lip-sync audio based on a preset silence duration, and then concatenating it before the lip-sync audio to obtain extended driving audio.

[0048] As shown above, by masking the facial areas of non-target individuals, the interference of their facial features on subsequent character localization and lip shape generation can be reduced. By setting a silent audio before the lip-sync audio, an initial time interval corresponding to the closed-mouth state can be provided for the lip shape generation process of the target individual.

[0049] In one embodiment, a specific implementation process for masking non-target face regions is described. Please refer to [reference needed]. Figure 3 , Figure 3 This is a flowchart illustrating a specific implementation process for masking non-target face regions, as provided in an embodiment of this application. Figure 3 As shown, the specific implementation steps for masking non-target face areas and generating a video that retains the target person are as follows: Step S301: Using the target face region as the initial tracking region, determine the face region corresponding to the target person in each frame of the original video frame by frame, and determine the remaining face regions in the same frame as the set of non-target face regions.

[0050] The initial tracking region can be the bounding box of the target person's face in the target frame image. Determining the face region corresponding to the target person frame by frame can be based on facial features, positional features, and facial key points in the initial tracking region, identifying face regions belonging to the same target person in adjacent frames. The set of non-target face regions can include one or more face regions other than the target person in the same frame image.

[0051] Optionally, the method of determining the target person's face region in each frame of the original video frame by frame, using the target face region as the initial tracking region, and defining the remaining face regions in the same frame as the non-target face region set, can be as follows: extract the target person's facial features and facial key points from the target face region; detect candidate face regions in each frame after the target frame; determine the target person's face region in the current frame based on the positional continuity, facial feature similarity, and key point distribution similarity between the candidate face regions and the target face region in the previous frame; and add the remaining face regions in the current frame to the non-target face region set. When the target person moves or their face turns to the side, the face region corresponding to the target person can be updated by combining the movement direction and changes in facial key points in the preceding and following frames.

[0052] Step S302: For each frame of the original video, perform blur masking on each face region in the corresponding set of non-target face regions, while keeping the face region corresponding to the target person in the corresponding frame unchanged, to obtain the corresponding masked image.

[0053] The blurring and masking process can reduce the sharpness of facial contours, features, and lips in non-target face regions, making it difficult for subsequent lip-syncing generation to extract effective facial information from these regions. The masked image can be the image obtained after blurring the non-target face regions in the current frame.

[0054] Optionally, for each frame of the original video, blurring and masking are performed on each face region in the corresponding set of non-target face regions, while keeping the face region corresponding to the target person in the corresponding frame unchanged. The method to obtain the corresponding masked image can be as follows: determine the masking range based on the face bounding box corresponding to the non-target face region; expand the masking range to the outside of the face bounding box according to a preset expansion ratio to cover the complete face outline; perform blurring on the pixels within the masking range and write the processed pixels back to the corresponding frame image; do not perform blurring on the face region corresponding to the target person to obtain the masked image.

[0055] For example, a Gaussian filter window of a preset size can be used to blur non-target face areas. The preset size can be 21×21 pixels, or it can be adaptively set according to the size of the non-target face area.

[0056] Step S303: Combine the masked images according to their temporal order in the original video to obtain the video of the target person.

[0057] Each masked image can retain its frame number and timestamp from the original video. The target person retention video can maintain the original video's frame rate, resolution, and temporal order, only changing the non-target face regions in each frame.

[0058] Optionally, the method of combining the mask images according to their temporal order in the original video to obtain the target person's retained video can be as follows: arrange the mask images in ascending order of their corresponding frame numbers; encode each mask image using the frame rate and resolution of the original video; and encapsulate the other unprocessed content in the original video with the encoded mask images to obtain the target person's retained video.

[0059] As can be seen from the above, by continuously tracking the target person in the original video, the consistency of the target person's identity can be maintained when the target person moves or changes posture; by blurring and masking non-target face areas frame by frame, the interference of other people's facial information on the target person's lip-syncing generation process can be reduced, and the generation process can be focused on the specified target person.

[0060] In one embodiment, a specific implementation process for adding silent audio at the beginning of lip-sync audio is described. Please refer to [reference needed]. Figure 4 , Figure 4 This is a flowchart illustrating a specific implementation process for adding silent audio at the beginning of lip-sync audio, as provided in an embodiment of this application. Figure 4 As shown, the specific implementation steps for adding silent audio at the beginning of the lip-sync audio to obtain extended driven audio are as follows: Step S401: Determine the silence duration based on the frame rate of the original video and the preset number of silent frames.

[0061] The frame rate can be used to represent the number of frames contained in the original video per unit of time, such as 25 or 30 frames per second. The preset number of silent frames can be the number of initial frames required for the lip-sync generation process to enter a stable closed state. The silence duration can be the length of time corresponding to the ratio between the preset number of silent frames and the frame rate of the original video.

[0062] Optionally, the silence duration can be determined based on the frame rate of the original video and the preset number of silence frames by: reading the frame rate parameter of the original video and obtaining the preset number of silence frames; dividing the preset number of silence frames by the frame rate of the original video to obtain the silence duration.

[0063] For example, if the original video has a frame rate of 30 frames per second and the preset number of silent frames is 30, then the silence duration can be set to 1 second. The preset number of silent frames can be set according to the initialization time required for the lip-sync generation process.

[0064] Step S402: Generate all-zero audio data corresponding to the silence duration according to the sampling parameters of the lip-sync audio to obtain the silence audio.

[0065] The sampling parameters may include at least one of the following: sampling rate, bit depth, number of channels, and audio encoding format. All-zero audio data can be audio data where the sample value at each audio sampling point is set to zero, so that the audio data does not produce audible speech during playback.

[0066] Optionally, all-zero audio data corresponding to the silence duration can be generated according to the sampling parameters of the lip-sync audio. The method to obtain the silence audio can be as follows: read the sampling rate, bit depth, and number of channels of the lip-sync audio; determine the number of sampling points contained in the silence audio according to the sampling rate and silence duration; create the corresponding number of all-zero sampling points according to the bit depth and number of channels of the lip-sync audio, and encapsulate them into silence audio with the same format as the lip-sync audio.

[0067] For example, when the sampling rate of the lip-sync audio is 48,000 Hz, the bit depth is 16 bits, and the silence duration is 1 second, it is possible to generate silent audio with 48,000 all-zero samples per channel.

[0068] Step S403: Concatenate the silent audio before the lip-sync audio and record the concatenation time of the silent audio and the lip-sync audio to obtain the extended driving audio.

[0069] The splicing time point can be the position where the silent audio ends and the lip-sync audio begins, used to identify the boundary between the silent portion and the speech portion in the extended driving audio. This splicing time point can also be used to subsequently determine the silent segments to be cut from the lip-sync video.

[0070] Optionally, the extended driving audio can be obtained by splicing the silent audio before the lip-sync audio and recording the splicing time point between the silent audio and the lip-sync audio. This can be achieved by: connecting the end sampling point of the silent audio with the start sampling point of the lip-sync audio in a continuous time sequence; recording the duration of the silent audio as the splicing time point; and re-encapsulating the spliced ​​audio data to obtain the extended driving audio.

[0071] Optionally, if the sampling parameters of the silent audio and the lip-sync audio are inconsistent, the sampling parameters of the silent audio can be set to be consistent with those of the lip-sync audio before audio splicing is performed to avoid audio format jumps at the splicing position.

[0072] As can be seen from the above, by determining the silence duration based on the video frame rate and the preset number of silence frames, the number of initial frames required for lip-sync generation can be matched with the number of silence audio frames. By recording the splicing time points, a time basis can be provided for the subsequent determination and trimming of silence segments.

[0073] Step S103: Based on the extended driving audio and the target face region, generate a new lip shape sequence for the target person in the video to obtain a lip-sync video, and remove the silent segments corresponding to the silent audio in the lip-sync video to obtain a lip-sync synchronized video of the target person.

[0074] The new lip shape sequence can be multiple target lip shapes arranged in video time sequence, each corresponding to the lip shape of the target person in different frame images. The lip-sync video can be a video generated by fusing the new lip shape sequence into the retained video of the target person. The silent segment can be a video segment and an audio segment in the lip-sync video that have a corresponding temporal relationship with the silent audio of the extended driving audio front end.

[0075] Optionally, based on the extended driving audio and the target face region, a new lip shape sequence is generated for the target person in the preserved video. The method to obtain the lip shape video can be as follows: determine the initial closed state of the target person based on the silent audio in the extended driving audio; extract the speech driving features corresponding to each time interval in the lip shape synchronization audio, and generate a new lip shape sequence by combining the lip key points of the target person in each frame image; and fuse the new lip shape sequence into the target face region in the preserved video of the target person in a temporal order to obtain the lip shape video.

[0076] Optionally, the method to obtain the lip-sync video of the target person by cutting out the silent segments corresponding to the silent audio in the lip-sync video can be as follows: based on the time interval of the silent audio, synchronously cut out the corresponding video and audio content from the beginning position of the lip-sync video, and reset the timestamp of the remaining content to obtain the lip-sync video of the target person.

[0077] As can be seen from the above, the embodiments of this application determine the target person in a multi-person video by using the person description information, reduce the interference of other faces by masking non-target persons, and establish a closed initial state by using silent audio, so that the generated new lip shape can stably act on the designated target person; after the lip shape is generated, the silent segment is removed, so that the effective content of the final video can keep the lip-sync audio in line with the lip shape.

[0078] In one embodiment, a specific implementation process for generating a new lip shape sequence for a target person and obtaining a lip-sync video is described. Please refer to [reference needed]. Figure 5 , Figure 5 This is a flowchart illustrating a specific implementation process for generating a new lip shape sequence for a target person and obtaining a lip-sync video, as provided in this application embodiment. Figure 5 As shown, the specific implementation steps for generating a new lip shape sequence and obtaining a lip-sync video for the target person are as follows: Step S501: According to the frame rate of the video of the target person, the extended driving audio is segmented to obtain silent audio segments and multiple speech audio segments, and each audio segment is matched with each frame image in the video of the target person in terms of time sequence.

[0079] Specifically, audio segments can be audio data units divided from extended driving audio based on the duration of a single frame image in the target person's retained video. Silent audio segments can be audio segments located at the beginning of the extended driving audio and divided from silent audio. Speech audio segments can be audio segments containing speech content divided from lip-sync audio.

[0080] Optionally, the extended driving audio is segmented according to the frame rate of the target person's preserved video, resulting in silent audio segments and multiple speech audio segments. The method for temporally mapping each audio segment to each frame in the target person's preserved video can be as follows: determine the duration of a single frame based on the frame rate of the target person's preserved video; set segment boundaries on the time axis of the extended driving audio according to the duration of that single frame, and extract each audio segment sequentially; establish a temporal correspondence between each audio segment and the frame image at the corresponding time position based on the start and end times of each audio segment. For the initialization time interval corresponding to the silent audio, the starting frame image of the target person's preserved video or the initialization frame sequence formed by the starting frame image can be mapped to the silent audio segments, thereby maintaining the stability of the target person's facial state before formal speech driving.

[0081] Step S502: Determine the silent audio segment as the initial audio segment, and based on the initial audio segment and the target face region in the corresponding frame image, determine the initial closed state of the target person.

[0082] The initial audio segment can be an audio segment used to initialize the lip state of the target person. The closed initial state can be the lip state of the target person when there is no effective speech drive, which can be represented by the positional relationship between the key points of the upper lip, the key points of the lower lip, and the key points of the corners of the mouth.

[0083] Optionally, the method for determining the initial audio segment as the silent audio segment and the target face region in the corresponding frame image to determine the initial closed state of the target person can be as follows: when the audio sampling value of the initial audio segment meets the silent condition, read the lip key points in the target face region in the corresponding frame image; form the lip key point reference corresponding to the closed state according to the distance between the upper lip and the lower lip and the position of the left and right corners of the mouth; and determine the lip key point reference as the initial closed state of the target person.

[0084] For example, the positions of the left and right corners of the target person's mouth in the initial frame image can be kept unchanged, and the distance between the center key point of the upper lip and the center key point of the lower lip can be made smaller than the preset closing distance to form the initial closed state.

[0085] Step S503: Based on each speech audio segment, the initial closed state, and the target face region in the corresponding frame image, generate a new lip shape sequence for the target person, and fuse the new lip shape sequence into the retained video of the target person to obtain the lip shape video of the target person.

[0086] The audio segmentation provides corresponding speech-driven information for each frame of the image; the new lip shape sequence describes the process of the target person's lips changing from an initial closed state according to the speech content and time sequence. Fusion can replace or overlay the generated new lip shape onto the target face region in the corresponding frame of the image.

[0087] Optionally, the method for generating a new lip shape sequence for the target person based on each speech audio segment, the initial closed state, and the target face region in the corresponding frame image can be as follows: extract the speech-driven features of each speech audio segment respectively; take the initial closed state as the starting state of the new lip shape sequence, and combine the lip key points of the target person in the corresponding frame image to generate the new lip shape corresponding to each frame image.

[0088] Optionally, the new lip shape sequence can be fused into the target person's preserved video to obtain the target person's lip shape video. This can be achieved by arranging the new lip shapes sequentially according to the frame images and fusion them into the corresponding target face region in the target person's preserved video. During the fusion process, the eyes, nose, facial contours, and skin texture in the target face region can be preserved, and only the local area containing the lips and the area around the mouth can be updated, thereby reducing the impact of the new lip shapes on other facial content of the target person.

[0089] As can be seen from the above, by using silent audio segments as initial input, the initial closed state of the target person can be established; on this basis, new lip shapes can be generated according to the temporal sequence of the speech audio segments, so that the new lip shape sequence can gradually transition from a stable state to a lip movement state corresponding to the lip shape synchronized with the audio.

[0090] In one embodiment, a specific implementation process for generating a new lip shape sequence for a target person is described. Please refer to [reference needed]. Figure 6 , Figure 6 This is a flowchart illustrating a specific implementation process for generating a new lip shape sequence for a target person, as provided in an embodiment of this application. Figure 6 As shown, the specific steps for generating a new lip shape sequence for the target person are as follows: Step S601: Extract the speech-driven features corresponding to each speech audio segment.

[0091] Among them, speech-driven features can be audio features used to characterize speech content, pronunciation state, and speech rhythm, and can include at least one of audio spectrum features, phoneme representation features, volume features, duration features, and pronunciation intensity features.

[0092] Optionally, the method for extracting the speech driving features corresponding to each speech audio segment can be as follows: pre-emphasize, frame, and perform spectral transformation on each speech audio segment to obtain the corresponding time-frequency features; encode the time-frequency features to obtain speech driving features containing pronunciation content and pronunciation intensity information; and configure time identifiers for each speech driving feature according to the time position of each speech audio segment in the extended driving audio.

[0093] For example, audio segments containing closed lip consonants can correspond to a smaller degree of lip opening, while audio segments containing open vowels can correspond to a larger degree of lip opening.

[0094] Step S602: Using the position of the lip key points corresponding to the initial closed state as the starting constraint, and based on each speech-driven feature, adjust the position of the lip key points in the target face region in the corresponding frame image to obtain the target lip key points corresponding to each frame image.

[0095] The initial constraint can be used to define the first lip state of a new lip shape sequence, so that the target person's lip movement starts from a closed state. The target lip key points can be lip key points adjusted according to speech-driven features, and can include upper lip contour points, lower lip contour points, left and right corner points of the mouth, and lip center points.

[0096] Optionally, taking the position of the lip keypoints corresponding to the initial closed state as the starting constraint, and based on various speech-driven features, the position of the lip keypoints within the target face region in the corresponding frame image is adjusted to obtain the target lip keypoints for each frame image. This can be achieved by: reading the original lip keypoints within the target face region in the corresponding frame image; determining the degree of lip opening, the lateral displacement of the corners of the mouth, and the upper and lower lip shape variables corresponding to that frame image based on the speech-driven features; and using the position of the lip keypoints corresponding to the initial closed state as the first frame constraint, sequentially adjusting the positions of the original lip keypoints in each frame image to obtain the target lip keypoints. To reduce abrupt changes in lip keypoints between adjacent frames, the target lip keypoints in the current frame image can be continuously corrected based on the target lip keypoints of the previous frame image, ensuring that the displacement between adjacent target lip keypoints remains within a preset range.

[0097] Step S603: According to the temporal sequence of each frame image, generate corresponding new lip shapes based on each target lip key point to obtain the new lip shape sequence of the target person.

[0098] The new lip shape can be a lip image shape constructed based on the key points of the target lips. The new lip shape sequence can be a sequence composed of the new lip shapes corresponding to each frame image in the video time sequence.

[0099] Optionally, the method for generating corresponding new lip shapes based on key points of each target lip according to the temporal sequence of each frame image to obtain the new lip shape sequence of the target person can be as follows: determine the upper lip boundary, lower lip boundary, and corner of the mouth boundary according to the key points of the target lip corresponding to each frame image; generate the new lip shape corresponding to the frame image according to the area enclosed by each boundary; sort the generated new lip shapes according to the time marker of each frame image to obtain the new lip shape sequence of the target person.

[0100] For example, in three consecutive frames of images, when the speech-driven features sequentially represent closed mouth, gradually open mouth, and fully open mouth, closed lip shape, half-open lip shape, and open lip shape can be generated respectively, and a continuous new lip shape sequence can be formed according to the temporal order of the three frames of images.

[0101] As can be seen from the above, by adjusting the key points of the lips of the target person through speech-driven features and restricting the initial state of the new lip shape sequence with the initial closed state, the continuity between adjacent lip shapes can be improved, so that the lip movement of the target person can change frame by frame with the speech content.

[0102] In one embodiment, a specific implementation process for trimming silent segments from a lip-syncing video is described. Please refer to [reference needed]. Figure 7 , Figure 7 This is a flowchart illustrating a specific implementation process for trimming silent segments from a lip-syncing video, as provided in an embodiment of this application. Figure 7 As shown, the specific steps for removing silent segments from a lip-sync video to obtain a synchronized lip-sync video of the target person are as follows: Step S701: Determine the corresponding silent segment in the lip-sync video based on the time interval of the silent audio in the extended driving audio.

[0103] The time interval of the silent audio can be the range between the start time of the extended driving audio and the splicing time point between the silent audio and the lip-sync audio. The silent segment can be the video and audio content within this time interval of the lip-sync video.

[0104] Optionally, the method for determining the corresponding silent segment in the lip-sync video based on the time interval of the silent audio in the extended driving audio can be as follows: read the start time, end time of the silent audio, or the splicing time point of the silent audio and the lip-sync audio; map the time interval of the silent audio to the lip-sync video according to the time axis correspondence between the lip-sync video and the extended driving audio; and determine the content between the mapped start time position and end time position as the silent segment.

[0105] For example, when the duration of the silent audio is 1 second, the video and audio content in the lip-sync video from second 0 to second 1 can be defined as a silent segment.

[0106] Step S702: Simultaneously cut out the video and audio content of silent segments from the lip-sync video, and reset the start time of the retained video and audio content to zero to obtain the lip-sync video of the target person.

[0107] Synchronous cropping can be achieved by cropping corresponding content from the video and audio streams at the same time boundary, thus maintaining the temporal relationship between the cropped video and audio content. Resetting the start time to zero can be achieved by adjusting the timestamps of the first video frame and the first audio sample point after cropping to zero.

[0108] Optionally, the method to obtain the lip-sync video of the target person by simultaneously cropping the video and audio content of silent segments from the lip-sync video and resetting the start time of the retained video and audio content to zero can be as follows: using the end time of the silent segment as the cropping starting point, retaining the content of the video stream and audio stream of the lip-sync video that is after the cropping starting point; subtracting the time value corresponding to the cropping starting point from each retained timestamp, so that the retained video and audio content all start from time zero; and re-encapsulating the processed video and audio streams to obtain the lip-sync video of the target person.

[0109] As can be seen from the above, by using the time interval corresponding to the silent audio to determine the silent segment to be cut, and simultaneously cutting the video content and audio content, the pre-content used only for the initialization of the closed-mouth state can be removed, so that the final lip-sync video starts playing from the effective speech content of the lip-sync audio.

[0110] Figure 8 This is a structural block diagram of a multi-person video lip-syncing device provided in an embodiment of this application. The device is configured to execute the multi-person video lip-syncing method provided in the above embodiment, and has corresponding functional modules and beneficial effects for executing the method. Figure 8 As shown, the device specifically includes: The determination module 801 is configured to acquire the original video, lip-sync audio, and character description information of the target person, and determine the target face region of the target person in the original video. The extension module 802 is configured to mask the face area outside the target face area in the original video, generate a video that retains the target person, and add silent audio at the beginning of the lip-sync audio to obtain extended driving audio. The lip-sync module 803 is configured to generate a new lip shape sequence for the target person in the video based on the extended driver audio and the target face region, thereby obtaining a lip-sync video. The silent segments corresponding to the silent audio are then removed from the lip-sync video to obtain a lip-sync video of the target person.

[0111] As described above, by performing face detection on multi-person videos and combining it with the target person's descriptive information to determine the target face region, and by masking non-target face regions, the subsequent lip-sync generation process focuses on the target person. Simultaneously, by adding silent audio at the beginning of the original audio, a stable closed-mouth initialization state is provided for the target person's lip-sync generation, and the corresponding silent segment is removed after generation. Therefore, this method achieves the generation of a new lip-sync sequence for a specified target person and audio-visual timing synchronization in multi-person scenarios, avoiding mis-driving of other people and improving the accuracy of target person localization and lip-sync synchronization.

[0112] In one possible embodiment, the extension module 802 is specifically configured as follows: The silence duration is determined based on the frame rate of the original video and the preset number of silence frames. The silent audio is obtained by generating all-zero audio data corresponding to the silence duration based on the sampling parameters of the lip-sync audio. The silent audio is spliced ​​before the lip-sync audio, and the splicing time of the silent audio and the lip-sync audio is recorded to obtain the extended driven audio.

[0113] In one possible embodiment, the lip-sync module 803 is specifically configured as follows: According to the frame rate of the video of the target person, the extended driving audio is segmented to obtain silent audio segments and multiple speech audio segments, and each audio segment is matched with each frame of the video of the target person in time sequence. The silent audio segment is determined as the initial audio segment, and the initial closed state of the target person is determined based on the initial audio segment and the target face region in the corresponding frame image. Based on each speech audio segment, the initial closed state, and the target face region in the corresponding frame image, a new lip shape sequence of the target person is generated, and the new lip shape sequence is fused into the target person's preserved video to obtain the target person's lip shape video.

[0114] In one possible embodiment, the lip-sync module 803 is further configured as follows: Extract the speech-driven features corresponding to each speech audio segment; Using the position of the lip key points corresponding to the initial closed state as the starting constraint, and based on each speech-driven feature, the position of the lip key points in the target face region in the corresponding frame image is adjusted to obtain the target lip key points corresponding to each frame image. Based on the temporal sequence of each frame image, a new lip shape is generated corresponding to each target lip key point, resulting in a new lip shape sequence for the target person.

[0115] In one possible embodiment, the lip-sync module 803 is further configured as follows: Based on the time interval of the silent audio in the extended driving audio, determine the corresponding silent segment in the lip-sync video; The video and audio content of silent segments are simultaneously removed from the lip-sync video, and the start time of the remaining video and audio content is reset to zero to obtain the lip-sync video of the target person.

[0116] In one possible embodiment, the determining module 801 is specifically configured as follows: Target frame images for character localization are extracted from the original video, and multi-scale facial feature extraction is performed on the target frame images to obtain the face bounding box, facial key points and detection confidence for each character. Filter out face bounding boxes with a detection confidence level lower than a preset confidence threshold to obtain multiple face regions. Sort the multiple face regions according to their horizontal positions in the target frame image and configure a person index label for each face region. Based on the person description information, a person location command is generated, and multimodal semantic analysis is performed on the person location command and the target frame image to determine the target person index identifier that matches the person description information. Based on the target person's index identifier, the corresponding face bounding boxes and facial key points are extracted from multiple face regions to obtain the target face region of the target person.

[0117] In one possible embodiment, the extension module 802 is further configured as follows: Using the target face region as the initial tracking region, the face region corresponding to the target person in each frame of the original video is determined frame by frame, and the remaining face regions in the same frame are determined as the set of non-target face regions. For each frame of the original video, blur and mask each face region in the corresponding set of non-target face regions, while keeping the face region corresponding to the target person in the corresponding frame unchanged, to obtain the corresponding masked image. The target person's video is preserved by combining the masked images according to their temporal order in the original video.

[0118] Figure 9 This application provides a schematic diagram of the structure of a multi-person video lip-syncing device, as shown in the embodiment of the present application. Figure 9 As shown, the device includes a processor 901, a memory 902, an input device 903, and an output device 904.

[0119] The number of processors 901 can be one or more. Figure 9 Taking a processor 901 as an example; the processor 901, memory 902, input device 903, and output device 904 can be connected via a bus or other means. Figure 9 Taking a bus connection as an example, the memory 902, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the multi-person video lip-syncing method in this embodiment. The processor 901 executes various functional applications and data processing of the device by running the software programs, instructions, and modules stored in the memory 902, thereby realizing the aforementioned multi-person video lip-syncing method. The input device 903 can be configured to receive input digital or character information and generate key signal inputs related to user settings and function control of the device. The output device 904 may include a display screen or other display device.

[0120] This application also provides a non-volatile storage medium containing computer-executable instructions, which, when executed by a computer processor, are configured to perform a method for lip-syncing in a multi-person video as described in the above embodiments. The method includes: acquiring an original video, lip-syncing audio, and description information of the target person; determining the target face region of the target person in the original video; masking the face regions outside the target face region in the original video to generate a retained video of the target person; adding silent audio at the beginning of the lip-syncing audio to obtain extended driving audio; generating a new lip-shape sequence for the target person in the retained video based on the extended driving audio and the target face region to obtain a lip-syncing video; and removing silent segments corresponding to the silent audio from the lip-syncing video to obtain the lip-syncing video of the target person.

[0121] It is worth noting that in the above embodiments of the multi-person video lip-syncing device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not configured to limit the protection scope of the embodiments of this application.

[0122] In some possible implementations, various aspects of the methods provided in this application can also be implemented as a program product comprising program code that, when run on a computer device, is configured to cause the computer device to perform the steps of the methods according to the various exemplary embodiments of this application described above. For example, the computer device may execute the method for lip-syncing in multi-person video as described in the embodiments of this application. The program product may be implemented using a combination of one or more readable media.

Claims

1. A method for synchronizing lip movements in a multi-person video, characterized in that, include: Acquire the original video, lip-synced audio, and character description information of the target person, and determine the target face region of the target person in the original video; Masking is performed on the face regions outside the target face region in the original video to generate a video that retains the target person, and silent audio is added at the beginning of the lip-sync audio to obtain extended driving audio; Based on the extended driving audio and the target face region, a new lip shape sequence is generated for the target person in the video, resulting in a lip-sync video. The silent segments corresponding to the silent audio are then removed from the lip-sync video to obtain a lip-sync synchronized video of the target person.

2. The method for synchronizing lip movements in multi-person video according to claim 1, characterized in that, Adding silent audio at the beginning of the lip-sync audio to obtain extended drive audio includes: The silence duration is determined based on the frame rate of the original video and the preset number of silence frames; Generate all-zero audio data corresponding to the silence duration according to the sampling parameters of the lip-sync audio, and obtain the silence audio; The silent audio is spliced ​​before the lip-sync audio, and the splicing time of the silent audio and the lip-sync audio is recorded to obtain the extended driving audio.

3. The method for synchronizing lip movements in multi-person video according to claim 1, characterized in that, The step of generating a new lip shape sequence for the target person in the video based on the extended driving audio and the target face region to obtain a lip-sync video includes: According to the frame rate of the video of the target person, the extended driving audio is segmented to obtain silent audio segments and multiple speech audio segments, and each audio segment is matched with each frame of the video of the target person in time sequence. The silent audio segment is determined as the initial audio segment, and based on the initial audio segment and the target face region in the corresponding frame image, the initial closed-mouth state of the target person is determined. Based on each speech audio segment, the initial closed state, and the target face region in the corresponding frame image, a new lip shape sequence of the target person is generated, and the new lip shape sequence is fused into the retained video of the target person to obtain the lip shape video of the target person.

4. The method for synchronizing lip movements in multi-person video according to claim 3, characterized in that, The step of generating a new lip shape sequence for the target person based on each speech audio segment, the initial closed-mouth state, and the target face region in the corresponding frame image includes: Extract the speech-driven features corresponding to each speech audio segment; Using the lip key point position corresponding to the initial closed state as the starting constraint, and based on each voice-driven feature, the position of the lip key point in the target face region in the corresponding frame image is adjusted to obtain the target lip key point corresponding to each frame image. Based on the temporal sequence of each frame image, a new lip shape is generated corresponding to each target lip key point, thus obtaining a new lip shape sequence for the target person.

5. The method for synchronizing lip movements in multi-person video according to claim 1, characterized in that, The step of trimming the silent segments corresponding to the silent audio in the lip-sync video to obtain the lip-sync video of the target person includes: Based on the time interval of the silent audio in the extended driving audio, determine the corresponding silent segment in the lip-sync video; The video and audio content of the silent segment are simultaneously cut out from the lip-sync video, and the start time of the retained video and audio content is reset to zero to obtain the lip-sync video of the target person.

6. The method for synchronizing lip movements in multi-person video according to claim 1, characterized in that, Determining the target face region of the target person in the original video includes: Target frame images for character localization are extracted from the original video, and multi-scale facial feature extraction is performed on the target frame images to obtain the face bounding box, facial key points and detection confidence for each character. Filter out face bounding boxes whose detection confidence is lower than a preset confidence threshold to obtain multiple face regions. Sort the multiple face regions according to their horizontal positions in the target frame image and configure a person index identifier for each face region. Based on the person description information, a person location instruction is generated, and multimodal semantic analysis is performed on the person location instruction and the target frame image to determine the target person index identifier that matches the person description information. Based on the target person index identifier, the corresponding face bounding box and face key points are extracted from the multiple face regions to obtain the target face region of the target person.

7. The method for synchronizing lip movements in multi-person video according to claim 6, characterized in that, The step of masking facial regions outside the target facial region in the original video to generate a video retaining the target person includes: Using the target face region as the initial tracking region, the face region corresponding to the target person in each frame of the original video is determined frame by frame, and the remaining face regions in the same frame are determined as a set of non-target face regions. For each frame of the original video, blurring and masking are performed on each face region in the corresponding set of non-target face regions, while keeping the face region corresponding to the target person in the corresponding frame unchanged, to obtain the corresponding masked image. The target person's video is obtained by combining the masked images according to their temporal order in the original video.

8. A device for synchronizing lip movements in multi-person video, characterized in that, include: The determination module is configured to acquire the original video, lip-sync audio, and character description information of the target person, and determine the target face region of the target person in the original video; The extension module is configured to mask the face region outside the target face region in the original video, generate a video retaining the target person, and add silent audio at the beginning of the lip-sync audio to obtain extended driving audio; The lip-sync module is configured to generate a new lip shape sequence for the target person in the video based on the extended driving audio and the target face region, to obtain a lip-sync video, and to remove the silent segments in the lip-sync video that correspond to the silent audio, to obtain a lip-sync video of the target person.

9. A multi-person video lip-syncing device, characterized in that, The device includes: one or more processors; and a storage device configured to store one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the method for lip-syncing in multi-person video according to any one of claims 1-7.

10. A non-volatile storage medium for storing computer-executable instructions, characterized in that, The computer-executable instructions, when executed by a computer processor, are configured to perform the method for lip-syncing in a multi-person video as described in any one of claims 1-7.

Images