Audio and video synchronization method, device, electronic device and storage medium

Abstract

The embodiments of the present invention propose a method, device, electronic device and storage medium for synchronizing audio and video, and relate to the technical field of network live broadcast. The method determines the synchronization time range corresponding to the face-changing image according to the timestamp of each frame of the face-changing image, obtains at least one target image and at least one target audio whose timestamp is within the synchronization time range, encodes the target image and the target audio, and obtains a face-changing video with synchronized audio and video. The present invention realizes dynamic adjustment of the synchronization time range according to the timestamp of each frame of the face-changing image, thereby ensuring that the delay error between the image and audio whose timestamp is within the synchronization time range and the face-changing image is extremely small, thereby ensuring that different anchors can obtain face-changing videos with synchronized audio and video in different network environments, thereby improving the live broadcast quality, user traffic and user experience.

Description

Technical Field

[0001] The present invention relates to the technical field of network live broadcasting, and in particular to an audio and video synchronization method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of Internet technology, live streaming has been accepted by more and more mainstream people. In order to attract more user traffic, online anchors can use face-changing technology to change their appearance and use the face-changing image for online live streaming. However, due to the delay in obtaining the face-changing image, the audio and video of the online anchor are out of sync during the live broadcast, which seriously affects the quality of the live broadcast. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a method, device, electronic device and storage medium for synchronizing audio and video, which can obtain face-changing videos with synchronized audio and video, and improve the live broadcast quality and user experience.

[0004] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0005] In a first aspect, the present invention provides a method for synchronizing audio and video, which is applied to an electronic device, and the method comprises:

[0006] Determine the synchronization time range corresponding to the face-changing image according to the timestamp of each frame of the face-changing image; the face-changing image represents the image obtained after the face-changing anchor is subjected to the face-changing operation according to the original captured image;

[0007] Acquire at least one target image and at least one target audio with a timestamp within the synchronization time range; the target image includes the face-swapped image;

[0008] Encoding is performed according to the target image and the target audio to obtain a face-changing video with synchronized audio and video.

[0009] In an optional implementation, the electronic device is communicatively connected to a cloud server, and before determining a synchronization time range corresponding to the face-swapped image according to a timestamp of each frame of the face-swapped image, the method further includes:

[0010] Sending the original captured image of the face-changing anchor to the cloud server, so that the cloud server performs anchor face-changing according to the original captured image to obtain the face-changing image of the face-changing anchor;

[0011] Receive the face-swapped image returned by the cloud server.

[0012] In an optional implementation, the acquiring of at least one target image and at least one target audio whose timestamp is within the synchronization time range includes:

[0013] When the electronic device records the face-swapped image and the audio of the face-swapped host, determining the face-swapped image as the target image;

[0014] In the audio of the face-changing host, the audio with a timestamp within the synchronization time range is determined as the target audio.

[0015] In an optional implementation, the acquiring of at least one target image and at least one target audio whose timestamp is within the synchronization time range includes:

[0016] When the electronic device records a face-swapped image, the audio of the face-swapped host, the image of the connected-microphone host, and the audio of the connected-microphone host, the image of the connected-microphone host with a timestamp within the synchronization time range and the face-swapped image are determined as the target image;

[0017] Among the audio of the face-changing host and the audio of the connected microphone host, the audio with a timestamp within the synchronization time range is determined as the target audio.

[0018] In an optional implementation, the acquiring of at least one target image and at least one target audio whose timestamp is within the synchronization time range includes:

[0019] When the electronic device records a face-swapped image, the audio of the face-swapped host, a game image, and a game audio, the game image with a timestamp within the synchronization time range and the face-swapped image are determined as the target image;

[0020] Among the audio of the face-changing anchor and the game audio, the audio with a timestamp within the synchronization time range is determined as the target audio.

[0021] In an optional implementation, the acquiring of at least one target image and at least one target audio whose timestamp is within the synchronization time range includes:

[0022] When the electronic device records a face-swapped image, the audio of the face-swapped host, the image of the connected-microphone host, the audio of the connected-microphone host, a game image, and a game audio, the image of the connected-microphone host with a timestamp within the synchronization time range, the game image with a timestamp within the synchronization time range, and the face-swapped image are determined as the target image;

[0023] Among the audio of the face-changing host, the audio of the connected microphone host, and the game audio, the audio with a timestamp within the synchronization time range is determined as the target audio.

[0024] In an optional implementation manner, encoding the target image and the target audio to obtain a face-changing video with synchronized audio and video includes:

[0025] When there are multiple target images and multiple target audios, the multiple target images are mixed to obtain an image to be played;

[0026] Mixing the plurality of target audios to obtain audio to be played;

[0027] Encoding is performed according to the image to be played and the audio to be played to obtain a face-changing video with synchronized audio and video.

[0028] In a second aspect, the present invention provides an audio-video synchronization device, which is applied to an electronic device, and the device comprises:

[0029] A processing module, used to determine a synchronization time range corresponding to the face-changing image according to a timestamp of each frame of the face-changing image; the face-changing image represents an image obtained after the face is changed according to the original shot image of the face-changing host;

[0030] An acquisition module, configured to acquire at least one target image and at least one target audio whose timestamp is within the synchronization time range; the target image includes the face-swapped image;

[0031] The synchronization module is used to encode the target image and the target audio to obtain a face-changing video with synchronized audio and video.

[0032] In a third aspect, the present invention provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the audio and video synchronization method described in any of the aforementioned implementations.

[0033] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the audio and video synchronization method as described in any one of the aforementioned embodiments is implemented.

[0034] Compared with the prior art, the audio and video synchronization method, device, electronic device and storage medium provided by the embodiments of the present invention are applied to the electronic device, determine the synchronization time range corresponding to the face-changing image according to the timestamp of each frame of the face-changing image, obtain at least one target image and at least one target audio whose timestamp is within the synchronization time range, encode the target image and the target audio, and obtain a face-changing video with synchronized audio and video.

[0035] The present invention realizes dynamic adjustment of the synchronization time range according to the timestamp of each frame of the face-changing image, thereby ensuring that the delay error between the image and audio whose timestamp is within the synchronization time range and the face-changing image is extremely small, thereby ensuring that different anchors can obtain face-changing videos with synchronized audio and video in different network environments, thereby improving the live broadcast quality, user traffic and user experience.

[0036] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0038] Figure 1 A schematic flow chart of a method for synchronizing audio and video provided by an embodiment of the present invention is shown.

[0039] Figure 2 Another schematic flow chart of the audio and video synchronization method provided by an embodiment of the present invention is shown.

[0040] Figure 3 Another schematic flow chart of the audio and video synchronization method provided by an embodiment of the present invention is shown.

[0041] Figure 4 A block diagram of an audio and video synchronization device provided by an embodiment of the present invention is shown.

[0042] Figure 5 A block diagram of an electronic device provided by an embodiment of the present invention is shown.

[0043] Icons: 100 - electronic device; 110 - memory; 120 - processor; 130 - communication module; 200 - cloud server; 300 - terminal device of the host connected to the microphone; 400 - user terminal; 500 - audio and video synchronization device; 501 - processing module; 502 - acquisition module; 503 - synchronization module. DETAILED DESCRIPTION

[0044] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0045] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0046] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0047] In live broadcast software, beautification is a common technical means, which can usually be achieved by running a beautification algorithm on the host's live broadcast equipment. Face swapping is an algorithm that requires extremely high computing power, so there will be a delay in obtaining the face swapped image based on the host's original shot image, which will cause the face swapped image to be out of sync with the audio of the microphone, seriously affecting the quality of the live broadcast.

[0048] The inventors have found that when the host previews the face-changing image, there is a delay of about 500 to 1000 milliseconds. In order to solve the problem of audio and video asynchrony, a common processing method is to add a buffer queue of fixed length to the audio, so that the buffer queue can cache audio with a fixed delay (for example, 800 milliseconds), which is equivalent to adding a fixed delay to the audio, thereby improving the audio and video asynchrony during the live broadcast as much as possible.

[0049] Since the length of the audio buffer queue is estimated and fixed, the delay in obtaining the face-changing image varies for different users and network environments, resulting in different audio and video synchronization effects. If the audio buffer is too large, the sound will lag behind the picture, and if the audio buffer is too small, the sound will be ahead of the picture, making it impossible to ensure the synchronization of the live audio.

[0050] Even for the same user, the unstable latency of the live broadcast network will also cause the latency of the face-changing link to fluctuate. The audio latency is fixed, which will cause the audio and video synchronization effect to fluctuate continuously. Sometimes the audio is ahead, and sometimes the audio is behind, but the jitter amplitude may not be too large and the user will not feel it obviously.

[0051] Based on this, the audio and video synchronization method and device provided by the embodiments of the present invention dynamically adjust the synchronization time range according to the timestamp of each frame of the face-changing image, thereby ensuring that the delay error between the image and audio with the timestamp within the synchronization time range and the face-changing image is extremely small, thereby ensuring that different anchors can obtain face-changing videos with synchronized audio and video in different network environments, thereby improving the live broadcast quality, user traffic and user experience.

[0052] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0053] Please refer to Figure 1 , Figure 1 A schematic flow chart of a method for synchronizing audio and video provided by an embodiment of the present invention is shown. The method is applied to an electronic device and comprises the following steps:

[0054] Step 10, determining the synchronization time range corresponding to the face-changing image according to the timestamp of each frame of the face-changing image; the face-changing image represents the image obtained after the face is changed according to the original shot image of the face-changing host.

[0055] In the embodiment of the present invention, a preset synchronization delay can be initialized according to the requirement for audio and video synchronization, and the synchronization time range corresponding to the face-changing image can be determined according to the timestamp of each frame of the face-changing image and the preset synchronization delay.

[0056] The timestamp of the face-swapped image is the timestamp of the corresponding original captured image, that is, the timestamp of the original captured image captured by the camera. The preset synchronization delay can be set according to the delay range in which humans can perceive audio asynchrony. According to experience, when the delay between audio and picture exceeds 200 milliseconds, it can be clearly perceived by people, resulting in the phenomenon of audio and picture asynchrony. For example, in order to ensure that the live broadcast process always has a good audio and picture synchronization effect, the preset synchronization delay can be initialized to 100 milliseconds.

[0057] As an implementation method, the difference between the timestamp of the face-swapped image and the preset synchronization delay is determined as the synchronization start time, the sum of the timestamp of the face-swapped image and the preset synchronization delay is determined as the synchronization end time, and the synchronization time range is determined according to the synchronization start time and the synchronization end time. The synchronization start time is less than the synchronization end time.

[0058] Specifically, assuming that the timestamp of the face-swapped image is 1000 and the preset synchronization delay is 100, the synchronization start time and the synchronization end time are 900 and 1100 respectively, and the synchronization time range corresponding to the face-swapped image is 900-1100.

[0059] Step 20, obtaining at least one target image and at least one target audio whose timestamp is within the synchronization time range; the target image includes a face-swapped image.

[0060] Step 30, encoding is performed according to the target image and the target audio to obtain a face-changing video with synchronized audio and video.

[0061] In an embodiment of the present invention, each audio or each frame of image is recorded with a timestamp, and the audio whose timestamp is within the synchronization time range of the face-changing image is determined as the target audio, and the image whose timestamp is within the synchronization time range of the face-changing image is determined as the target image. Since the time delay between the target audio and the target image meets the requirements of audio and video synchronization, the target image and the target video can be encoded to obtain a face-changing video with synchronized audio and video.

[0062] In summary, the audio and video synchronization method provided by the embodiment of the present invention determines the synchronization time range corresponding to the face-changing image according to the timestamp of each frame of the face-changing image, obtains at least one target image and at least one target audio whose timestamp is within the synchronization time range, and encodes according to the target image and the target audio to obtain a face-changing video with synchronized audio and video. The present invention realizes the dynamic adjustment of the synchronization time range according to the timestamp of each frame of the face-changing image, thereby ensuring that the time delay error between the image and audio whose timestamp is within the synchronization time range and the face-changing image is extremely small, thereby ensuring that different anchors can obtain face-changing videos with synchronized audio and video in different network environments, thereby improving the live broadcast quality, user traffic and user experience.

[0063] Optionally, the electronic device is connected to a cloud server for communication, and the cloud server can perform face-changing on the portrait in the original captured image by running a face-changing algorithm. Figure 1 Before the step 10, the method further comprises the following steps:

[0064] The original shot image of the face-changing anchor is sent to the cloud server, so that the cloud server can perform the anchor's face-changing according to the original shot image and obtain the face-changing image of the face-changing anchor; and the face-changing image returned by the cloud server is received.

[0065] In the embodiment of the present invention, since face-changing is an algorithm that requires high computing power, the performance of the anchor's live broadcast device (i.e., electronic device) is usually average and cannot smoothly run the face-changing algorithm to achieve the face-changing function. Therefore, the face-changing function can be deployed on a cloud server with powerful computing power.

[0066] Specifically, the original captured image containing the face-changing host's portrait is collected by the camera, and the original captured image is sent to the cloud server. The cloud server runs the face-changing algorithm to convert the original captured image of the face-changing host into a face-changing image, and sends the face-changing image of the face-changing host to the electronic device, so that the electronic device generates a preview screen based on the face-changing image for the host to view, and caches the received face-changing image so that the audio and video synchronization can be performed according to each frame of the face-changing image later.

[0067] Optionally, in practical applications, the audio and video appeal of the video can be improved by mixing audio and video. Figure 1 The sub-steps of step 30 may include:

[0068] When there are multiple target images and multiple target audios, the multiple target images are mixed to obtain the image to be played; the multiple target audios are mixed to obtain the audio to be played; encoding is performed according to the image to be played and the audio to be played to obtain a face-changing video with synchronized audio and image.

[0069] In an embodiment of the present invention, when there are multiple target images and one target audio, the multiple target images are mixed to obtain an image to be played, and the image to be played and the target audio are encoded to obtain a face-changing video with synchronized audio and video. When there is one target image and multiple target audios, the multiple target audios are mixed to obtain an audio to be played, and the target image and the audio to be played are encoded to obtain a face-changing video with synchronized audio and video.

[0070] Optionally, in actual applications, when the host who turns on the face-changing function only does a live broadcast, it is necessary to achieve audio and video synchronization for the face-changing image and the audio of the face-changing host. Figure 1 The sub-steps of step 20 may include:

[0071] When the electronic device records the face-swapped image and the audio of the face-swapped host, the face-swapped image is determined as the target image; in the audio of the face-swapped host, the audio with a timestamp within the synchronization time range is determined as the target audio.

[0072] As an implementation method, when the host turns on the face-changing function for live broadcast, a face-changing image queue and a face-changing host audio queue are created for the host. The face-changing image queue is used to store the face-changing image, and the face-changing host audio queue is used to store the face-changing host's audio. The original image of the face-changing host contains the timestamp when the camera captured the image. When the cloud server performs face-changing, it will not re-collect the timestamp and retain the timestamp of the original image.

[0073] When the electronic device receives the face-changing image, the difference between the current system time of the electronic device and the timestamp of the face-changing image is the delay caused by the host's face-changing, which is constantly changing. The face-changing images returned by the cloud server are saved in the face-changing image queue in sequence, and the audio of the face-changing host collected by the microphone is saved in the face-changing host audio queue in sequence.

[0074] Assume that an audio and video synchronization task is created to detect whether there is a face-changing image in the face-changing image queue. Since the face-changing image has a time delay, the first frame of the face-changing image is later than the audio of the face-changing host. Therefore, before the first frame of the face-changing image is received, at least one audio of the face-changing host has been saved in the face-changing host audio queue. Until it is detected that there is a face-changing image in the face-changing image queue, the face-changing images are obtained from the face-changing image queue in sequence, and the taken face-changing images are used as the target images.

[0075] Determine the synchronization time range based on the timestamp of the face-changing image, and check the timestamp of each audio starting from the head of the face-changing host audio queue. Determine the audio corresponding to the earliest timestamp in the synchronization time range as the synchronization starting point audio. Take the audio before the synchronization starting point audio out of the face-changing host audio queue and discard it. This may be the sound data collected by turning on the microphone first. At this time, the camera is not turned on and it is invalid audio.

[0076] Starting from the synchronization starting point audio, check the audio of the next timestamp one by one, and determine the audio with a timestamp within the synchronization time range as the target audio, which includes the synchronization starting point audio. By screening the synchronization time range corresponding to each frame of the face-changing image, the timestamps corresponding to the target audio and target image sent to the audio and video encoder for encoding are almost at the same time, which will ensure that the video obtained after encoding is a face-changing video with synchronized audio and video. When the final user decodes and plays the video, there will be no audio and video asynchronism.

[0077] Optionally, in actual applications, when a host who has turned on the face-changing function is connected to other hosts during a live broadcast, it is necessary to achieve audio and video synchronization for the face-changing image, the audio of the face-changing host, the image of the connected host, and the audio of the connected host. Figure 1 The sub-steps of step 20 may include:

[0078] When an electronic device records a face-swapped image, the audio of a face-swapped host, the image of a connected microphone host, and the audio of a connected microphone host, the image of the connected microphone host whose timestamp is within the synchronization time range and the face-swapped image are determined as target images; among the audio of the face-swapped host and the audio of the connected microphone host, the audio whose timestamp is within the synchronization time range is determined as the target audio.

[0079] As an implementation method, Figure 2 For example, the electronic device 100 is respectively connected to the cloud server 200, the terminal device 300 of the connected microphone anchor, and the user terminal 400, and the cloud server 200 is connected to the terminal device 300 of the connected microphone anchor. When the anchor turns on the face-changing function to live broadcast with other anchors, a face-changing image queue, a face-changing anchor audio queue, a connected microphone anchor image queue, and a connected microphone anchor audio queue are created accordingly. Among them, the connected microphone anchor image queue is used to store the image of the connected microphone anchor, and the connected microphone anchor is the anchor who is connected to the face-changing anchor. The connected microphone anchor audio queue is used to store the audio of the connected microphone anchor.

[0080] The voice of the face-changing host is collected through a microphone, and the audio of the face-changing host is saved in the face-changing host audio queue in the order of collection. The audio of the face-changing host carries the timestamp of the collected sound.

[0081] The original captured image containing the face-changing anchor portrait is collected by the camera, and the original captured image is encoded and sent to the cloud server 200 through the anchor network, so that the cloud server 200 decodes the original captured image obtained by pulling the stream, performs face-changing (such as face shape adjustment) after decoding to obtain the face-changing image, enhances, encodes and pushes the face-changing image to obtain a high-definition face-changing image, and sends the high-definition face-changing image to the electronic device 100 through the anchor network. In addition, the face-changing image is enhanced, scaled, encoded and pushed to obtain a low-definition face-changing image, and the low-definition face-changing image is sent to the terminal device 300 of the connected microphone anchor through the anchor network.

[0082] The electronic device 100 of the face-changing anchor decodes the high-definition face-changing image, the low-definition image of the connected microphone anchor, and the audio of the connected microphone anchor, and saves the decoded face-changing image, the image of the connected microphone anchor, and the audio of the connected microphone anchor to the face-changing image queue, the connected microphone anchor image queue, and the connected microphone anchor audio queue, respectively.

[0083] The decoded face-changing image and the image of the connected microphone host are rendered to obtain a preview screen of the connected microphone live broadcast, so that the face-changing host can view the face-changing effect through the electronic device 100. When the face-changing image queue is not empty, the face-changing images are taken out in sequence, the synchronization time range is determined according to the timestamp of the face-changing image, and the target image and target audio with timestamps within the synchronization time range are obtained from the image of the connected microphone host, the connected microphone host audio queue and the face-changing host audio queue. The mixed target image and the mixed target audio are encoded to obtain a face-changing video with synchronized audio and video, and finally sent to the user terminal 400 through the content distribution network for the user to watch.

[0084] Optionally, in actual applications, when a game host turns on the face-changing function for live broadcast, it is necessary to achieve audio and video synchronization for the face-changing image, the audio of the face-changing host, the game image and the game audio. Figure 1 The sub-steps of step 20 may include:

[0085] When an electronic device records face-changing images, audio of a face-changing host, game images and game audio, the game images with timestamps within the synchronization time range and the face-changing images are determined as target images; among the audio of the face-changing host and the game audio, the audio with timestamps within the synchronization time range is determined as the target audio.

[0086] As an implementation method, when a game anchor who turns on the face-changing function broadcasts a game live through screen capture, the anchor image is generally embedded in the lower left corner of the game screen, that is, the picture-in-picture mode. When the game anchor broadcasts live, a face-changing image queue, a face-changing anchor audio queue, a game image queue, and a game audio queue are created accordingly. Among them, the game image queue is used to store game images, and the game audio queue is used to store game sound effects.

[0087] The screen-captured game images and corresponding game sound effects are saved in the game image queue and game audio queue respectively. The face-swapped images obtained after face-swapped the original captured images of the face-swapped host are saved in the face-swapped image queue in sequence. When the face-swapped image queue is not empty, each frame of the face-swapped image is obtained in sequence, and the target image and target audio whose timestamps are within the synchronization time range are filtered out from the face-swapped host audio queue, the game image queue, and the game audio queue.

[0088] Optionally, in actual applications, when a game anchor who turns on the face-changing function connects with other anchors during a live broadcast, it is necessary to achieve audio and video synchronization for the face-changing image, the audio of the face-changing anchor, the image of the connected anchor, the audio of the connected anchor, the game image and the game audio. Figure 1 The sub-steps of step 20 may include:

[0089] When an electronic device records face-swapped images, audio of a face-swapped host, images of a connected microphone host, audio of a connected microphone host, game images, and game audio, images of the connected microphone host with timestamps within a synchronization time range, game images with timestamps within a synchronization time range, and the face-swapped images are determined as target images; among the audio of the face-swapped host, the audio of the connected microphone host, and the game audio, the audio with timestamps within the synchronization time range is determined as the target audio.

[0090] As an implementation method, Figure 3 For example, the electronic device 100 is respectively connected to the cloud server 200, the terminal device 300 of the connected microphone anchor, and the user terminal 400, and the cloud server 200 is connected to the terminal device 300 of the connected microphone anchor. When the game anchor turns on the face-changing function to live broadcast with other anchors, a face-changing image queue, a face-changing anchor audio queue, a connected microphone anchor image queue, a connected microphone anchor audio queue, a game image queue, and a game sound effect queue are created accordingly.

[0091] The game image and the game sound effects are saved in the game image queue and the game sound effect queue in turn, and the original shot image of the face-changing host captured by the camera is sent to the cloud server 200 through the host network, so that the cloud server 200 generates a face-changing image based on the original shot image and sends it to the electronic device 100. The electronic device 100 decodes the face-changing image and saves it in the face-changing image queue, and saves the received image and audio of the connected microphone host in the connected microphone host image queue and the connected microphone host audio queue respectively.

[0092] When the face-changing image queue is not empty, each frame of the face-changing image is obtained in sequence, and the target image and target audio whose timestamps are within the synchronization time range are screened out from the face-changing host audio queue, the game image queue, the game audio queue, the microphone host image queue, and the microphone host audio queue. The mixed target image and the mixed target audio are encoded to obtain a face-changing video with synchronized audio and video, and finally sent to the user terminal 400 through the content distribution network for the user to watch.

[0093] Based on the same inventive concept, the embodiment of the present invention further provides an audio-visual synchronization device, the basic principle and technical effects of which are the same as those of the above embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference may be made to the corresponding contents of the above embodiment.

[0094] Please refer to Figure 4 , Figure 4 A block diagram of an audio-video synchronization device 500 provided in an embodiment of the present invention is shown. The audio-video synchronization device 500 includes a processing module 501 , an acquisition module 502 and a synchronization module 503 .

[0095] The processing module 501 is used to determine the synchronization time range corresponding to the face-changing image according to the timestamp of each frame of the face-changing image; the face-changing image represents the image obtained after the face is changed according to the original shot image of the face-changing anchor.

[0096] The acquisition module 502 is used to acquire at least one target image and at least one target audio whose timestamp is within the synchronization time range; the target image includes a face-swapped image.

[0097] The synchronization module 503 is used to encode the target image and the target audio to obtain a face-changing video with synchronized audio and video.

[0098] In summary, the audio and video synchronization device provided by the embodiment of the present invention determines the synchronization time range corresponding to the face-changing image according to the timestamp of each frame of the face-changing image, obtains at least one target image and at least one target audio whose timestamp is within the synchronization time range, and encodes according to the target image and the target audio to obtain a face-changing video with synchronized audio and video. The present invention realizes the dynamic adjustment of the synchronization time range according to the timestamp of each frame of the face-changing image, thereby ensuring that the time delay error between the image and audio whose timestamp is within the synchronization time range and the face-changing image is extremely small, thereby ensuring that different anchors can obtain face-changing videos with synchronized audio and video in different network environments, thereby improving the live broadcast quality, user traffic and user experience.

[0099] Optionally, the electronic device is communicatively connected to the cloud server, and the processing module 501 is also used to send the original captured image of the face-changing anchor to the cloud server, so that the cloud server can perform the anchor's face-changing according to the original captured image to obtain the face-changing image of the face-changing anchor; and receive the face-changing image returned by the cloud server.

[0100] Optionally, the acquisition module 502 is specifically used to determine the face-swapped image as the target image when the electronic device records the face-swapped image and the audio of the face-swapped host; in the audio of the face-swapped host, determine the audio with a timestamp within the synchronization time range as the target audio.

[0101] Optionally, the acquisition module 502 is specifically used to determine the image of the connected microphone host with a timestamp within a synchronization time range and the face-swapped image as target images when the electronic device records a face-swapped image, the audio of the face-swapped host, the image of the connected microphone host and the audio of the connected microphone host; among the audio of the face-swapped host and the audio of the connected microphone host, the audio with a timestamp within the synchronization time range is determined as the target audio.

[0102] Optionally, the acquisition module 502 is specifically used to determine the game images with timestamps within a synchronization time range and the face-changing images as target images when the electronic device records face-changing images, audio of a face-changing host, game images and game audio; and among the audio of the face-changing host and the game audio, determine the audio with timestamps within the synchronization time range as the target audio.

[0103] Optionally, the acquisition module 502 is specifically used to determine the image of the connected microphone host with a timestamp within the synchronization time range, the game image with a timestamp within the synchronization time range, and the face-swapped image as target images when the electronic device records a face-swapped image, the audio of the face-swapped host, the image of the connected microphone host, the audio of the connected microphone host, the game image and the game audio; and determine the audio with a timestamp within the synchronization time range as the target audio among the audio of the face-swapped host, the audio of the connected microphone host and the game audio.

[0104] Optionally, the synchronization module 503 is specifically used to mix the multiple target images to obtain the image to be played; mix the multiple target audios to obtain the audio to be played; and encode the image to be played and the audio to be played to obtain a face-changing video with synchronized audio and video.

[0105] Please refer to Figure 5 , Figure 5 A block diagram of an electronic device 100 provided in an embodiment of the present invention. The electronic device 100 may be a personal computer, a laptop computer, or a server, etc. The electronic device 100 includes a memory 110, a processor 120, and a communication module 130. The memory 110, the processor 120, and the communication module 130 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0106] The memory 110 is used to store programs or data. The memory 110 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0107] The processor 120 is used to read / write data or programs stored in the memory 110 and execute corresponding functions. For example, when the computer program stored in the memory 110 is executed by the processor 120, the audio and video synchronization method disclosed in the above embodiments can be implemented.

[0108] The communication module 130 is used to establish a communication connection between the electronic device 100 and other communication terminals through a network, and to send and receive data through the network.

[0109] It should be understood that Figure 5 The structure shown is only a schematic diagram of the structure of the electronic device 100. The electronic device 100 may also include Figure 5 More or fewer components as shown, or with Figure 5 Different configurations shown. Figure 5 Each component shown in the figure can be implemented by hardware, software or a combination thereof.

[0110] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 120, the audio and video synchronization method disclosed in the above embodiments is implemented.

[0111] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0113] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0114] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for synchronizing audio and video, characterized in that: Applied to electronic equipment, the method comprises: Determine the synchronization time range corresponding to the face-changing image according to the timestamp of each frame of the face-changing image; the face-changing image represents the image obtained after the face-changing anchor is subjected to the face-changing operation according to the original captured image; Acquire at least one target image and at least one target audio with a timestamp within the synchronization time range; the target image includes the face-swapped image; Encoding is performed according to the target image and the target audio to obtain a face-changing video with synchronized audio and video.

2. The method for synchronizing audio and video according to claim 1, characterized in that: The electronic device is connected to the cloud server for communication. Before determining the synchronization time range corresponding to the face-swapped image according to the timestamp of each frame of the face-swapped image, the method further includes: Sending the original captured image of the face-changing anchor to the cloud server, so that the cloud server performs anchor face-changing according to the original captured image to obtain the face-changing image of the face-changing anchor; Receive the face-swapped image returned by the cloud server.

3. The method for synchronizing audio and video according to claim 1, characterized in that: The acquiring of at least one target image and at least one target audio with a timestamp within the synchronization time range comprises: When the electronic device records the face-swapped image and the audio of the face-swapped host, determining the face-swapped image as the target image; In the audio of the face-changing host, the audio with a timestamp within the synchronization time range is determined as the target audio.

4. The method for synchronizing audio and video according to claim 1, characterized in that: The acquiring of at least one target image and at least one target audio with a timestamp within the synchronization time range comprises: When the electronic device records a face-swapped image, the audio of the face-swapped host, the image of the connected-microphone host, and the audio of the connected-microphone host, the image of the connected-microphone host with a timestamp within the synchronization time range and the face-swapped image are determined as the target image; Among the audio of the face-changing host and the audio of the connected microphone host, the audio with a timestamp within the synchronization time range is determined as the target audio.

5. The method for synchronizing audio and video according to claim 1, characterized in that: The acquiring of at least one target image and at least one target audio with a timestamp within the synchronization time range comprises: When the electronic device records a face-swapped image, the audio of the face-swapped host, a game image, and a game audio, the game image with a timestamp within the synchronization time range and the face-swapped image are determined as the target image; Among the audio of the face-changing anchor and the game audio, the audio with a timestamp within the synchronization time range is determined as the target audio.

6. The method for synchronizing audio and video according to claim 1, characterized in that: The acquiring of at least one target image and at least one target audio with a timestamp within the synchronization time range comprises: When the electronic device records a face-swapped image, the audio of the face-swapped host, the image of the connected-microphone host, the audio of the connected-microphone host, a game image, and a game audio, the image of the connected-microphone host with a timestamp within the synchronization time range, the game image with a timestamp within the synchronization time range, and the face-swapped image are determined as the target image; Among the audio of the face-changing host, the audio of the connected microphone host, and the game audio, the audio with a timestamp within the synchronization time range is determined as the target audio.

7. The method for synchronizing audio and video according to claim 1, characterized in that: The encoding according to the target image and the target audio to obtain a face-changing video with synchronized audio and video includes: When there are multiple target images and multiple target audios, the multiple target images are mixed to obtain an image to be played; Mixing the plurality of target audios to obtain audio to be played; Encoding is performed according to the image to be played and the audio to be played to obtain a face-changing video with synchronized audio and video.

8. An audio and video synchronization device, characterized in that: Applied to electronic equipment, the device comprises: A processing module, used to determine a synchronization time range corresponding to the face-changing image according to a timestamp of each frame of the face-changing image; the face-changing image represents an image obtained after the face is changed according to the original shot image of the face-changing host; An acquisition module, configured to acquire at least one target image and at least one target audio whose timestamp is within the synchronization time range; the target image includes the face-swapped image; The synchronization module is used to encode the target image and the target audio to obtain a face-changing video with synchronized audio and video.

9. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the audio and video synchronization method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the audio and video synchronization method according to any one of claims 1 to 7 is implemented.

Images