Video processing method, video processing device, electronic equipment and medium

Abstract

The invention discloses a video processing method, a video processing device, electronic equipment and a medium, and belongs to the technical field of computers. The video processing method comprises the steps that N frames of images and N audios in a first video are determined, the N frames of images are images with action features in the first video, the N audios are audios with preset audio features in the first video, and N is a positive integer; determining a first output delay based on the N frames of images and the N audios; adjusting the output time of the image or the output time of the audio in the second video based on the first output delay; wherein the image and the audio in the adjusted second video are synchronized.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and specifically relates to a video processing method, a video processing device, an electronic device and a medium. Background Art

[0002] With the increasing popularity of electronic devices with shooting functions such as mobile phones and digital cameras, it is becoming more and more common to use electronic devices to record concerts.

[0003] At present, at concerts, there is usually a large screen that magnifies the scene of the concert. When the audience watching the concert is far away from the performers, the audience can see the performance of the performers clearly through the large screen. When using electronic equipment to record the concert, the display picture on the large screen and the sound of the concert can be recorded. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a video processing method, a video processing device, an electronic device and a medium, which can synchronously record pictures and sounds.

[0005] In a first aspect, an embodiment of the present application provides a video processing method, the method comprising:

[0006] Determine N frames of images and N audios in the first video, where the N frames of images are images with motion features in the first video, and the N audios are audios with preset audio features in the first video, where N is a positive integer;

[0007] Determine a first output delay based on N frames of images and N audios;

[0008] Based on the first output delay, the output time of the image or the output time of the audio in the second video is adjusted; wherein the image and the audio in the adjusted second video are synchronized.

[0009] In a second aspect, an embodiment of the present application provides a video processing device, which includes: a determination module and an adjustment module; wherein:

[0010] A determination module, used to determine N frames of images and N audios in the first video, where the N frames of images are images with action features in the first video, and the N audios are audios with preset audio features in the first video, where N is a positive integer;

[0011] A determination module, used to determine a first output delay based on N frames of images and N audios;

[0012] An adjustment module is used to adjust the output time of the image or the output time of the audio in the second video based on the first output delay; wherein the image and audio in the adjusted second video are synchronized.

[0013] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the video processing method described in the first aspect are implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the video processing method described in the first aspect are implemented.

[0015] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the video processing method as described in the first aspect.

[0016] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the video processing method as described in the first aspect.

[0017] In an embodiment of the present application, by determining N frames of images and N audios in a first video, the N frames of images are images with motion features in the first video, and the N audios are audios with preset audio features in the first video, where N is a positive integer; based on the N frames of images and the N audios, a first output delay is determined; based on the first output delay, the output time of the image or the output time of the audio in the second video is adjusted; wherein the image and audio in the adjusted second video are synchronized. In this way, by determining the first output delay based on N frames of images and N audios, the output delay between the image with motion features in the video and the audio with preset audio features can be determined, and then based on the first output delay, when the output time of the image or the output time of the audio in the second video is adjusted, the image with motion features in the second video can be matched with the audio with preset audio features, thereby synchronizing the picture and sound in the second video. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of a concert scene provided by an embodiment of the present application;

[0019] Figure 2 This is a flow diagram of a video processing method provided in an embodiment of the present application. Figure 1 ;

[0020] Figure 3 This is a flow diagram of a video processing method provided in an embodiment of the present application. Figure 2 ;

[0021] Figure 4 This is a flow diagram of a video processing method provided in an embodiment of the present application. Figure 3 ;

[0022] Figure 5 This is a flow diagram of a video processing method provided in an embodiment of the present application. Figure 4 ;

[0023] Figure 6 is a schematic diagram of a preview image provided in an embodiment of the present application;

[0024] Figure 7 This is a flow diagram of a video processing method provided in an embodiment of the present application. Figure 5 ;

[0025] Figure 8 This is a flow diagram of a video processing method provided in an embodiment of the present application. Figure 6 ;

[0026] Fig. 9 This is a schematic diagram of the structure of a video processing device provided in an embodiment of the present application. Figure 1 ;

[0027] Fig.10 This is a schematic diagram of the structure of a video processing device provided in an embodiment of the present application. Figure 2 ;

[0028] Fig.11 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0029] Fig.12 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0031] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0032] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0033] The video processing method, video processing device, electronic device and medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0034] The video processing method, video processing device, electronic device and medium provided in the embodiments of the present application can be applied to video recording scenarios, such as recording live videos of some large-scale stage activities.

[0035] In an application scenario, the video processing method provided in the embodiment of the present application can be applied to the scenario of recording a live concert video.

[0036] At a concert, there is usually a large screen that magnifies the scene of the concert, so that the audience can see the performance of the actors at the concert clearly through the large screen. Figure 1 As shown, at the concert site, while the actors are performing on the stage 101, the large screen 102 above the concert stage will magnify the actors on the stage, so that the audience can clearly see the performance of the actors on the stage 101 through the large screen 102. In order to facilitate the later viewing of the concert, the audience watching at the concert site can use electronic devices with recording functions such as mobile phones and digital cameras to record the video of the concert site. When the electronic device records the video of the concert site, it can simultaneously record the display screen on the large screen 102 and the sound of the concert site.

[0037] However, due to the display delay of the large screen at the concert, the display image of the large screen recorded by the electronic device may be out of sync with the recorded live sound, that is, the image and sound recorded by the electronic device are out of sync.

[0038] To this end, the embodiments of the present application provide a video processing method, a video processing device, an electronic device, and a medium, by determining N frames of images and N audios in a first video, wherein the N frames of images are images with motion features in the first video, and the N audios are audios with preset audio features in the first video; based on the N frames of images and the N audios, a first output delay is determined; based on the first output delay, the output time of the image or the output time of the audio in the second video is adjusted; wherein the image and audio in the adjusted second video are synchronized. In this way, by determining the first output delay based on the N frames of images and the N audios, the output delay between the image with motion features in the video and the audio with preset audio features can be determined, and then based on the first output delay, when the output time of the image in the second video or the output time of the audio in the second video is adjusted, the image with motion features in the second video can be matched with the audio with preset audio features, thereby synchronizing the picture and sound in the second video.

[0039] The execution subject of the video processing method provided in the embodiment of the present application may be a video processing device. Exemplarily, the video processing device may be an electronic device, or a functional component or functional entity in the electronic device. The video processing device provided in the embodiment of the present application will be described exemplarily below by taking the execution subject as an electronic device as an example.

[0040] Figure 2 is a flow chart of a video processing method provided in an embodiment of the present application, such as Figure 2 As shown, the video processing method provided in the embodiment of the present application may include the following steps 201 to 203.

[0041] Step 201: The electronic device determines N frames of images and N audios in a first video.

[0042] In some embodiments of the present application, the N frames of images are images with action features in the first video, the N audios are audios with preset audio features in the first video, and N is a positive integer.

[0043] In some embodiments of the present application, the first video is a video obtained by an electronic device capturing a target environment, the target environment includes a first display screen and a first object, the first display screen is used to display the first object captured by a camera, the first video includes M frames of images, each frame of image includes a first image area and a second image area, the first image area is an image area corresponding to the first display screen, and the second image area is an image area corresponding to the first object;

[0044] Among them, the above-mentioned N frames of images are images in which the first image area of ​​the above-mentioned M frames of images has motion features, M is a positive integer, and M is greater than or equal to N.

[0045] In some embodiments of the present application, the target environment may be a live environment of some large-scale stage activities, such as a live environment of a concert, and the first display screen is used to magnify and display the first object.

[0046] In some embodiments of the present application, the first object may be a person, an animal, or an object that can actively or passively perform an action.

[0047] In some embodiments of the present application, the above-mentioned characters include but are not limited to: men, women, and children.

[0048] In some embodiments of the present application, the above-mentioned animals include but are not limited to: livestock, wild animals, pets, birds, fish, insects, etc.

[0049] In some embodiments of the present application, the above-mentioned objects include but are not limited to: clothing accessories, daily necessities, office supplies, toys, labor tools, entertainment tools and other inanimate objects.

[0050] In some embodiments of the present application, the electronic device may perform a recognition operation on each of the M frames of images to determine a first image region and a second image region of each frame of image.

[0051] In some embodiments of the present application, the electronic device may perform action recognition on the first object displayed in an enlarged manner in the first image region of each of the M image frames to determine the N image frames from the M image frames.

[0052] In some embodiments of the present application, the first video also includes M audios, and the audio with the preset audio feature refers to the audio whose frequency is greater than or equal to the preset frequency among the M audios, that is, the N audios are the audios in the first video whose frequency is greater than or equal to the preset frequency.

[0053] Step 202: The electronic device determines a first output delay based on N frames of images and N audios.

[0054] In some embodiments of the present application, Figure 2 ,like Figure 3 As shown, the above step 202 can be implemented through the following steps 202a to 202d:

[0055] Step 202a: The electronic device determines the output time difference between the i-th frame image and the (i+1)-th frame image in N frames of images, and obtains K first output time differences.

[0056] In some embodiments of the present application, i∈[1, 3, ..., N), K is N / 2 and K is a positive integer.

[0057] In some embodiments of the present application, after the electronic device determines the N frames of images from the first video, it can sort the N frames of images according to the order of the output time of the images.

[0058] In some embodiments of the present application, the above-mentioned i-th frame image can be the image whose output time ranks i-th among the above-mentioned N frame images, and the above-mentioned i+1-th frame image can be the image whose output time ranks i+1th among the above-mentioned N frame images.

[0059] In some embodiments of the present application, the output time difference between the i-th frame image and the (i+1)-th frame image may be the difference between the output time of the (i+1)-th frame image and the output time of the i-th frame image.

[0060] For example, when N is 6, K is 3. If the output time of the first frame of the six frames is 10:24:23.05 on September 24, 2024, the output time of the second frame is 10:24:23.26 on September 24, 2024, the output time of the third frame is 10:24:23.47 on September 24, 2024, the output time of the fourth frame is 10:24:23.90 on September 24, 2024, the output time of the fifth frame is 10:25:24.32 on September 24, 2024, and the output time of the sixth frame is 10:25:24.88 on September 24, 2024, then the difference between the output time of the second frame and the output time of the first frame is 0.21 seconds, which is the first output time difference among the three first output time differences. The difference between the output time of the 4th frame image and the output time of the 3rd frame image is 0.43 seconds, which is the second output time difference among the three first output time differences. The difference between the output time of the 6th frame image and the output time of the 5th frame image is 0.56 seconds, which is the third output time difference among the three first output time differences.

[0061] Step 202b: The electronic device determines the output time difference between the ith audio and the (i+1)th audio among the N audios, and obtains K second output time differences.

[0062] In some embodiments of the present application, after the electronic device determines the N audios from the first video, it can sort the N audios according to the order of the output time of the audios.

[0063] In some embodiments of the present application, the above-mentioned i-th audio can be the audio whose output time ranks i-th among the above-mentioned N audios, and the above-mentioned i+1-th audio can be the audio whose output time ranks i+1th among the above-mentioned N audios.

[0064] In some embodiments of the present application, the output time difference between the i-th frame image and the (i+1)-th frame image may be the difference between the output time of the (i+1)-th frame image and the output time of the i-th frame image.

[0065] For example, when N is 6, K is 3. If the output time of the first audio among the 6 audios is 10:24:22.82 seconds on September 24, 2024, the output time of the second audio is 10:24:23.06 seconds on September 24, 2024, the output time of the third audio is 10:24:23.23 seconds on September 24, 2024, the output time of the fourth audio is 10:24:23.70 seconds on September 24, 2024, the output time of the fifth audio is 10:24:24.11 seconds on September 24, 2024, and the output time of the sixth audio is 10:25:24.68 seconds on September 24, 2024, then the difference between the output time of the second audio and the output time of the first audio, 0.24 seconds, is the first output time difference among the three second output time differences. The difference between the output time of the 4th audio and the output time of the 3rd audio is 0.47 seconds, which is the 2nd output time difference among the 3 second output time differences. The difference between the output time of the 6th audio and the output time of the 5th audio is 0.57 seconds, which is the 3rd output time difference among the 3 second output time differences.

[0066] Step 202c: The electronic device determines the output delay between the image corresponding to the third output time difference and the audio corresponding to the fourth output time difference, and obtains K groups of output delays.

[0067] In some embodiments of the present application, the third output time difference is any one of the K first output time differences, and the fourth output time difference is the output time difference with the smallest difference between the K second output time differences and the third output time difference.

[0068] For example, when K is 3, and the three first output time differences are 0.21 seconds, 0.43 seconds and 0.56 seconds respectively, and the three second output time differences are 0.24 seconds, 0.47 seconds and 0.57 seconds respectively, if the third output time difference is 0.21 seconds, the difference between 0.24 seconds and 0.21 seconds among the three second output time differences is the smallest, and 0.24 seconds is the above-mentioned fourth output time difference.

[0069] In some embodiments of the present application, Figure 3 ,like Figure 4 As shown, the above step 202c can be implemented by the following step 202c1 and the following step 202c2:

[0070] Step 202c1: The electronic device determines the output delay between the i-th frame image corresponding to the third output time difference and the i-th audio corresponding to the fourth output time difference as one of a group of output delays.

[0071] In some embodiments of the present application, the output delay between the i-th frame image and the i-th audio may be the difference between the output time of the i-th frame image and the output time of the i-th audio.

[0072] For example, when the third output time difference is 0.21 seconds and the fourth output time difference is 0.24 seconds, the output time of the first frame image corresponding to the third output time difference is 10:24:23.05 seconds on September 24, 2024, and the output time of the first audio corresponding to the fourth output time difference is 10:24:22.82 seconds on September 24, 2024. Then, the difference between the output time of the first frame image and the output time of the first audio, 0.23 seconds, is one of a group of output delays.

[0073] It can be understood that the action feature of the first image area in the i-th frame image matches the audio feature of the i-th audio.

[0074] Step 202c2: The electronic device determines the output delay between the i+1th frame image corresponding to the third output time difference and the i+1th audio corresponding to the fourth output time difference as another output delay in a set of output delays.

[0075] In some embodiments of the present application, the output delay between the i+1th frame image corresponding to the third output time difference and the i+1th audio corresponding to the fourth output time difference may be the difference between the output time of the i+1th frame image corresponding to the third output time difference and the output time of the i+1th audio corresponding to the fourth output time difference.

[0076] For example, when the third output time difference is 0.21 seconds and the fourth output time difference is 0.24 seconds, the output time of the second frame image corresponding to the third output time difference is 10:24:23.26 seconds on September 24, 2024, and the output time of the second audio corresponding to the fourth output time difference is 10:24:23.06 seconds on September 24, 2024. Then the difference between the output time of the second frame image and the output time of the second audio, 0.20 seconds, is another output delay in a set of output delays.

[0077] It can be understood that the motion feature of the first image area in the above i+1th frame image matches the audio feature of the above i+1th audio.

[0078] It should be noted that there is no order in which the electronic device performs the above step 202c1 and the above step 202c2. The electronic device may first perform the above step 202c1 and then perform the above step 202c2. Alternatively, the electronic device may first perform the above step 202c2 and then perform the above step 202c1. Alternatively, the electronic device may perform the above step 202c1 and the above step 202c2 simultaneously.

[0079] In this way, the electronic device determines the output delay between the i-th frame image corresponding to the third output time difference and the i-th audio corresponding to the fourth output time difference as one of the output delays in a group of output delays, and determines the output delay between the i+1-th frame image corresponding to the third output time difference and the i+1-th audio corresponding to the fourth output time difference as another output delay in a group of output delays, and then determines the first output delay based on K groups of output delays. It can adjust the output time of the image or audio in the second video according to the output delay between the image and audio whose action features match the audio features, thereby matching the action features of the image in the second video with the audio features of the audio, and then achieving synchronization of the display screen and the sound of the second video.

[0080] Step 202d: The electronic device determines a first output delay based on the K groups of output delays.

[0081] In some embodiments of the present application, Figure 3 ,like Figure 5 As shown, the above step 202d can be implemented by the following step 202d1 and the following step 202d2:

[0082] Step 202d1: The electronic device determines a maximum output delay and a minimum output delay from K groups of output delays.

[0083] In some embodiments of the present application, a group of output delays includes two output delays. The electronic device can sort K groups of output delays, a total of N output delays in order from small to large or from large to small, and determine the maximum output delay and the minimum output delay from the K groups of output delays.

[0084] For example, when N is 6 and K is 3, the three groups of output delays are 0.23 seconds and 0.20 seconds, 0.24 seconds and 0.20 seconds, and 0.21 seconds and 0.20 seconds respectively. Then, among the total 6 output delays of the three groups of output delays, the maximum output delay is 0.24 seconds and the minimum output delay is 0.20 seconds.

[0085] Step 202d2: The electronic device determines the output time difference between the first image and the second image in the M frames of images that meet the first condition as the first output delay.

[0086] In some embodiments of the present application, the first condition includes: the lip shape features of the first image area in one frame image among the M frame images match the lip shape features of the second image area in another frame image among the M frame images, and the output time difference between the one frame image and the other frame image is greater than or equal to the minimum output delay and less than or equal to the maximum output delay.

[0087] In some embodiments of the present application, the electronic device may perform facial feature extraction on the first object displayed in an enlarged manner in the first image area of ​​each of the M frame images to obtain lip shape features of the first image area of ​​each of the M frame images, and perform facial feature extraction on the first object in the second image area of ​​each of the M frame images to obtain lip shape features of the second image area of ​​each of the M frame images.

[0088] In some embodiments of the present application, the electronic device may perform lip shape feature matching on the lip shape features of the first image area of ​​each frame image in the above-mentioned M frame images, and on the lip shape features of the second image area of ​​each frame image in the above-mentioned M frame images, and determine the first image and the second image that meet the above-mentioned first condition from the above-mentioned M frame images.

[0089] In some embodiments of the present application, if the lip shape features of the first image area of ​​the first image match the lip shape features of the second image area of ​​the second image, then the output time difference between the second images may be the difference between the output time of the second image and the output time of the first image; if the lip shape features of the second image area of ​​the first image match the lip shape features of the first image area of ​​the second image, then the output time difference between the first image and the second image may be the difference between the output time of the first image and the output time of the second image.

[0090] It is understandable that, since the second image area of ​​each frame image in the first video is the area corresponding to the first object, there is no display delay in the second image area, and the second image area is synchronized with the audio. Therefore, when the electronic device adjusts the output time of the image or audio in the second video according to the output time difference between the first image and the second image that meet the first condition, it can synchronize the first image area of ​​each frame image in the second video with the audio, thereby realizing synchronization of the display screen in the second video with the audio.

[0091] In this way, the electronic device determines the output time difference between the first image and the second image that meet the first condition in the M frame images as the first output delay, and then adjusts the output time of the image or audio in the second video according to the output time difference between the first image and the second image, thereby achieving synchronization of the display picture in the second video with the audio.

[0092] In this way, the electronic device determines the output delay between the image corresponding to the third output time difference and the audio corresponding to the fourth output time difference through the electronic device to obtain K groups of output delays, and determines the first output delay based on the K groups of output delays, so that when the electronic device adjusts the output time of the image or video in the second video based on the first output delay, it can adjust the output time of the image or audio in the second video according to the output delay between the image and audio whose action features match the audio features, thereby matching the action features of the image in the second video with the audio features of the audio, and then realizing synchronization of the display screen of the second video with the sound.

[0093] In some embodiments of the present application, the above step 202d may also be implemented by the following step 202d3 and the following step 202d4:

[0094] Step 202d3, the electronic device calculates the average value or median value of all output delays in the K groups of output delays;

[0095] Step 202d4: The electronic device determines the average value or the median value as the first output delay.

[0096] In some embodiments of the present application, a group of output delays includes 2 output delays, and K groups of output delays include 2K output delays in total. The electronic device can calculate the average or median of the 2K output delays and use the average or median of the 2K output delays as the first output delay.

[0097] In this way, the electronic device calculates the average or median of all output delays in K groups of output delays, and determines the average or median as the first output delay, so that when the electronic device adjusts the output time of the image or video in the second video based on the first output delay, it can adjust the output time of the image or audio in the second video according to the output delay between the image and audio that matches the action features and the audio features, thereby matching the action features of the image in the second video with the audio features of the audio, and further achieving synchronization of the display screen of the second video with the sound.

[0098] Step 203: The electronic device adjusts the output time of the image or the output time of the audio in the second video based on the first output delay.

[0099] In some embodiments of the present application, the output time of the image in the adjusted second video matches the output time of the audio.

[0100] In some embodiments of the present application, the second video may be the first video and a video of the target environment captured by the electronic device after capturing the first video, or the second video may be a video of the target environment captured by the electronic device after capturing the first video. This embodiment does not make specific restrictions here.

[0101] It can be understood that the output time of the image in the adjusted second video matches the output time of the audio, which means that the output time of the third image in the second video and the third audio in the second video are the same. The third image and the third audio are images and audio whose motion features of the first image area of ​​the image in the second video match the audio features of the audio in the second video.

[0102] In some embodiments of the present application, the above step 203 may be implemented by the following step 203a or the following step 203b:

[0103] Step 203a: The electronic device reduces the output time of the image in the second video by the first output delay.

[0104] In some embodiments of the present application, the electronic device may reduce the output time of each frame of the second video by the first output delay.

[0105] Step 203b: The electronic device increases the output time of the audio in the second video by the first output delay.

[0106] In some embodiments of the present application, the electronic device may increase the output time of each audio in the second video by the first output delay.

[0107] In some embodiments of the present application, after the electronic device performs the above step 203a or 203b, the second video may be encoded using a video encoder. Exemplarily, when the electronic device encodes the second video using a video encoder, the image and audio with the same output time may be encoded as a pair of image and audio. In this way, the display delay of the large screen in the first image area in the second video image can be eliminated, thereby synchronizing the picture and sound in the second video.

[0108] In this way, the electronic device can eliminate the display delay of the large screen in the first image area of ​​the second video image by reducing the output time of the image in the second video by the first output delay, or increasing the output time of the audio in the second video by the first output delay, thereby synchronizing the picture and sound in the second video.

[0109] In the video processing method provided in the embodiment of the present application, by determining N frames of images and N audios in the first video, the N frames of images are images with action features in the first video, and the N audios are audios with preset audio features in the first video; based on the N frames of images and the N audios, a first output delay is determined; based on the first output delay, the output time of the image or the output time of the audio in the second video is adjusted; wherein the output time of the image in the adjusted second video matches the output time of the audio. In this way, by determining the first output delay based on the N frames of images and the N audios, the output delay between the image with action features in the video and the audio with preset audio features can be determined, and then based on the first output delay, when adjusting the output time of the image or the output time of the audio in the second video, the image with action features in the second video can be matched with the audio with preset audio features, thereby synchronizing the picture and sound in the second video.

[0110] In some embodiments of the present application, after the above step 204d2, the video processing method provided by the embodiment of the application may further include the following step 204 or the following step 205:

[0111] Step 204: The electronic device updates the second image region in the first image to the second image region in the second image.

[0112] In some embodiments of the present application, the electronic device may segment and extract the second image, determine the second image region of the second image from the second image, and segment and extract the first image, determine the second image region of the first image from the first image. Further, the electronic device may update the second image region in the first image to the second image region in the second image, and then merge the image region in the first image except the second image region and the second image region in the second image into a new first image.

[0113] Step 205: The electronic device updates the first image area in the second image to the first image area in the first image.

[0114] In some embodiments of the present application, the electronic device may segment and extract the second image, determine the first image region of the second image from the second image, and segment and extract the first image, determine the first image region of the first image from the first image. Further, the electronic device may update the first image region in the second image to the first image region in the second image, and then merge the image region in the second image except the first image region and the first image region in the first image into a new second image.

[0115] In this way, the electronic device can match the lip features of the two image areas in the image by updating the second image area in the first image to the second image area in the second image, or updating the first image area in the second image to the first image area in the first image, thereby synchronizing the display screens of the two image areas in the image.

[0116] In some embodiments of the present application, before the above step 201, the video processing method provided in the embodiment of the present application may further include the following step 206 and the following step 207. The above step 201 may be implemented by the following step 201a and the following step 201b:

[0117] Step 206: The electronic device displays a preview image.

[0118] In some embodiments of the present application, the preview image includes at least one object identifier and at least one audio identifier.

[0119] In some embodiments of the present application, the object identifier may be an image of the object indicated by the object identifier.

[0120] In some embodiments of the present application, the audio identifier may be the audio of the object indicated by the audio identifier.

[0121] For example, when the electronic device is a mobile phone, when the user uses the mobile phone to record a video of a concert, a preview image of the concert can be displayed on the mobile phone. Figure 6 As shown, the preview image 60 of the mobile phone includes a sub-area 601 and a sub-area 602. The sub-area 601 displays an image of the person 1 at the concert, and the sub-area 602 displays an image of the person 2 at the concert. When the person 1 is speaking, the mobile phone can collect the voice of the person 1 and display the audio identifier 603 of the person 1 on the preview image 60, for example Figure 6When person 2 is speaking, the mobile phone can collect the voice of person 2 and display the audio identifier 604 of person 2 on the preview image 60, for example Figure 6 The sound 2.

[0122] Step 207: The electronic device associates the first object identifier with the first audio identifier in response to the first input of the user.

[0123] In some embodiments of the present application, the first object identifier is an object identifier among the at least one object identifier, and the first audio identifier is an audio identifier among the at least one audio identifier.

[0124] In some embodiments of the present application, the above-mentioned first input may include any of the following: user's click input, sliding input, pressing input, voice input, gesture input, or other feasible input, which is not limited in the embodiments of the present application.

[0125] In some embodiments of the present application, the above-mentioned gesture input may include but is not limited to at least one of the following: click gesture, sliding gesture, drag gesture, pressure recognition gesture, long press gesture, area change gesture, double press gesture, double click gesture, specific gesture input or other possible gesture inputs. The specific gesture input form can be determined according to actual needs and is not limited to this in some embodiments.

[0126] In some embodiments of the present application, the above-mentioned click input can be a single-click input, a double-click input, or any number of click inputs, etc., and can also be a long-press input or a short-press input, which is not limited in some embodiments.

[0127] In some embodiments of the present application, the above-mentioned sliding input can be a sliding input in any direction, such as sliding up, sliding down, sliding to the left or sliding to the right, etc., which is not limited in some embodiments.

[0128] In some embodiments of the present application, the first input may include two consecutive first sub-inputs and second sub-inputs. The first sub-input is a user input of the first object identifier, the second sub-input is a user input of the first audio identifier, and the first sub-input and the second sub-input are used to indicate the association between the first object identifier and the first audio identifier.

[0129] In some embodiments of the present application, the electronic device may associate the first object identifier with the first audio identifier in response to the first sub-input and the second sub-input.

[0130] For example, when the first sub-input and the second sub-input are respectively a pressing input, as Figure 6As shown, the user can continuously press and input the image of person 1 and the audio identifier 603 of person 1 in the sub-area 601 on the preview image to associate the image of person 1 with the audio identifier 603 of person 1 .

[0131] Step 201a: The electronic device determines N frames of images corresponding to the first object from the first video based on the first object identifier.

[0132] In some embodiments of the present application, the first object identifier is an image of the first object, and the electronic device may perform feature extraction on the image of the first object to obtain features of the first object.

[0133] In some embodiments of the present application, the electronic device can perform feature extraction on each frame image in the above-mentioned first video, and determine N frame images that include features of the above-mentioned first object from the above-mentioned first video. The N frame images that include features of the above-mentioned first object in the above-mentioned first video are the N frame images corresponding to the above-mentioned first object.

[0134] Step 201b: The electronic device determines N audios corresponding to the first object from the first video based on the first audio identifier.

[0135] In some embodiments of the present application, the first object is an object indicated by the first object identifier and the first audio identifier.

[0136] In some embodiments of the present application, the first audio identifier is the audio of the first object, and the electronic device can perform voiceprint extraction on the audio of the first object to obtain the voiceprint of the first object.

[0137] In some embodiments of the present application, the electronic device can perform voiceprint extraction on each audio in the above-mentioned first video, and determine N audios including the voiceprint of the above-mentioned first object from the above-mentioned first video. The N audios including the voiceprint of the above-mentioned first object in the above-mentioned first video are the N audios corresponding to the above-mentioned first object.

[0138] In this way, the electronic device determines N frames of images corresponding to the first object from the first video based on the first object identifier, and determines N audios corresponding to the first object from the first video based on the first audio identifier, so that when the electronic device determines the first output delay for the N frames of images and the N audios, it can determine the output delay between the image with the action characteristics of the existing first object and the audio with the preset audio characteristics of the existing first object, and then based on the first output delay, when adjusting the output time of the image or the output time of the audio in the second video, it can match the image with the action characteristics of the first object in the second video with the audio with the preset audio characteristics of the first object, thereby synchronizing the picture of the first object and the sound of the first object in the second video.

[0139] Combine the following Figure 7 , taking the electronic device as a mobile phone, and the first video and the second video as live concert videos as an example, the video processing method provided in this application is described in detail. Figure 7 FIG. 1 is a flow chart of a video processing method provided in an embodiment of the present application. Figure 7 As shown, the method may include the following steps 701 to 705.

[0140] Step 701: The mobile phone receives the video input of the user.

[0141] Exemplarily, the video recording input may be a click input of a video recording mode button on a mobile phone camera by a user.

[0142] Step 702: The mobile phone records a first video in response to the video input.

[0143] Exemplarily, the first video recorded by the mobile phone includes the image sensor of the mobile phone collecting M frames of images at the concert site, and the speaker of the mobile phone collecting M audios at the concert site. Each of the M frames of images includes a first image area and a second image area, the first image area is the area corresponding to the large screen of the concert, and the second image area is the area corresponding to the characters at the concert site, that is, the area corresponding to the actors.

[0144] For example, the mobile phone may send the images collected in real time through the image sensor to the ISP, and the ISP may process the images collected in real time by the image sensor.

[0145] Step 703: The mobile phone calculates the audio and video delay range of the first video based on the character movements in the first image area of ​​the image in the first video and the audio features of the audio in the first video, and calculates the audio and video delay of the first video (i.e., the above-mentioned first output delay) based on the character lip shapes in the second image area of ​​the image in the first video and the character lip shapes in the first image area within the audio and video delay range.

[0146] Exemplarily, the mobile phone obtains N frames of images from the M frames of images of the captured first video, and obtains N audios from the M audios of the first video. The N frames of images are images in which the first image area of ​​the M frames of images has a human action feature, and the N videos are audios in the M audios whose audio frequency is greater than or equal to a preset frequency.

[0147] Exemplarily, the mobile phone may determine the audio and video delay range, that is, the range of the minimum output delay and the maximum output delay, based on N frames of images and N audios.

[0148] Exemplarily, the mobile phone can determine that the lip shape features of the first image area in one frame of M frames match the second image area in another frame of M frames, as well as two frames of images (i.e., the first image and the second image) whose output time difference between one frame of image and another frame of image is within the audio and video delay range, and then calculate the output timestamps of the two frames of image as the audio and video delay of the first video.

[0149] Step 704: The mobile phone uses the audio and video delay of the first video to modify the output time of the image in the second video or the output time of the image in the second video.

[0150] Exemplarily, the second video may be the first video and a video captured by the electronic device in a live concert environment after capturing the first video, or the second video may be a video captured by the electronic device in a live concert environment after capturing the first video. This embodiment does not make specific restrictions here.

[0151] Exemplarily, the mobile phone may reduce the output delay of each frame of image in the second video by the above-mentioned audio-visual delay, or increase the output delay of each audio in the second video by the above-mentioned audio-visual delay.

[0152] Step 705: The mobile phone encodes the image and audio with the same output time in the second video as a pair of image and audio.

[0153] For example, after the mobile phone modifies the output time of the image in the second video or the output time of the image in the second video, the encoder can be used to encode the second video. When encoding, the encoder can encode the audio and image with aligned output time to obtain a finished video with complete alignment of audio and video. In this way, the display delay of the large screen in the first image area in the second video image can be eliminated, thereby synchronizing the picture and sound in the second video.

[0154] The video processing method provided in the embodiment of the present application aligns the output time of the image and audio in the user's video by capturing and calculating the output delay of the image and audio of the video, thereby reducing or eliminating the problem of audio and video asynchrony in the recorded concert video and improving the user experience.

[0155] Combine the following Figure 8 , taking the electronic device as a mobile phone, the first video and the second video as videos of a concert scene, and there are multiple actors at the concert scene as an example, the video processing method provided by this application is described in detail. Figure 8 FIG. 1 is a flow chart of a video processing method provided in an embodiment of the present application. Figure 8 As shown, the method may include the following steps 801 to 808.

[0156] Step 801: The mobile phone displays a preview interface (ie, the preview image mentioned above), wherein the preview interface includes at least one character identifier and at least one audio identifier.

[0157] Exemplarily, the character identification may be an image of the character, and the audio identification may be the audio of the character.

[0158] For example, at the opening stage of a stage show, when the actors introduce themselves, the preview interface of the mobile phone will display the image of each actor at the concert and the audio of each actor at the concert.

[0159] Step 802: The mobile phone associates a first person identifier with a first audio identifier in response to a first input from a user.

[0160] Exemplarily, the first object identifier is a character identifier among the at least one character identifier, and the first audio identifier is an audio identifier among the at least one audio identifier.

[0161] For example, when an actor at a concert is speaking, the mobile phone can collect the actor's voice and display the actor's audio identifier on the preview interface. At the same time, the actor's speaking status will be displayed on the actor's image on the preview interface. At this time, the user can make a first input of the actor's image and the actor's audio representation to associate the actor's image with the actor's audio.

[0162] Step 803: The mobile phone receives the user's video input.

[0163] Step 804: The mobile phone records a first video in response to the video input.

[0164] It should be noted that the above steps 803 and 804 are similar to the above steps 701 and 702, and to avoid repetition, they are not described again in this embodiment.

[0165] Step 805: The mobile phone calculates the audio and video delay range of the first video based on the character movements in the first image area of ​​the third image in the first video and the audio features of the third audio in the first video, and calculates the audio and video delay of the first video based on the lip shape of the first character in the second image area of ​​the image in the first video and the lip shape of the first character in the first image area within the audio and video delay range.

[0166] Exemplarily, the third image is N frames of images in which the first character has character action features in the first image area among M frames of images, and the N videos are audios in which the audio frequency of the first character is greater than or equal to a preset frequency among M audios.

[0167] Exemplarily, the mobile phone may determine the audio and video delay range, that is, the range of the minimum output delay and the maximum output delay, based on N frames of images and N audios.

[0168] Exemplarily, the mobile phone can determine that the lip features of the first person in the first image area in one frame of M frames match the lip features of the first person in the second image area in another frame of M frames, and that the output time difference between one frame of image and another frame of image is within the audio and video delay range of two frames of image (i.e., the first image and the second image mentioned above), and then calculate the output timestamps of the two frames of image as the audio and video delay of the first video.

[0169] Step 806: The mobile phone updates the first image area of ​​one frame of the two images in which the lip features of the first person in the second image area match the lip features of the first person in the first image area, to the first image area of ​​the other image, or updates the second image area of ​​one frame of the image to the second image area of ​​the other image.

[0170] Exemplarily, the electronic device may segment and extract each frame of the first video, and determine the first image area and the second image area of ​​each frame of the image. Furthermore, the mobile phone may update the first image area of ​​one frame of the image to the first image area of ​​another frame of the image, and then fuse the image area of ​​the above-mentioned frame of the image except the first image area and the first image area of ​​the other frame of the image into a new frame of the image. The mobile phone may update the second image area of ​​another frame of the image to the second image area of ​​the above-mentioned frame of the image, and then fuse the image area of ​​the above-mentioned other frame of the image except the second image area and the second image area of ​​the above-mentioned frame of the image into a new frame of the image.

[0171] In this way, the mobile phone can achieve full scene alignment of real person-screen-sound in the image of the first video by updating the first image area of ​​one frame to the first image area of ​​another image, or updating the second image area of ​​one frame to the second image area of ​​another frame in two frames of images that match the lip features of the first person in the second image area with the lip features of the first person in the first image area, thereby improving the overall video viewing experience.

[0172] Step 807: The mobile phone uses the audio and video delay of the first video to modify the output time of the image in the second video or the output time of the image in the second video.

[0173] Step 808: The mobile phone encodes the image and audio with the same output time in the second video as a pair of image and audio.

[0174] It should be noted that the implementation process of the above step 807 and the above step 808 is similar to the above step 704 and the above step 705. To avoid repetition, this embodiment will not be repeated here.

[0175] In the video processing method provided in the embodiment of the present application, on the one hand, by displaying the character identification and audio identification of multiple actors on the stage in the preview interface, the object identification and audio identification of different actors can be associated, so as to more accurately identify the lip shapes and audio of different actors, and then the lip shapes and audio of different actors can be aligned, thereby increasing the audio and video synchronization accuracy of the video. On the other hand. By segmenting and extracting the image, the different image areas of the real person and the large screen in the image can be determined, and then the real person and the large screen can be segmented and aligned, which can not only align the large screen and sound in the video, but also align the output delay difference between the real person and the large screen, thereby achieving full synchronization of the entire scene.

[0176] It should be noted that the specific implementation process of the video processing method in the above steps can refer to the relevant description of the above embodiment. To avoid repetition, this embodiment will not be repeated here.

[0177] It should be noted that the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or any two or more can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not impose any restrictions on this.

[0178] The video processing method provided in the embodiment of the present application can be executed by a video processing device. In the embodiment of the present application, the video processing device provided in the embodiment of the present application is described by taking the video processing method executed by the video processing device as an example.

[0179] Fig. 9 is a structural diagram of a video processing device provided in an embodiment of the present application. The video processing device 900 includes: a determination module 901 and an adjustment module 902.

[0180] The determination module 901 is used to determine N frames of images and N audios in the first video, where the N frames of images are images with action features in the first video, and the N audios are audios with preset audio features in the first video, and N is a positive integer;

[0181] The determination module 901 is further configured to determine a first output delay based on the N frames of images and the N audios;

[0182] The adjustment module 902 is used to adjust the output time of the image or the output time of the audio in the second video based on the first output delay; wherein the image and the audio in the adjusted second video are synchronized.

[0183] In some embodiments of the present application, the determination module 901 is specifically configured to:

[0184] Determine the output time difference between the i-th frame image and the i+1-th frame image in N frames of images, and obtain K first output time differences, i∈[1, 3, ..., N), K is N / 2 and K is a positive integer;

[0185] Determine the output time difference between the i-th audio and the (i+1)-th audio among the N audios, and obtain K second output time differences;

[0186] Determine an output delay between an image corresponding to a third output time difference and an audio corresponding to a fourth output time difference, and obtain K groups of output delays, wherein the third output time difference is any one of the K first output time differences, and the fourth output time difference is an output time difference with the smallest difference between the K second output time differences and the third output time difference;

[0187] Based on the K groups of output delays, a first output delay is determined.

[0188] In some embodiments of the present application, the determination module 901 is specifically configured to:

[0189] Determine the output delay between the i-th frame image corresponding to the third output time difference and the i-th audio corresponding to the fourth output time difference as one of a group of output delays;

[0190] The output delay between the i+1th frame image corresponding to the third output time difference and the i+1th audio corresponding to the fourth output time difference is determined as another output delay in a group of output delays.

[0191] In some embodiments of the present application, the first video is a video acquired from a target environment, the target environment includes a first display screen and a first object, the first display screen is used to display the first object acquired by a camera, the first video includes M frames of images, each frame of the image includes a first image area and a second image area, the first image area is an image area corresponding to the first display screen, and the second image area is an image area corresponding to the first object;

[0192] The N-frame image is an image in which the first image region of the M-frame image has motion features, M is a positive integer, and M is greater than or equal to N.

[0193] In some embodiments of the present application, the determination module 901 is specifically configured to:

[0194] Determine the maximum output delay and the minimum output delay from the K groups of output delays;

[0195] Determine the output time difference between the first image and the second image satisfying the first condition in the M frames of images as the first output delay;

[0196] The first condition includes: the lip shape features of the first image area in one frame image match the lip shape features of the second image area in another frame image, and the output time difference between one frame image and the other frame image is greater than or equal to the minimum output delay and less than or equal to the maximum output delay.

[0197] In some embodiments of the present application, Fig. 9 ,like Fig.10 As shown, the device 900 also includes:

[0198] The updating module 903 is configured to update the second image region in the first image to the second image region in the second image; or update the first image region in the second image to the first image region in the first image.

[0199] In some embodiments of the present application, the determination module 901 is specifically configured to:

[0200] Calculate the average or median value of all output delays in the K groups of output delays;

[0201] The average value or the median value is determined as the first output delay.

[0202] In some embodiments of the present application, the apparatus 900 further includes:

[0203] A display module 904 is used to display a preview image, the preview image includes at least one object identifier and at least one audio identifier; an association module is used to associate a first object identifier and a first audio identifier in response to a first input of a user, the first object identifier is an object identifier in the at least one object identifier, and the first audio identifier is an audio identifier in the at least one audio identifier;

[0204] The determination module 901 is specifically used to: determine N frames of images corresponding to the first object from the first video based on the first object identifier;

[0205] Based on the first audio identifier, determine N audios corresponding to the first object from the first video;

[0206] The first object is an object indicated by the first object identifier and the first audio identifier.

[0207] In some embodiments of the present application, the adjustment module 902 is specifically used to:

[0208] reducing the output time of the image in the second video by the first output delay; or,

[0209] Increase the output time of the audio in the second video by the first output delay.

[0210] In the video processing device provided by the present application, by determining N frames of images and N audios in the first video, the N frames of images are images with action features in the first video, and the N audios are audios with preset audio features in the first video; based on the N frames of images and the N audios, a first output delay is determined; based on the first output delay, the output time of the image or the output time of the audio in the second video is adjusted; wherein the image and audio in the adjusted second video are synchronized. In this way, by determining the first output delay based on the N frames of images and the N audios, the output delay between the image with action features in the video and the audio with preset audio features can be determined, and then based on the first output delay, when adjusting the output time of the image or the output time of the audio in the second video, the image with action features in the second video can be matched with the audio with preset audio features, thereby synchronizing the picture and sound in the second video.

[0211] The video processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or a device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device, an augmented reality / virtual reality device, a robot, a wearable device, a super mobile personal computer, a netbook or a personal digital assistant, etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0212] The video processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0213] The video processing device provided in the embodiment of the present application can implement each process implemented by each embodiment of the above-mentioned video processing method. To avoid repetition, it will not be described again here.

[0214] Alternatively, if Fig.11 As shown, an embodiment of the present application further provides an electronic device 1100, including a processor 1101 and a memory 1102, wherein the memory 1102 stores programs or instructions that can be executed on the processor 1101, and when the program or instructions are executed by the processor 1101, the various steps of the above-mentioned video processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0215] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0216] Fig.12 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0217] The electronic device 1200 includes but is not limited to: a radio frequency unit 1201, a network module 1202, an audio output unit 1203, an input unit 1204, a sensor 1205, a display unit 1206, a user input unit 1207, an interface unit 1208, a memory 1209, and a processor 1210 and other components.

[0218] Those skilled in the art will appreciate that the electronic device 1200 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1210 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0219] Among them, the processor 1210 is used to determine N frames of images and N audios in the first video, where the N frames of images are images with motion features in the first video, and the N audios are audios with preset audio features in the first video, and N is a positive integer; based on the N frames of images and the N audios, determine the first output delay; based on the first output delay, adjust the output time of the image or the output time of the audio in the second video; wherein the image and audio in the adjusted second video are synchronized.

[0220] In some embodiments of the present application, the processor 1210 is specifically used to determine the output time difference between the i-th frame image and the i+1-th frame image in N frames of images, and obtain K first output time differences, i∈[1, 3, ..., N), K is N / 2 and K is a positive integer; determine the output time difference between the i-th audio and the i+1-th audio in N audios, and obtain K second output time differences; determine the output delay between the image corresponding to the third output time difference and the audio corresponding to the fourth output time difference, and obtain K groups of output delays, the third output time difference is any one of the K first output time differences, and the fourth output time difference is the output time difference among the K second output time differences with the smallest difference from the third output time difference; based on the K groups of output delays, determine the first output delay.

[0221] In some embodiments of the present application, the processor 1210 is specifically used to determine the output delay between the i-th frame image corresponding to the third output time difference and the i-th audio corresponding to the fourth output time difference as one output delay in a group of output delays; and determine the output delay between the i+1-th frame image corresponding to the third output time difference and the i+1-th audio corresponding to the fourth output time difference as another output delay in a group of output delays.

[0222] In some embodiments of the present application, the first video is a video obtained by capturing a target environment, the target environment includes a first display screen and a first object, the first display screen is used to display the first object captured by a camera, the first video includes M frames of images, each frame of image includes a first image area and a second image area, the first image area is an image area corresponding to the first display screen, and the second image area is an image area corresponding to the first object; wherein, the N frame of image is an image in which the first image area of ​​the M frame of image has motion features, M is a positive integer, and M is greater than or equal to N.

[0223] In some embodiments of the present application, the processor 1210 is specifically used to determine the maximum output delay and the minimum output delay from K groups of output delays; determine the output time difference between the first image and the second image that meets the first condition in M ​​frames of images as the first output delay; the first condition includes: the lip shape features of the first image area in one frame of image match the lip shape features of the second image area in another frame of image, and the output time difference between one frame of image and another frame of image is greater than or equal to the minimum output delay, and less than or equal to the maximum output delay.

[0224] In some embodiments of the present application, the processor 1210 is further used to update the second image area in the first image to the second image area in the second image; or, to update the first image area in the second image to the first image area in the first image.

[0225] In some embodiments of the present application, the processor 1210 is specifically configured to calculate an average value or a median value of all output delays in the K groups of output delays; and determine the average value or the median value as the first output delay.

[0226] In some embodiments of the present application, the display unit 1206 is used to display a preview image, which includes at least one object identifier and at least one audio identifier; the processor 1210 is used to associate a first object identifier and a first audio identifier in response to a first input from a user, the first object identifier being an object identifier among at least one object identifier, and the first audio identifier being an audio identifier among at least one audio identifier; the processor 1210 is specifically used to determine, from a first video, N frames of images corresponding to a first object based on the first object identifier; and to determine, from the first video, N audios corresponding to the first object based on the first audio identifier; wherein the first object is the object indicated by the first object identifier and the first audio identifier.

[0227] In some embodiments of the present application, the processor 1210 is specifically configured to reduce the output time of the image in the second video by the first output delay; or increase the output time of the audio in the second video by the first output delay.

[0228] In the electronic device provided by the present application, by determining N frames of images and N audios in a first video, the N frames of images are images with motion features in the first video, and the N audios are audios with preset audio features in the first video; based on the N frames of images and the N audios, a first output delay is determined; based on the first output delay, the output time of the image or the output time of the audio in the second video is adjusted; wherein the image and audio in the adjusted second video are synchronized. In this way, by determining the first output delay based on the N frames of images and the N audios, the output delay between the image with motion features in the video and the audio with preset audio features can be determined, and then based on the first output delay, when the output time of the image or the output time of the audio in the second video is adjusted, the image with motion features in the second video can be matched with the audio with preset audio features, thereby synchronizing the picture and sound in the second video.

[0229] It should be understood that in the embodiment of the present application, the input unit 1204 may include a graphics processor (Graphics Processing Unit, GPU) 12041 and a microphone 12042, and the graphics processor 12041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1206 may include a display panel 12061, and the display panel 12061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1207 includes a touch panel 12071 and at least one of other input devices 12072. The touch panel 12071 is also called a touch screen. The touch panel 12071 may include two parts: a touch detection device and a touch controller. Other input devices 12072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0230] The memory 1209 can be used to store software programs and various data. The memory 1209 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1209 may include a volatile memory or a non-volatile memory, or the memory 1209 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1209 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0231] The processor 1210 may include one or more processing units; optionally, the processor 1210 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1210.

[0232] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned video processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0233] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0234] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned video processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0235] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0236] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned video processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0237] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0238] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0239] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Determine N frames of images and N audios in a first video, where the N frames of images are images with action features in the first video, and the N audios are audios with preset audio features in the first video, where N is a positive integer; Determine a first output delay based on the N frames of images and the N audios; Based on the first output delay, the output time of the image or the output time of the audio in the second video is adjusted; wherein the image and the audio in the adjusted second video are synchronized.

2. The method according to claim 1, characterized in that The determining a first output delay based on the N frames of images and the N audios includes: Determine the output time difference between the i-th frame image and the i+1-th frame image in the N frames of images, and obtain K first output time differences, i∈[1, 3, ..., N), K is N / 2 and K is a positive integer; Determine the output time difference between the i-th audio and the (i+1)-th audio among the N audios, and obtain K second output time differences; Determine an output delay between an image corresponding to a third output time difference and an audio corresponding to a fourth output time difference, and obtain K groups of output delays, wherein the third output time difference is any one of the K first output time differences, and the fourth output time difference is an output time difference with the smallest difference between the K second output time differences and the third output time difference; Based on the K groups of output delays, the first output delay is determined.

3. The method according to claim 2, characterized in that The step of determining the output delay between the image corresponding to the third output time difference and the audio corresponding to the fourth output time difference to obtain K groups of output delays includes: Determine the output delay between the i-th frame image corresponding to the third output time difference and the i-th audio corresponding to the fourth output time difference as one of a group of output delays; The output delay between the i+1th frame image corresponding to the third output time difference and the i+1th audio corresponding to the fourth output time difference is determined as another output delay in the set of output delays.

4. The method according to claim 2 or 3, characterized in that: The first video is a video acquired from a target environment, the target environment includes a first display screen and a first object, the first display screen is used to display the first object acquired by a camera, the first video includes M frames of images, each frame of image includes a first image area and a second image area, the first image area is an image area corresponding to the first display screen, and the second image area is an image area corresponding to the first object; The N-frame image is an image in which a motion feature exists in the first image region of the M-frame image, M is a positive integer, and M is greater than or equal to N.

5. The method according to claim 4, characterized in that The determining the first output delay based on the K groups of output delays includes: Determining a maximum output delay and a minimum output delay from the K groups of output delays; Determine the output time difference between the first image and the second image satisfying the first condition in the M frames of images as the first output delay; The first condition includes: the lip shape features of the first image area in one frame image match the lip shape features of the second image area in another frame image, and the output time difference between the one frame image and the other frame image is greater than or equal to the minimum output delay and less than or equal to the maximum output delay.

6. The method according to claim 5, characterized in that The method further comprises: updating the second image region in the first image to the second image region in the second image; or, The first image region in the second image is updated to the first image region in the first image.

7. The method according to claim 4, characterized in that The determining the first output delay based on the K groups of output delays includes: Calculate the average or median value of all output delays in the K groups of output delays; The average value or the median value is determined as the first output delay.

8. The method according to claim 1, characterized in that The method further comprises: displaying a preview image, the preview image including at least one object identifier and at least one audio identifier; In response to a first input from a user, associating a first object identifier and a first audio identifier, wherein the first object identifier is an object identifier among the at least one object identifier, and the first audio identifier is an audio identifier among the at least one audio identifier; The determining of N frames of images and N audios in the first video includes: Based on the first object identifier, determining the N frames of images corresponding to the first object from the first video; Based on the first audio identifier, determining the N audios corresponding to the first object from the first video; The first object is the object indicated by the first object identifier and the first audio identifier.

9. The method according to claim 1, characterized in that: The adjusting the output time of the image or the output time of the audio in the second video based on the first output delay includes: reducing the output time of the image in the second video by the first output delay; or The output time of the audio in the second video is increased by the first output delay.

10. A video processing device, characterized in that: The device comprises: a determination module and an adjustment module; wherein, a determination module, configured to determine N frames of images and N audios in a first video, wherein the N frames of images are images with action features in the first video, and the N audios are audios with preset audio features in the first video, where N is a positive integer; The determining module is used to determine a first output delay based on the N frames of images and the N audios; An adjustment module is used to adjust the output time of the image or the output time of the audio in the second video based on the first output delay; wherein the image and audio in the adjusted second video are synchronized.

11. The device according to claim 10, characterized in that The determination module is specifically used for: Determine the output time difference between the i-th frame image and the i+1-th frame image in the N frames of images, and obtain K first output time differences, i∈[1, 3, ..., N), K is N / 2 and K is a positive integer; Determine an output time difference between the i-th audio and the (i+1)-th audio among the N audios, and obtain K second output time differences; Determine an output delay between an image corresponding to a third output time difference and an audio corresponding to a fourth output time difference, and obtain K groups of output delays, wherein the third output time difference is any one of the K first output time differences, and the fourth output time difference is an output time difference with the smallest difference between the K second output time differences and the third output time difference; Based on the K groups of output delays, the first output delay is determined.

12. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the video processing method according to any one of claims 1 to 9 are implemented.

13. A readable storage medium, characterized in that: The readable storage medium stores a program or an instruction, and when the program or the instruction is executed by a processor, the steps of the video processing method according to any one of claims 1 to 9 are implemented.

Images