Audio and video separation shooting method, device and computer readable storage medium

By separating and recombining the image and audio features of video data in video shooting, the video shooting problem restricted by audio and video synchronization in the prior art is solved, and audio and video separation shooting is realized, and user experience is improved.

CN114095637BActive Publication Date: 2024-11-26NUBIA TECHNOLOGY CO LTD
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
CN202111405668.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-24
Publication Date
2024-11-26
Estimated Expiration
2041-11-24

AI Technical Summary

Technical Problem

In the prior art, the fun of video shooting is limited to audio and video content or audio and video special effects, and there is a lack of a high-interesting and unique video shooting solution that can achieve audio and video out of synchronization based on the shooting content, and the user experience is insufficient.

Method used

By acquiring image features and audio features in the video data, determining the segmentation point of the video, separating the video data into image data and audio data, and recombining it according to the characteristics and timing conditions to generate a video that is separated and recombined.

Benefits of technology

A humanized audio and video separation shooting solution is realized, allowing users to adaptively generate interesting special effects videos that are separated and reorganized during the video shooting process, greatly improving users' video shooting and production experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114095637B_ABST
    Figure CN114095637B_ABST
Patent Text Reader

Abstract

The present invention discloses an audio and video separation shooting method, device and computer-readable storage medium, wherein the method comprises: obtaining video data of a shot video, determining at least one first segmentation point related to the image and at least one second segmentation point related to the audio of the shot video according to a first image feature and / or a first audio feature in the video data; separating the video data into image data and audio data, and segmenting the image data into multiple image segments through the first segmentation point, and segmenting the audio data into multiple audio segments through the second segmentation point; combining all the multiple image segments and all the multiple audio segments through the combination point and the preset timing conditions. A humanized audio and video separation shooting scheme is realized, which can adaptively generate interesting special effects videos with separated and recombined audio and video, greatly improving the user's video shooting and production experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of mobile communications, and in particular to an audio and video separation shooting method, device and computer-readable storage medium. Background Art

[0002] In the prior art, with the continuous development of intelligent terminal devices and the gradual improvement of living standards, devices such as smart phones are well-known to everyone. In particular, using smart phones to take photos has become an indispensable part of people's daily lives. Whether it is a gathering with friends, a family gathering or an outing, a funny and fun video can make a person feel happy and add color to life.

[0003] There are many ways to shoot short videos that are popular and well-known in our daily lives, and they can also produce short videos with many effects.

[0004] However, the current video interest is limited to audio and video content or audio and video special effects, and there is no video shooting solution that can obtain highly interesting and unique video with asynchronous audio and video according to the shooting content, and the user experience needs to be improved. Summary of the invention

[0005] In order to solve the above technical defects in the prior art, the present invention proposes an audio and video separation shooting method, which includes:

[0006] Video data of a captured video is acquired, and at least one first segmentation point associated with an image and at least one second segmentation point associated with an audio of the captured video are determined according to a first image feature and / or a first audio feature in the video data.

[0007] The video data is separated into image data and audio data, and the image data is divided into a plurality of image segments by the first division point, and the audio data is divided into a plurality of audio segments by the second division point.

[0008] At least one second image feature is determined in at least one image segment of the plurality of image segments, and at least one second audio feature is determined in at least one audio segment of the plurality of audio segments.

[0009] A joining point of at least one image segment and one audio segment is determined according to the first number of segments of the multiple image segments, the second number of segments of the multiple audio segments, the second image feature, and the second audio feature, and all of the multiple image segments and all of the multiple audio segments are combined through the joining point and a preset timing condition.

[0010] Optionally, the acquiring video data of the captured video, and determining at least one first segmentation point related to the image and at least one second segmentation point related to the audio of the captured video according to a first image feature and / or a first audio feature in the video data, comprises:

[0011] The video data is obtained when shooting of the video is finished.

[0012] Image content and audio content in the video data are identified.

[0013] Optionally, the acquiring video data of the captured video, and determining at least one first segmentation point related to the image and at least one second segmentation point related to the audio of the captured video according to the first image feature and / or the first audio feature in the video data, further includes:

[0014] The first image feature is determined according to a transition picture of the image content, wherein the transition picture includes one or more of a character face switch, a character action switch, and a picture background switch.

[0015] The first audio feature is determined according to a transition segment of the audio content, wherein the transition segment includes one or more of a character speech switch, a character tone switch, and a background music switch.

[0016] Optionally, the acquiring video data of the captured video, and determining at least one first segmentation point related to the image and at least one second segmentation point related to the audio of the captured video according to the first image feature and / or the first audio feature in the video data, further includes:

[0017] The first segmentation point corresponding to at least one of the character face switching, at least one of the character action switching, or at least one of the screen background switching is selected.

[0018] A second segmentation point corresponding to at least one of the character speech switching, at least one of the character tone switching, or at least one of the background music switching is selected.

[0019] Optionally, the determining at least one second image feature in at least one image segment of the multiple image segments, and determining at least one second audio feature in at least one audio segment of the multiple audio segments, comprises:

[0020] At least one to-be-determined second image feature is determined in each of the multiple image segments, wherein the to-be-determined second image feature is a switching image frame of the character face switching, the character action switching, and the screen background switching.

[0021] At least one undetermined second audio feature is determined in each of the multiple audio segments, wherein the undetermined second audio feature is a switching audio frame of the character's speech switching, the character's tone switching, and the background music switching.

[0022] Optionally, the method is characterized in that the determining of at least one second image feature in at least one image segment of the multiple image segments, and the determining of at least one second audio feature in at least one audio segment of the multiple audio segments, further comprises:

[0023] Each pending second image feature and each pending second audio feature are matched.

[0024] The image frames and the audio frames that match the character face switching, the character action switching, and the picture background switching, and the character speech switching, the character tone switching, and the background music switching are obtained.

[0025] Optionally, determining a joining point of at least one image segment and one audio segment according to a first number of segments of the multiple image segments, a second number of segments of the multiple audio segments, the second image feature, and the second audio feature, and combining all of the multiple image segments and all of the multiple audio segments through the joining point and a preset timing condition includes:

[0026] The combination point is obtained according to each group of the image frame and the audio frame that match each other.

[0027] All of the multiple image segments and all of the multiple audio segments are recombined respectively through the combination points.

[0028] Optionally, the determining a joining point of at least one image segment and one audio segment according to the first number of segments of the multiple image segments, the second number of segments of the multiple audio segments, the second image feature, and the second audio feature, and combining all of the multiple image segments and all of the multiple audio segments through the joining point and a preset timing condition, further includes:

[0029] A timing chain corresponding to the total duration of the video data, the first number of segments, and the second number of segments is determined.

[0030] All of the multiple segments of images and all of the multiple segments of audio that have been reassembled are merged into a new captured video according to the time sequence chain.

[0031] The present invention also proposes an audio and video separation shooting device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the audio and video separation shooting method as described in any one of the above items when executed by the processor.

[0032] The present invention also proposes a computer-readable storage medium, on which an audio and video separation shooting program is stored. When the audio and video separation shooting program is executed by a processor, the steps of the audio and video separation shooting method as described in any one of the above items are implemented.

[0033] The audio and video separation shooting method, device and computer-readable storage medium of the present invention are implemented by acquiring video data of the shot video, determining at least one first segmentation point related to the image and at least one second segmentation point related to the audio of the shot video according to the first image feature and / or the first audio feature in the video data; separating the video data into image data and audio data, and segmenting the image data into multiple image segments through the first segmentation point, and segmenting the audio data into multiple audio segments through the second segmentation point; determining at least one second image feature in at least one image segment of the multiple image segments, and determining at least one second audio feature in at least one audio segment of the multiple audio segments; determining the junction point of at least one image segment and one audio segment according to the first number of the multiple image segments, the second number of the multiple audio segments, the second image feature and the second audio feature, and combining all the multiple image segments and all the multiple audio segments through the junction point and the preset timing conditions. A humanized audio and video separation shooting scheme is realized, so that users can adaptively generate interesting special effects videos with audio and video separation and recombination during video shooting, greatly improving the user's video shooting and production experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0035] Figure 1 It is a hardware structure diagram of a mobile terminal involved in the present invention;

[0036] Figure 2 is a communication network system architecture diagram provided by an embodiment of the present invention;

[0037] Figure 3 is a flow chart of a first embodiment of the audio and video separation shooting method of the present invention;

[0038] Figure 4 is a flow chart of a second embodiment of the audio and video separation shooting method of the present invention;

[0039] Figure 5 is a flow chart of a third embodiment of the audio and video separation shooting method of the present invention;

[0040] Figure 6 is a flow chart of a fourth embodiment of the audio and video separation shooting method of the present invention;

[0041] Figure 7 is a flow chart of a fifth embodiment of the audio and video separation shooting method of the present invention;

[0042] Figure 8 is a flow chart of a sixth embodiment of the audio and video separation shooting method of the present invention;

[0043] Fig. 9 is a flow chart of a seventh embodiment of the audio and video separation shooting method of the present invention;

[0044] Fig.10 It is a flow chart of an eighth embodiment of the audio and video separation shooting method of the present invention. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0046] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.

[0047] The terminal may be implemented in various forms. For example, the terminal described in the present invention may include mobile terminals such as mobile phones, tablet computers, laptop computers, PDAs, portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, etc., as well as fixed terminals such as digital TVs and desktop computers.

[0048] The following description will be made by taking a mobile terminal as an example, and those skilled in the art will appreciate that, in addition to components specifically used for mobile purposes, the configuration according to the embodiments of the present invention can also be applied to fixed-type terminals.

[0049] See also Figure 1 , which is a schematic diagram of the hardware structure of a mobile terminal for implementing various embodiments of the present invention, the mobile terminal 100 may include: RF (Radio Frequency, radio frequency) unit 101, WiFi module 102, audio output unit 103, A / V (audio / video) input unit 104, sensor 105, display unit 106, user input unit 107, interface unit 108, memory 109, processor 110, and power supply 111 and other components. Those skilled in the art can understand that Figure 1The structure of the mobile terminal shown in the figure does not constitute a limitation on the mobile terminal. The mobile terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0050] Combine the following Figure 1 The following is a detailed introduction to the various components of the mobile terminal:

[0051] The radio frequency unit 101 can be used for receiving and sending signals during information transmission or communication. Specifically, after receiving the downlink information of the base station, it is sent to the processor 110 for processing; in addition, the uplink data is sent to the base station. Generally, the radio frequency unit 101 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc. In addition, the radio frequency unit 101 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA2000 (Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution) and TDD-LTE (Time Division Duplexing-Long Term Evolution), etc.

[0052] WiFi is a short-range wireless transmission technology. Mobile terminals can help users send and receive emails, browse web pages, and access streaming media through WiFi module 102. It provides users with wireless broadband Internet access. Figure 1 The WiFi module 102 is shown, but it is understandable that it is not an essential component of the mobile terminal and can be omitted as required without changing the essence of the invention.

[0053] The audio output unit 103 can convert the audio data received by the RF unit 101 or the WiFi module 102 or stored in the memory 109 into an audio signal and output it as sound when the mobile terminal 100 is in a call signal reception mode, a talk mode, a recording mode, a voice recognition mode, a broadcast reception mode, etc. Moreover, the audio output unit 103 can also provide audio output related to a specific function performed by the mobile terminal 100 (for example, a call signal reception sound, a message reception sound, etc.). The audio output unit 103 may include a speaker, a buzzer, etc.

[0054] The A / V input unit 104 is used to receive audio or video signals. The A / V input unit 104 may include a graphics processor (GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The processed image frame can be displayed on the display unit 106. The image frame processed by the graphics processor 1041 can be stored in the memory 109 (or other storage medium) or sent via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 can receive sound (audio data) via the microphone 1042 in a telephone call mode, a recording mode, a voice recognition mode, and other operating modes, and can process such sound into audio data. The processed audio (voice) data can be converted into a format output that can be sent to a mobile communication base station via the radio frequency unit 101 in the case of a telephone call mode. The microphone 1042 can implement various types of noise elimination (or suppression) algorithms to eliminate (or suppress) noise or interference generated in the process of receiving and sending audio signals.

[0055] The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 1061 according to the brightness of the ambient light, and the proximity sensor can turn off the display panel 1061 and / or the backlight when the mobile terminal 100 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can also be configured on the mobile phone, such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0056] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0057] The user input unit 107 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile terminal. Specifically, the user input unit 107 may include a touch panel 1071 and other input devices 1072. The touch panel 1071, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1071 or near the touch panel 1071 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 110, and can receive and execute the command sent by the processor 110. In addition, the touch panel 1071 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1071, the user input unit 107 may also include other input devices 1072. Specifically, the other input devices 1072 may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, etc., which are not specifically limited here.

[0058] Furthermore, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the processor 110 to determine the type of the touch event. Then, the processor 110 provides a corresponding visual output on the display panel 1061 according to the type of the touch event. Figure 1 In the figure, the touch panel 1071 and the display panel 1061 are used as two independent components to implement the input and output functions of the mobile terminal. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to implement the input and output functions of the mobile terminal, which is not limited here.

[0059] The interface unit 108 serves as an interface through which at least one external device can be connected to the mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, etc. The interface unit 108 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more elements within the mobile terminal 100 or may be used to transmit data between the mobile terminal 100 and an external device.

[0060] The memory 109 can be used to store software programs and various data. The memory 109 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 109 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0061] The processor 110 is the control center of the mobile terminal. It uses various interfaces and lines to connect various parts of the entire mobile terminal. It executes various functions of the mobile terminal and processes data by running or executing software programs and / or modules stored in the memory 109, and calling data stored in the memory 109, so as to monitor the mobile terminal as a whole. The processor 110 may include one or more processing units; preferably, the processor 110 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 110.

[0062] The mobile terminal 100 may also include a power supply 111 (such as a battery) for supplying power to various components. Preferably, the power supply 111 may be logically connected to the processor 110 via a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system.

[0063] although Figure 1 Not shown, the mobile terminal 100 may further include a Bluetooth module, etc., which will not be described in detail here.

[0064] To facilitate understanding of the embodiments of the present invention, the communication network system on which the mobile terminal of the present invention is based is described below.

[0065] See also Figure 2 , Figure 2 A communication network system architecture diagram is provided for an embodiment of the present invention. The communication network system is an LTE system of universal mobile communication technology. The LTE system includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network) 202, an EPC (Evolved Packet Core) 203 and an operator's IP service 204, which are sequentially connected for communication.

[0066] Specifically, UE201 may be the above-mentioned terminal 100, which will not be described in detail here.

[0067] E-UTRAN 202 includes eNodeB 2021 and other eNodeBs 2022 , etc. Among them, eNodeB 2021 can be connected to other eNodeBs 2022 through a backhaul (eg, an X2 interface), and eNodeB 2021 is connected to EPC 203 , and eNodeB 2021 can provide UE 201 with access to EPC 203 .

[0068] EPC203 may include MME (Mobility Management Entity) 2031, HSS (Home Subscriber Server) 2032, other MMEs 2033, SGW (Serving Gate Way) 2034, PGW (PDN Gate Way) 2035 and PCRF (Policy and Charging Rules Function) 2036, etc. Among them, MME2031 is a control node that processes signaling between UE201 and EPC203, and provides bearer and connection management. HSS2032 is used to provide some registers to manage functions such as home location register (not shown in the figure), and save some user-specific information such as service features and data rates. All user data can be sent through SGW2034, PGW2035 can provide IP address allocation and other functions for UE 201, PCRF2036 is the policy and charging control policy decision point for service data flow and IP bearer resources, which selects and provides available policy and charging control decisions for the policy and charging execution function unit (not shown in the figure).

[0069] The IP service 204 may include the Internet, an intranet, an IMS (IP Multimedia Subsystem) or other IP services.

[0070] Although the above introduction is made using the LTE system as an example, those skilled in the art should know that the present invention is not only applicable to the LTE system, but also to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA and future new network systems, etc., which are not limited here.

[0071] Based on the above mobile terminal hardware structure and communication network system, various embodiments of the method of the present invention are proposed.

[0072] Embodiment 1

[0073] Figure 3 1 is a flow chart of a first embodiment of an audio and video separation shooting method of the present invention. A method for separating audio and video from shooting, the method comprising:

[0074] S1. Obtain video data of a captured video, and determine at least one first segmentation point related to the image and at least one second segmentation point related to the audio of the captured video according to a first image feature and / or a first audio feature in the video data.

[0075] S2. Separate the video data into image data and audio data, and divide the image data into multiple image segments through the first segmentation point, and divide the audio data into multiple audio segments through the second segmentation point.

[0076] S3. Determine at least one second image feature in at least one image segment of the multiple image segments, and determine at least one second audio feature in at least one audio segment of the multiple audio segments.

[0077] S4. Determine a joining point of at least one image segment and one audio segment according to the first number of segments of the multiple image segments, the second number of segments of the multiple audio segments, the second image feature, and the second audio feature, and combine all of the multiple image segments and all of the multiple audio segments through the joining point and a preset timing condition.

[0078] In this embodiment, considering that we generally work in a multi-threaded synchronous way, whether in daily life or in current film and television shooting, we can use expressions and actions to describe our behavior at the same time. We can make two independent actions, action and audio, at the same time. Therefore, the technical idea of ​​this embodiment is that when expressing communication, we can describe it in an all-round way. For example, for a common action analysis of introducing oneself, the action content is "waving to say hello", "smile", "bow", and the corresponding sound content is "Hello everyone", "My name is xxx", "Please give me more advice!" In this embodiment, in order to improve the fun of the video, the audio and picture are separated, and short videos are shot. That is, when shooting a funny video with audio and picture separation, the audio and video are separated independently during shooting. When generating the total video, you can choose to put the audio first or the video first, and finally capture the last frame of static video through the change of mouth shape to generate a new video, and then connect the newly generated video in front of or behind the video with only video but no audio. Thus, a funny short video that describes the audio first or the video first is generated. Similarly, as described in the above example, the above original video is divided into three time nodes. The first time node corresponds to "waving to say hello" and "hello everyone", the second time node corresponds to "smile" and "my name is xxx", and the third time node corresponds to "bowing" and "please give me more advice!". After processing, there can be six time nodes. In the first, second and third time nodes, they are "waving to say hello", "smile" and "bowing", and then in the fourth, fifth and sixth time nodes, they are "hello everyone", "my name is xxx" and "please give me more advice!", that is, when the "bowing" action ends, the last image frame is maintained while playing "hello everyone", "my name is xxx" and "please give me more advice!". For another example, it is also divided into six time nodes. In the first, second and third time nodes, they are respectively "Hello everyone", "My name is xxx", and "Please give me your advice!". Then, in the fourth, fifth and sixth time nodes, they are respectively "Waving to say hello", "Smile" and "Bow". Among them, the first image frame of "Waving to say hello" is taken and kept, and at the same time, "Hello everyone", "My name is xxx", and "Please give me your advice!" are played. After the playback is completed, the first image frame of "Waving to say hello" is used as the starting point, and the actions of "Waving to say hello", "Smile" and "Bow" are played.

[0079] It can be seen that in this embodiment, there will be a difference in length between the shot video and the ordinary video. The difference between the ordinary video and the audio and video separated video is that the ordinary video has parallel audio and video, while the audio and video separated audio and video are in a serial state. The effect of the shot is to first use language to describe the action to be performed, or the action is performed first and then described in words. It looks like a person cannot do two things at the same time and can only complete one thing at a time in serial, which is very interesting.

[0080] The beneficial effect of this embodiment is that, by acquiring the video data of the captured video, at least one first segmentation point related to the image and at least one second segmentation point related to the audio of the captured video are determined according to the first image feature and / or the first audio feature in the video data; the video data is separated into image data and audio data, and the image data is segmented into multiple image segments through the first segmentation point, and the audio data is segmented into multiple audio segments through the second segmentation point; at least one second image feature is determined in at least one image segment of the multiple image segments, and at least one second audio feature is determined in at least one audio segment of the multiple audio segments; according to the first segment number of the multiple image segments, the second segment number of the multiple audio segments, the second image feature and the second audio feature, the combination point of at least one image segment and one audio segment is determined, and all the multiple image segments and all the multiple audio segments are combined through the combination point and the preset timing conditions. A humanized audio and video separation shooting scheme is realized, so that users can adaptively generate interesting special effects videos with audio and video separation and recombination during video shooting, which greatly improves the user's video shooting and production experience.

[0081] Embodiment 2

[0082] Figure 4 : is a flowchart of a second embodiment of the audio and video separation shooting method of the present invention. Based on the above embodiment, the acquiring of video data of the shot video, and determining at least one first segmentation point related to the image and at least one second segmentation point related to the audio of the shot video according to the first image feature and / or the first audio feature in the video data, include:

[0083] S11. Acquire the video data when shooting the video is finished.

[0084] S12: Identify image content and audio content in the video data.

[0085] Optionally, in this embodiment, when shooting a video, the shooting mode of the current video is provided to the user, wherein the shooting mode includes person shooting and scene shooting, and the like.

[0086] Optionally, in this embodiment, the recognition basis of the image content and audio content in the video data is determined according to the shooting mode, that is, when it is determined that the shooting is of a person, the person in the video data is used as the target object to identify and analyze the image content and audio content.

[0087] The beneficial effect of this embodiment is that, by acquiring the video data at the end of video shooting and identifying the image content and audio content in the video data, a humanized audio and video separation shooting solution is realized, so that the user can adaptively generate an interesting special effect video with audio and video separation and recombination during the video shooting process, which greatly improves the user's video shooting and production experience.

[0088] Embodiment 3

[0089] Figure 5 : is a flowchart of a third embodiment of the audio and video separation shooting method of the present invention. Based on the above embodiment, the acquiring of video data of the shot video, determining at least one first segmentation point related to the image and at least one second segmentation point related to the audio of the shot video according to the first image feature and / or the first audio feature in the video data, further comprises:

[0090] S13. Determine the first image feature according to a transition picture of the image content, wherein the transition picture includes one or more of a character face switch, a character action switch, and a picture background switch.

[0091] S14. Determine the first audio feature according to a transition segment of the audio content, wherein the transition segment includes one or more of a character's speech switching, a character's tone switching, and a background music switching.

[0092] Optionally, in this embodiment, the switching of the character's face includes switching between the front and side of the character's face, switching between looking up and looking down, etc.

[0093] Optionally, in this embodiment, the character's speech switching includes a pause between two sentences of the character.

[0094] The beneficial effect of this embodiment is that the first image feature is determined by the transition picture of the image content, wherein the transition picture includes one or more of character face switching, character action switching, and picture background switching; the first audio feature is determined according to the transition paragraph of the audio content, wherein the transition paragraph includes one or more of character speech switching, character tone switching, and background music switching. A humanized audio and video separation shooting solution is realized, so that users can adaptively generate interesting special effects videos with audio and video separation and recombination during video shooting, which greatly improves the user's video shooting and production experience.

[0095] Embodiment 4

[0096] Figure 6 4 is a flowchart of a fourth embodiment of the audio and video separation shooting method of the present invention. Based on the above embodiment, the acquiring of video data of the shot video, determining at least one first segmentation point related to the image and at least one second segmentation point related to the audio of the shot video according to the first image feature and / or the first audio feature in the video data, further comprises:

[0097] S15, selecting the first segmentation point corresponding to at least one of the character face switching, at least one of the character action switching, or at least one of the picture background switching.

[0098] S16: Select a second segmentation point corresponding to at least one of the character's speech switching, at least one of the character's tone switching, or at least one of the background music switching.

[0099] Optionally, in this embodiment, each complete switch or each switch whose duration after switching exceeds a preset duration is used as a division basis for the corresponding division point.

[0100] The beneficial effect of this embodiment is that by selecting the first segmentation point corresponding to at least one of the character face switching, at least one of the character action switching, or at least one of the picture background switching; selecting the second segmentation point corresponding to at least one of the character speech switching, at least one of the character tone switching, or at least one of the background music switching, a humanized audio and video separation shooting solution is realized, so that the user can adaptively generate an interesting special effect video with audio and video separation and recombination during the video shooting process, which greatly improves the user's video shooting and production experience.

[0101] Embodiment 5

[0102] Figure 7 4 is a flowchart of a fifth embodiment of the audio and video separation shooting method of the present invention. Based on the above embodiment, determining at least one second image feature in at least one image of the multiple image segments, and determining at least one second audio feature in at least one audio of the multiple audio segments, includes:

[0103] S31. Determine at least one to-be-determined second image feature in each of the multiple image segments, wherein the to-be-determined second image feature is a switching image frame of the character face switching, the character action switching, and the screen background switching.

[0104] S32. Determine at least one pending second audio feature in each of the multiple audio segments, wherein the pending second audio feature is a switching audio frame of the character's speech switching, the character's tone switching, and the background music switching.

[0105] Optionally, in this embodiment, a switching image frame including a characteristic mouth shape or a characteristic action is determined.

[0106] Optionally, in this embodiment, a switching audio frame including when a voice just falls or when a voice just begins is determined.

[0107] The beneficial effect of this embodiment is that by determining at least one second image feature to be determined in each of the multiple image segments, wherein the second image feature to be determined is the switching image frame of the character face switching, the character action switching, and the screen background switching; and by determining at least one second audio feature to be determined in each of the multiple audio segments, wherein the second audio feature to be determined is the switching audio frame of the character speech switching, the character tone switching, and the background music switching. A humanized audio and video separation shooting solution is realized, so that the user can adaptively generate an interesting special effects video with audio and video separation and recombination during the video shooting process, which greatly improves the user's video shooting and production experience.

[0108] Embodiment 6

[0109] Figure 8 : is a flowchart of a sixth embodiment of the audio and video separation shooting method of the present invention. Based on the above embodiment, the determining of at least one second image feature in at least one image of the multiple image segments, and the determining of at least one second audio feature in at least one audio of the multiple audio segments, further include:

[0110] S33: Match each pending second image feature with each pending second audio feature.

[0111] S34, obtaining the image frames and the audio frames that match the character face switching, the character action switching, and the picture background switching, and the character speech switching, the character tone switching, and the background music switching.

[0112] Optionally, in this embodiment, the switching image frame of the characteristic mouth shape or characteristic action is matched with the switching audio frame when the voice just falls or when the voice begins.

[0113] Optionally, in this embodiment, the basis for matching is not whether the content and the picture are consistent (it can be understood that if they are consistent, it is a video with original audio and video synchronization), but the basis for matching is existence, that is, when any characteristic mouth shape exists, it can be matched with the switching audio frame when the voice just falls or when the voice starts.

[0114] The beneficial effect of this embodiment is that by matching each pending second image feature and each pending second audio feature, the image frames and audio frames that match each other in the switching of the character's face, the switching of the character's action, and the switching of the picture background, and the switching of the character's speech, the switching of the character's tone, and the switching of the background music are obtained. A humanized audio and video separation shooting solution is realized, so that the user can adaptively generate an interesting special effects video with audio and video separation and recombination during the video shooting process, which greatly improves the user's video shooting and production experience.

[0115] Embodiment 7

[0116] Fig. 9 : is a flowchart of a seventh embodiment of the audio and video separation shooting method of the present invention. Based on the above embodiment, the method determines the combination point of at least one image segment and one audio segment according to the first number of the multiple image segments, the second number of the multiple audio segments, the second image feature, and the second audio feature, and combines all the multiple image segments and all the multiple audio segments through the combination point and a preset timing condition, including:

[0117] S41, obtaining the combination point according to each group of the matching image frames and the audio frames.

[0118] S42. Recombining all of the multiple image segments and all of the multiple audio segments respectively through the combination points.

[0119] Optionally, in this embodiment, all of the multiple image segments and all of the multiple audio segments are recombined through the combination point, that is, multiple combinations are obtained, each of which includes all of the multiple image segments and all of the multiple audio segments.

[0120] The beneficial effect of this embodiment is that the combination point is obtained through each group of matching image frames and audio frames; all the multiple image segments and all the multiple audio segments are recombined through the combination point. A humanized audio and video separation shooting solution is realized, so that the user can adaptively generate an interesting special effect video with audio and video separation and recombination during the video shooting process, which greatly improves the user's video shooting and production experience.

[0121] Embodiment 8

[0122] Fig.10 The flowchart of the eighth embodiment of the audio and video separation shooting method of the present invention is based on the above embodiment, the method further comprises: determining the combination point of at least one image segment and one audio segment according to the first number of the multiple image segments, the second number of the multiple audio segments, the second image feature and the second audio feature, and combining all the multiple image segments and all the multiple audio segments through the combination point and the preset timing condition;

[0123] S43: Determine a timing chain corresponding to the total duration of the video data, the first number of segments, and the second number of segments.

[0124] S44, merging all the reassembled multiple image segments and all the multiple audio segments into a new shot video according to the time sequence chain.

[0125] Optionally, in this embodiment, the number of time nodes in the timing chain is determined according to the number of combination points, the number of the first segments and / or the number of the second segments.

[0126] Optionally, in this embodiment, each combination is merged according to a set timing chain to obtain a new shot video list for the user to select.

[0127] The beneficial effect of this embodiment is that by determining the timing chain corresponding to the total duration of the video data, the first number of segments, and the second number of segments; and merging all the reassembled multiple image segments and all the multiple audio segments into a new shot video according to the timing chain, a humanized audio and video separation shooting solution is realized, so that the user can adaptively generate an interesting special effect video with audio and video separation and reassembly during the video shooting process, which greatly improves the user's video shooting and production experience.

[0128] Embodiment 9

[0129] Based on the above embodiments, the present invention also proposes an audio and video separation shooting device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the audio and video separation shooting method as described in any one of the above items when executed by the processor.

[0130] It should be noted that the above-mentioned device embodiment and method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, and the technical features in the method embodiment are correspondingly applicable in the device embodiment, which will not be repeated here.

[0131] Embodiment 10

[0132] Based on the above embodiments, the present invention also proposes a computer-readable storage medium, which stores an audio and video separation shooting program. When the audio and video separation shooting program is executed by a processor, the steps of the audio and video separation shooting method as described in any one of the above items are implemented.

[0133] It should be noted that the above-mentioned medium embodiment and method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, and the technical features in the method embodiment are correspondingly applicable in the medium embodiment, which will not be repeated here.

[0134] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0135] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0137] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for separating audio and video, characterized in that: The method comprises: Acquire video data of a captured video, and determine at least one first segmentation point related to an image and at least one second segmentation point related to an audio of the captured video according to a first image feature and / or a first audio feature in the video data; Separating the video data into image data and audio data, dividing the image data into multiple image segments through the first segmentation point, and dividing the audio data into multiple audio segments through the second segmentation point; Determining at least one second image feature in at least one of the plurality of image segments, and determining at least one second audio feature in at least one of the plurality of audio segments; Determine a junction point of at least one image segment and one audio segment according to a first number of segments of the plurality of image segments, a second number of segments of the plurality of audio segments, the second image feature, and the second audio feature, and combine all of the plurality of image segments and all of the plurality of audio segments through the junction point and a preset timing condition; The acquiring video data of the captured video, and determining at least one first segmentation point related to the image and at least one second segmentation point related to the audio of the captured video according to the first image feature and / or the first audio feature in the video data, comprises: Acquiring the video data when shooting the video ends; identifying image content and audio content in the video data; Determining the first image feature according to a transition picture of the image content, wherein the transition picture includes one or more of a character face switch, a character action switch, and a picture background switch; Determine the first audio feature according to a transition segment of the audio content, wherein the transition segment includes one or more of a character speech switch, a character tone switch, and a background music switch; Selecting the first segmentation point corresponding to at least one of the character face switching, at least one of the character action switching, or at least one of the picture background switching; Selecting a second segmentation point corresponding to at least one of the character speech switching, at least one of the character tone switching, or at least one of the background music switching; The determining of at least one second image feature in at least one image segment of the plurality of image segments, and the determining of at least one second audio feature in at least one audio segment of the plurality of audio segments, comprises: Determining at least one to-be-determined second image feature in each of the plurality of image segments, wherein the to-be-determined second image feature is a switching image frame of the character face switching, the character action switching, and the screen background switching; Determine at least one undetermined second audio feature in each of the multiple audio segments, wherein the undetermined second audio feature is a switching audio frame of the character's speech switching, the character's tone switching, and the background music switching; matching each pending second image feature with each pending second audio feature; Acquire the image frames and the audio frames that match the switching of the character's face, the switching of the character's action, and the switching of the picture background, and the switching of the character's speech, the switching of the character's tone, and the switching of the background music; The step of determining a junction point of at least one image segment and one audio segment according to the first number of segments of the multiple image segments, the second number of segments of the multiple audio segments, the second image feature, and the second audio feature, and combining all of the multiple image segments and all of the multiple audio segments through the junction point and a preset timing condition includes: Obtaining the combination point according to each set of matched image frames and audio frames; Recombining all of the multiple image segments and all of the multiple audio segments respectively through the combination points; Determine a timing chain corresponding to the total duration of the video data, the first number of segments, and the second number of segments; Merging all the reassembled multiple image segments and all the multiple audio segments into a new shot video according to the time sequence chain; in, When filming, audio and video are separated independently; When recombining, select audio first or video first, so as to generate a new shot video that describes the audio first or the video first.

2. An audio and video separation shooting device, characterized in that: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the audio and video separation shooting method as claimed in claim 1 are implemented.

3. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an audio and video separation shooting program, and when the audio and video separation shooting program is executed by the processor, the steps of the audio and video separation shooting method according to claim 1 are implemented.

Citation Information

Patent Citations

  • Catalytic dehydrohalogenation process

    EP0002021A1

  • Method based on audio / video combination for detecting highlight events in football video

    CN101650722A

  • Multimedia file splicing method and apparatus

    CN105791938A

  • Video system avoiding non-synchronization of images and sound

    CN107613358A

  • Method applicable to synthesis of audio and video

    CN107750004A