Video processing method and device, computer equipment and storage medium

Abstract

The invention relates to a video processing method and device, computer equipment, a storage medium and a computer program product. The video processing method comprises the following steps: acquiring a to-be-processed video, and when original background music of the video belongs to a to-be-replaced type, determining at least one candidate replacement audio for the original background music; the candidate replacement audio is matched with the video and does not belong to the to-be-replaced type; in response to a to-be-replaced audio selected by the terminal in the at least one candidate replacement audio, determining an audio replacement time point of the to-be-replaced audio for the original background music based on the video content matching time point of the video and the audio matching time point of the original background music; and through the to-be-replaced audio, replacing the original background music in the video according to the audio replacement time point, and obtaining the video after background music replacement processing. By adopting the method, the accuracy of original background music replacement can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology, video has become an important carrier for people to obtain information and enjoy entertainment in their daily lives. The specific forms of video can include movies, television, short videos, etc. In addition to watching videos from the Internet, such as watching movies, television or short videos through online video platforms, users can also make their own videos. Specifically, users can make videos by shooting through terminals and upload them to online video platforms for sharing.

[0003] When users make their own videos, in order to improve the video effect, users often add music to the homemade videos, such as adding background music to create an atmosphere. However, the music selected by users may be at risk of illegal use. For example, if the selected music does not have copyright, it is often necessary to replace the music of the video to ensure the legality of the homemade video. However, it is currently difficult to accurately replace the music of a video. Summary of the invention

[0004] Based on this, it is necessary to provide a video processing method, device, computer equipment, computer-readable storage medium and computer program product that can accurately replace the background music of a video to ensure the legal use of the video in response to the above technical problems.

[0005] In a first aspect, the present application provides a video processing method. The method comprises:

[0006] Acquire a video to be processed, and when the original background music of the video belongs to the type to be replaced, determine at least one candidate replacement audio for the original background music; the candidate replacement audio is compatible with the video and does not belong to the type to be replaced;

[0007] In response to the terminal selecting the audio to be replaced from at least one candidate replacement audio, determining an audio replacement time point of the audio to be replaced for the original background music based on a video content matching time point of the video and an audio matching time point of the original background music;

[0008] The original background music in the video is replaced by the audio to be replaced according to the audio replacement time point to obtain the video after the background music is replaced.

[0009] In a second aspect, the present application further provides a video processing device. The device comprises:

[0010] A candidate replacement audio determination module is used to obtain a video to be processed, and when the original background music of the video belongs to the type to be replaced, determine at least one candidate replacement audio for the original background music; the candidate replacement audio is compatible with the video and does not belong to the type to be replaced;

[0011] An audio replacement time point determination module, configured to determine, in response to the terminal selecting the audio to be replaced from at least one candidate replacement audio, an audio replacement time point of the audio to be replaced for the original background music based on a video content matching time point of the video and an audio matching time point of the original background music;

[0012] The replacement module is used to replace the original background music in the video with the audio to be replaced according to the audio replacement time point to obtain the video after the background music is replaced.

[0013] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the above video processing method are implemented.

[0014] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the steps in the above video processing method when executed by a processor.

[0015] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps in the above video processing method are implemented.

[0016] The above-mentioned video processing method, apparatus, computer equipment, storage medium and computer program product, when the original background music of the video to be processed belongs to the type to be replaced, at least one candidate replacement audio that is compatible with the video and does not belong to the type to be replaced is determined, in response to the terminal selecting the audio to be replaced from at least one candidate replacement audio, based on the video content matching time point of the video and the audio matching time point of the original background music, the audio replacement time point of the audio to be replaced for the original background music is determined, and the original background music in the video is replaced according to the audio replacement time point by the audio to be replaced, so as to obtain the video after the background music replacement processing. In the video processing process, for the video to be processed, if the original background music of the video belongs to the type to be replaced, at least one candidate replacement audio that is compatible with the video and does not belong to the type to be replaced is determined, and when the terminal selects the audio to be replaced from at least one candidate replacement audio, the audio replacement time point is determined in combination with the video content matching time point and the audio matching time point of the original background music, so as to achieve the replacement of the original background music according to the audio to be replaced and the audio replacement time point. By replacing the original background music with the audio to be replaced that is compatible with the video and does not belong to the type to be replaced according to the determined audio replacement time point, the legitimacy of the video after the background music replacement can be ensured, and the video content in the video after the background music replacement process can be adapted to the respective time points of the background music, thereby improving the accuracy of the original background music replacement. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A diagram showing an application environment of a video processing method in an embodiment;

[0018] Figure 2 is a schematic flow chart of a video processing method in one embodiment;

[0019] Figure 3 A schematic diagram of determining an audio replacement time point in a time dimension in one embodiment;

[0020] Figure 4 It is a schematic diagram of replacing the original background music according to the audio replacement time point in the time dimension in one embodiment;

[0021] Figure 5 A schematic diagram of a process for determining an audio replacement time point in one embodiment;

[0022] Figure 6 A schematic diagram of determining an audio replacement time point in a time dimension in another embodiment;

[0023] Figure 7 A schematic diagram of adjusting the audio replacement time point in the time dimension in one embodiment;

[0024] Figure 8It is a schematic diagram of replacing the original background music according to the audio replacement time point in the time dimension in another embodiment;

[0025] Fig. 9 A schematic diagram of audio replacement time point correction in the time dimension in one embodiment;

[0026] Fig.10 A schematic diagram of replacing the original background music according to the audio replacement time point in the time dimension in yet another embodiment;

[0027] Fig.11 is a schematic diagram of an interface of a video processing method in an embodiment;

[0028] Fig.12 is a schematic diagram of an interface of a video processing method in another embodiment;

[0029] Fig.13 A schematic diagram of a business process of a video processing method in an embodiment;

[0030] Fig.14 is a flowchart of a video processing method in one embodiment;

[0031] Fig.15 A schematic diagram of a business process of a video processing method in another embodiment;

[0032] Fig.16 is a timing diagram of a video processing method in one embodiment;

[0033] Fig.17 is a structural block diagram of a video processing device in another embodiment;

[0034] Fig.18 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0036] The video processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, can be integrated on the server 104, or can be placed on the cloud or other servers. When performing video processing, the user can shoot a video through the terminal 102 and edit the shot video. For example, a soundtrack can be added to the shot video to synthesize a homemade video, and the homemade video can be used as the video to be processed. The user can send the self-made video to the server 104 through the terminal 102, and the server 104 can obtain the video to be processed sent by the terminal 102. When the original background music of the video to be processed belongs to the type to be replaced, such as the original background music has the risk of illegal use, specifically when the original background music does not have copyright or does not comply with legal regulations, it means that the original background music belongs to the type to be replaced, and at least one candidate replacement audio for the original background music is determined. The candidate replacement audio is compatible with the video and does not belong to the type to be replaced. The server 102 can send the at least one candidate replacement audio to the terminal 102, and the user selects from the at least one candidate replacement audio through the terminal 102. When the server 104 determines that the user selects the audio to be replaced from the at least one candidate replacement audio through the terminal 102, the server 104 determines the audio replacement time point of the audio to be replaced for the original background music based on the video content matching time point of the video and the audio matching time point of the original background music. The server 104 replaces the original background music in the video according to the audio replacement time point through the audio to be replaced, and obtains the video after the background music is replaced. In addition, the server 104 can return the video after the background music is replaced to the terminal 102, so that the user can share it to the video platform through the terminal 102, ensuring the legal use of the video.

[0037] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0038] In one embodiment, Figure 2 As shown, a video processing method is provided, which is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0039] Step 202, obtaining a video to be processed, and when the original background music of the video belongs to the type to be replaced, determining at least one candidate replacement audio for the original background music; the candidate replacement audio is compatible with the video and does not belong to the type to be replaced.

[0040] Among them, the video can be a continuous picture determined by a group of dynamic images whose content changes over time. The specific form of the video can be a movie, a TV series, or a short video with a shorter duration. The video can carry audio, that is, the video can be obtained after processing based on dynamic images and audio. The video to be processed is a video that needs to be processed, which can be obtained by the user. For example, the user can shoot according to his own preferences to obtain the video, and the user can also use video editing software to synthesize the video to obtain a self-made video. Background music can refer to audio for atmosphere adjustment in the video to be processed, and the original background music can specifically be the initial background music carried in the video to be processed. Among them, the original background music can be obtained by the user through self-configuration in combination with the actual video screen content in the self-made video. For example, the user can select an audio file as the background music of the self-made video in combination with the text, characters or scenes included in the video screen content, and in combination with his own preferences, that is, as the original background music of the video.

[0041] The type to be replaced may refer to the type of background music that needs to be replaced. The type to be replaced can be set according to actual needs. For example, the soothing style can be determined as the type to be replaced, that is, the background music of the soothing style can be determined as the type to be replaced and needs to be replaced. The type to be replaced can also be a type with risks, such as a type with risks of illegal use. Specifically, when the original background music has the risk of not complying with the set rules or violating the law, it can be determined that the original background music has the risk of illegal use, that is, it belongs to the type to be replaced. If the original background music does not have music copyright, or the original background music is not suitable for a certain group of people, such as teenagers, children, etc.; or the original background music does not comply with the use regulations due to the negative content, etc., the original background music can be considered to belong to the type to be replaced. For the original background music belonging to the type to be replaced, the video itself may be illegal. For example, for the original background music that does not have copyright, the video itself will also be illegally used due to copyright infringement. Among them, the candidate replacement audio can refer to the audio determined to replace the original background music. The candidate replacement audio is compatible with the video and does not belong to the type to be replaced, such as there is no risk of illegal use. The candidate replacement audio and video are compatible, which means that the candidate replacement audio and video are matched in some features, such as the video tonality of the candidate replacement audio and the rhythm and style of the original background music of the video. The candidate replacement audio and video are compatible, and when the background music of the video is replaced with the candidate replacement audio, the video effect after the replacement can be ensured.

[0042] Specifically, the server can obtain the video to be processed, and the video to be processed can be sent from the terminal to the server, such as a user making a video through the terminal and then sending it to the server. The video to be processed can also be extracted by the server from a video database. The server analyzes the original background music of the video to determine whether the original background music of the video belongs to the type to be replaced. When it is determined that the original background music belongs to the type to be replaced, the server determines at least one candidate replacement audio that is compatible with the video and does not belong to the type to be replaced. The data source of the video to be processed may include various applications running on the terminal, such as live broadcast applications, editing applications, etc. Specifically, the user can generate a homemade video using the data generated by various applications during operation. In addition, the video can also be directly shot by the user through the terminal, such as the user can shoot a real scene through the terminal to obtain a homemade video.

[0043] In a specific application, when determining whether the original background music belongs to the type to be replaced, the server can extract audio features based on the original background music, analyze the audio features, and specifically search in a pre-established music library database based on the audio features. If no music matching the original background music is found in the music library database, it can be considered that the original background music belongs to the type to be replaced. Alternatively, the server finds music matching the original background music in the music library database, but the music has copyright issues, then it can be considered that the original background music has a risk of illegal use, that is, the original background music belongs to the type to be replaced; or the server finds music matching the original background music in the music library database, and although the music has no copyright issues, the video is for teenagers to learn, and the music is not suitable for learning scenarios, then it can be considered that the original background music has a risk of illegal use, that is, the original background music belongs to the type to be replaced.

[0044] In addition, when the server determines that the original background music of the video does not belong to the type to be replaced, such as when the original background music has copyright and complies with the usage regulations, the server can directly end the video processing process, such as feeding back a prompt message to the terminal to prompt that the original background music in the video to be processed does not belong to the type to be replaced, and the video can be used legally, such as being legally uploaded to a video platform for sharing and dissemination. Furthermore, the server can also determine the adapted replacement audio that is compatible with the video in combination with the original background music of the video, the video screen content, etc., and send the adapted replacement audio to the terminal, so that the user can adaptively select the audio to be replaced from the adapted replacement audio according to their own needs. For example, the server can determine the candidate recommended audio that is compatible with the video and does not belong to the type to be replaced based on the original background music and the video screen content of the video, and return the candidate recommended audio to the terminal, so that the terminal user can replace the background music of the video with the candidate recommended audio, thereby quickly obtaining a video with a different soundtrack.

[0045] Step 204, in response to the terminal selecting the audio to be replaced from at least one candidate replacement audio, determining the audio replacement time point of the audio to be replaced for the original background music based on the video content matching time point of the video and the audio matching time point of the original background music.

[0046] The audio to be replaced is the audio selected by the user from the candidate replacement audio based on the terminal. When the terminal selects the audio to be replaced from at least one candidate replacement audio, the number of audios to be replaced can be one or more, and the specific number of audios to be replaced can be determined according to the actual selection of the terminal. The video content matching time point can be a time point determined according to the video screen content in the video, wherein the video screen content can include text, action, and subtitles, etc., and the video content matching time point can be determined specifically according to the time point at which the text, action, and subtitles involved in the video screen content appear. For example, the time point at which a specific action appears in the video screen content can be determined as the video content matching time point; the time point at which the text appears in the video screen content can also be determined as the video content matching time point; the time point at which the subtitle appears in the video screen content can also be determined as the video content matching time point, etc. Different videos include different video screen contents, and different video screen contents appear at different time points in the video, that is, different video content matching time points can be provided.

[0047] The audio matching time point may be a time point determined based on the original background music. By analyzing the original background music, the original background music starting point and the original background music ending point of the original background music may be determined, and the music rhythm points of the original background music may also be analyzed, wherein the original background music starting point may also be one of the music rhythm points. The audio matching time point may be determined based on the original background music starting point, the original background music ending point, and the music rhythm points of the original background music. The audio replacement time point refers to the time point when the original background music is replaced with the audio to be replaced in the video, and specifically may be the starting point when the audio to be replaced is played in the video after the original background music is replaced with the audio to be replaced.

[0048] Specifically, the server can detect the terminal's selection operation on at least one candidate replacement audio, and when it is determined that the terminal selects the audio to be replaced from at least one candidate replacement audio, the server can obtain the video content matching time point of the video and the audio matching time point of the original background music, and determine the audio replacement time point of the audio to be replaced for the original background music based on the video content matching time point of the video and the audio matching time point of the original background music. In a specific implementation, for at least one determined candidate replacement audio, the server can send at least one candidate replacement audio to the terminal to instruct the user of the terminal to select from at least one candidate replacement audio, the user can select from at least one candidate replacement audio through the terminal, and feed back the selection result to the server, the server can determine the audio to be replaced selected by the terminal user from at least one candidate replacement audio according to the selection result, and further determine the audio replacement time point. The server determines the audio replacement time point for the original background music according to the audio to be replaced selected by the user through the terminal, according to the video content matching time point and the audio matching time point.

[0049] In a specific application, there may be only one video content matching time point of the video and one audio matching time point of the original background music. When there is only one video content matching time point and one audio matching time point, such as the video content matching time point can be the action time point when a specific action occurs in the video, and the audio matching time point can be the background music playback starting point when the original background music is played in the video, the server can analyze the action time point and the background music playback starting point to determine the audio replacement time point. Specifically, when the action time point and the background music playback starting point are the same, the background music playback starting point can be determined as the audio replacement time point for the original background music.

[0050] The video content matching time points of the video and the audio matching time points of the original background music may also have at least some of multiples, such as multiple video content matching time points or multiple audio matching time points, or multiple video content matching time points and multiple audio matching time points. In the case where there are multiple video content matching time points and audio matching time points, such as the video content matching time points include the action time points of the specific action 1 in the video, the time points of the specific action 2 in the video, and the time points of the text; the audio matching time points include the background music playback start point, the time point of the music rhythm point 1, and the time point of the music rhythm point 2 when the original background music is played in the video; the server can conduct a comprehensive analysis of the action time points of the specific action 1, the time points of the specific action 2, and the time points of the text, the background music playback start point, the time point of the music rhythm point 1, and the time point of the music rhythm point 2, such as comparing the action time points of the specific action 1 with the background music playback start point, comparing the time points of the specific action 2 with the music rhythm point 1, and comparing the time points of the text with the music rhythm point 2, respectively determining the corresponding comparison results, and the server can combine the comparison results to determine the audio replacement time point. For example, when the action time point of the specific action 1 does not match the background music playback start point, but matches the time point of the music rhythm point 1, the time point of the music rhythm point 1 can be determined as the audio replacement time point for the original background music.

[0051] Step 206, replace the original background music in the video with the audio to be replaced according to the audio replacement time point to obtain the video after the background music is replaced.

[0052] Specifically, the server can determine the audio clip that replaces the original background music based on the audio to be replaced, and based on the audio replacement time point, use the audio clip to replace the original background music in the video to obtain the video after the background music is replaced. For example, if the audio replacement time point determined by the server is the 20th second in the video, the server can replace the original background music of the video with the audio to be replaced starting from the 20th second of the video, thereby obtaining the video after the background music is replaced. In the video after the background music is replaced, the audio to be replaced can be played starting from the 20th second of the video.

[0053] In a specific application, such as Figure 3As shown, for the video to be processed, which can specifically be a video sent by the terminal, in the video progress bar with time as the dimension, for the video screen content, there are three video content matching time points, namely, video content matching time point 1 at 2S (seconds), video content matching time point 2 at 10S, and video content matching time point 3 at 16S; and the video carries the original background music, and the audio matching time points of the original background music also include three, namely, audio matching time point 1 at 2S, audio matching time point 2 at 10S, and audio matching time point 3 at 16S. The server can perform a risk analysis on the original background music in the video sent by the terminal. When it is determined that the original background music is of the type to be replaced, if there is a risk of illegal use, it indicates that the original background music needs to be replaced. The server can determine at least one candidate replacement audio. The candidate replacement audio is compatible with the video and does not belong to the type to be replaced, that is, the candidate replacement audio does not have a risk of illegal use. The server can send at least one candidate replacement audio to the terminal, and the user of the terminal selects the audio to be replaced. The server can determine the audio replacement time point of the original background music based on the three video content matching time points and the three audio matching time points in the video, and replace the original background music according to the audio replacement time point to obtain the video after the background music is replaced.

[0054] like Figure 4 As shown, in the video after the background music is replaced, the background music carried by it is replaced with the audio to be replaced, and the audio replacement time point of the audio to be replaced is 2S, that is, the server starts to replace the original background music in the video at 2S through the audio to be replaced, thereby obtaining the video after the background music is replaced. In the video after the background music is replaced, 10S and 16S correspond to the audio rhythm time point 1 and the audio rhythm time point 2 of the audio to be replaced, respectively.

[0055] In a specific embodiment, when the server determines that the terminal has selected more than one audio to be replaced, such as determining that the terminal has selected two audios to be replaced, audio 1 to be replaced and audio 2 to be replaced, the server can use audio 1 to be replaced and audio 2 to be replaced to jointly replace the original background music, and obtain the video after the background music is replaced. Among them, the server can segment the original background music according to the time length to obtain at least two segments of segmented background music. For example, the server can segment the original background music according to a time length of 10S. When the playing time of the original background music is 20S, the segmented background music 1 with a playing time period of 1-10S and the segmented background music 2 with a playing time period of 11S-20S can be obtained respectively. The server can extract at least one first audio segment with a duration of 10 seconds from audio 1 to be replaced, and extract at least one second audio segment with a duration of 10S from audio 2 to be replaced. Furthermore, the server can perform matching analysis on at least one first audio clip and at least one second audio clip respectively with the segmented background music 1, and select the audio clip with the highest matching degree to replace the segmented background music 1, so as to obtain background replacement video 1. The server can perform matching analysis on at least one first audio clip and at least one second audio clip respectively with the segmented background music 2, and select the audio clip with the highest matching degree to replace the segmented background music 2, so as to obtain background replacement video 2. The server can perform splicing processing based on background replacement audio 1 and background replacement audio 2 to obtain a video after background replacement. Through this video processing method, the video after background music replacement processing can be made more diversified while satisfying the adaptability and legality, thereby improving the flexibility of video processing.

[0056] In a specific application, when the server replaces the original background music with the audio to be replaced according to the audio replacement time point, it can also be set to change the volume of the audio to be replaced at different music rhythm points. For example, if the original background music includes music rhythm point 1 and music rhythm point 2 in addition to the audio replacement time point, the server can set the volume to increase every time a music rhythm point is played, such as the volume of the audio played to music rhythm point 1 can be set to be higher than the volume played to music rhythm point 2. For another example, when the server replaces the original background music with the audio to be replaced according to the audio replacement time point, it can also adjust the volume according to the video screen content. The server analyzes the video screen content and determines that the video screen content includes specific action 1, specific action 2 and subtitle 1. The server can set different volumes of music at specific action 1, specific action 2 and subtitle 1, such as setting the volume of music played to specific action 1 to be higher than the volume played to specific action 2, and the volume of music played to specific action 2 to be higher than the volume played to subtitle 1. Through this volume adjustment, the video can be processed in a more fine-grained manner, which improves the flexibility of video processing.

[0057] In the above-mentioned video processing method, when the original background music of the video to be processed belongs to the type to be replaced, at least one candidate replacement audio that is compatible with the video and does not belong to the type to be replaced is determined, in response to the terminal selecting the audio to be replaced from at least one candidate replacement audio, based on the video content matching time point of the video and the audio matching time point of the original background music, the audio replacement time point of the audio to be replaced for the original background music is determined, and the original background music in the video is replaced according to the audio replacement time point through the audio to be replaced, so as to obtain the video after the background music replacement processing. In the video processing process, for the video to be processed, if the original background music of the video belongs to the type to be replaced, at least one candidate replacement audio that is compatible with the video and does not belong to the type to be replaced is determined, and when the terminal selects the audio to be replaced from at least one candidate replacement audio, the audio replacement time point is determined in combination with the video content matching time point and the audio matching time point of the original background music, so as to achieve the replacement of the original background music according to the audio to be replaced and the audio replacement time point. By replacing the original background music with the audio to be replaced that is compatible with the video and does not belong to the type to be replaced according to the determined audio replacement time point, the legitimacy of the video after the background music replacement can be ensured, and the video content in the video after the background music replacement process can be adapted to the respective time points of the background music, thereby improving the accuracy of the original background music replacement.

[0058] In one embodiment, Figure 5As shown, the process of determining the audio replacement time point, that is, determining the audio replacement time point of the audio to be replaced for the original background music based on the video content matching time point of the video and the audio matching time point of the original background music, includes the following steps:

[0059] Step 502, obtaining the video content matching time point of the video and the audio matching time point of the original background music.

[0060] Specifically, the server can obtain the video content matching time point of the video and the audio matching time point of the original background music. When obtaining the video content matching time point, the server can determine the video screen content from the video, and select any time point as the video content matching time point according to the time point at which a specific action appears in the video screen content, the time point at which text appears in the video screen content, or the time point at which subtitles appear. The server can also determine all or multiple time points as the video screen content time point from the time point at which a specific action appears in the video screen content, the time point at which text appears in the video screen content, or the time point at which subtitles appear. When obtaining the audio matching time point, the server can also select any one or more of the time points at which the original background music starting point of the original background music is located, and the time points of other music rhythm points other than the background music starting point as the audio matching time point, or the time point at which the background music starting point is located, and the time points of other music rhythm points other than the background music starting point are all used as the audio matching time point.

[0061] Step 504: When the audio matching time point matches the video content matching time point, the audio matching time point is used as the audio replacement time point of the audio to be replaced for the original background music.

[0062] Among them, the audio matching time point matches the video content matching time point, which means that the matching degree between the audio matching time point and the video content matching time point meets certain matching degree requirements. For example, the matching degree can be specifically characterized by the time difference between the audio matching time point and the video content matching time point. When the time difference is less than a threshold, it is considered that the audio matching time point matches the video content matching time point. When the audio matching time point includes the time point at which the original background music starts, and the video content matching time point is the time point at which a certain action in the video appears, what is determined is the matching degree between the time point at which the original background music starts and the time point at which a certain action in the video appears; when the audio matching time point includes the music rhythm point of the original background music, and the video content matching time point is the time point at which a certain action in the video appears or the time point at which text appears, what is compared is the matching degree between the music rhythm point of the original background music and the time point at which a certain action in the video appears or the time point at which text appears.

[0063] Specifically, the server can conduct comparative analysis based on the video content matching time point and the audio matching time point of the original background music to determine whether the audio matching time point and the video content matching time point meet the set matching degree conditions. When the set matching degree determination conditions are met, the audio matching time point is determined to match the video content matching time point, and the audio matching time point is directly used as the audio replacement time point for the original background music to be replaced. Among them, the matching degree determination condition refers to the condition set to determine whether the matching degree meets the matching degree requirements. For example, it can be set that the matching degree determination condition is met only when the audio matching time point and the video content matching time point are exactly the same. It can also be set that the time difference between the audio matching time point and the video content matching time point does not exceed the set time difference matching threshold, then it can be determined that the matching degree determination condition is met. The specific matching degree determination condition can be set adaptively in combination with the accuracy requirements of video processing.

[0064] In a specific application, the audio matching time point may include multiple, such as audio matching time point 1, audio matching time point 2, and audio matching time point 3. The video content matching time point may also include multiple, such as video content matching time point 1, video content matching time point 2, and video content matching time point 3. The server may determine whether there is a match based on the matching degree of multiple time points, such as comparing the audio matching time point 1 with the video content matching time point 1 to obtain comparison result 1, comparing the audio matching time point 2 with the video content matching time point 2 to obtain comparison result 2, and comparing the audio matching time point 3 with the video content matching time point 3 to obtain comparison result 3. When the server determines whether the audio matching time point matches the video content matching time point, it may be determined that the audio matching time point matches the video content matching time point when comparison result 1, comparison result 2, and comparison result 3 all meet the matching degree determination condition, or it may be determined that the audio matching time point matches the video content matching time point when most of the comparison results 1, comparison result 2, and comparison result 3 meet the matching degree determination condition, such as comparison result 1 and comparison result 2 meet the matching degree determination condition, or comparison result 2 and comparison result 3 meet the matching degree determination condition, etc. Further, when determining that the audio matching time point matches the video content matching time point, the server may select any one of audio matching time point 1, audio matching time point 2, and audio matching time point 3 as the audio replacement time point.

[0065] In this embodiment, the server obtains the video content matching time point of the video and the audio matching time point of the original background music, and uses the audio matching time point as the audio replacement time point of the audio to be replaced for the original background music when it is determined that the audio matching time point matches the video content matching time point. Since the audio start time point is determined when the audio matching time point matches the video content matching time point, the respective time points of the video content and the background music in the video after the background music replacement process can be adapted, thereby improving the accuracy of the original background music replacement.

[0066] In one embodiment, the audio matching time point includes the audio starting time point; the video content matching time point includes the video picture time point of the video picture to be matched with music; when the audio matching time point matches the video content matching time point, the audio matching time point is used as the audio replacement time point of the audio to be replaced for the original background music, including: when the audio starting time point is the same as the video picture time point, the audio starting time point is used as the audio replacement time point of the audio to be replaced for the original background music.

[0067] The audio matching time point may include the audio start time point, and the audio start time point may include the audio playback start point, that is, the time point when the original background music starts to play in the video. The video screen to be matched with music may refer to the video screen that needs to be matched with music, that is, the video screen that needs to add background music. The video screen to be matched with music may specifically include specific actions, text, subtitles, and the like. The video screen time point may be related to the video screen to be matched with music. When the video screen to be matched with music is a specific action, the video screen time point may be the time point when the specific action appears. When the video screen to be matched with music is text, the video screen time point may be the time point when the text appears.

[0068] Specifically, the server can compare the audio start time point with the video screen time point of the video screen to be matched with music. When the audio start time point is the same as the video screen time point of the video screen to be matched with music, the audio start time point is used as the audio replacement time point of the audio to be replaced for the original background music. In specific implementation, there can be only one video screen time point or multiple video screen time points. When there is only one video screen time point, such as when the video screen time point is the time point when a specific screen appears in the video, the server can compare the audio start time point with the time point when the specific screen appears in the video to determine whether the time of the two is the same. When there are multiple video screen time points, such as when the video screen time point includes the time point when text appears and the time point when a specific screen appears, and the specific screen may include the time point when a character appears, the time point when a subtitle appears, etc., the server can compare the importance of multiple video screen time points to the video presentation effect, and select one of the video screen time points as the target video screen time point according to the importance to compare with the audio start time point.

[0069] In a specific application, such as Figure 6 As shown, for the video to be processed, in the video progress bar of the video with time as the dimension, for the video screen content, there are 3 video content matching time points, namely, video screen time point 1 at 2S, video screen time point 2 at 10S, and video screen time point 3 at 16S; and the video carries the original background music, and the audio matching time points of the original background music include the audio start time point at 2S, that is, the original background music starts playing from 2S in the video. The server can determine that the audio start time point is the same as the video screen time point 1, that is, both are 2S, then the server can use the audio start time point at 2S as the audio replacement time point, that is, starting from 2S, the original background music in the video is replaced with the audio to be replaced selected by the terminal, thereby obtaining the video after the background music replacement processing.

[0070] In this embodiment, when the server determines that the audio starting time point is the same as the video screen time point, the server uses the audio starting time point as the audio replacement time point for the original background music to be replaced. Since the audio starting time point is determined when the audio matching time point is the same as the video content matching time point, after the background music replacement process, the new background music can be more compatible with the video, thereby improving the accuracy of the replacement of the original background music.

[0071] In one embodiment, the audio matching time point also includes at least one audio rhythm time point; the video picture time point includes at least two; the video processing method also includes: when at least one audio rhythm time point does not match the video picture time point in the video, respectively determining the time difference between at least one audio rhythm time point and the respective mismatched video picture time point; when the time difference meets the replacement time point adjustment condition, determining the adjustment time based on the time difference; adjusting the audio replacement time point according to the adjustment time to obtain the adjusted audio replacement time point.

[0072] Among them, the audio matching time point may also include an audio rhythm time point other than the audio start time point, and the number of audio rhythm time points may be only one or more. The video screen time point may include multiple. At least one music rhythm time point also needs to be compared and analyzed with the video screen time point of the video to determine whether it matches. The time difference may refer to the difference between the audio rhythm time point and its corresponding unmatched video screen time point. When there are multiple audio rhythm time points, each audio rhythm time point corresponds to a video screen time point. The replacement time point adjustment replacement condition may refer to the set condition for determining whether to adjust and replace the audio replacement time point. The adjustment time may refer to the time for adjusting the audio replacement time point, and the adjustment time may specifically include adjusting the audio replacement time point forward by a certain time, such as the audio replacement time point is 11 minutes and 45 seconds of the video playback, then by adjusting the time, the audio replacement time point is adjusted to 11 minutes and 35 seconds. The adjustment time may also include adjusting the audio replacement time point backward by a certain time, such as the audio replacement time point is 12 minutes and 20 seconds of the video playback, then by adjusting the time, the audio replacement time point is adjusted to 12 minutes and 27 seconds.

[0073] Specifically, the server can compare and analyze the audio rhythm time point with the video screen time point in the video to obtain a comparison result. When the comparison result is that the audio rhythm time point does not match the video screen time point in the video, the time difference between the audio rhythm time point and the video screen time point in the video is determined, and whether the replacement time point adjustment replacement condition is met according to the time difference. When the time difference meets the replacement time point adjustment replacement condition, the adjustment time is obtained, and the audio replacement time point is replaced according to the adjustment time. Among them, when the server sets the replacement time point adjustment replacement condition, it can be set according to the time adjustment threshold, that is, the time difference needs to reach the time adjustment threshold to meet the replacement time point adjustment condition; for the case where there are multiple time differences, it can be set that the replacement time adjustment replacement condition is met only when all time differences are the same, or it can be set that when the number of the same time differences reaches 80% of the total number of time differences, the replacement time point adjustment replacement condition is met. When setting the replacement time point adjustment replacement condition, it can be adaptively set in combination with the number of time differences, the adjustment accuracy requirements of the replacement time point, etc.

[0074] In a specific application, the audio rhythm time point includes audio rhythm time point 1, audio rhythm time point 2 and audio rhythm time point 3. The server can compare audio rhythm time point 1, audio rhythm time point 2 and audio rhythm time point 3 with the corresponding video screen time point. When there is a time mismatch between the audio rhythm time point and the corresponding video screen time point, such as the audio rhythm time point 1 and the audio rhythm time point 2 do not match the corresponding video screen time point, the server can calculate the time difference 1 between the audio rhythm time point 1 and the corresponding video screen time point, and the time difference 2 between the audio rhythm time point 2 and the corresponding video screen time point. When the server determines that the time difference 1 and the time difference 2 are the same, it can be determined that the time difference meets the replacement time point adjustment replacement condition.

[0075] Furthermore, the original background music in the video is replaced according to the audio replacement time point through the audio to be replaced, and the video after the background music replacement is obtained, including: the original background music in the video is replaced according to the adjusted audio replacement time point through the audio to be replaced, and the video after the background music replacement is obtained.

[0076] Specifically, the server can replace the original background music in the video with the audio to be replaced based on the adjusted audio replacement time point, that is, determine the adjusted audio replacement time point as the audio playback start point to insert the audio to be replaced, and obtain the video after the background music is replaced.

[0077] In a specific application, such as Figure 7As shown, for the video to be processed, in the video progress bar with time as the dimension of the video, for the video screen content, 4 video content matching time points are included, namely, video screen time point 1 at 2S, video screen time point 2 at 8S, video screen time point 3 at 14S, and video screen time point 4 at 20S; and the original background music is carried in the video, and the audio matching time points of the original background music include the audio start time point at 2S, the audio rhythm time point 1 at 8.5S, the audio rhythm time point 2 at 14.5S, and the audio rhythm time point 3 at 20.5S, that is, the original background music is played from 2S in the video, and there are music rhythm points at 8.5S, 14.5S, and 20.5S respectively. The server can determine that the audio start time point is the same as the time point of the video screen time point 1, and the audio start time point can be used as the audio replacement time point, that is, the audio replacement time point is set to 2S. However, the time points of audio rhythm time point 1 and video screen time point 2, audio rhythm time point 2 and video screen time point 3, and audio rhythm time point 3 and video screen time point 4 are all different, and the difference is 0.5S. The server can consider that the replacement time point adjustment conditions are met, and can determine that the adjustment time is 0.5S in advance, so that the audio replacement time point can be adjusted from 2S to 1.5S.

[0078] Furthermore, if Figure 8 As shown, the server can replace the original background music according to the adjusted audio replacement time point. In the video after the background music replacement, although the audio replacement time point is 1.5S, which is different from the 2S of the video screen time point 1, the audio rhythm time point 1 and the video screen time point 2, the audio rhythm time point 2 and the video screen time point 3, and the audio rhythm time point 3 and the video screen time point 4 are the same, thereby ensuring the adaptation of the background music and the video screen to a greater extent.

[0079] In this implementation, when the server determines that at least one audio rhythm time point does not match the video picture time point in the video, it will respectively determine the time difference between at least one audio rhythm time point and the respective mismatched video picture time point, and when the time difference meets the replacement time point adjustment replacement condition, the server determines the adjustment time based on the time difference, adjusts the audio replacement time point according to the adjustment time, and obtains the adjusted audio replacement time point. By adjusting the audio replacement time point, the adaptability of the audio to be replaced and the video can be improved, and the accuracy of the original background music replacement can be improved.

[0080] In one embodiment, the video processing method also includes: when the audio matching time point does not match the video content matching time point, correcting the audio matching time point based on the video content matching time point to obtain a corrected audio matching time point; and using the corrected audio matching time point as the audio replacement time point for the original background music to be replaced.

[0081] The mismatch between the audio matching time point and the video content matching time point may refer to that the matching degree between the audio matching time point and the video content matching time point does not meet the matching degree requirement, such as the audio matching time point and the video content matching time point are different and there is a time difference. Correction may be a process of adjusting the audio matching time point.

[0082] Specifically, when the server determines that the audio matching time point does not match the video content matching time point, if it is determined that the audio matching time point and the video content matching time point are different, then it is determined that the audio matching time point and the video content matching time point do not match; or if it is determined that the time difference between the audio matching time point and the video content matching time point is greater than the set time difference correction threshold, then it is determined that the audio matching time point and the video content matching time point do not match. The server can adjust the audio matching time point based on the video content matching time point, obtain the adjusted audio matching time point, and use the adjusted audio matching time point as the audio replacement time point for the original background music of the audio to be replaced. Among them, when the server adjusts the audio matching time point, it can adjust the audio matching time point to the left of the time progress bar of the video. For example, if the original video matching time point is 1 minute and 10 seconds, by adjusting the audio matching time point to the left of the time progress bar of the video, the adjusted audio matching time point is 1 minute and 2 seconds. The audio matching time point can also be adjusted to the right of the video time progress bar. For example, if the original video matching time point is 1 minute and 10 seconds, by adjusting the audio matching time point to the right of the video time progress bar, the adjusted audio matching time point will be 1 minute and 20 seconds.

[0083] In a specific application, such as Fig. 9As shown, for the video to be processed, in the video progress bar of the video with time as the dimension, there are three video content matching time points for the video screen content, namely, video content matching time 1 at 2S, video content matching time 2 at 10S, and video content matching time 3 at 16S; and the video carries the original background music, which includes the audio matching time point at 1.5S. The server can compare the audio matching time point with each video content matching time to determine the matching relationship between the audio matching time point and each video content matching time. If it is found that the audio matching time point does not match each video content matching time, and the time difference between the audio matching time point and the video content matching time point 1 is the smallest, specifically 0.5S, then the server can correct the audio matching time point, specifically, the audio matching time point can be pushed back by 0.5S to 2S, so that the audio matching time point is also 2S. Further, as Fig.10 As shown, after the audio matching time point is corrected to 2S, the corrected audio matching time point at 2S is determined as the audio replacement time point, and according to the audio replacement time point, the original background music in the video is replaced by the audio to be replaced.

[0084] In this embodiment, when the server determines that the audio matching time point does not match the video content matching time point, it will correct the audio matching time point based on the video content matching time point to obtain the corrected audio matching time point, and use the corrected audio matching time point as the audio replacement time point of the audio to be replaced for the original background music. By adjusting the audio matching time point, the server can obtain a more accurate audio matching time point, improve the adaptability of the audio to be replaced and the video, and improve the accuracy of audio processing.

[0085] In one embodiment, when the original background music of a video belongs to the type to be replaced, determining at least one candidate replacement audio for the original background music includes: determining the original background music of the video; when the original background music belongs to the type to be replaced, determining at least one candidate replacement audio based on the audio features of the original background music and the video features of the video; wherein the candidate replacement audios are respectively adapted to the audio features and the video features, and the candidate replacement audios do not belong to the type to be replaced.

[0086] Among them, audio features may refer to features obtained after audio analysis of the audio content of the original background music, such as the melody, rhythm, chords, structure, etc. of the original background music. Video features may refer to features extracted from the video screen content of the video by analyzing the video screen content of the video. Video features may include video type, video tonality, scenery and characters in the video, etc. Video types may include dance, drama, etc. Video tonality may include festive and funny, black humor, and fresh, etc.

[0087] Specifically, the server can determine the original background music of the video, and when the original background music belongs to the type to be replaced, the server can determine the candidate replacement audio that matches both the audio and video features of the original background music and does not belong to the type to be replaced by combining the audio features and video features of the original background music. In a specific application, when the server determines that the original background music has negative content, it determines that the original background music belongs to the type to be replaced. The server can further determine the candidate replacement audio by combining the audio features and the video features. The server can first determine the first reference replacement audio that matches the audio features in a pre-set legal music database based on the audio features, and further combine the video features to select the candidate replacement audio from the first reference replacement audio; of course, the server can also first determine the second reference replacement audio that matches the video features in a pre-set legal music database based on the video features, and further combine the audio features to select the candidate replacement audio from the second reference replacement audio; the server can also fuse the audio features and the video features to obtain a fused audio and video feature, and directly search for at least one candidate replacement audio from the preset legal music database based on the fused audio and video feature.

[0088] In this embodiment, the server determines that the original background music belongs to the type to be replaced, and then determines at least one candidate replacement audio that does not belong to the type to be replaced and is compatible with the audio features and video features of the original background music. This allows the server to recommend to the user audio that is legal and matches the original background music, thereby improving the accuracy of audio recommendations.

[0089] In one embodiment, when the original background music belongs to the type to be replaced, at least one candidate replacement audio is determined based on the audio features of the original background music and the video features of the video, including: extracting audio features from the original background music; based on the audio features, searching a music library database for recognized audio that matches the original background music; when the recognized audio does not have copyright, determining copyrighted audio that matches the audio features and has copyright; when the recognized audio has copyright, determining the recognized audio as copyrighted audio; and obtaining at least one candidate replacement audio that matches the video features of the video from the determined copyrighted audio.

[0090] Among them, the audio features are features extracted from the original background music. The music library database stores various types of audio. Through the audio features of the original background music, the music database is searched to obtain the recognized audio. The recognized audio can refer to the audio that matches the original background music, that is, the audio with a high degree of matching. Copyrighted audio can refer to audio that is not only compatible with the original background music, but also has copyright. Copyrighted audio will not have the risk of illegal use, that is, it does not belong to the type to be replaced.

[0091] Specifically, the server can extract audio features from the original background music, find the recognized audio that matches the audio features in the music library database, that is, find the recognized audio that matches the original background music to a high degree, and determine whether the recognized audio is copyrighted. If it is not copyrighted, redetermine the copyrighted audio; when the recognized audio is determined to be copyrighted, the recognized audio can be directly determined as copyrighted audio, and at least one candidate replacement audio that matches the video features of the video can be determined from the copyrighted audio. In specific implementation, the server can perform similarity matching based on the audio features of the original background music and the audio features of the music library of each audio in the music library database, determine the similarity between the original background music and each audio in the music library database, and select the audio with the highest similarity as the recognized audio based on the similarity. When performing similarity matching, the server can perform similarity calculation based on the audio features and the audio fingerprint of the music library audio features. Furthermore, when the server determines that the recognized audio is not copyrighted, it can sort the original background music based on the similarity between the audio in the music library database and the original background music, and reselect the audio in the music library database that is highly similar to the original background music as the new recognized audio according to the similarity, until the copyrighted audio is found.

[0092] When the server determines that the recognized audio does not have copyright, it can also reselect the copyright music library database, and directly query the audio with high similarity to the original background music from the copyright music library database as the copyright audio. The audio stored in the copyright music library database is all copyrighted audio. When the server determines that the recognized audio does not have copyright, it can also reset the similarity threshold, which is used to determine the recognized audio from the music library database, that is, the audio in the music library database whose similarity to the original background music exceeds the similarity threshold can be determined as the recognized audio matching the original background music. When the first recognized audio determined according to the similarity threshold does not have copyright, the similarity threshold can be adjusted, such as the value of the similarity threshold can be reduced to re-determine the second recognized audio. If the similarity threshold is set to 95%, when the recognized audio determined from the music library database based on the similarity threshold does not have copyright, or the number of copyrighted audios is less than the number of candidate replacement audios, the server can adjust the similarity threshold to 85%, so as to re-determine more recognized audios from the music library database based on the similarity threshold, so as to obtain candidate replacement audios according to the re-determined recognized audios.

[0093] In this embodiment, the server determines whether the recognized audio of the original background music has copyright by querying in the music library database. If the recognized audio does not have copyright, the server determines copyrighted audio that matches the audio features and has copyright. If the recognized audio has copyright, the server directly uses the recognized audio as copyrighted audio, and further determines at least one candidate replacement audio from the copyrighted audio in combination with the video features. By first determining the copyrighted audio that matches the audio and has copyright, and then using the video features to screen the copyrighted audio, the server can ensure that at least one candidate replacement audio has copyright and has a high degree of compatibility with the video, thereby improving the accuracy of replacing the original background music.

[0094] In one embodiment, at least one candidate replacement audio that matches the video features of a video is obtained from the determined copyrighted audio, including: determining a login account corresponding to the video, and obtaining historical soundtrack features of the login account; fusing the historical soundtrack features with the video features of the video to obtain fused features; and determining at least one candidate replacement audio that matches the fused features from the copyrighted audio.

[0095] Among them, the login account may refer to a set of information used to confirm the identity of the user on a website, application or computer device. The login account may specifically be the account name of the login terminal, that is, the identification code of the user at the terminal. The login account corresponding to the video may be the account corresponding to the login on the terminal that sends the video. The historical soundtrack feature may be a feature that records the user description data. The user description data may include the music data selected by the user in the past, and may also include description information describing the user himself. The fusion feature may refer to the feature obtained by fusing the video feature and the historical soundtrack feature. When performing feature fusion in practice, multiple feature fusion methods such as feature-level fusion, decision-level fusion, hybrid-level fusion and model-level fusion may be adopted.

[0096] Specifically, the server can determine the historical soundtrack features of the login account corresponding to the video, such as the login account of the terminal that sends the video, and use the selected feature fusion method to fuse the historical soundtrack features and video features to obtain a fusion feature, thereby determining at least one candidate replacement audio from the copyrighted audio according to the fusion feature. In specific implementation, the server can set corresponding proportions for the historical soundtrack features and video features based on the importance of the historical soundtrack features and video features relative to the soundtrack of the video, and carry out fusion in combination with the set proportions when performing feature fusion to obtain a fusion feature. The proportions of the historical soundtrack features and video features can be the same or different. If the historical soundtrack features and video features have the same importance for the soundtrack of the video, the proportions of the historical soundtrack features and video features can be set to be the same; if the historical soundtrack features are more important to the soundtrack of the video than the video features, the historical soundtrack features can be set to have a higher proportion than the video features; if the video features are more important to the soundtrack of the video than the historical soundtrack features, the video features can be set to have a higher proportion than the historical soundtrack features.

[0097] In this embodiment, the server obtains a fusion feature by fusing the video feature and the historical soundtrack feature, and further determines at least one candidate replacement audio that matches the fusion feature from the copyrighted audio through the fusion feature. When searching for at least one candidate replacement audio, the user description data is combined, so that the candidate replacement audio found can better meet the needs of the user and improve the accuracy of replacing the original background music.

[0098] In one embodiment, the video processing method further includes: performing feature extraction on the video according to at least one content element dimension to obtain content element features under at least one content element dimension; and obtaining video features of the video according to the content element features.

[0099] The content element dimension may refer to the element dimension divided according to the video screen content. For example, the characters, objects, texts, scenery, and tonality included in the video screen content may belong to different element dimensions. The content element features may be features obtained by extracting features from the content element dimensions. The content element features may be image features, text features, and tonality features obtained by extracting features from characters, objects, texts, scenery, and tonality.

[0100] Specifically, the server can extract features of the video screen content according to at least one content element dimension to obtain content element features under at least one screen content dimension, and determine the video features based on at least one content element feature. In specific implementation, the server can perform frame processing on the video, such as framing the video according to a preset framing interval to obtain multiple frames of video screen content images, select at least one frame of video screen content image from each frame image, and determine the selected video screen content image as the target video screen content image to be identified, wherein the server can obtain the target video screen content image in sequence from at least one frame of video screen content image according to the framing order, and can also randomly extract the target video screen content image from at least one frame of video screen content image. Further, the server can extract content element features based on the target video screen content image. Among them, the server can carry out feature extraction according to at least one content element dimension. For example, for the character dimension, the extracted content element features may include character expressions, character actions, etc.; for the object dimension, the extracted content element features may include cakes, fireworks, etc.; for the scenery dimension, the extracted content element features may include flowers, grass, and rivers, etc.; for the tonality dimension, the extracted content element features may include lazy rural tonality, festive tonality, etc.

[0101] In this embodiment, the server extracts features from different content element dimensions to obtain content element features under different content element dimensions, and obtains video features of the video based on the content element features. The video features formed by multiple content element features have a wider coverage, can more comprehensively reflect the video screen content, and to a certain extent improve the accuracy of determining candidate replacement audio.

[0102] In one embodiment, in response to a terminal selecting an audio to be replaced from at least one candidate replacement audio, an audio replacement time point of the audio to be replaced relative to the original background music is determined based on a video content matching time point of the video and an audio matching time point of the original background music, including: sending at least one candidate replacement audio to the terminal; at least one candidate replacement audio is used to be displayed in a selectable manner at the terminal; obtaining an audio selection instruction fed back by the terminal, and determining the audio to be replaced selected by the terminal according to the audio selection instruction; and determining an audio replacement time point of the audio to be replaced relative to the original background music based on a video content matching time point of the video and an audio matching time point of the original background music.

[0103] The selectable method may refer to the form in which the candidate replacement audio is displayed on the terminal, such as the name of the music corresponding to the candidate replacement audio can be presented on the terminal, or letters, numbers or feature identification codes can be directly used to represent the candidate replacement audio, as long as different candidate replacement audios can be distinguished. The audio selection instruction may refer to an instruction generated by the terminal based on the selected audio to be replaced. By obtaining the audio selection instruction sent by the terminal, the server can clarify the candidate replacement audio corresponding to the audio to be replaced selected by the terminal.

[0104] Specifically, the server can send at least one determined candidate replacement audio to the terminal so that the user can select the required audio to be replaced based on the terminal. When the user determines the audio to be replaced based on the audio to be replaced, the terminal can generate an audio selection instruction based on the audio to be replaced, such as generating an audio selection instruction based on the identification information, name information, etc. of the audio to be replaced, and send the audio selection instruction to the server. The server can parse the audio selection instruction according to the received audio selection instruction to determine the audio selected by the user. Further, the server determines the audio replacement time point of the audio selected by the user for the original background music according to the video content matching time point of the video and the audio matching time point of the original background music. In a specific application, the candidate replacement audio can be displayed on the terminal using an identifier. For example, when the candidate replacement audio can use letters as identifiers, letters a can correspond to candidate replacement audio 1, letters b can correspond to candidate replacement audio 2, and letters c can correspond to candidate replacement audio 3. When the terminal selects the audio to be replaced based on the identifier, the terminal can directly generate an audio selection instruction based on the identifier. The server determines the identifier by parsing the audio selection instruction, and can determine the candidate replacement audio selected by the terminal based on the determined identifier.

[0105] In this embodiment, after the terminal selects the audio to be replaced, it will feedback the audio selection instruction to the server. The server can determine the audio to be replaced selected by the user based on the audio selection instruction. The server further determines the audio replacement time point of the audio to be replaced for the original background music based on the video content matching time point of the video and the audio matching time point of the original background music. Through the information interaction between the server and the terminal, the efficiency of video processing can be improved.

[0106] In one embodiment, the video processing method further includes: determining candidate soundtrack video frames from the video; determining the soundtrack video frames from the candidate soundtrack video frames according to the soundtrack conditions of the video screen; and obtaining the video content matching time point of the video according to the time point of the soundtrack video frame.

[0107] Among them, the candidate music video frame may refer to a video frame including a specific action, subtitles, text, etc., and the video music frame may refer to a video frame selected from the candidate music video for determining the time point of the video content matching. When selecting the video music frame, the selection may be made according to the video picture music condition, that is, the candidate music video frame that meets the video picture music condition is determined as the video music frame; or the similarity analysis method may be adopted to perform inter-frame similarity analysis on the candidate music video frame, and select the video frame with a large action change amplitude from the candidate music video frame as the video music frame. Among them, the video picture music condition may refer to the condition set for music matching for a specific video picture. The video picture music condition may be set by the user, or may support flexible modification by the user, for example, the user may set a specific action in the video to be matched with music, or may set a group of text appearing in the video to be matched with music. The video picture music condition may also be set by the server, such as the server may perform video analysis on the video, obtain the video analysis result, and determine which content needs to be matched with music based on the video analysis result.

[0108] Specifically, the server can determine candidate music video frames based on the video, and determine the video frames that meet the video screen matching conditions in the candidate music video frames as music video frames, so that the video content matching time point of the video can be obtained according to the time point of the music video frames. In specific implementation, the server can perform frame processing on the video according to the set frame interval to obtain multiple video screen image frames, and select some from the multiple video screen image frames as candidate music video frames. For example, the server can select from the video screen image frames according to the content in the video screen content that may need music. For example, the actions, texts or subtitles included in the video screen content may need music. The server can determine the video screen image frames with actions, texts or subtitles in the video screen image frames as candidate music video frames. Furthermore, the server can select from the candidate music video frames according to the set music conditions of the video screen. For example, if the user sets that music needs to be played when certain specific actions appear, the server can filter out the candidate music video frames that do not contain the specific actions, retain the candidate music video frames that contain the specific actions as the music video frames, and directly use the time points of the music video frames as the video content matching time points of the video. For another example, if the user sets that music needs to be played when certain specific words or certain specific subtitles appear, the server can filter out the candidate music video frames that do not contain the certain specific words or certain specific subtitles, retain the candidate music video frames that contain the certain specific words or certain subtitles as the music video frames, and directly use the time points of the music video frames as the video content matching time points of the video.

[0109] In this embodiment, the server can determine the video content matching time point based on the candidate music video frame, so that the determined video content matching time point meets the video picture music condition. Since the video picture music condition can be flexibly set based on the user, the video after music replacement can meet the user's needs, which improves the flexibility of audio replacement to a certain extent.

[0110] In one embodiment, the video processing method further includes: determining the audio rhythm points of the original background music based on the audio features of the original background music; and obtaining the audio matching time points of the original background music according to the time points of the audio rhythm points.

[0111] Among them, the audio rhythm point can refer to the point for marking the rhythm of the original background music, and for the original background music, it can include multiple audio rhythm points. Specifically, when determining the audio matching time point, the server can determine the audio rhythm point of the original background music based on the audio features of the original background music, and obtain the audio matching time point of the original background music according to the time point of the audio rhythm point. In the specific implementation, when the server determines the audio rhythm point, if there are many audio rhythm points determined, the audio rhythm points can also be screened, only some audio rhythm points are retained, and the audio matching time point is obtained by using the time points of some audio rhythm points. When the server screens the audio rhythm points, it can be screened according to the set rhythm point interval time condition. If the time interval between two adjacent audio rhythm points reaches the time interval threshold, it is considered that the rhythm point interval time condition is met. If the two adjacent audio rhythm points do not reach the time interval threshold, it means that the rhythm point interval time condition is not met, and one of the audio rhythm points needs to be discarded.

[0112] In this embodiment, the server can determine the audio matching time point in combination with the audio features, that is, the determined audio matching time point matches the audio rhythm point, so that when the background music is replaced, the replacement can be carried out according to the music rhythm, thereby improving the accuracy of replacing the original background music.

[0113] In one embodiment, the original background music in the video is replaced according to the audio replacement time point with the audio to be replaced, and the video after the background music replacement is obtained, including: determining the playback duration of the original background music; determining the audio replacement segment from the audio to be replaced based on the playback duration; according to the audio replacement time point, replacing the original background music in the video with the audio replacement segment, and obtaining the video after the background music replacement is obtained; wherein, in the video after the background music replacement, the audio replacement time point serves as the audio playback start time point of the audio replacement segment.

[0114] Among them, the playing duration may refer to the duration of the original background music from the start to the stop of the playing. The audio replacement segment is determined based on the audio to be replaced, and the playing duration of the audio replacement segment is the same as the playing duration of the original background music. Specifically, the server can determine the playing duration of the original background music through the music playing starting point of the original background music and the music playing ending point of the original background music, and determine the audio replacement segment from the audio to be replaced according to the playing duration. When the duration of the audio to be replaced is the same as the playing duration, the audio to be replaced can be directly determined as the audio replacement segment. When the duration of the audio to be replaced is greater than the playing duration, audio interception processing can be performed based on the audio to be replaced to obtain an audio replacement segment with the same duration as the original background music. The server can further replace the original background music in the video with the audio replacement segment according to the audio replacement time point, and obtain the video after the background music replacement processing. In the video after the background music replacement processing, the audio playing starting time point of the audio replacement segment is the audio replacement time point.

[0115] In this embodiment, the server determines the audio replacement segment according to the playback duration, and replaces the original background music in the video with the audio replacement segment according to the audio replacement time point, thereby obtaining the video after the background music replacement processing, thereby ensuring the compatibility of the video after the audio replacement and its background music, and improving the accuracy of the replacement of the original background music.

[0116] The present application also provides an application scenario, and the application scenario applies the above-mentioned video recognition method. Specifically, the application of the video recognition method in the application scenario is as follows:

[0117] With the development of computer technology, the rise of short videos has led to the emergence of many popular videos. However, many videos uploaded by users often come with music. With the growing awareness of copyright, music copyright issues may arise and lead to some disputes. Since music belongs to the type to be replaced, there is a risk of illegal use, which eventually leads to the removal of videos uploaded by users.

[0118] Currently, in dealing with copyright issues, there are some methods that identify the scene elements of the video. For example, if the items in the video that contain Spring Festival elements such as lanterns are identified, songs with a Spring Festival atmosphere will be recommended. Although this can solve copyright issues, the accuracy of the recommendation is not high, which makes the new background music of the replaced video not very compatible with the video.

[0119] Based on this, the video processing method provided in this embodiment determines the audio to be replaced that is compatible with the video and does not pose a risk of illegal use, and replaces the original background music according to the determined audio replacement time point. This not only makes the video after the background music replacement no longer pose a risk of illegal use, but also makes the replaced video more compatible with the background music it includes, thereby improving the accuracy of background music replacement.

[0120] When the user sends the video to the server through the terminal, the server can upload the video to the editing interface. The specific display of the editing page is as follows Fig.11 As shown, the server can extract the original background music for the video on the editing page, and perform frame processing on the video to obtain multiple frames of images. For the extracted original background music, the server can extract the audio features of the original background music, determine the melody of the original background music, and detect whether the original music has copyright. When it is determined that the original background music has no copyright, the server will recommend a series of music to the terminal for selection. Fig.12 As shown, when the server detects that the original background music has no copyright, it will recommend replacement music 1 and replacement music 2 for the user to choose. The user only needs to click the button where the replacement music is located to replace the original background music. For example, if you click the button where replacement music 1 is located, you can use replacement music 1 to replace the original background music. If you click replacement music 2, you can use replacement music 2 to replace the original background music.

[0121] Among them, Fig.13 As shown, it is a business flow chart of the video processing method provided by an embodiment of the present application. In the business process for video processing, step 1301, the user uploads a video. Among them, the video uploaded by the user can be obtained by the user himself, such as the user can shoot and obtain the video according to his own preferences. The terminal sends the user-made video to the server for processing step 1302. The server can obtain the video sent by the terminal, and identify the original background music according to the melody, and detect whether the original background music is copyrighted. When specifically detecting whether the original background music is copyrighted, the server can first determine the audio features of the original background music, and find the recognition audio that is adapted to the audio features in the music library database, that is, find the recognition audio that has a high degree of match with the original background music, and determine whether the recognition audio is copyrighted.

[0122] In step 1303, the server determines whether the original background music has copyright. If it is determined that the original background music has copyright, the video processing flow can be terminated. If it is determined that the original background music does not have copyright, the process proceeds to step 1304. In step 1304, the server can determine audio features and video features, wherein audio features can include music rhythm, melody, chords, structure, and style, etc., and video features can include video tonality, image features, and text features, etc. In step 1305, the server can also identify the music start time point and video action time point, wherein the music start time point can be the time when the original background music in the video starts to play, and the video action time point can refer to the time point when a specific action appears in the video. The music start time point and the video action time point can be used to set the music replacement time point of the music to be replaced before the music is replaced.

[0123] In step 1306, the server may determine a recommended music list in combination with the audio features and the video features, wherein the recommended music list may include the music to be replaced. When determining the music list based on the audio features and the video features, the server may first determine a reference music list that matches the audio features in a pre-set legal music database based on the audio features, and then determine a recommended music list from the reference music list based on the video features. The server may also first determine a reference music list that matches the video features in a pre-set legal music database based on the video features, and then determine a recommended music list from the reference music list in combination with the audio features. The server may also perform feature fusion based on the audio features and the video features to obtain fused audio and video features, and determine a recommended music list from the legal music database based on the fused audio and video features.

[0124] Step 1307, replace the music according to the music selected by the user and the recorded music start time point. After determining the recommended music list, the server can send the recommended music list to the terminal, and the user can select music based on the recommended music list. After the user selects music based on the recommended music list, the server can determine the music replacement time point of the music according to the determined music start time point and video action time point, and replace the original background music in the video with the music selected by the user according to the music replacement time point, and obtain the video after the background music is replaced. Step 1308, the server can return the video after the background music is replaced to the user.

[0125] Among them, Fig.14 As shown, it is a flowchart of determining the copyright of the original background music according to the audio features of the original background music. Fig.14The flowchart shown includes: extracting audio content; converting audio analog signals to digital signals; extracting audio features; constructing audio fingerprints; retrieving similarity calculations; returning corresponding song lists and matching rankings; finding the copyright of the song with the highest matching degree; if there is no copyright, then finding the copyright in the song list based on the similarity. Specifically, the server can process the video and extract the audio content in the video. The audio content is an audio analog signal. The server can input the proposed audio content into a pre-set audio analysis algorithm or audio analysis model for processing to convert the audio analog signal into a digital signal. Based on the digital signal, the server can extract audio features. The extracted audio features may include melody, rhythm, chords, structure, etc. The server can determine the audio fingerprint based on the audio features and perform similarity calculations. When performing similarity calculations, the server can perform similarity calculations based on the audio fingerprints of the original background music and the audio fingerprints of the music in the music library database, obtain the similarity between the original background music and each piece of music in the music library database, and select some music from the music library database according to the similarity to obtain a song list. The server may first select the music in the song list that is most similar to the original background music according to the degree of similarity between each piece of music in the song list and the original background music to determine whether there is copyright. If the music in the song list that is most similar to the original background music does not have copyright, the server may search for the copyright of the music in the song list in order according to the similarity until the copyrighted music is found.

[0126] like Fig.15As shown, it is a flowchart of the video processing method provided by an embodiment of the present application, and the flowchart includes a result display part, a behavior log part, a feature engineering part, a recall part, and a sorting part. Among them, the result display part. It is used to display the recommended music that finally passes the result. The user can select the music he needs from the recommended music based on the result display part. After the user selects the music he needs, the result display part can record the user's selection record and output the selection record to the behavior log part. The behavior log part is used to receive the historical data of the user's past music selection sent by the result display part, and the historical data of the user's past music selection can be output to the feature engineering part. The feature engineering part can extract labels for various features and provide data for various recommendation algorithms later. The feature engineering part can specifically extract labels for video features, audio features, and historical data of users' past music selections, and use these feature labels as input data for various subsequent recommendation algorithms. After extracting the labels, the feature engineering part can send them to the recall part. The recall part can use data engineering and algorithm methods as well as the feature tags input by the feature engineering part to lock in a specific candidate set from the music library database and complete the preliminary screening of the music recommendation list, which to a certain extent determines the efficiency of the sorting stage and the quality of the recommendation results. The sorting part can perform more refined scoring and sorting on the music recommendation list to obtain a final set of recommended music.

[0127] like Fig.16 As shown, it is a timing diagram of the video processing method provided by the present application. Among them, in the video processing method provided by the present application, in the video processing scenario, a terminal and a server may be involved, wherein a music recommendation system may be run in the server, and the determination of the music recommendation list and the identification of the copyright involved in the video processing process may be completed by the music recommendation system in the server.

[0128] Among them, the terminal can upload the material to be identified to the server, and the material to be identified can refer to the training material. The developer can upload the training material to the terminal in advance, and the terminal can further send it to the server. Specifically, after receiving the training material uploaded by the developer, the terminal can forward the training material to the server. Among them, the training material can include character expressions extracted from various films, and the expressions can include disgust, fear, sadness, surprise, happiness, etc., and can also include character actions, key items, scenery, etc.

[0129] The server can perform feature training based on the material that needs to be identified. Considering the need to identify the video features, audio features, etc. of the video, the server can train the neural network model based on the training material to obtain a neural network model that can be used for video feature and audio feature recognition. When performing video feature recognition and audio feature recognition, it is only necessary to input the image, audio data, etc. into the pre-trained neural network model to determine the corresponding video features, audio features, etc.

[0130] Furthermore, the user can upload the video to be identified to the server through the terminal, and the video to be identified can be obtained by the user, such as the user can synthesize the video through video editing software to obtain the self-made video. The server can extract audio features from the video to be identified and store the audio features, and extract video features from the video to be identified and store the video features.

[0131] Among them, when the server identifies the video features, it can select video clips based on the video. The selected video clips need to be able to identify the emotions of the main characters, scenery, key items, and video tonality. The server can first identify the characters in the video clips. When performing character recognition, it can perform video face detection, preprocessing of video face images, and extract features of video face images. For example, the main characters can be found first, such as the protagonist A in the crowd, the main face in the video is her, and the proportion of other characters is very small, and their emotions are identified, and the emotional characteristics of the main characters are saved, such as happy, sad, indifferent, etc. The server can also extract features of the actions of the video characters, such as street dance, parachuting, heart-shaped, etc. The server can also identify key items in the video, such as cakes, fireworks, roses, and red wine. Among them, the features corresponding to cakes and fireworks can be birthday celebrations, and the features corresponding to roses and red wine can be romance. The server can further identify scene features, such as ski resorts, beaches, seas, rural fields, etc.

[0132] Among them, there are relatively mature interfaces for physical object recognition. The corresponding interfaces can be directly called for background recognition, and the recognized video background style content features can be stored on the server. When determining the tonality features, the server can understand the video content and the tonality of the corresponding music style, such as festive, lazy, sports, ancient style, aesthetic, etc. For example, if there are many people in the video and they all have happy expressions and fireworks, the festive tonality can be determined through these features; for example, riding a bicycle in a rural field can be determined as a lazy rural tonality.

[0133] The music recommendation system in the server can detect the copyright of the background music in the video to be identified based on the extracted detection features, including video features and audio features, and use the recommendation algorithm to match and find a recommended music list that meets the features. When the music recommendation system detects that the background music in the video to be identified is not copyrighted, it can directly return the recommended music list to the terminal. The terminal can display the recommended music list to the user, the user selects the corresponding music, and sends the selected music to the server based on the terminal. For example, the music identifier corresponding to the selected music can be sent to the server. The server can replace the new music according to the music selected by the user and the starting time point of the detected music, and synthesize a new video. The synthesized new video is returned to the terminal so that the user can share it to the video platform through the terminal.

[0134] The video processing method provided in this application will not only identify characteristic elements of music, such as rhythm, style and other characteristic scenes, but will also classify the tone of the video, such as festive and funny, design style, fresh and clean, etc., so that recommendations can be more accurate.

[0135] The video processing method provided by the present application is intelligently identified by the terminal during the upload stage. First, the corresponding song is found through melody recognition, and then the copyright of the song is checked. If not, the music is identified. The type label of the music is determined by identifying the audio features. The audio features may include music rhythm and style, etc.; then the video features of the video are identified. The video features may include video type, video tonality, action, etc., such as whether the video type is dance or plot, such as whether the video tonality is festive and funny, black humor, fresh, etc. In the subsequent audio replacement, in order to accurately replace the card points of the subsequent music, considering that the rhythm of the action and the music need to be consistent, the time points of the actions corresponding to the video, such as dance actions and the rhythm of the music are processed. The recommended music list is found through the previous music tags, video features, etc. After the user selects, the intelligent music replacement is performed according to the time point determined later to ensure that the rhythm points of the music match the video.

[0136] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0137] Based on the same inventive concept, the embodiment of the present application also provides a video processing device for implementing the above-mentioned video processing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more video processing device embodiments provided below can refer to the limitations on the video processing method above, and will not be repeated here.

[0138] In one embodiment, Fig.17 As shown, a video processing device 1700 is provided, comprising: a candidate replacement audio determination module 1702, an audio replacement time point determination module 1704 and a replacement module 1706, wherein:

[0139] The candidate replacement audio determination module 1702 is used to obtain the video to be processed, and when the original background music of the video belongs to the type to be replaced, determine at least one candidate replacement audio for the original background music; the candidate replacement audio is compatible with the video and does not belong to the type to be replaced.

[0140] The audio replacement time point determination module 1704 is used to determine the audio replacement time point of the audio to be replaced for the original background music based on the video content matching time point of the video and the audio matching time point of the original background music in response to the terminal selecting the audio to be replaced from at least one candidate replacement audio.

[0141] The replacement module 1706 is used to replace the original background music in the video with the audio to be replaced according to the audio replacement time point to obtain the video after the background music is replaced.

[0142] In one embodiment, the audio replacement time point determination module 1704 is also used to obtain the video content matching time point of the video and the audio matching time point of the original background music; when the audio matching time point matches the video content matching time point, the audio matching time point is used as the audio replacement time point of the audio to be replaced for the original background music.

[0143] In one embodiment, the audio matching time point includes the audio starting time point; the video content matching time point includes the video picture time point of the video to be matched with music; the audio replacement time point determination module 1704 is also used to use the audio starting time point as the audio replacement time point of the audio to be replaced for the original background music when the audio starting time point is the same as the video picture time point.

[0144] In one embodiment, the audio matching time point also includes at least one audio rhythm time point; the video screen time point includes at least two; the audio replacement time point determination module 1704 includes an audio replacement time point adjustment module; the audio replacement time point adjustment module is used to determine the time difference between at least one audio rhythm time point and the respective unmatched video screen time point when at least one audio rhythm time point does not match the video screen time point in the video; when the time difference meets the replacement time point adjustment replacement condition, determine the adjustment time based on the time difference; adjust the audio replacement time point according to the adjustment time to obtain the adjusted audio replacement time point. The replacement module 1706 is also used to replace the original background music in the video according to the adjusted audio replacement time point through the audio to be replaced, and obtain the video after the background music replacement processing.

[0145] In one embodiment, the video processing device also includes a correction module; the correction module is used to correct the audio matching time point based on the video content matching time point when the audio matching time point does not match the video content matching time point to obtain the corrected audio matching time point; and use the corrected audio matching time point as the audio replacement time point of the audio to be replaced for the original background music.

[0146] In one embodiment, the candidate replacement audio determination module 1702 is also used to determine the original background music of the video; when the original background music belongs to the type to be replaced, at least one candidate replacement audio is determined based on the audio features of the original background music and the video features of the video; wherein the candidate replacement audios are respectively adapted to the audio features and the video features, and the candidate replacement audios do not belong to the type to be replaced.

[0147] In one embodiment, the candidate replacement audio determination module 1702 is also used to extract audio features from the original background music; based on the audio features, query the music library database for recognized audio that matches the original background music; when the recognized audio does not have copyright, determine copyrighted audio that matches the audio features and has copyright; when the recognized audio has copyright, determine the recognized audio as copyrighted audio; and obtain at least one candidate replacement audio that matches the video features of the video from the determined copyrighted audio.

[0148] In one embodiment, the candidate replacement audio determination module 1702 determines the login account corresponding to the video and obtains the historical soundtrack features of the login account; fuses the historical soundtrack features with the video features of the video to obtain fusion features; and determines at least one candidate replacement audio that matches the fusion features from the copyrighted audio.

[0149] In one embodiment, the candidate replacement audio determination module 1702 includes an audio feature extraction module; the audio feature extraction module is used to extract features of the video according to at least one content element dimension to obtain content element features under at least one content element dimension; and obtain video features of the video based on the content element features.

[0150] In one embodiment, the audio replacement time point determination module 1704 is also used to send at least one candidate replacement audio to the terminal; at least one candidate replacement audio is used to be displayed in a selectable manner at the terminal; an audio selection instruction fed back by the terminal is obtained, and the audio to be replaced selected by the terminal is determined according to the audio selection instruction; based on the video content matching time point of the video and the audio matching time point of the original background music, the audio replacement time point of the audio to be replaced relative to the original background music is determined.

[0151] In one embodiment, the video processing device also includes a video content matching time determination module; the video content matching time determination module is used to determine candidate music video frames from the video; determine the music video frames from the candidate music video frames according to the music conditions of the video screen; and obtain the video content matching time of the video according to the time point of the music video frame.

[0152] In one embodiment, the video processing device also includes an audio matching time point determination module; the audio matching time point determination module is used to determine the audio rhythm point of the original background music based on the audio features of the original background music; and obtain the audio matching time point of the original background music according to the time point of the audio rhythm point.

[0153] In one embodiment, the replacement module 1706 is also used to determine the playback duration of the original background music; determine the audio replacement segment from the audio to be replaced based on the playback duration; replace the original background music in the video with the audio replacement segment according to the audio replacement time point, and obtain the video after the background music replacement processing; wherein, in the video after the background music replacement processing, the audio replacement time point is used as the audio playback start time point of the audio replacement segment.

[0154] Each module in the above video processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0155] In one embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.18 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video processing method is implemented.

[0156] Those skilled in the art will understand that Fig.18 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0157] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0158] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0159] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0160] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0161] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0162] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video processing method, characterized in that: The method comprises: Acquire a video to be processed, and when the original background music of the video belongs to the type to be replaced, determine at least one candidate replacement audio for the original background music; the candidate replacement audio is compatible with the video and does not belong to the type to be replaced; In response to the terminal selecting the audio to be replaced from the at least one candidate replacement audio, determining an audio replacement time point of the audio to be replaced for the original background music based on a video content matching time point of the video and an audio matching time point of the original background music; The original background music in the video is replaced by the audio to be replaced according to the audio replacement time point, so as to obtain the video after the background music is replaced.

2. The method according to claim 1, characterized in that The step of determining the audio replacement time point of the audio to be replaced for the original background music based on the video content matching time point of the video and the audio matching time point of the original background music includes: Obtaining a video content matching time point of the video and an audio matching time point of the original background music; When the audio matching time point matches the video content matching time point, the audio matching time point is used as the audio replacement time point of the audio to be replaced for the original background music.

3. The method according to claim 2, characterized in that The audio matching time point includes the audio starting time point; the video content matching time point includes the video picture time point of the video picture to be matched with music in the video; When the audio matching time point matches the video content matching time point, using the audio matching time point as the audio replacement time point of the audio to be replaced for the original background music includes: When the audio start time point is the same as the video picture time point, the audio start time point is used as the audio replacement time point of the audio to be replaced for the original background music.

4. The method according to claim 3, characterized in that The audio matching time point further includes at least one audio rhythm time point; the video picture time point includes at least two; the method further includes: When the at least one audio rhythm time point does not match the video picture time point in the video, respectively determining the time difference between the at least one audio rhythm time point and the respective non-matching video picture time point; When the time difference satisfies a replacement time point adjustment condition, determining an adjustment time based on the time difference; Adjusting the audio replacement time point according to the adjustment time to obtain an adjusted audio replacement time point; The replacing the original background music in the video according to the audio to be replaced at the audio replacement time point to obtain the video after the background music is replaced includes: The original background music in the video is replaced by the audio to be replaced according to the adjusted audio replacement time point, so as to obtain the video after the background music is replaced.

5. The method according to claim 2, characterized in that: The method further comprises: When the audio matching time point does not match the video content matching time point, correcting the audio matching time point based on the video content matching time point to obtain a corrected audio matching time point; The corrected audio matching time point is used as the audio replacement time point of the audio to be replaced for the original background music.

6. The method according to claim 1, characterized in that When the original background music of the video belongs to the type to be replaced, determining at least one candidate replacement audio for the original background music includes: Determining the original background music of the video; When the original background music belongs to the type to be replaced, determining at least one candidate replacement audio according to the audio features of the original background music and the video features of the video; The candidate replacement audio is respectively adapted to the audio feature and the video feature, and the candidate replacement audio does not belong to the type to be replaced.

7. The method according to claim 6, characterized in that When the original background music belongs to the type to be replaced, determining at least one candidate replacement audio according to the audio features of the original background music and the video features of the video, comprises: Extracting audio features from the original background music; Based on the audio features, searching a music library database for recognized audio that matches the original background music; When the identified audio does not have copyright, determining copyrighted audio that matches the audio feature and has copyright; When the identified audio has copyright, determining the identified audio as copyrighted audio; At least one candidate replacement audio that matches the video feature of the video is obtained from the determined copyrighted audio.

8. The method according to claim 7, characterized in that The step of obtaining at least one candidate replacement audio that matches the video features of the video from the determined copyrighted audio comprises: Determine the login account corresponding to the video, and obtain the historical soundtrack features of the login account; Fusion the historical soundtrack feature with the video feature of the video to obtain a fusion feature; At least one candidate replacement audio that matches the fusion feature is determined from the copyrighted audio.

9. The method according to claim 6, characterized in that The method further comprises: Extracting features of the video according to at least one content element dimension to obtain content element features under the at least one content element dimension; The video feature of the video is obtained according to the content element feature.

10. The method according to claim 1, characterized in that In response to the terminal selecting the audio to be replaced from the at least one candidate replacement audio, determining the audio replacement time point of the audio to be replaced for the original background music based on the video content matching time point of the video and the audio matching time point of the original background music, comprises: Sending the at least one candidate replacement audio to the terminal; the at least one candidate replacement audio is used to be displayed in a selectable manner at the terminal; Acquire an audio selection instruction fed back by the terminal, and determine the audio to be replaced selected by the terminal according to the audio selection instruction; Based on the video content matching time point of the video and the audio matching time point of the original background music, the audio replacement time point of the audio to be replaced relative to the original background music is determined.

11. The method according to any one of claims 1 to 10, characterized in that: The method further comprises at least one of the following: Determine candidate soundtrack video frames from the video; determine soundtrack video frames from the candidate soundtrack video frames according to the soundtrack conditions of the video screen; obtain the video content matching time point of the video according to the time point of the soundtrack video frame; Based on the audio features of the original background music, the audio rhythm points of the original background music are determined; and according to the time points of the audio rhythm points, the audio matching time points of the original background music are obtained.

12. The method according to any one of claims 1 to 10, characterized in that: The replacing the original background music in the video according to the audio to be replaced at the audio replacement time point to obtain the video after the background music is replaced includes: Determining the playing time of the original background music; Determining an audio replacement segment from the audio to be replaced based on the playback duration; According to the audio replacement time point, the original background music in the video is replaced with the audio replacement segment to obtain a video after the background music is replaced; Among them, in the video after the background music replacement processing, the audio replacement time point is used as the audio playback start time point of the audio replacement segment.

13. A video processing device, characterized in that: The device comprises: A candidate replacement audio determination module is used to obtain a video to be processed, and when the original background music of the video belongs to the type to be replaced, determine at least one candidate replacement audio for the original background music; the candidate replacement audio is compatible with the video and does not belong to the type to be replaced; An audio replacement time point determination module, configured to determine, in response to the terminal selecting the audio to be replaced from the at least one candidate replacement audio, an audio replacement time point of the audio to be replaced for the original background music based on a video content matching time point of the video and an audio matching time point of the original background music; The replacement module is used to replace the original background music in the video with the audio to be replaced according to the audio replacement time point to obtain the video after the background music is replaced.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images