Multimedia data processing method, device, equipment and computer readable storage medium

Abstract

The present application provides a multimedia data processing method, device, equipment and computer-readable storage medium; the method includes: obtaining video data to be matched with music and music data to be processed, performing source separation on the music data to obtain a singing track and an accompaniment track; determining non-singing time information based on the singing track, and determining a reference start and end point set based on the accompaniment track; determining a target start and end point set from the reference start and end point set based on the non-singing time information; determining a music matching start point and a music matching end point from the target start and end point set based on the first playback duration of the video data to be matched with music; determining the target music data between the music matching start point and the music matching end point in the music data as the target music matching data for the video data. Through the present application, the efficiency, accuracy and versatility of video music matching can be improved.

Description

Technical Field

[0001] The present application relates to information processing technology, and in particular to a multimedia data processing method, device, equipment and computer-readable storage medium. Background Art

[0002] In recent years, with the development of mobile networks and Internet technologies, smart terminals have become a necessity in people's daily lives and work. Along with this, film and television works and short videos have emerged and prospered, and then video intelligent understanding and editing technology has emerged. For example, in order to increase the effect of short videos, music and cartoon avatars can be added to the produced short videos.

[0003] In the application scenario of adding music to a video, it is particularly important to select the start and end points of the music. In the related art, when determining the start and end points of the music, the method of manual annotation can be used, or the method of determining the start and end points based on the paragraphs of the lyrics can be used. However, the manual annotation method is too inefficient, and determining the start and end points based on the paragraphs of the lyrics is not suitable for self-created songs and other music without lyrics, and has poor versatility. Summary of the invention

[0004] The embodiments of the present application provide a multimedia data processing method, device and computer-readable storage medium, which can improve the efficiency, accuracy and versatility of video soundtrack.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a multimedia data processing method, including:

[0007] Acquire the video data to be matched with music and the music data to be processed, perform source separation on the music data, and obtain a singing track and an accompaniment track;

[0008] Determine non-singing time information based on the singing track, and determine a reference start and end point set based on the accompaniment track;

[0009] Based on the non-singing time information, determining a target start-end point set from the reference start-end point set;

[0010] Based on the first playback duration of the video data to be matched with music, determining a music matching start point and a music matching end point from the target start and end point set;

[0011] The target music data between the music start point and the music end point in the music data is determined as the target music data of the video data.

[0012] The present application provides a multimedia data processing device, including:

[0013] A source separation module is used to obtain the video data to be matched with music and the music data to be processed, and perform source separation on the music data to obtain a singing track and an accompaniment track;

[0014] A first determination module is used to determine non-singing time information based on the singing track, and to determine a reference start and end point set based on the accompaniment track;

[0015] A second determination module is used to determine a target start-end point set from the reference start-end point set based on the non-singing time information;

[0016] A third determining module is used to determine a music matching start point and a music matching end point from the target start and end point set based on a first playback duration of the video data to be matched with music;

[0017] The fourth determination module is used to determine the target music data between the soundtrack start point and the soundtrack end point in the music data as the target soundtrack data of the video data.

[0018] In some embodiments, the information source separation module is further used to:

[0019] Performing time-frequency conversion on the music data to obtain a frequency spectrum amplitude spectrum of the music data;

[0020] Extracting features from the frequency spectrum amplitude spectrum to obtain singing features and accompaniment features, and merging the singing features and the accompaniment features to obtain merged features;

[0021] Determining a singing mask and an accompaniment mask based on the combined feature, the singing feature and the accompaniment feature;

[0022] Using the singing mask and the accompaniment mask to respectively perform mask calculations on the frequency spectrum amplitude spectrum, and correspondingly obtain the singing frequency spectrum amplitude and the accompaniment frequency spectrum amplitude;

[0023] The singing spectrum amplitude and the accompaniment spectrum amplitude are respectively converted into frequency-time, and a singing track and an accompaniment track are obtained accordingly.

[0024] In some embodiments, the first determining module is further configured to:

[0025] Performing voice activity detection on the singing track, positioning the impulse signal in the singing track, and determining the singing time information in the singing track;

[0026] Based on the singing time information, non-singing time information in the singing track is determined.

[0027] In some embodiments, the first determining module is further configured to:

[0028] Get the trained audio event detection model;

[0029] Inputting the singing audio track into the audio event detection model to obtain singing time information in the singing audio track;

[0030] Based on the singing time information, non-singing time information in the singing track is determined.

[0031] In some embodiments, the first determining module is further configured to:

[0032] Obtaining a frequency spectrum feature sequence and a preset sliding window length of the accompaniment audio track;

[0033] Perform sliding window processing on the spectrum feature sequence based on the sliding window length to obtain multiple sliding window results;

[0034] Divide each sliding window result into N sub-results, and determine the energy mean of the N sub-results in each sliding window result;

[0035] Determining a target sliding window result from the multiple sliding window results based on the energy averages of the N sub-results in the respective sliding window results;

[0036] A reference start and end point set is determined based on the target sliding window result.

[0037] In some embodiments, the first determining module is further configured to:

[0038] Determine the sliding window result whose energy mean value of N sub-results satisfies the decreasing condition as the first target sliding window result;

[0039] Determine the sliding window result whose energy mean value of the N sub-results satisfies the increasing condition as the second target sliding window result;

[0040] Determine the right boundary point of the first target sliding window result as a reference end point;

[0041] The left boundary point of the second target sliding window result is determined as a reference starting point.

[0042] In some embodiments, the non-singing time information includes a non-singing time interval, and the second determination module is further used to:

[0043] A target start-end point set is determined based on reference start-end point in the reference start-end point set that falls within the non-singing time interval, wherein the target start-end point set includes a target starting point and a target ending point.

[0044] In some embodiments, the third determination module is further used to:

[0045] Determining a first playing duration of the video data to be matched with music;

[0046] Determine each interval duration between a target starting point and each target ending point after the target starting point in the target starting and ending point set;

[0047] Determine each duration difference between each interval duration and the first playback duration;

[0048] When there is at least one target interval duration whose duration difference is less than a preset threshold, the target starting point corresponding to the minimum value of the at least one target interval duration is determined as the soundtrack starting point, and the target ending point corresponding to the minimum value is determined as the soundtrack ending point.

[0049] In some embodiments, the apparatus further comprises:

[0050] A first acquisition module, configured to acquire the track type of the music data to be processed when there is no at least one target interval duration whose duration difference is less than a preset threshold;

[0051] An output module, configured to determine a plurality of candidate music data based on the track type, and output a plurality of music identifiers of the plurality of candidate music data;

[0052] A fifth determination module, configured to determine a selected target music identifier based on receiving a selection operation for the music identifier;

[0053] The sixth determination module is used to determine the music data to be processed based on the target music identifier.

[0054] In some embodiments, the apparatus further comprises:

[0055] A seventh determination module, configured to determine a second playback duration of the target music data based on the music start point and the music end point;

[0056] An adjustment module, configured to adjust the playing time of the video data to be matched with music based on the second playing time to obtain adjusted video data;

[0057] The synthesis module is used to synthesize the adjusted video data and the target music data to obtain the video data with music.

[0058] An embodiment of the present application provides a computer device, including:

[0059] A memory for storing executable instructions;

[0060] The processor is used to implement the multimedia data processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0061] An embodiment of the present application provides a computer-readable storage medium storing executable instructions. When the executable instructions are executed by a processor, the multimedia data processing method provided by the embodiment of the present application is implemented.

[0062] A computer program product provided in an embodiment of the present application includes a computer program or an instruction. When the computer program or the instruction is executed by a processor, the multimedia data processing method provided in an embodiment of the present application is implemented.

[0063] The embodiments of the present application have the following beneficial effects:

[0064] After obtaining the video data to be matched with music and the music data to be processed, the music data is firstly separated from the source to obtain the singing track and the accompaniment track, and then the non-singing time information is determined based on the singing track, and the reference start and end point set is determined based on the accompaniment track, and then the target start and end point set is determined from the reference start and end point set based on the non-singing time information, and the music start point and the music end point are determined from the target start and end point set based on the first playback time of the video data to be matched with music, and finally the target music data between the music start point and the music end point in the music data is determined as The target soundtrack data of the video data; in this way, the singing track and the accompaniment track are separated for the music data to be processed, so that the non-singing time information in the whole music can be accurately located even in scenes without lyrics, thereby improving the versatility of the video soundtrack; then the energy fade-in and fade-out time position of the accompaniment is determined based on the accompaniment track, so as to obtain the reference starting and ending points suitable for the soundtrack, and finally the final soundtrack starting and ending points are selected from the reference starting points based on the detected non-singing time segments, which can both ensure the integrity of the singing and locate the position where the energy fades in and out of the song rhythm, thereby improving the efficiency and accuracy of the video soundtrack. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 is a schematic diagram of the network architecture of the multimedia data processing system 100 provided in an embodiment of the present application;

[0066] Figure 2 is a schematic diagram of the structure of the terminal 400 provided in an embodiment of the present application;

[0067] Figure 3 is a flowchart of a multimedia data processing method provided in an embodiment of the present application;

[0068] Figure 4 It is a schematic diagram of the implementation process of source separation of music data provided by an embodiment of the present application;

[0069] Figure 5 A schematic diagram of another implementation flow of the multimedia data processing method provided in an embodiment of the present application;

[0070] Figure 6 The audio graph and spectrum graph of the Vocal track extracted from the first version of the short video provided in the embodiment of the present application;

[0071] Figure 7 The audio graph and spectrum graph of the BGM track extracted from the first version of the short video provided in the embodiment of the present application;

[0072] Figure 8 An audio graph of the final short video clip generated after the soundtrack provided in the embodiment of the present application;

[0073] Fig. 9 A schematic diagram of another implementation flow of the multimedia data processing method provided in the embodiment of the present application;

[0074] Fig.10 This is the Unet network architecture diagram;

[0075] Fig.11 A schematic diagram of the implementation process of source separation using the Unet model provided in an embodiment of the present application;

[0076] Fig.12 A schematic diagram of a non-silent segment located by using VAD provided in an embodiment of the present application;

[0077] Fig.13 A schematic diagram of the termination point positioning based on energy detection provided in an embodiment of the present application;

[0078] Fig.14 A schematic diagram of non-vocal paragraph screening points provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0080] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0081] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0082] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0083] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0084] 1) Source separation: In an entire audio, there may be multiple audio signals mixed together, so the entire audio is generated by mixing multiple audio signals. Source separation is to separate the mixed audio signals through signal processing or other algorithms, extract the specified type of audio signal sequence from the mixed signal, and finally generate a separate audio file.

[0085] 2) Vocals refers to the audio track of the singing part, which is written into a waveform sound file (wav, wave) after the signal source is separated from the entire mixed audio. The file stores the human singing signal separated from the entire music.

[0086] 3) Background music (Bgm) refers to the accompaniment track, which is the accompaniment signal separated from the audio track of the entire song through the signal source and written into the wav file.

[0087] 4) Unet is one of the earliest algorithms that used a fully convolutional network for semantic segmentation. The use of a symmetrical U-shaped structure containing a compression path and an expansion path was very innovative at the time and influenced the design of several subsequent segmentation networks to a certain extent. The name of the network is also taken from its U-shaped shape.

[0088] 5) Voice Activity Detection (VAD) is widely used in speech coding, noise reduction and ASR scenarios. Here we are talking about speech / non-speech (non-speech / silence) detection. A VAD system usually consists of two parts: feature extraction and speech / non-speech judgment.

[0089] 6) Mel frequency, a nonlinear frequency scale based on the human ear's sensory judgment of equidistant pitch changes. It is an artificially set frequency scale that can better cater to changes in the human ear's auditory perception threshold during signal processing. In the field of audio processing, many basic audio features are calculated using mel frequency.

[0090] 7) WebRTC Voice Activity Detector (VAD) python interface (webrtcvad), which can effectively support python2 and python3, and it can distinguish between silent frames and non-silent frames in a speech segment.

[0091] In order to better understand the multimedia data processing method provided in the embodiment of the present application, the multimedia data processing method for determining the start and end points of the soundtrack in the related art and the existing shortcomings are first explained.

[0092] In the related art, there are at least three multimedia data processing methods for determining the start and end points of music:

[0093] First, the method relies on manual labeling. After determining the highlights or chorus sections of the music, the starting and ending points of the music are manually selected before and after the determined sections, and then the music is cut according to this position.

[0094] Second, perform audio energy detection on the entire original music audio to determine the start and end points of the soundtrack, locate the entire original music audio, and select fade-in and fade-out points on the original audio track.

[0095] Third, in some intelligent production and editing of short video music solutions, in order to avoid singing truncation when selecting the start and end points of the music, the timing of the lyrics is often taken into consideration. The time segments of the lyrics are used to determine which time segments of the entire song do not contain the singer's singing, so as to avoid truncation.

[0096] The disadvantages of the above three technical solutions include the following aspects:

[0097] First, the start and end points of the music marked by human beings are more suitable for the selection of the start and end points of the music for the intelligent production of short video clips. However, the manual marking method is less efficient for the task, increases the cost of the entire intelligent production, and cannot be industrialized, nor can it guarantee real-time performance.

[0098] Second, the start and end points calculated by audio energy detection based on the original audio track of the entire song can locate the position of the fade-in and fade-out effect in terms of overall energy, but it cannot avoid the situation where the selected start and end points are exactly where a line of lyrics has not been sung. If the song is cut according to the selected start and end points, the music of the short video will be cut off at the beginning and end.

[0099] Third, not all songs have lyrics information, especially on short video editing apps, where users upload their own music or recorded songs as background music, which lack lyrics information. Therefore, the third technical solution cannot be universal. In addition, only non-singing segments are used as the starting and ending points, which cannot guarantee a fade-in and fade-out effect at the beginning and end of the soundtrack, and cannot achieve a good audience audio-visual experience.

[0100] Based on the above problems, the embodiments of the present application provide a multimedia data processing method, apparatus, device and computer-readable storage medium, which determines the start and end points of the music by combining the source separation model and the audio energy detection. This can not only avoid the singing truncation phenomenon caused by the start and end points of the music, but also ensure the experience of fading in at the beginning and fading out at the end when editing the music for the intelligently produced short video.

[0101] See also Figure 1 , Figure 1 is a schematic diagram of the network architecture of the multimedia data processing system 100 provided in an embodiment of the present application, such as Figure 1 As shown, the network architecture includes a server 200, a network 300 and a terminal 400. The terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0102] Terminal 400 first obtains the video data to be matched with music, which can be collected by its own image acquisition device, or generated by multiple pictures using a video production tool, or downloaded from the Internet, or sent to the terminal 400 by other terminals. Terminal 400 also obtains the music data to be processed. After obtaining the video data and music data to be matched with music, terminal 400 can separate the source of the music data by itself to obtain the singing track and the accompaniment track, and then determine the time information of the non-singing part based on the singing track, and determine the reference start and end point set of fade-in and fade-out based on the accompaniment track, and then select the target start and end point set that falls into the non-singing time interval from the reference start and end point set, and finally determine the music starting point and music ending point according to the first playback duration of the video data to be matched with music and the target starting point set. After obtaining the music starting point and music ending point, the target music data can be intercepted from the music data, and finally the target music data and the video data to be matched with music are synthesized and processed to obtain the video data after matching with music. Afterwards, the terminal 400 may send the video data with music to the server 200 , so as to publish the video data with music on the video viewing platform.

[0103] based on Figure 1 The network architecture shown can also be a process in which the server 200 implements multimedia data processing. At this time, after acquiring the video data to be matched with music and the music to be processed, the terminal 400 sends the video data and the music data to the server 200. The server 200 performs source separation on the music data to obtain the singing track and the accompaniment track, and then determines the time information of the non-singing part based on the singing track, determines the reference start and end point set of fade-in and fade-out based on the accompaniment track, and then selects the target start and end point set that falls into the non-singing time interval from the reference start and end point set. Finally, the first playback of the video data to be matched with music is performed. The duration and the target starting point set determine the music starting point and the music ending point. After obtaining the music starting point and the music ending point, the target music data can be intercepted from the music data, and finally the target music data and the video data to be synthesized to obtain the video data with music. Then the server 200 sends the video data with music to the terminal 400. When the terminal 400 determines that the video data with music meets its own requirements, it can save the video data with music to the local storage space, and can also send the video data with music to the server 200 to publish the video data with music on the video platform.

[0104] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, a smart car terminal, etc., but is not limited thereto. The terminal 400 may be installed with an application client that can realize the video soundtrack function, for example, a video playback application (A PP, Application), a video editing APP, etc. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0105] Figure 2 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application, and the computer device may be Figure 1 The server 200 or the terminal 400 shown in the embodiment of the present application is described by taking the computer device as the terminal 400 as an example.

[0106] like Figure 2 As shown, the terminal 400 includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 2 Various buses are labeled as bus system 440 .

[0107] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0108] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0109] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0110] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0111] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0112] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0113] A network communication module 452, for reaching other computing devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Authentication (WiFi), and Universal Serial Bus (USB);

[0114] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0115] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0116] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2The multimedia data processing device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a source separation module 4551, a first determination module 4552, a second determination module 4553, a third determination module 4554 and a fourth determination module 4555. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0117] In other embodiments, the device provided in the embodiments of the present application can be implemented in hardware. As an example, the device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the multimedia data processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs) or other electronic components.

[0118] In some embodiments, the terminal or server can implement the multimedia data processing method provided in the embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a video playback APP, a video editing APP, etc.; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0119] The multimedia data processing method provided in the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the terminal provided in the embodiment of the present application.

[0120] An embodiment of the present application provides a multimedia data processing method for composing music for a video. The multimedia data processing method can be executed by a terminal or a server, or by both the terminal and the server. Figure 3 is a flowchart of a multimedia data processing method provided in an embodiment of the present application, which will be combined with Figure 3The various steps of the multimedia data processing method provided in the embodiment of the present application are described.

[0121] Step S101, obtaining video data to be matched with music and music data to be processed, performing source separation on the music data to obtain a singing track and an accompaniment track.

[0122] When step S101 is implemented by the terminal, the video data may be collected and acquired by the terminal's own image acquisition device, or may be generated by the terminal using multiple images, or may be downloaded from a network server, or may be sent to the terminal by other terminals. The music data to be processed may be stored locally in the terminal, or may be downloaded from a network server, or may be recorded by the terminal user himself, etc. When step S101 is implemented by the server, the video data and the music data may be sent by the terminal, or may be obtained by searching and querying based on the video identifier and the music identifier sent by the terminal.

[0123] The video data includes at least a plurality of video frame images, and may include audio data or may not include audio data.

[0124] The music data is separated from the source. In practice, the music data is input into a trained source separation model, and the trained source separation model separates the music data into an accompaniment track and a singing track. Each track defines the properties of the track, such as the timbre, number of channels, input / output ports, volume, etc. The accompaniment track only includes background music data, and the singing track only includes vocal data.

[0125] Step S102, determining non-singing time information based on the singing track, and determining a reference start and end point set based on the accompaniment track.

[0126] When determining the non-singing time information based on the singing audio track, the impulse signal can be detected by VAD, and then the position where the impulse signal does not appear is determined as the non-singing segment. The non-singing time information includes the non-singing time interval, that is, the start time and end time of each non-singing segment.

[0127] When determining the reference start and end point set based on the accompaniment audio track, the fade-in and fade-out positions in the music data can be determined by performing audio energy detection on the accompaniment audio track, that is, determining the reference start and end point set, so as to ensure the experience of fading in at the beginning and fading out at the end of the video data after the music. The reference start and end point set includes a reference start point and a reference end point, and both the reference start point and the reference end point can be represented by time points.

[0128] Step S103, based on the non-singing time information, determining a target start and end point set from the reference start and end point set.

[0129] Since the reference start and end points are determined by energy detection of the accompaniment track without considering the singing part, in order to avoid the phenomenon of singing truncation, when implementing step S103, the reference start and end points in the reference start and end point set that fall into the non-singing time interval can be determined as the target start and end point set, and the target start and end point set includes a target start point and a target end point. The target start and end point set is a subset of the reference start and end point set.

[0130] Step S104: determining a music matching start point and a music matching end point from the target start and end point set based on the first play duration of the video data to be matched with music.

[0131] When this step is implemented, the interval duration between each target starting point and each target ending point after the target starting point is calculated in turn, and the duration difference between each interval duration and the first playback duration is determined in turn, and then the target starting and ending points corresponding to the duration difference with the smallest duration difference and less than a preset threshold are determined as the soundtrack starting and ending points. In this way, the matching degree of the target music data and video data playback duration between the determined soundtrack starting point and soundtrack ending point can be ensured.

[0132] Step S105: determining the target music data between the music start point and the music end point in the music data as the target music data of the video data.

[0133] The music start point and the music end point are respectively a time point, for example, the music start point is 1 minute 50 seconds, and the music end point is 2 minutes 40 seconds, then the music data between 1 minute 50 seconds and 2 minutes 40 seconds is determined as the target music data. The target music data is used to compose music for the video data. In some embodiments, after the target music data for the video data is determined, the target music data is synthesized with the video data to obtain the video data after the music is composed, thereby improving the entertainment and viewing quality of the video.

[0134] In the multimedia data processing method provided in the embodiment of the present application, after obtaining the video data to be matched with music and the music data to be processed, the music data is firstly separated from the source to obtain the singing track and the accompaniment track, and then the non-singing time information is determined based on the singing track, and the reference start and end point set is determined based on the accompaniment track, and then based on the non-singing time information, the target start and end point set is determined from the reference start and end point set, and based on the first playback time of the video data to be matched with music, the music start point and the music end point are determined from the target start and end point set, and finally the music start point and the music end point in the music data are separated. The target music data in the time interval is determined as the target soundtrack data of the video data; in this way, the singing track and the accompaniment track are separated for the music data to be processed, so that the non-singing time information in the whole music can be accurately located even in scenes without lyrics, thereby improving the versatility of video soundtracks; then the energy fade-in and fade-out time position of the accompaniment is determined based on the accompaniment track, so as to obtain a reference start and end point suitable for the soundtrack, and finally the final soundtrack start and end point is selected from the reference starting point based on the detected non-singing time segment, which can both ensure the integrity of the singing and locate the position where the energy fades in and out of the song rhythm, thereby improving the efficiency and accuracy of video soundtracks.

[0135] In some embodiments, the "source separation of the music data to obtain the singing track and the accompaniment track" in the above step S101 can be performed by Figure 4 Steps S1011 to S1015 shown below are combined Figure 4 Each step is explained.

[0136] Step S1011, performing time-frequency conversion on the music data to obtain the frequency spectrum amplitude spectrum of the music data.

[0137] In the embodiment of the present application, the acquired music data is a time-domain audio signal, so in this step, the music data is converted into a time-frequency signal, for example, the time-domain audio signal can be converted into a frequency domain by Fourier transform to obtain a frequency-domain signal of the music data. The frequency-domain signal includes phase information and amplitude information, and in this step, the phase information in the frequency-domain signal needs to be eliminated to obtain the frequency spectrum amplitude spectrum of the music data.

[0138] Step S1012, extracting features from the frequency spectrum amplitude spectrum to obtain singing features and accompaniment features, and merging the singing features and the accompaniment features to obtain merged features.

[0139] When implementing this step, the frequency spectrum amplitude spectrum can be input into the accompaniment feature extraction model and the singing feature extraction model respectively to obtain the accompaniment feature and the singing feature respectively. The accompaniment feature extraction model and the singing feature extraction model can be a Unet model or other types of neural network models. The frequency spectrum amplitude spectrum is processed by convolution and pooling through the accompaniment feature extraction model to obtain the accompaniment feature; similarly, the frequency spectrum amplitude spectrum is processed by convolution and pooling through the singing feature extraction model to obtain the singing feature.

[0140] Both the accompaniment feature and the singing feature are feature vectors, for example, they can be vectors with a dimension of 1*N. The accompaniment feature and the singing feature are merged. In implementation, the feature values ​​corresponding to the accompaniment feature and the singing feature are added to obtain the merged feature.

[0141] For example, the accompaniment feature is [a 1 ,a 2 ,...,a N ], the singing characteristics are [b 1 ,b 2 ,...,b N ], the merged features are [a 1 +b 1 ,a 2 +b 2 ,...,a N +b N ].

[0142] Step S1013, determining a singing mask and an accompaniment mask based on the combined features, the singing features and the accompaniment features.

[0143] Here, the proportion of the accompaniment feature in the merged feature may be determined as the accompaniment mask, and the proportion of the singing feature in the merged feature may be determined as the singing mask.

[0144] Continuing with the above example, the singing feature and the accompaniment feature are both 1*N vectors, and the accompaniment mask and the singing mask are also 1*N vectors, where the accompaniment mask is Singing mask is

[0145] Step S1014, using the singing mask and the accompaniment mask to perform mask calculations on the frequency spectrum amplitude spectrum respectively, and correspondingly obtain the singing frequency spectrum amplitude and the accompaniment frequency spectrum amplitude.

[0146] Continuing with the above example, the spectrum amplitude spectrum is also a 1*N vector, assuming [L 1 ,L 2 ,...,L NIn this step, when performing mask calculation, the accompaniment mask and the spectrum amplitude spectrum are dot-multiplied to obtain the accompaniment spectrum amplitude, that is, The singing mask and the spectrum amplitude are dot-multiplied to obtain the singing spectrum amplitude, that is,

[0147] Step S1015, performing frequency-time conversion on the singing spectrum amplitude and the accompaniment spectrum amplitude respectively, and obtaining a singing track and an accompaniment track accordingly.

[0148] When implementing this step, the singing spectrum amplitude and the accompaniment spectrum amplitude can be respectively subjected to inverse Fourier transform, thereby realizing the conversion from frequency domain to time domain, and obtaining the singing audio track and the accompaniment audio track accordingly.

[0149] In some embodiments, the "determining the non-singing time information based on the singing audio track" in the above step S102 can be implemented in the following two ways, which are explained below respectively.

[0150] The first method can be achieved by following the steps below:

[0151] Step S1021A, performing voice activity detection on the singing track, positioning the impulse signal in the singing track, and determining the singing time information in the singing track.

[0152] Through the source separation process, it is found that the singing track no longer contains any audio signal other than human voice, so VAD is used to locate the impulse signal of human voice in the singing track, thereby obtaining the singing time information in the singing track. In the embodiment of the present application, the singing time information is also the singing time interval, including the singing start time and the singing end time.

[0153] Step S1022A, based on the singing time information, determine the non-singing time information in the singing track.

[0154] After the singing time information is determined, the singing time interval can be removed from the singing track to obtain the non-singing time interval, thereby determining the non-singing time information.

[0155] The second implementation can be achieved through the following steps:

[0156] Step S1021B, obtaining a trained audio event detection model.

[0157] The audio event detection model is a neural network model, which can be obtained by multiple iterations of training singing track data with labeled singing parts and non-singing parts.

[0158] Step S1022B, input the singing audio track into the audio event detection model to obtain the singing time information in the singing audio track.

[0159] After the singing track is input into the audio event detection model, the audio event detection model can perform singing event detection on the singing track and output the singing time information of each singing segment in the singing track.

[0160] Step S1023B, based on the singing time information, determine the non-singing time information in the singing track.

[0161] Similar to step S1022A, after the singing time information is determined, the singing time interval can be removed from the singing track to obtain the non-singing time interval, thereby determining the non-singing time information.

[0162] Through the above two implementation methods, after extracting the singing track, the part of the entire singing track where the human voice appears can be located. The located singing segment is the segment where the singer speaks in the music data, so as to determine the non-vocal part of the entire song, so as to avoid cutting off the human voice and lyrics when intercepting the song.

[0163] In some embodiments, the “determining a reference start and end point set based on the accompaniment track” in step S102 may be implemented by the following steps:

[0164] Step S1024, obtaining the frequency spectrum feature sequence and the preset sliding window length of the accompaniment audio track.

[0165] When implementing this step, the separated accompaniment audio track can first be subjected to Fourier transformation, for example, short-time Fourier transformation, to obtain the spectrum of the accompaniment audio track, and then subjected to Mel transformation and logarithmic operation to obtain the spectrum feature sequence of the accompaniment audio track. In the embodiment of the present application, the spectrum feature sequence can be a logarithmic Mel spectrum sequence. The sliding window length W can be preset, and W is a positive integer, for example, 18, 24, 36, etc.

[0166] Step S1025: performing sliding window processing on the frequency spectrum feature sequence based on the sliding window length to obtain a plurality of sliding window results.

[0167] In some embodiments, it is also necessary to obtain a sliding step length S. In the actual implementation process, the sliding step length can be determined based on the sliding window length, for example, it can be one-half or one-third of the sliding window length W, or the sliding step length can also be the sliding window length. When implementing step S1025, the spectrum feature sequence can be subjected to sliding window processing according to the sliding window length and the sliding step length, thereby obtaining multiple sliding window results. Each sliding window result includes W spectrum feature values. There will be (WS) overlapping spectrum feature values ​​in adjacent sliding results.

[0168] Step S1026: divide each sliding window result into N sub-results, and determine the energy mean of the N sub-results in each sliding window result.

[0169] Where N is a positive integer, for example, N can be 2, 3, 4, etc. In order to ensure that the sliding window result can be evenly divided, when setting the sliding window length, W can be set to an integer multiple of N. Of course, it is not limited to evenly dividing the sliding window result, so it is not necessary to set W to an integer multiple of N. In the embodiment of the present application, W can be set to 18, N to 3, and S to 6.

[0170] When implementing step S1026, all frequency domain energies in each sliding window result may be accumulated to generate a one-dimensional energy sequence, and then the one-dimensional energy sequence is divided into N sub-results, and then the energy mean value in each sub-result is determined.

[0171] Step S1027: determining a target sliding window result from the multiple sliding window results based on the energy averages of the N sub-results in the respective sliding window results.

[0172] After the above steps S1024 to S1027, the N sub-results corresponding to each sliding window result can be determined, and then the N sub-results corresponding to each sliding window are compared in size. If the energy mean of the N sub-results meets the decreasing condition, the sliding window result corresponding to the N sub-results is determined as the first target sliding window result; if the energy mean of the N sub-results meets the increasing condition, the sliding window result corresponding to the N sub-results is determined as the second target sliding window result. If the N sub-results do not meet the increasing condition or the decreasing condition, the sliding window result corresponding to the N sub-results is not the target sliding window result.

[0173] It should be noted that the energy mean of the N sub-results meets the decreasing condition, which means that the energy mean of the N sub-results is compared in chronological order. If P 1 >P 2 >P 3 >…>P N , then it is determined that the energy mean of the N sub-results meets the decreasing condition, that is, when comparing the energy mean, it is necessary to sort the sub-results in chronological order. The energy mean of the N sub-results meets the increasing condition in a similar way, which means that the energy mean of the N sub-results is compared in chronological order. If P is satisfied 1 <P 2 <P 3 <… <P N , then it is determined that the energy mean of the N sub-results meets the increasing condition.

[0174] Step S1028: determining a reference start and end point set based on the target sliding window result.

[0175] Following the implementation of step S1027, since the N sub-results in the first target sliding window result meet the decreasing condition, there is an energy fade-out effect in the first target sliding window result. At this time, in step S1028, the right boundary point of the first target sliding window result is determined as the reference end point; and the N sub-results in the second target sliding window result meet the increasing condition, there is an energy fade-in effect in the first target sliding window result. At this time, in step S1028, the left boundary point of the second target sliding window result is determined as the reference starting point.

[0176] Through the above steps S1024 to S1028, energy detection can be performed on the accompaniment audio track to determine the energy fade-in position and the energy fade-out position in the accompaniment audio track, and the energy fade-in position is determined as the reference starting point, and the energy fade-out position is determined as the reference ending point, so that the time information of the reference starting and ending points in the accompaniment audio track that are suitable for the soundtrack can be accurately located.

[0177] It should be noted that step S102 can be implemented through steps S1021A, S1022A, and S1024 to S1028, or can be implemented through steps S1021B to S1023B, and S1024 to S1028, but the above two implementation methods are not shown in the accompanying drawings.

[0178] Based on the above embodiments, the present application further provides a multimedia data processing method, which is applied to Figure 1 The network architecture shown, Figure 5 Another implementation flow diagram of the multimedia data processing method provided in the embodiment of the present application is as follows: Figure 5 As shown, the process includes:

[0179] Step S201: the terminal obtains video data to be matched with music and music data to be processed.

[0180] During implementation, the terminal may determine the video data to be matched with music and the music data to be processed based on the received operation instruction for matching music to the video.

[0181] Step S202: The terminal sends the video data and the music data to the server.

[0182] Step S203: the server performs source separation on the music data to obtain a singing track and an accompaniment track.

[0183] Step S204, the server determines the non-singing time information based on the singing track, and determines the reference start and end point set based on the accompaniment track.

[0184] In step S205, the server determines a target start-end point set based on the reference start-end point set that falls within the non-singing time interval.

[0185] The implementation process of the above steps S203 to S205 is similar to the implementation process of steps S101 to S103, and in actual application, reference may be made to steps S101 to S103.

[0186] Step S206: the server determines a first playing duration of the video data to be matched with music.

[0187] The first playing time is the time taken to play the video data to be matched with music at a normal speed (1x speed).

[0188] Step S207: the server determines the duration of each interval between the target starting point and each target ending point after the target starting point in the target starting and ending point set.

[0189] In this step, the server determines the duration of each interval between each target starting point in the target starting and ending point set and each target ending point located after the target starting point.

[0190] For example, in chronological order, they are the first starting point 52 seconds, the second starting point 1 minute 3 seconds, the first end point 1 minute 42 seconds, the third starting point 1 minute 58 seconds, the second end point 2 minutes 20 seconds, and the third end point 2 minutes 36 seconds. Then, it is necessary to calculate the interval between the first starting point and the first end point (50 seconds), the interval between the first starting point and the second end point (1 minute 28 seconds), the interval between the first starting point and the third end point (1 minute 43 seconds), the interval between the second starting point and the second end point (1 minute 17 seconds), the interval between the second starting point and the third end point (1 minute 33 seconds), and the interval between the third starting point and the third end point (38 seconds).

[0191] Step S208: The server determines each duration difference between each interval duration and the first playback duration.

[0192] In this step, the duration difference between each interval duration and the first playback duration may be a value obtained by taking the absolute value of the difference obtained by subtracting the interval duration from the first playback duration.

[0193] Continuing with the above example, assuming that the first playback duration is 1 minute and 30 seconds, the differences between each interval duration and the first playback duration are 40 seconds, 2 seconds, 13 seconds, 13 seconds, 3 seconds, and 52 seconds respectively.

[0194] Step S209: The server determines whether there is at least one target interval duration whose duration difference is less than a preset threshold.

[0195] When there is at least one target interval duration whose duration difference is less than the preset threshold, the process proceeds to step S210; when there is not at least one target interval duration whose duration difference is less than the preset threshold, the process proceeds to step S215.

[0196] In step S210, the server determines the target starting point corresponding to the minimum value of at least one target interval duration as the music start point, and determines the target ending point corresponding to the minimum value as the music ending point.

[0197] Assuming that the preset threshold is 5 seconds, the duration difference of 2 seconds and the duration difference of 3 seconds are less than 5 seconds. Therefore, step S209 proceeds to step S210. In step S210, the target starting point corresponding to 2 seconds is determined as the soundtrack starting point, and the target ending point corresponding to 2 seconds is determined as the soundtrack ending point, that is, the first starting point is determined as the soundtrack starting point, and the second ending point is determined as the soundtrack ending point.

[0198] Step S211: The server determines the target music data between the start point of the music score and the end point of the music score as the target music score data, and determines the second playing time length of the target music score data.

[0199] The music start point and the music end point each correspond to a time point. In this step, the second playback duration of the target music data can be determined by subtracting the time point corresponding to the music start point from the music end point.

[0200] Continuing with the above example, the starting point of the music is 52 seconds, and the ending point of the music is 2 minutes and 20 seconds, so the second playback time is 1 minute and 28 seconds.

[0201] Step S212: The server adjusts the playing time of the video data to be matched with music based on the second playing time to obtain adjusted video data.

[0202] When implementing this step, the playback time can be adjusted by adjusting the playback speed of the video data to be matched with music, so that the playback time of the adjusted video data is the same as the second playback time. It can also be achieved by editing the video data to be matched with music. For example, when the second playback time is less than the first playback time, the video data to be matched with music is trimmed to the second playback time. When the second playback time is greater than the first playback time, a trailer can be added to the video data to be matched with music so that the playback time of the adjusted video data reaches the second playback time.

[0203] Step S213: the server synthesizes the adjusted video data and the target music data to obtain the video data with music.

[0204] When implementing this step, the target music data and the adjusted video data can be directly synthesized to obtain the video data with music. Different processing methods can also be adopted based on whether the adjusted video data itself has audio data. For example, when the adjusted video data itself does not have audio data, it is directly synthesized with the target music data. When the adjusted video data itself has reference data, there can be different implementation methods. For example, the audio data in the adjusted video data can be filtered out and the muted video data can be synthesized with the target music data. Alternatively, the volume of the audio data in the adjusted video data can be lowered and then synthesized with the target music data.

[0205] Step S214: the server sends the video data with music to the terminal.

[0206] Step S215: the server obtains the track type of the music data to be processed.

[0207] In an embodiment of the present application, if none of the multiple duration differences obtained is less than a preset threshold, then it is determined that there is no suitable music clip in the music data to be processed to compose music for the video data to be accompanied by music. In subsequent steps, the server will recommend music to the terminal based on the track type of the music data to be processed.

[0208] Step S216: The server determines a plurality of candidate music data based on the track type.

[0209] Step S217: the server outputs multiple music identifiers of the multiple candidate music data.

[0210] In the embodiment of the present application, the server outputs multiple music identifiers of multiple candidate music data, that is, the server sends the multiple music identifiers to the terminal.

[0211] Step S218: The terminal determines the selected target music identifier based on the received selection operation for the music identifier.

[0212] In some embodiments, after the terminal receives the music identifier sent by the server, multiple music identifiers are presented, and music playback controls may also be presented. Each music identifier corresponds to a music playback control. If the terminal receives a touch operation for a certain music playback control, the corresponding music data is played for the user to determine whether to select the music identifier. When the terminal receives a selection operation for a certain music identifier, the selected target music identifier is determined.

[0213] Step S219: the terminal sends the selected target music identifier to the server.

[0214] Step S220: The server determines the music data to be processed based on the target music identifier.

[0215] During implementation, the server determines the music data corresponding to the target music identifier as the updated music data to be processed. After this step, the above steps S203 to S220 are repeatedly performed until the music matching process of the video data is completed.

[0216] It should be noted that the music data processing and video soundtrack process provided in the embodiment of the present application is implemented by a server, and in actual implementation, it can also be implemented by a terminal.

[0217] In the multimedia data processing method provided in the embodiment of the present application, after obtaining the video data to be matched with music and the music data to be processed, the music data is firstly separated from the source to obtain the singing track and the accompaniment track, and then the non-singing time information is determined based on the singing track, and the reference start and end point set is determined based on the accompaniment track, and then the target start and end point set is determined from the reference start and end point set based on the non-singing time information, and then the second playback time between the target starting point and each target end point after the target starting point is calculated respectively, and then the interval time between each second playback time and the first playback time is calculated, and the interval time less than the preset threshold and the minimum interval time is calculated. The target starting and ending points corresponding to the length are determined as the starting points of the music, so as to ensure the matching degree between the music playback time and the video playback time between the finally determined music start and end points, and ensure the integrity of the music. In addition, if the final music start and end points are not screened out in the embodiment of the present application, the server can screen the music that matches the track type from the track library based on the track type of the music data and determine it as the music data to be processed, and send it to the terminal for confirmation. If the server receives the notification message that the terminal determines to use the music data to be processed, the music data to be processed is used to re-compose the video data, so as to ensure that the target music data that matches the video data can be determined, and the robustness of the video soundtrack is guaranteed.

[0218] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0219] The multimedia data processing method provided in the embodiment of the present application can be applied to the scenario of intelligently producing short video clips. By combining the two modules of source separation model and audio energy detection, the starting and ending points of the soundtrack are located, providing a reference for music cutting points for editing the soundtrack.

[0220] In the multimedia data processing method provided in the embodiment of the present application, the first version of the short video is input into the source separation model to obtain the Vocal track (corresponding to the singing track in other embodiments) and the BGM track (corresponding to the accompaniment track in other embodiments), and then the VAD algorithm is used to determine the position information of the singing section in the music. The non-singing position of the whole song is located using the position information of the singing section, and the audio energy detection is performed on the BGM track to determine the fade-in and fade-out position in the whole song. Finally, the non-singing section and energy detection dual selection method are used to determine the start and end points of the soundtrack. This can avoid the phenomenon of singing truncation and accurately locate the time point position of the audio gradient, and fully automatically provide the reference cutting start and end points of the music for intelligent production of soundtracks.

[0221] Figure 6 The audio graph and spectrum graph of the Vocal track extracted from the first version of the short video provided in the embodiment of the present application, wherein 601 is the audio graph and 602 is the spectrum graph. Figure 7 The audio graph and spectrum graph of the BGM track extracted from the first version of the short video provided in the embodiment of the present application. Among them, 701 is the audio graph, 702 is the spectrum graph, Figure 8 This is an audio graph of the final short video editing product generated after the music is provided in the embodiment of the present application.

[0222] The following is an explanation of the technical implementation process of the multimedia data processing method provided in the embodiment of the present application. The multimedia data processing method provided in the embodiment of the present application may include the following three processes when implemented: non-vocal segment detection based on source separation and VAD, start and end point positioning based on energy detection, and final screening of the start and end points combined with the non-vocal segment.

[0223] Fig. 9 A schematic diagram of another implementation flow of the multimedia data processing method provided in the embodiment of the present application. The multimedia data processing method can be applied to a server or a terminal. In the embodiment of the present application, the method is described by taking application to a terminal as an example. Fig. 9 As shown, the process includes:

[0224] Step S901: the terminal performs source separation on the input music to obtain a vocal track and an accompaniment track.

[0225] In implementation, the entire input music can be first passed through a source separation model built with a dual U-net network to extract the audio accompaniment track and the singing vocal track from the entire music.

[0226] Fig.10 This is the Unet network architecture diagram, such as Fig.10As shown, the U-shaped network (Unet) is a U-shaped symmetrical structure, with 1001 on the left being the convolution layer and 1002 on the right being the upsampling layer. Fig.10 The Unet structure shown in the figure contains 4 convolutional layers and corresponding 4 upsampling layers. When implementing, you can either implement the network from scratch and initialize the weights, and then train the model; you can also borrow the convolutional layer structure and the corresponding trained weight files of some existing networks, and add the subsequent upsampling layers for training calculations, such as the Residual Ne twork (ResNet), the super-resolution test sequence network (VGGNet, Visual Geometry Group Network), etc., because in the model training of deep learning, if the existing weight model files can be used, the training speed can be greatly accelerated. Another feature is that the feature maps obtained by each convolutional layer of the Unet network will be connected (concatenate) to the corresponding upsampling layer, so that each layer of feature maps can be effectively used in subsequent calculations. That is, skip connection. In this way, compared with some other network structures such as the Fully Convolution Network (FCN), Unet avoids direct supervision and loss calculation in high-level feature maps, but combines the features in low-level feature maps, so that the final feature map contains both high-level features and many low-level features, realizing feature fusion at different sizes and improving the accuracy of the model results.

[0227] Fig.11 A schematic diagram of the implementation process of source separation using the Unet model provided in the embodiment of the present application is shown in FIG. Fig.11 As shown, the process includes:

[0228] Step S1101: the terminal obtains the music to be processed and determines the frequency spectrum of the music track.

[0229] Step S1102: the terminal eliminates the phase in the frequency spectrum of the audio track to obtain the frequency spectrum amplitude spectrum of the audio track.

[0230] In step S1103, the terminal inputs the frequency spectrum amplitude spectrum of the audio track into the background music U-type network (BGM-Unet, BackGround Music Unethical) and the singing U-type network (VOCAL-Unet).

[0231] BGM-Unet and VOCAL-Unet can be used to extract the accompaniment features and singing voice features in music.

[0232] Step S1104: the terminal merges the accompaniment feature and the singing voice feature to obtain a merged feature.

[0233] In step S1105, the terminal performs mask calculation on the merged features to calculate the accompaniment mask and the singing mask respectively.

[0234] In step S1106, the terminal calculates the corresponding positions of the accompaniment mask map and the singing mask map with the original track spectrum amplitude spectrum respectively, and then generates the corresponding accompaniment track and singing track after inverse spectrum transformation.

[0235] Step S902: The terminal inputs the singer's vocal track into the VAD model to locate the non-vocal segment position.

[0236] The source separation model can be used to separate the accompaniment track and the singing track. The singing track does not contain any audio signals other than the human voice, so VAD is used to locate the place where the human voice appears in the vocal track. In actual application, the open source VAD algorithm can be used to locate the position of the non-silent segment. Fig.12 A schematic diagram of a non-silent segment located by using VAD provided in an embodiment of the present application, Fig.12 It can be seen that in the entire singing track, except for the sections marked by the boxes where high audio impulse signals appear, the rest are silent. The VAD algorithm is then used to locate the impulse signals. After that, the remaining area of ​​the entire track is the set of non-vocal sections, and the time position of the non-vocal sections in the original music audio is calculated.

[0237] It should be noted that, in some embodiments, the time position information of non-vocal segments can also be determined through an audio event detection positioning model, and the singing segments in the song can be located by locating the singing position, thereby locating the non-vocal segment time set.

[0238] Step S903: the terminal performs audio energy detection on the accompaniment audio track to determine a reference starting point set of the accompaniment music.

[0239] This step can be implemented by performing a short-time Fourier transform (STFT) on the accompaniment audio track obtained through source separation to obtain the spectrum of the accompaniment audio track, and then performing a Mel transform and obtaining the logarithm to generate a characteristic log-mel spectrum (log-melspectrogram) required for the final detection.

[0240] On the entire generated spectrum feature sequence, a sliding window with a duration of T is set. The sliding window is a fixed filter for judging the energy gradient. Fig.13 A schematic diagram of the end point positioning based on energy detection provided in the embodiment of the present application, such as Fig.13 As shown, in each sliding window, all frequency domain energies are accumulated to generate a one-dimensional energy sequence, and then the entire sequence is evenly divided into three segments according to time T / 3, and the average value of the energy sequence in each segment is obtained. Fig.13 This is a schematic diagram of locating the end point. In the short video editing and music matching task of intelligent production, the choice of the music end point is preferably a fade-out effect, so the three average energy values ​​in the entire sliding window should decrease from left to right (the opposite is true for the starting point, increasing from left to right), that is, E1>E2>E3. If it fully meets the performance of decreasing energy in the window, the right boundary of the window meets the choice of the end point (if the starting point meets, it is the left boundary).

[0241] During implementation, the sliding distance of the sliding window in the entire detection process is T / 3, and detection is performed at a granularity of every T / 3.

[0242] Step S904: the terminal uses the non-vocal segment time set to filter the reference start and end points of the accompaniment to obtain a final set of soundtrack start and end points.

[0243] Through the above-mentioned steps S901 to S904, the non-vocal time segment set can be located on the singing track, and the preliminary start and end point time positions can be located on the accompaniment track. Then, the time threshold of the non-vocal segment is used to filter the points on the accompaniment track, and the points that are not on the non-vocal segment are eliminated. The remaining points whose time is included in the non-vocal segment are the start and end points of the soundtrack detected by the system.

[0244] Fig.14 A schematic diagram of non-vocal segment selection points provided in an embodiment of the present application, wherein the non-vocal time position 1401 in the singing track and the reference start and end points selected by the accompaniment track 1402 are as shown in FIG. Fig.14 As shown in the figure, in the final screening process, the start and end points that fall into the non-vocal time interval are retained, and the Fig.14 The target start and end points shown below include start point 1403, end point 1404, end point 1405, start point 1406 and end point 1407. Then calculate the interval duration between start point 1403 and end point 1404, the interval duration between start point 1403 and end point 1405, the interval duration between start point 1403 and end point 1407, and the interval duration between start point 1406 and end point 1407, and then determine the duration difference between each of the above interval durations and the playback of the video, and determine the minimum duration difference. If the minimum duration difference is less than a preset threshold, the start point and end point corresponding to the minimum duration difference are determined as the final soundtrack start and end points.

[0245] In the multimedia data processing method provided in the embodiment of the present application, a source separation model is used to separate the singing track and the accompaniment track of the entire song, which can be applied to the situation without lyrics, and can accurately locate the non-singing time segment set in the entire song, so as to avoid the phenomenon of singing truncation during the selection process. Then, audio energy detection is performed on the accompaniment track to locate the energy fade-in and fade-out time position of the accompaniment, so as to accurately locate the time information of the song accompaniment suitable for the start and end point position of the soundtrack cutting. Finally, the detected non-singing time segment and the start and end point position of the soundtrack cutting detected on the accompaniment track are used to jointly screen the final start and end point positioning, so as to avoid the phenomenon of singing truncation and locate the position of the gradual fade-in and fade-out of the song rhythm energy, so as to adapt to the intelligent production of short video editing models.

[0246] The following is a description of an exemplary structure of the multimedia data processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the multimedia data processing device 455 of the memory 450 may include:

[0247] The information source separation module 4551 is used to obtain the video data to be matched with music and the music data to be processed, and perform information source separation on the music data to obtain the singing track and the accompaniment track;

[0248] A first determination module 4552 is used to determine non-singing time information based on the singing track, and to determine a reference start and end point set based on the accompaniment track;

[0249] A second determination module 4553 is used to determine a target start-end point set from the reference start-end point set based on the non-singing time information;

[0250] A third determining module 4554 is used to determine a music matching start point and a music matching end point from the target start and end point set based on the first playback duration of the video data to be matched with music;

[0251] The fourth determination module 4555 is used to determine the target music data between the soundtrack start point and the soundtrack end point in the music data as the target soundtrack data of the video data.

[0252] In some embodiments, the information source separation module is further used to:

[0253] Performing time-frequency conversion on the music data to obtain a frequency spectrum amplitude spectrum of the music data;

[0254] Extracting features from the frequency spectrum amplitude spectrum to obtain singing features and accompaniment features, and merging the singing features and the accompaniment features to obtain merged features;

[0255] Determining a singing mask and an accompaniment mask based on the combined feature, the singing feature and the accompaniment feature;

[0256] Using the singing mask and the accompaniment mask to respectively perform mask calculations on the frequency spectrum amplitude spectrum, and correspondingly obtain the singing frequency spectrum amplitude and the accompaniment frequency spectrum amplitude;

[0257] The singing spectrum amplitude and the accompaniment spectrum amplitude are respectively converted into frequency-time, and a singing track and an accompaniment track are obtained accordingly.

[0258] In some embodiments, the first determining module is further configured to:

[0259] Performing voice activity detection on the singing track, positioning the impulse signal in the singing track, and determining the singing time information in the singing track;

[0260] Based on the singing time information, non-singing time information in the singing track is determined.

[0261] In some embodiments, the first determining module is further configured to:

[0262] Get the trained audio event detection model;

[0263] Inputting the singing audio track into the audio event detection model to obtain singing time information in the singing audio track;

[0264] Based on the singing time information, non-singing time information in the singing track is determined.

[0265] In some embodiments, the first determining module is further configured to:

[0266] Obtaining a frequency spectrum feature sequence and a preset sliding window length of the accompaniment audio track;

[0267] Perform sliding window processing on the spectrum feature sequence based on the sliding window length to obtain multiple sliding window results;

[0268] Divide each sliding window result into N sub-results, and determine the energy mean of the N sub-results in each sliding window result;

[0269] Determining a target sliding window result from the multiple sliding window results based on the energy averages of the N sub-results in the respective sliding window results;

[0270] A reference start and end point set is determined based on the target sliding window result.

[0271] In some embodiments, the first determining module is further configured to:

[0272] Determine the sliding window result whose energy mean value of N sub-results satisfies the decreasing condition as the first target sliding window result;

[0273] Determine the sliding window result whose energy mean value of the N sub-results satisfies the increasing condition as the second target sliding window result;

[0274] Determine the right boundary point of the first target sliding window result as a reference end point;

[0275] The left boundary point of the second target sliding window result is determined as a reference starting point.

[0276] In some embodiments, the non-singing time information includes a non-singing time interval, and the second determination module is further used to:

[0277] A target start-end point set is determined based on reference start-end point in the reference start-end point set that falls within the non-singing time interval, wherein the target start-end point set includes a target starting point and a target ending point.

[0278] In some embodiments, the third determination module is further used to:

[0279] Determining a first playing duration of the video data to be matched with music;

[0280] Determine each interval duration between a target starting point and each target ending point after the target starting point in the target starting and ending point set;

[0281] Determine each duration difference between each interval duration and the first playback duration;

[0282] When there is at least one target interval duration whose duration difference is less than a preset threshold, the target starting point corresponding to the minimum value of the at least one target interval duration is determined as the soundtrack starting point, and the target ending point corresponding to the minimum value is determined as the soundtrack ending point.

[0283] In some embodiments, the apparatus further comprises:

[0284] A first acquisition module, configured to acquire the track type of the music data to be processed when there is no at least one target interval duration whose duration difference is less than a preset threshold;

[0285] An output module, configured to determine a plurality of candidate music data based on the track type, and output a plurality of music identifiers of the plurality of candidate music data;

[0286] A fifth determination module, configured to determine a selected target music identifier based on receiving a selection operation for the music identifier;

[0287] The sixth determination module is used to determine the music data to be processed based on the target music identifier.

[0288] In some embodiments, the apparatus further comprises:

[0289] A seventh determination module, configured to determine a second playback duration of the target music data based on the music start point and the music end point;

[0290] An adjustment module, configured to adjust the playing time of the video data to be matched with music based on the second playing time to obtain adjusted video data;

[0291] The synthesis module is used to synthesize the adjusted video data and the target music data to obtain the video data with music.

[0292] It should be noted here that the description of the above multimedia data processing device embodiment is similar to the above method description and has the same beneficial effects as the method embodiment. For technical details not disclosed in the multimedia data processing device embodiment of the present application, those skilled in the art should refer to the description of the method embodiment of the present application for understanding.

[0293] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the multimedia data processing method described in the embodiment of the present application.

[0294] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the multimedia data processing method provided by the present application embodiment, for example, Figure 3 , Figure 4 , Figure 5 The multimedia data processing method is shown.

[0295] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0296] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0297] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (for example, files storing one or more modules, subroutines, or code portions).

[0298] By way of example, executable instructions may be deployed to be executed on one computer device, or on multiple computer devices located at one site, or on multiple computer devices distributed across multiple sites and interconnected by a communication network.

[0299] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A multimedia data processing method, characterized in that: The method comprises: Acquire the video data to be matched with music and the music data to be processed, perform source separation on the music data, and obtain a singing track and an accompaniment track; Determine the non-singing time information based on the singing track, and determine the reference start and end point set based on the accompaniment track; wherein, determining the reference start and end point set based on the accompaniment track includes: obtaining a frequency spectrum feature sequence and a preset sliding window length of the accompaniment track; Performing sliding window processing on the spectrum feature sequence based on the sliding window length to obtain multiple sliding window results; dividing each sliding window result into N sub-results, and determining the energy mean of the N sub-results in each sliding window result; determining a target sliding window result from the multiple sliding window results based on the energy mean of the N sub-results in each sliding window result; and determining a reference start and end point set based on the target sliding window result; Based on the non-singing time information, determining a target start-end point set from the reference start-end point set; Based on the first playback duration of the video data to be matched with music, determining a music matching start point and a music matching end point from the target start and end point set; The target music data between the music start point and the music end point in the music data is determined as the target music data of the video data.

2. The method according to claim 1, characterized in that The step of performing source separation on the music data to obtain a singing track and an accompaniment track comprises: Performing time-frequency conversion on the music data to obtain a frequency spectrum amplitude spectrum of the music data; Extracting features from the frequency spectrum amplitude spectrum to obtain singing features and accompaniment features, and merging the singing features and the accompaniment features to obtain merged features; Determining a singing mask and an accompaniment mask based on the combined feature, the singing feature and the accompaniment feature; Using the singing mask and the accompaniment mask to respectively perform mask calculations on the frequency spectrum amplitude spectrum, and correspondingly obtain the singing frequency spectrum amplitude and the accompaniment frequency spectrum amplitude; The singing spectrum amplitude and the accompaniment spectrum amplitude are respectively converted into frequency-time, and a singing track and an accompaniment track are obtained accordingly.

3. The method according to claim 1, characterized in that The determining of non-singing time information based on the singing track includes: Performing voice activity detection on the singing track, positioning the impulse signal in the singing track, and determining the singing time information in the singing track; Based on the singing time information, non-singing time information in the singing track is determined.

4. The method according to claim 1, characterized in that The determining of non-singing time information based on the singing track includes: Get the trained audio event detection model; Inputting the singing audio track into the audio event detection model to obtain singing time information in the singing audio track; Based on the singing time information, non-singing time information in the singing track is determined.

5. The method according to claim 1, characterized in that The determining a target sliding window result from the multiple sliding window results based on the energy average of the N sub-results in each sliding window result includes: Determine the sliding window result whose energy mean value of N sub-results satisfies the decreasing condition as the first target sliding window result; Determine the sliding window result whose energy mean value of the N sub-results satisfies the increasing condition as the second target sliding window result; The determining of a reference start and end point set based on the target sliding window result includes: Determine the right boundary point of the first target sliding window result as a reference end point; The left boundary point of the second target sliding window result is determined as a reference starting point.

6. The method according to claim 1, characterized in that The non-singing time information includes a non-singing time interval, and determining a target start-end point set from the reference start-end point set based on the non-singing time information includes: A target start-end point set is determined based on reference start-end point in the reference start-end point set that falls within the non-singing time interval, wherein the target start-end point set includes a target starting point and a target ending point.

7. The method according to claim 6, characterized in that The step of determining the music matching start point and the music matching end point from the target start and end point set based on the first playback duration of the video data to be matched with music comprises: Determining a first playing duration of the video data to be matched with music; Determine each interval duration between a target starting point and each target ending point after the target starting point in the target starting and ending point set; Determine each duration difference between each interval duration and the first playback duration; When there is at least one target interval duration whose duration difference is less than a preset threshold, the target starting point corresponding to the minimum value of the at least one target interval duration is determined as the soundtrack starting point, and the target ending point corresponding to the minimum value is determined as the soundtrack ending point.

8. The method according to claim 7, characterized in that The method further comprises: When there is no at least one target interval duration whose duration difference is less than a preset threshold, obtaining the track type of the music data to be processed; Determining a plurality of candidate music data based on the track type, and outputting a plurality of music identifiers of the plurality of candidate music data; Based on receiving a selection operation for the music identifier, determining a selected target music identifier; The music data to be processed is determined based on the target music identifier.

9. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Determining a second playback duration of the target music data based on the music start point and the music end point; Adjusting the playing time of the video data to be matched with music based on the second playing time to obtain adjusted video data; The adjusted video data and the target music data are synthesized to obtain the video data with music.

10. A multimedia data processing device, characterized in that: The device comprises: A source separation module is used to obtain the video data to be matched with music and the music data to be processed, and perform source separation on the music data to obtain a singing track and an accompaniment track; The first determination module is used to determine the non-singing time information based on the singing track, and to determine the reference start and end point set based on the accompaniment track; it is also used to obtain the spectrum feature sequence and the preset sliding window length of the accompaniment track; based on the sliding window length, the spectrum feature sequence is subjected to sliding window processing to obtain multiple sliding window results; each sliding window result is divided into N sub-results, and the energy mean of the N sub-results in each sliding window result is determined; based on the energy mean of the N sub-results in each sliding window result, a target sliding window result is determined from the multiple sliding window results; and a reference start and end point set is determined based on the target sliding window result; A second determination module is used to determine a target start-end point set from the reference start-end point set based on the non-singing time information; A third determining module is used to determine a music matching start point and a music matching end point from the target start and end point set based on a first playback duration of the video data to be matched with music; The fourth determination module is used to determine the target music data between the soundtrack start point and the soundtrack end point in the music data as the target soundtrack data of the video data.

11. A computer device, characterized in that: The computer device comprises: A memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 9 when executing the executable instructions stored in the memory.

12. A computer-readable storage medium storing executable instructions, characterized in that: When the executable instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images