Audio processing method, device, equipment, and computer-readable storage medium
Through the audio processing method, the matching probability is calculated using timing characteristics and deep networks, and the music chorus is intelligently positioned, solving the problems of large positioning and large errors in the existing technology, and achieving fast and accurate chorus positioning.
Patent Information
- Application Number
- CN202210082760.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-24
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-01-24
AI Technical Summary
It is difficult to quickly and accurately locate music choruses in the prior art, especially when large amounts of music processing, there are problems of human operation error and time-consuming.
By determining the candidate interval based on the timing characteristics of the audio to be processed, and computing the match probability based on the starting point probability, the end point probability and the interval probability. If the match probability is greater than the preset threshold, the interval is determined as the chorus interval and the features are extracted using the deep network. To achieve intelligent positioning.
It realizes the rapid and accurate positioning of music choruses, avoids overlapping of multiple chorus intervals, improves the accuracy and efficiency of positioning, and is suitable for batch processing.
Smart Images

Figure CN114420075B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computers, and in particular to an audio processing method and apparatus, a device, and a computer-readable storage medium. Background Art
[0002] The chorus is a repeated musical passage in a song, usually between several verses. Most choruses are intense and can stimulate the singer's emotions to the maximum extent. Therefore, in order to quickly release emotions, memorize and master the rhythm of the song, singers often focus on practicing the chorus and start practicing from the beginning of the chorus interval.
[0003] The chorus section is a complete section formed by the start and end times of the chorus part. Singers usually manually adjust the progress of the song to quickly enter the chorus section, which is extremely inconvenient. The existing method of locating the chorus of music is generally through manual marking. However, the total length of each song, the chorus length and the number of chorus loops are not necessarily the same. Manual marking is often time-consuming and there are human operation errors. It is difficult to automatically locate the chorus of music while processing a large amount of music in batches.
[0004] With the rise of online karaoke platforms and the explosive demand for mass singing entertainment, especially the demand for singing the chorus of a song, how to intelligently, quickly and accurately locate the chorus of music has become a technical problem that needs to be solved urgently in the field of audio processing. Summary of the invention
[0005] To solve the above technical problems, the embodiments of the present application respectively provide an audio processing method and apparatus, an audio processing device, and a computer-readable storage medium, which obtain the chorus starting point probability, interval probability, and ending point probability according to the timing characteristics of the audio to be processed, thereby intelligently, quickly, and accurately locating the music chorus.
[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.
[0007] According to one aspect of an embodiment of the present application, there is provided an audio processing method, including:
[0008] Determining a candidate interval included in the audio to be processed according to a time series feature of the audio to be processed;
[0009] Calculating the matching probability of the candidate interval based on the starting point probability, the ending point probability and the interval probability corresponding to the candidate interval;
[0010] If the matching probability is greater than a first preset threshold, the candidate interval is determined to be the chorus interval included in the audio to be processed.
[0011] According to one aspect of an embodiment of the present application, there is provided an audio processing device, including:
[0012] A determination module, configured to determine a candidate interval included in the audio to be processed according to a time sequence feature of the audio to be processed;
[0013] The calculation and positioning module is configured to calculate the matching probability of the candidate interval based on the starting point probability, the ending point probability and the interval probability corresponding to the candidate interval; if the matching probability is greater than a first preset threshold, the candidate interval is determined to be the chorus interval contained in the audio to be processed.
[0014] In another exemplary embodiment, the audio processing device further includes:
[0015] The subtask module is configured to construct a first deep network, which is composed of a convolutional neural network layer, a multi-layer long short-term LSTM network and a fully connected network layer stacked in sequence; input the audio to be processed into the first deep network; extract deep features from the first deep network as the time series features of the audio to be processed.
[0016] In another embodiment, the subtask module further includes:
[0017] The extraction unit is configured to extract the features output by the last LSTM network layer as the time series features of the audio to be processed.
[0018] In another embodiment, the determining module includes:
[0019] A time series modeling unit is configured to input the time series features of the audio to be processed into a second deep network, and obtain the chorus position probability at each moment contained in the audio to be processed output by the second deep network, wherein the chorus position probability includes a starting point probability and an ending point probability;
[0020] The candidate interval construction unit is configured to determine a candidate interval from the audio to be processed according to the obtained starting point probability and ending point probability.
[0021] In another embodiment, the candidate interval construction unit is configured to select the first moment as the starting moment of the candidate interval if the starting point probability corresponding to the first moment of the audio to be processed is greater than a second preset threshold or the starting point probability of all moments within a preset interval; if the ending point probability corresponding to the second moment of the audio to be processed is greater than a third preset threshold or the ending point probability of all moments within a preset interval, then select the second moment as the ending time of the candidate interval.
[0022] In another embodiment, the calculation and positioning module includes:
[0023] An interval feature extraction unit, configured to determine the interval feature of the candidate interval according to the time series feature of the audio to be processed;
[0024] The interval probability extraction unit is configured to input the interval feature into the third deep network to obtain the interval probability of the candidate interval output by the third deep network.
[0025] In another embodiment, the calculation and positioning module includes:
[0026] an interval feature extraction unit, configured to extract features from the candidate interval, an interval near the starting point of the candidate interval, and an interval near the ending point of the candidate interval, respectively, to obtain corresponding candidate interval features, wherein the feature lengths of each candidate interval feature are the same; and to concatenate the features of each candidate interval to obtain the interval feature of the candidate interval;
[0027] The interval probability extraction unit is configured to input the interval feature into the third deep network to obtain the interval probability of the candidate interval output by the third deep network.
[0028] According to one aspect of an embodiment of the present application, there is provided an audio processing device, comprising: a receiving device for receiving audio to be processed; and a controller electrically connected to the receiving device to execute the above method.
[0029] According to one aspect of an embodiment of the present application, a computer-readable storage medium is further provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the above method.
[0030] According to one aspect of the embodiments of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above method.
[0031] In the technical solution provided in the embodiments of the present application, candidate intervals are determined according to the timing features of the audio to be processed, and the matching probability of the candidate intervals is calculated based on the starting point probability, the ending point probability and the interval probability corresponding to the candidate intervals. Since the interval probability can accurately determine the middle position of the chorus interval, the overlap of multiple chorus intervals can be avoided, so that a complete and accurate chorus interval is determined based on the comparison result of the matching probability with the first preset threshold.
[0032] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0034] Figure 1 It is a schematic diagram of an implementation environment involved in this application;
[0035] Figure 2 is a flow chart of an audio processing method shown in an exemplary embodiment of the present application;
[0036] Figure 3 is based on Figure 2 A flowchart of another audio processing method proposed in the illustrated embodiment;
[0037] Figure 4 is a flowchart of determining a candidate interval included in the audio to be processed, shown in another exemplary embodiment of the present application;
[0038] Figure 5 is a flowchart of determining a candidate interval included in the audio to be processed, shown in another exemplary embodiment of the present application;
[0039] Figure 6 is a flowchart of another exemplary embodiment of the present application for obtaining the interval probability of the candidate interval contained in the audio to be processed;
[0040] Figure 7 is a flowchart of another exemplary embodiment of the present application showing another method of obtaining the interval probability of the candidate interval contained in the audio to be processed;
[0041] Figure 8 is a structural schematic diagram of an audio processing device shown in an exemplary embodiment of the present application;
[0042] Fig. 9 is a schematic diagram of a flow chart of audio processing performed by an audio processing device according to an exemplary embodiment of the present application;
[0043] Fig.10 An exemplary embodiment of the present application shows a schematic diagram of the structure of a computer system of an audio processing device. DETAILED DESCRIPTION
[0044] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.
[0045] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0046] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0047] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.
[0048] First of all, it should be explained that artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0049] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0050] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.
[0051] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or a local area network to achieve data calculation, storage, processing and sharing. The method shown in the embodiment of the present application can perform audio processing in the cloud, and store the processed music chorus in a remote database, which can be directly sent to other ports.
[0052] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model. It can form a resource pool, which can be used on demand and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the rapid development and application of the Internet industry, in the future, each item may have its own identification mark, which needs to be transmitted to the backend system for logical processing. Data of different levels will be processed separately. All kinds of industry data need strong system backing support, which can only be achieved through cloud computing.
[0053] The audio processing method and apparatus, audio processing device, and computer-readable storage medium proposed in the embodiments of the present application involve the artificial intelligence technology, machine learning technology, and cloud technology recorded above, and these embodiments will be described in detail below.
[0054] First see Figure 1 , Figure 1 1 is a schematic diagram of an implementation environment involved in the present application. The implementation environment includes a terminal 100 and a server 200, and the terminal 100 and the server 200 communicate with each other via a wired or wireless network.
[0055] The terminal 100 is used to receive a video or audio file and transmit the received video or audio file to the server 200; the server 200 obtains the audio to be processed according to the video or audio file, determines that the audio to be processed includes a chorus interval, and sends the determined chorus interval to the terminal 100.
[0056] Exemplarily, after the server 200 obtains the audio to be processed based on the video or audio file, it determines the candidate interval contained in the audio to be processed based on the timing characteristics of the audio to be processed, and then calculates the matching probability of the selected interval based on the starting point probability, ending point probability and interval probability corresponding to the candidate interval. If the matching probability is greater than the first preset threshold, the candidate interval is determined to be the chorus interval contained in the audio to be processed.
[0057] Among them, the user terminal 100 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, car terminals, etc., such as smart phones, tablets, laptops, computers, and any other electronic devices that can realize image visualization, which are not limited here. The server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, wherein multiple servers can form a blockchain, and the server is a node on the blockchain. The server 200 can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network) and big data and artificial intelligence platforms, which are not limited here.
[0058] See also Figure 2 , Figure 2 is a flowchart of an audio processing method shown in an exemplary embodiment of the present application. The method can be Figure 1 The method is specifically performed by the server 200 in the implementation environment shown. Of course, the method can also be applied to other implementation environments and executed by server devices in other implementation environments, and this embodiment does not limit this. Figure 2 As shown, the method at least includes steps S210 to S230, which are described in detail as follows:
[0059] S210: Determine candidate intervals included in the audio to be processed according to the time series features of the audio to be processed.
[0060] The audio to be processed in this embodiment may be obtained by preprocessing a video or music file.
[0061] Exemplarily, an audio track in a video or music file is extracted as the audio to be processed, wherein the audio to be processed includes audio timing and audio duration, etc. The audio timing indicates that audio parameters are arranged in a time sequence. For example, a tone is a music parameter of the audio to be processed, and multiple tones are arranged and combined in a certain time sequence to form a piece of music, i.e., an audio to be processed. In addition, the audio duration includes multiple audio moments, and each audio moment has a corresponding tone.
[0062] The timing feature of this embodiment is a feature related to the audio timing obtained after feature extraction of the audio to be processed, and the timing feature is a feature related to the audio duration or audio moment, which can be used to determine the candidate interval in the audio to be processed.
[0063] The candidate interval of this embodiment is not constructed arbitrarily, but is determined from the audio interval of the audio to be processed according to the timing characteristics. It can be understood that a section of the audio interval is extracted or divided from the audio interval of the audio to be processed using the timing characteristics. For example, the starting point and the end point of the candidate interval in the audio interval of the audio to be processed are obtained using the timing characteristics, and the candidate interval can be divided from the starting point and the end point. It is worth noting that step S210 of this embodiment can determine multiple candidate intervals, which is not limited here.
[0064] Exemplarily, the time series features of the audio to be processed are input into the deep network to obtain the probability matrix P(T*3), which represents the probability of each moment belonging to the start, inside, and end of the chorus. Based on this, the moment with the highest probability of the start of the chorus is used as the start time of the candidate interval, and the moment with the highest probability of the end of the chorus is used as the end time of the candidate interval, thereby determining the candidate interval. Since there may be multiple chorus intervals in a song, such as choruses A, B, and C, multiple start and end times can be determined accordingly, so multiple candidate intervals can be determined, and then multiple candidate intervals are judged and screened to determine one or more accurate chorus intervals.
[0065] S220: Calculate the matching probability of the candidate interval based on the starting point probability, the ending point probability and the interval probability corresponding to the candidate interval.
[0066] The starting point probability indicates the probability of the candidate interval belonging to the starting position or time of the chorus interval of the audio to be processed, and the value range of the starting point probability is [0, 1]. For example, the starting point probability of the candidate interval is 0.9, which means that the probability of this position or time of the candidate interval belonging to the starting position or time of the chorus interval of the audio to be processed is 0.9. The larger the probability, the higher the matching degree.
[0067] The end point probability indicates the probability of the candidate interval belonging to the end position or time of the chorus interval of the audio to be processed, and the value range of the end point probability is [0, 1]. For example, if the end point probability of the candidate interval is 0.8, it means that the probability of this position or time belonging to the end position or time of the chorus interval of the audio to be processed is 0.8. Similarly, the greater the probability, the higher the matching degree; the smaller the probability, the lower the matching degree.
[0068] In general, because the candidate interval has been determined in the above step S210, the starting point probability and the ending point probability corresponding to the candidate interval in this step are both relatively high, and the probability values of both are at least greater than 0.5. The reason is that in the previous step, the candidate interval will not be determined with a smaller starting point probability or ending point probability, that is, the interval position point or time with a relatively small starting point probability or a relatively small ending point probability will not be used as the starting position point (starting time) or the ending position point (ending time) of the candidate interval. However, in special cases, the starting point probability or the ending point probability corresponding to the determined candidate interval may be relatively small due to the uncertain factors in the previous step, which will not be described in detail here. Therefore, the description of the starting point probability and the ending point probability in general cases here does not limit the value range of the starting point probability and the ending point probability of this embodiment.
[0069] The interval probability represents the probability that the candidate interval belongs to the middle interval of the chorus interval of the audio to be processed, and the value range of the interval probability is [0, 1]. The interval probability can show the correlation between the starting point and the ending point of the candidate interval, and can accurately and completely determine all the chorus intervals of the audio to be processed.
[0070] The "calculation" in this embodiment refers to multiplying the starting point probability, the ending point probability and the interval probability, and the corresponding product is the matching probability. For example, the starting point probability of a candidate interval is 0.9, the ending point probability is 0.8, and the interval probability is 0.7, then the matching probability of the candidate interval is 0.504.
[0071] The matching probability indicates the degree of matching between the candidate interval and the chorus interval. The matching probability of each candidate interval is not necessarily the same. The higher the matching probability, the more likely the candidate interval is to be the chorus interval in the audio to be processed.
[0072] It should be noted that the interval probability introduced in this embodiment shows the correlation between the starting position point and the ending position point of the candidate interval. For example, there may be chorus intervals A, B, and C in an audio to be processed. When determining the candidate interval, the starting position point of the chorus interval A is used as the starting position point of the candidate interval, and the ending position point of the chorus interval B is used as the ending position point of the candidate interval. Because there may be other audio intervals between the chorus interval A and the chorus interval B that do not belong to the chorus interval, the obtained candidate interval is not a true chorus interval. In this case , the starting point probability and the ending point probability of the candidate interval are relatively high, such as 0.97 and 0.95, and the product of the two is as high as 0.9215. However, because the starting position point and the ending position point of the candidate interval do not belong to the same chorus interval, and there is an audio interval that does not belong to the chorus interval, the correlation between the two is low, and the interval probability will be relatively small, such as 0.12. The matching probability of the candidate interval is about 0.11, which obviously lowers the value of the matching probability, avoiding confusion between the starting position points and the ending position points of different chorus intervals, which makes the candidate interval inaccurate.
[0073] In addition, since there may be at least one chorus interval in an audio to be processed, the introduction of interval probability can determine all candidate intervals that may be the chorus intervals of the audio to be processed at one time, and accurately determine the starting and ending points of the chorus intervals A, B, and C. At the same time, it can accurately determine the middle interval of the corresponding chorus interval, and better determine at least 3 candidate intervals with a high degree of match with the chorus intervals A, B, and C.
[0074] S230: If the matching probability is greater than the first preset threshold, determining that the candidate interval is the chorus interval included in the audio to be processed.
[0075] The first prediction threshold represents the critical value of the matching probability corresponding to the chorus interval, which is usually an empirical value. In this embodiment, the first preset threshold is a number greater than 0 and less than 1, such as 0.5. If the matching probability of a candidate interval is greater than 0.5, the candidate interval is output, and the candidate interval is determined to be the chorus interval contained in the audio to be processed. The corresponding matching probability range is [0, 1]. Here, it is necessary to explain the special case where the matching probability is 1, that is, a very ideal critical state value, which can indicate that the candidate interval is 100% the chorus interval contained in the audio to be processed. In the deep network or related algorithm model introduced in the subsequent embodiments of this application, the intermediate value in the process cannot obtain the relevant critical value, and the starting point probability, the end point probability and the interval probability cannot all be 1, but it cannot be limited to the matching probability of the candidate interval of this application because of the selected deep network or related algorithm model. The critical value of 0 or 1, that is, the description of the embodiment of this application is not limited to this.
[0076] By comparing the matching probability of each candidate interval with the first preset threshold, direct numerical judgment is more intuitive and convenient. If the parameters in the above steps are adjusted, it is only necessary to fine-tune the value of the first preset threshold, that is, to change the value of the first preset threshold, and this method can be used to locate the chorus interval.
[0077] The interval probability of this embodiment can accurately determine the middle position of the chorus interval, avoid the overlap of multiple chorus intervals, and determine a complete and accurate chorus interval based on the comparison result of the matching probability and the first preset threshold.
[0078] See also Figure 3 , Figure 3 is based on Figure 2 The flowchart of another audio processing method proposed in the embodiment shown in FIG. Figure 2 The step S210 also includes steps S310 to S330, which are described in detail below:
[0079] S310: Construct a first deep network, which is composed of a convolutional neural network layer, a multi-layer long short-term LSTM network and a fully connected network layer stacked in sequence.
[0080] The first deep network of this embodiment is formed by stacking a Conv2D network layer, an LSTM network layer and a FC in sequence.
[0081] Conv2D is a convolutional neural network (CNN) with two convolution kernels. Convolutional neural networks are a type of feedforward neural networks that include convolution calculations and have a deep structure. They are one of the representative algorithms of deep learning. Among them, Conv2D (batch, new_rows, new_cols, filters), new_rows and new_cols represent the number of rows and columns of the output matrix after a convolution kernel convolves the text (since the convolution kernel can move right and downward, a matrix is obtained), and there are as many matrices as there are convolution kernel filters.
[0082] LSTM (Long Short-Term Memory) is a time recurrent neural network that is specially designed to solve the long-term dependency problem of general RNN (recurrent neural network). All RNNs have a chain form of repeated neural network modules.
[0083] The function of FC (Fully connected) is to perform nonlinear combination of the extracted features to obtain the output. The fully connected layer is located at the last part of the hidden layer of the convolutional neural network and only transmits signals to other fully connected layers. The feature map loses its spatial topological structure in the fully connected layer, is expanded into a vector and passes through the activation function.
[0084] In the early stage, a large number of audio songs with lyrics information were collected. Each song was accompanied by the start and end time of each line of lyrics, and all the chorus intervals of each audio were annotated for training the first deep network. In order to prevent errors in the position of the interval boundary during annotation, it was agreed in advance that the boundary of the chorus interval must coincide with the start and end time of two lines of lyrics. Therefore, it is only necessary to determine which lyrics belong to the chorus interval, thereby improving the consistency and effectiveness of the annotation results.
[0085] Exemplarily, the first deep network can be constructed according to the following process: randomly extract a batch of audio intervals, where the length of each audio interval is the same, for example, 30 seconds. For each audio interval, first extract Mel (map the audio spectrogram to the Mel scale to obtain the Mel spectrum), convert the one-dimensional audio signal into a two-dimensional spectrum signal (time_steps, mel_bins); then construct the first deep network, input the spectrum signal, and output the audio attributes.
[0086] This embodiment adopts a stacked Conv2D, LSTM, and FC network structure to obtain a first deep network. On the one hand, by continuously modeling the spectrum information, not only can the depth of the network be increased, but also the complete time information can be retained; on the other hand, the first deep network can output the probability of belonging to the chorus interval at each moment, so as to facilitate the subsequent determination of candidate intervals.
[0087] For the training of the first deep network, the annotation information can be combined, and the CrossEntropyLoss (cross entropy loss function, commonly used in classification tasks) loss function can be adopted to achieve the binary classification effect at each moment by counting the results. It can be obtained that the accuracy of the network model can reach 89.4%.
[0088] It should be noted that the detailed introduction to the structure of the first deep network in this embodiment does not limit the structure of the first deep network. In actual application scenarios, the first deep network may have multiple LSTM networks.
[0089] S320: Input the audio to be processed into the first deep network.
[0090] The audio to be processed in this embodiment is a complete song or video that has undergone relevant preprocessing, can be used as an input signal of the first deep network, and carries audio interval information.
[0091] S330: Extracting deep features from the first deep network as temporal features of the audio to be processed.
[0092] After the new audio to be processed is input into the first deep network, this embodiment extracts deep features as the timing features of the audio to be processed. The dimension of the timing features can be expressed as T*D, where T represents the length of the music and D represents the feature dimension at the current moment, such as D=1024. At the same time, the timing features can train the first deep network to continuously improve its performance. The dimensions of the timing features exemplified here do not mean that the timing features in this application are limited to T*D. The dimensions or dimensional units may change according to the adjustment of the first deep network. This is not limited here, and therefore cannot limit the scope of protection of this application.
[0093] In addition, the deep features extracted from the first deep network are features of the LSTM network layer, which may be features of the last LSTM network layer or features of the middle LSTM network layer. This embodiment does not specifically limit the extraction location.
[0094] This embodiment further clarifies how to obtain the timing features of the audio to be processed and introduces the composition of the first deep network. The first deep network model used in this embodiment can achieve an 89.4% prediction accuracy rate for timing features. Therefore, the audio to be processed is processed by the first deep network, and the accuracy of extracting deep features as the timing features of the audio to be processed is high. Subsequently, determining the candidate intervals contained in the audio to be processed based on this timing feature can greatly improve the accuracy of chorus positioning.
[0095] In another exemplary embodiment, based on the above step S330, specifically, the features output by the last LSTM network layer are extracted as the time series features of the audio to be processed.
[0096] The timing features of this embodiment are the features of the last LSTM network layer extracted, not the features output by the fully connected layer. The deeper the network layer features extracted, the more accurate they are. The timing features of this embodiment are more accurate in locating the subsequent chorus.
[0097] Figure 4 FIG. 1 is a flowchart of another exemplary embodiment of the present application for determining a candidate interval included in the audio to be processed. Figure 2 As shown, the process of determining the candidate intervals contained in the audio to be processed according to the time series characteristics of the audio to be processed includes steps S410 to S420, which are described in detail below:
[0098] S410: Inputting the time series features of the audio to be processed into the second deep network, obtaining the chorus position probability at each moment contained in the audio to be processed output by the second deep network, where the chorus position probability includes the starting point probability and the ending point probability.
[0099] The second deep network of this embodiment is a deep network formed by stacking Conv1D, where Conv1D has a convolutional neural network with one convolution kernel. Among them, Conv1D (batch, newsteps, filters), the number of rows of the output column vector after one convolution kernel convolves the text (of course, since the convolution kernel can only move downward, the result is one column vector), and the number of column vectors is the same as the number of convolution kernel filters.
[0100] Input the time series feature (T*D) to the second deep network, and the second deep network outputs the probability matrix P(T*3), which represents the probability of each moment belonging to the beginning, inner point, and end point of the chorus. The probability matrix P(T*3) used as an example here does not mean that the time series feature in this application can only be P(T*3). It can change its dimension according to the adjustment of the second deep network. It is not limited here and cannot limit the scope of protection of this application.
[0101] During the training phase, the corresponding label matrix G(T*3) and the probability matrix P predicted by the network are formed according to the labeled data, and the loss function is calculated according to the Cross Entropy Loss.
[0102] For example, the value of the (t, 0) element of the probability matrix P may represent the probability that the chorus starts at time t, and the value of the (t, 1) element may represent the probability that the chorus ends at time t.
[0103] S420: Determine a candidate interval from the audio to be processed according to the obtained starting point probability and ending point probability.
[0104] According to the probability matrix P(T*3), it can be determined that there are three types of probabilities at all moments in the audio interval to be processed, namely the probability of the chorus starting point, the internal point probability, and the ending point probability. Therefore, the candidate interval can be determined from the audio to be processed based on the starting point probability and the ending point probability.
[0105] This embodiment illustrates how to determine the candidate intervals contained in the audio to be processed, input the time series features into the second deep network, output the probability matrix to obtain the chorus position probability, and further determine the candidate intervals from the audio to be processed based on the starting point probability and the ending point probability. The time series features are trained in the second deep network so that the output probability matrix is more closely related to the chorus position probability, the obtained candidate intervals are more accurate, and the matching between the candidate intervals and the chorus intervals is more accurate.
[0106] Figure 5This is a flowchart of another exemplary embodiment of the present application for determining a candidate interval contained in the audio to be processed. The process of determining the candidate interval from the audio to be processed according to the obtained starting point probability and ending point probability includes steps S510 to S520, which are described in detail below:
[0107] S510: If the starting point probability corresponding to the first moment of the audio to be processed is greater than a second preset threshold or the starting point probability of all moments in the preset interval, the first moment is selected as the starting time of the candidate interval.
[0108] The preset interval represents an interval of fixed length near the position point corresponding to the first moment of the audio to be processed. For example, if the first moment of the audio to be processed is 1 minute and 30 seconds, and the length of the preset interval is 20 seconds, then the position point corresponding to the moment of 1 minute and 20 seconds in the preset interval is the starting point of the preset interval, and the position point corresponding to the moment of 1 minute and 40 seconds is the end point of the preset interval. In addition, it is only necessary to limit the first moment to be within the preset interval, and the specific starting point and end point of the preset interval are not limited. For example, if the first moment is 1 minute and 30 seconds, the position point corresponding to the moment of 1 minute and 28 seconds is the starting point of the preset interval, and the position point corresponding to the moment of 1 minute and 48 seconds is the end point of the preset interval. The length of the preset interval and the starting and end positions of the preset interval are limited here, so the protection scope of this application cannot be limited.
[0109] According to the probability matrix P(T*3) in the above step S420, if the starting point probability corresponding to a certain moment is greater than the preset second preset threshold, or greater than the starting point probability of all moments in the preset interval, the moment is defined as the starting point moment of the candidate interval.
[0110] Exemplarily, the second preset threshold is 0.9, and the starting point probability of 2 minutes and 10 seconds is 0.92, that is, greater than 0.9, so 2 minutes and 10 seconds is defined as the starting point of the candidate interval. Or the preset interval is 10 seconds, the starting point is the position corresponding to 2 minutes and 5 seconds, and the end point is the position corresponding to 2 minutes and 15 seconds, where 1 second is a moment, and there are 10 moments from 2 minutes and 5 seconds to 2 minutes and 15 seconds, and each moment corresponds to a starting point probability, which is 0.2, 0.25, 0.50, 0.6, 0.66, 0.32, 0.8, 0.5, 0.4, and 0.7, respectively. The starting point probability of 2 minutes and 10 seconds, 0.92, is greater than the starting point probabilities of all moments in the preset interval, so 2 minutes and 10 seconds is selected as the starting time of the candidate interval.
[0111] S520: If the end point probability corresponding to the second moment of the audio to be processed is greater than the third preset threshold or the end point probability of all moments in the preset interval, the second moment is selected as the end time of the candidate interval.
[0112] Similarly, if the end point probability corresponding to a certain moment is greater than a preset third preset threshold, or greater than the end point probability of all moments in the preset interval, the moment is defined as the end point moment of the candidate interval.
[0113] The second preset threshold and the third preset threshold in this embodiment can be adjusted according to different deep networks, and are not necessarily fixed and unique values. The second preset threshold and the third preset threshold can be the same or different, and the description of this example does not limit their scope. Similarly, the aforementioned first preset threshold can also be adjusted according to different deep networks, and is not necessarily a fixed and unique value. The first preset threshold can be the same as or different from the second preset threshold and / or the third preset threshold.
[0114] In a preferred embodiment, after determining the candidate intervals contained in the audio to be processed, the candidate intervals can also be evaluated. The evaluation can be performed using the starting point probability, the ending point probability and the interval probability, wherein the starting point probability and the ending point probability are respectively taken from the corresponding elements of the probability matrix P, and the interval probability needs to be further modeled.
[0115] This embodiment further illustrates how to use the starting point probability and the ending point probability to determine the candidate interval included in the audio to be processed, and the probability matrix obtains the probability of the starting point and the ending point of the audio interval belonging to the chorus, and judges by a preset threshold, or takes the moment with the largest corresponding probability of all moments in the preset interval, and takes this moment as the starting moment or the ending moment of the candidate interval, thereby determining the candidate interval included in the audio to be processed. The judgment process is more intelligent, does not require human control, and has a higher accuracy rate.
[0116] Figure 6 This is a flowchart of another exemplary embodiment of the present application for obtaining the interval probability of the candidate interval contained in the audio to be processed. After the candidate interval process of the audio to be processed is determined according to the time series characteristics of the audio to be processed, steps S610 to S620 are included, which are described in detail below:
[0117] S610: Determine interval features of candidate intervals according to time series features of the audio to be processed.
[0118] In order to further evaluate the candidate intervals, it is necessary to construct interval features to obtain interval probabilities, which are used to evaluate candidate intervals. The interval features here are features that can represent the entire audio interval within the candidate interval, and they are highly correlated with the candidate interval. In addition, the interval features are intermediate process parameters for obtaining the interval probability of the candidate interval.
[0119] S620: Input the interval feature into the third deep network to obtain the interval probability of the candidate interval output by the third deep network.
[0120] This embodiment constructs a deep network of stacked Conv1D and FC, that is, the third deep network in this embodiment. The interval features obtained by the above splicing are input into the third deep network, and the third deep network outputs the intersection and union ratio of the interval features and the real chorus interval, that is, the interval probability in this embodiment. The larger the interval probability, the higher the probability that the interval is a complete chorus interval.
[0121] This embodiment illustrates how to obtain the interval probability of the candidate interval contained in the audio to be processed, determine the interval feature of the candidate interval using the time series feature, and input the interval feature into the third deep network to obtain the interval probability of the candidate interval. This embodiment uses the time series feature to determine the candidate interval and obtain the interval feature, which increases the relevance of the pre-data, and uses the time series feature as the basic parameter for network training output to obtain other intermediate parameters.
[0122] Figure 7 This is a flowchart of another exemplary embodiment of the present application showing another method for obtaining the interval probability of the candidate interval contained in the audio to be processed. After the candidate interval process of the audio to be processed is determined according to the time series characteristics of the audio to be processed, it includes steps S710 to S730, which are described in detail below:
[0123] S710: extracting features from the candidate interval, the interval near the starting point of the candidate interval, and the interval near the ending point of the candidate interval to obtain corresponding candidate interval features, wherein the feature length of each candidate interval feature is the same.
[0124] Among them, the interval near the starting point of the candidate interval and the interval near the ending point of the candidate interval can be preset in advance so that the interval lengths of the two are fixed. For example, after determining the candidate interval, the corresponding starting point and ending point are obtained, and an interval with a preset interval length is automatically selected with the starting point as the midpoint as the interval near the starting point of the candidate interval. Alternatively, an interval with a preset interval length can be selected with the starting point as the 1 / 3 point as the interval near the starting point of the candidate interval. It is not limited here. Similarly, the interval near the ending point of the candidate interval is obtained.
[0125] This embodiment extracts features of a candidate interval, an interval near a starting point of the candidate interval, and an interval near an ending point of the candidate interval to obtain features of a fixed length.
[0126] S720: Concatenate the features of each candidate interval to obtain an interval feature of the candidate interval.
[0127] In a preferred embodiment, the features of the candidate interval, the interval near the starting point of the candidate interval, and the interval near the ending point of the candidate interval can be extracted as three fixed-length features to facilitate the subsequent connection of the fully connected layer of the third deep network. The three fixed-length features are concatenated to obtain the interval features of the candidate interval.
[0128] S730: Input the interval feature into the third deep network to obtain the interval probability of the candidate interval output by the third deep network.
[0129] This step is similar to the above step S620. The third deep network is a network formed by stacking Conv1D and FC. The interval probability obtained in this step can be used to evaluate the candidate interval.
[0130] This embodiment provides another way to obtain interval probability, which is to extract features from candidate intervals, positions near the starting point and the end point of the candidate interval, splice the obtained fixed-length features into interval features of the candidate interval, and input the interval features into the third deep network to obtain the interval probability of the candidate interval. This embodiment extracts features from the relevant intervals separately, and the constructed interval features can be better closely connected with the fully connected layer of the third deep network, and the intersection-over-union ratio of the output interval features and the real chorus interval, that is, the interval probability, is more accurate.
[0131] It should be noted that a song may have choruses A, B, and C. If the candidate interval is determined only based on the starting point probability and the ending point probability, the starting point or ending point of the candidate interval may be wrong. For example, the starting point of the chorus B is used as the starting point of the candidate interval, and the ending point of the chorus C is used as the ending point of the candidate interval. Because there may be a music interval between the chorus B and the chorus C that does not belong to the chorus interval, the candidate interval is not an accurate chorus interval. The introduction of interval probability can better avoid this situation. In addition, in the subsequent screening of candidate intervals, there may be candidate intervals with a large number of overlaps, which can be searched and screened using NMS (Non-Maximum Suppression).
[0132] Secondly, there may be multiple chorus intervals in a song, namely, chorus A, chorus B and chorus C. If during the recognition process, the chorus interval of chorus A is determined according to the chronological order, it is possible that the chorus intervals of chorus B and chorus C will no longer be recognized subsequently. In this way, for music with multiple chorus intervals, the complete chorus interval is not determined. The present application introduces interval probability to solve the technical problem of not being able to completely identify all chorus intervals of the audio to be processed.
[0133] Therefore, the introduction of interval probability in the audio processing method of the present application can not only avoid positioning errors of the starting points of candidate intervals, but also ensure that all chorus intervals of the audio to be processed are identified.
[0134] The method of the present application can be applied to artificial intelligence audio processing equipment and can be processed in batches. After the audio is input, the device can automatically and accurately locate all the chorus intervals of the music. The mAP (mean Average Precision, the average accuracy of all categories) of this method can reach 88.6%, the fault tolerance rate is only 2s, and the accuracy of determining the start and end times of the chorus reaches 93%, which is more accurate than the existing technology.
[0135] mAP is a very important metric in the field of object detection in machine learning, used to measure the performance of object detection algorithms. Generally speaking, mAP (also known as average accuracy) is the weighted average of the average accuracy (AP) of all categories of detection.
[0136] Another aspect of the present application also provides an audio processing device, such as Figure 8 As shown, Figure 8 : is a schematic diagram of the structure of an audio processing device shown in an exemplary embodiment of the present application. The audio processing device includes:
[0137] A determination module 810 is configured to determine a candidate interval included in the audio to be processed according to a time sequence feature of the audio to be processed;
[0138] The calculation and positioning module 820 is configured to calculate the matching probability of the candidate interval based on the starting point probability, the ending point probability and the interval probability corresponding to the candidate interval; if the matching probability is greater than the first preset threshold, the candidate interval is determined to be the chorus interval included in the audio to be processed.
[0139] In a preferred embodiment, the interface returns the music chorus interval information to other ports in the form of Json. This lightweight data exchange format is more convenient and quick. Here, only an example is given to illustrate in which data exchange format the music chorus interval information is returned. It does not mean that the device of this embodiment can only return the music chorus interval information through this data exchange format. It is not limited here.
[0140] In another embodiment, the audio processing device further comprises:
[0141] The subtask module is configured to construct a first deep network, which is composed of a convolutional neural network layer, a multi-layer long-short term LSTM network and a fully connected network layer stacked in sequence; input the audio to be processed into the first deep network; extract deep features from the first deep network as the time series features of the audio to be processed.
[0142] In addition, the subtask module can classify audio attributes and output audio attribute information.
[0143] In another embodiment, the subtask module further includes:
[0144] The extraction unit is configured to extract the features output by the last LSTM network layer as the time series features of the audio to be processed.
[0145] In another embodiment, the determination module 810 includes:
[0146] A time series modeling unit is configured to input the time series features of the audio to be processed into a second deep network, and obtain the chorus position probability at each moment contained in the audio to be processed output by the second deep network, where the chorus position probability includes a starting point probability and an ending point probability;
[0147] The candidate interval construction unit is configured to determine a candidate interval from the audio to be processed according to the obtained starting point probability and ending point probability.
[0148] In another embodiment, the candidate interval construction unit is configured to select the first moment as the starting moment of the candidate interval if the starting point probability corresponding to the first moment of the audio to be processed is greater than a second preset threshold or the starting point probability of all moments within the preset interval; if the ending point probability corresponding to the second moment of the audio to be processed is greater than a third preset threshold or the ending point probability of all moments within the preset interval, then select the second moment as the ending time of the candidate interval.
[0149] In another embodiment, the calculation positioning module 820 includes:
[0150] An interval feature extraction unit configured to determine the interval features of the candidate intervals according to the time series features of the audio to be processed;
[0151] The interval probability extraction unit is configured to input the interval feature into the third deep network to obtain the interval probability of the candidate interval output by the third deep network.
[0152] In another embodiment, the calculation positioning module 820 includes:
[0153] An interval feature extraction unit is configured to extract features from a candidate interval, an interval near a starting point of the candidate interval, and an interval near an ending point of the candidate interval, respectively, to obtain corresponding candidate interval features, wherein the feature lengths of each candidate interval feature are the same; and to concatenate the features of each candidate interval to obtain an interval feature of the candidate interval;
[0154] The interval probability extraction unit is configured to input the interval feature into the third deep network to obtain the interval probability of the candidate interval output by the third deep network.
[0155] Fig. 9This is a flowchart of an audio processing device for audio processing shown in an exemplary embodiment of the present application. It simply represents the entire audio processing process, involving relevant modules of the audio processing device. The description of the relevant modules in this embodiment is somewhat different from that in the above embodiment, but their functions and effects are basically the same. In order to implement the audio processing method of the present application, the module structure of the audio processing device can be re-divided or combined according to actual conditions. The description here cannot limit the module structure of the audio processing device of the present application.
[0156] The audio processing device of this embodiment includes:
[0157] Subtask module: Build a deep learning network to classify the local audio attributes of the input music. The subtask module is mainly used to extract timing features in the embodiment of the present application.
[0158] Exemplarily, a video or audio file URL (Uniform Resource Locator) is input, the audio track is extracted as the algorithm input music to obtain the audio to be processed, the audio to be processed is input into the first deep network, and the features output by the last LSTM network layer of the first deep network are extracted as the time series features of the audio to be processed.
[0159] Timing modeling module: Processing the obtained timing features in the second deep network to obtain the chorus position probability at each moment contained in the audio to be processed output by the second deep network, the chorus position probability includes the starting point probability and the ending point probability, which is similar to the timing modeling unit in the above-mentioned determination module; determining the candidate interval from the audio to be processed according to the obtained starting point probability and the ending point probability, that is, determining the candidate interval contained in the audio to be processed, which is similar to the candidate interval construction unit in the above-mentioned determination module.
[0160] The time series modeling module of this embodiment is equivalent to the above-mentioned determination module, and is configured to determine the candidate intervals included in the audio to be processed according to the time series features of the audio to be processed.
[0161] Candidate interval evaluation module: selects candidate intervals with higher evaluation scores as chorus intervals.
[0162] The candidate interval evaluation module of this embodiment is equivalent to the above-mentioned calculation and positioning module, and is configured to calculate the matching probability of the candidate interval based on the starting point probability, the ending point probability and the interval probability corresponding to the candidate interval; if the matching probability is greater than the first preset threshold, the candidate interval is determined to be the chorus interval contained in the audio to be processed, and finally the information of the audio chorus interval is returned to the client in the form of Json (JavaScriptObject Notation, a lightweight data exchange format). The hierarchical structure of the data exchange format is concise and clear, and it is an ideal data exchange language. It is easy for people to read and write, and it is also easy for machines to parse and generate, which can effectively improve network transmission efficiency.
[0163] In particular, the present application is an independent task of extracting timing features, that is, in the subtask module, the audio to be processed is input into the first deep network, and the deep features are extracted from the first deep network as the timing features of the audio to be processed, and then the timing features are input into the second deep network in the determination module to obtain the chorus position probability of each moment contained in the audio to be processed. Subsequently, the third deep network in the positioning module can be calculated based on the timing feature input to obtain a dense interval probability, so that the complete chorus interval can be accurately determined later. Three deep networks are used for modeling and learning in the audio processing process of the present application to make the matching probability more accurate.
[0164] It should be noted that the audio processing device provided in the above embodiment and the audio processing method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here.
[0165] Another aspect of the present application further provides an audio processing device, comprising: a receiving device, for receiving audio to be processed;
[0166] The controller is electrically connected to the receiving device to execute the audio processing methods in the above embodiments.
[0167] See also Fig.10 , Fig.10 1 is a schematic diagram of the structure of a computer system of an audio processing device shown in an exemplary embodiment of the present application, which shows a schematic diagram of the structure of a computer system of an audio processing device suitable for implementing an embodiment of the present application.
[0168] It should be noted that Fig.10 The computer system 1000 of the audio processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0169] like Fig.10As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003, such as executing the method in the above embodiment. In the RAM 1003, various programs and data required for system operation are also stored. The CPU 1001, the ROM 1002 and the RAM 1003 are connected to each other through the bus 1004. The input / output (I / O) interface 1005 is also connected to the bus 1004.
[0170] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed so that a computer program read therefrom is installed into the storage section 1008 as needed.
[0171] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1009, and / or installed from a removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.
[0172] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.
[0173] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0174] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0175] Another aspect of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned audio processing method is implemented. The computer-readable storage medium may be included in the audio processing device described in the above embodiment, or may exist independently without being assembled into the audio processing device.
[0176] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the audio processing method provided in each of the above embodiments.
[0177] According to one aspect of an embodiment of the present application, a computer system is also provided, including a central processing unit (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM), such as executing the method in the above embodiment. In RAM, various programs and data required for system operation are also stored. CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.
[0178] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.
[0179] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.
Claims
1. An audio processing method, characterized in that: include: Determining a candidate interval included in the audio to be processed according to a time series feature of the audio to be processed; The matching probability of the candidate interval is calculated based on the starting point probability corresponding to the starting position point of the candidate interval, the ending point probability corresponding to the ending position point, and the interval probability; wherein the interval probability represents the probability that the candidate interval belongs to the middle interval of the chorus interval of the audio to be processed, and the interval probability represents the correlation between the starting position point and the ending position point of the candidate interval; If the matching probability is greater than a first preset threshold, the candidate interval is determined to be the chorus interval included in the audio to be processed.
2. The method according to claim 1, characterized in that Before determining the candidate intervals included in the audio to be processed according to the time sequence features of the audio to be processed, the method includes: Constructing a first deep network, wherein the first deep network is formed by stacking a convolutional neural network layer, a multi-layer long short-term LSTM network, and a fully connected network layer in sequence; Inputting the audio to be processed into the first deep network; Deep features are extracted from the first deep network as time series features of the audio to be processed.
3. The method according to claim 2, characterized in that The step of extracting deep features from the first deep network as time series features of the audio to be processed includes: The features output by the last LSTM network layer are extracted as the time series features of the audio to be processed.
4. The method according to claim 1, characterized in that: The step of determining the candidate intervals included in the audio to be processed according to the time sequence features of the audio to be processed includes: Inputting the time series features of the audio to be processed into a second deep network, obtaining the chorus position probability at each moment contained in the audio to be processed output by the second deep network, wherein the chorus position probability includes a starting point probability and an ending point probability; A candidate interval is determined from the audio to be processed according to the obtained starting point probability and ending point probability.
5. The method according to claim 4, characterized in that The step of determining a candidate interval from the audio to be processed according to the obtained starting point probability and ending point probability includes: If the starting point probability corresponding to the first moment of the audio to be processed is greater than the second preset threshold or the starting point probability of all moments in the preset interval, selecting the first moment as the starting time of the candidate interval; If the end point probability corresponding to the second moment of the audio to be processed is greater than a third preset threshold or the end point probability of all moments in a preset interval, the second moment is selected as the end time of the candidate interval.
6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Determining the interval features of the candidate intervals according to the time series features of the audio to be processed; The interval feature is input into a third deep network to obtain the interval probability of the candidate interval output by the third deep network.
7. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Extracting features from the candidate interval, an interval near the starting point of the candidate interval, and an interval near the ending point of the candidate interval to obtain corresponding candidate interval features, wherein the feature lengths of each candidate interval feature are the same; Concatenating the features of each candidate interval to obtain the interval feature of the candidate interval; The interval feature is input into a third deep network to obtain the interval probability of the candidate interval output by the third deep network.
8. An audio processing device, characterized in that: include: A determination module, configured to determine a candidate interval included in the audio to be processed according to a time sequence feature of the audio to be processed; A calculation and positioning module is configured to calculate the matching probability of the candidate interval based on the starting point probability corresponding to the starting position point of the candidate interval, the ending point probability corresponding to the ending position point, and the interval probability; if the matching probability is greater than a first preset threshold, the candidate interval is determined to be the chorus interval contained in the audio to be processed; wherein the interval probability represents the probability that the candidate interval belongs to the middle interval of the chorus interval of the audio to be processed, and the interval probability expresses the correlation between the starting position point and the ending position point of the candidate interval.
9. An audio processing device, characterized in that: include: A receiving device, used for receiving audio to be processed; A controller is electrically connected to the receiving device to execute the method described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 7.
11. A computer program product, characterized in that The method comprises computer instructions, which implement the method according to any one of claims 1 to 7 when executed by a processor.
Citation Information
Patent Citations
Audio band extraction method and extraction device
CN103824555A
Refrain extracting apparatus and method
CN106409311A
Multimedia data processing method and device, equipment and storage medium
CN113573161A
Grammar generation support program for speech recognition
JP2011191752A
Method for extracting the key word of dialogic style voice
KR100286293B1
