Audio-based special effect generation method and device, equipment and storage medium
By identifying chorus clips in music and generating special effects control strategies, the problem of poor user experience in the existing technology is solved, and the special effects generation matching the music is achieved, which improves the richness and user experience of audio special effects.
Patent Information
- Application Number
- CN202410970197.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-15
AI Technical Summary
The prior art is difficult to effectively identify and utilize chorus clips in music to generate special effects matching the music, resulting in poor user experience.
By identifying the chorus clips in the target audio and generating special effects control strategies based on their characteristics, including building a self-similar matrix to filter chorus clips, combining audio and text features, using machine learning models to predict the high-pitched and pleasantness, and generating rich audio effects .
It improves the richness and user experience of audio special effects, can more accurately identify chorus clips, enhance the sense of musical atmosphere, and is suitable for controlling lights and other special effects.
Smart Images

Figure CN118969019A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to the technical fields of artificial intelligence, voice technology, large models, etc. Background Art
[0002] Music can be played through the speakers. When the speakers play music, they can be displayed with lighting effects to enhance the atmosphere when listening to music.
[0003] In the related art, different lighting effects are mainly displayed by the sound pressure level of the music audio. For example, when the sound pressure level is large, the light changes faster; when the sound pressure level is small, the light changes slower. Summary of the invention
[0004] The present disclosure provides a method, apparatus, device and storage medium for generating special effects based on audio.
[0005] According to one aspect of the present disclosure, a method for generating special effects based on audio is provided, comprising:
[0006] Identify the chorus segment in the target audio;
[0007] Based on the chorus segment, the corresponding special effects control strategy is generated.
[0008] According to another aspect of the present disclosure, there is provided an audio-based special effect generation device, comprising:
[0009] A recognition module, used to identify the chorus segment in the target audio;
[0010] The generation module is used to generate corresponding special effect control strategies based on the chorus segment.
[0011] According to another aspect of the present disclosure, there is provided an electronic device, comprising:
[0012] at least one processor; and
[0013] a memory communicatively connected to the at least one processor; wherein,
[0014] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.
[0015] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.
[0016] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.
[0017] In the disclosed embodiment, the characteristics of the audio can be controlled based on the chorus segment so that the special effects can switch the theme and core content of the audio, so that the generated special effects can better highlight the characteristics of the audio, thereby improving the user experience.
[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0020] Figure 1 is a flowchart of a method for generating special effects based on audio according to an embodiment of the present disclosure;
[0021] Figure 2 is a schematic diagram of a process of identifying a chorus segment in a target audio according to an embodiment of the present disclosure;
[0022] Figure 3 is a schematic diagram of a process of selecting chorus segments based on a constructed audio self-similarity matrix according to an embodiment of the present disclosure;
[0023] Figure 4 is a schematic diagram of similarity matrices according to an embodiment of the present disclosure;
[0024] Figure 5 is a structural schematic diagram of an audio-based special effect generation device according to an embodiment of the present disclosure;
[0025] Figure 6 It is a block diagram of an electronic device used to implement the audio-based special effect generation method of the embodiment of the present disclosure. DETAILED DESCRIPTION
[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0027] The terms "first", "second", etc. in this disclosure are used to distinguish similar objects, and are not necessarily used to describe a specific order or precedence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, including a series of steps or units. The method, system, product or device is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0028] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0029] When the speaker plays music, it will be displayed with lighting effects to enhance the atmosphere when listening to music. Not only can the lighting effects set off the atmosphere of the music, but other special effects controls can also set off the atmosphere of the music.
[0030] In view of this, the embodiment of the present disclosure provides a method for generating special effects based on audio, which can identify the chorus part in music and automatically generate special effects according to the characteristics of the chorus.
[0031] It should be noted that the special effects generated by the embodiments of the present disclosure can not only be used to control lighting effects, but also can be used to control other special effects, such as robot dance movements, robot expression changes, etc. The embodiments of the present disclosure are applicable to any scene that produces special effects based on the rhythm of music.
[0032] like Figure 1 FIG. 1 is a flow chart of a method for generating special effects based on audio according to an embodiment of the present disclosure, comprising:
[0033] S101, identifying a chorus segment in a target audio.
[0034] The target audio is the song to be played when the speaker is used. The chorus segment is often repeated multiple times, or summarizes the theme or emotion of the song, and is often the essence and climax of the entire audio. Therefore, identifying the chorus segment can well match the rhythm of the audio to generate special effects.
[0035] S102, generating a corresponding special effect control strategy based on the chorus segment.
[0036] Among them, in the special effect control strategy, the special effect of the chorus segment is different from the special effect of the audio part other than the chorus segment in the target audio. When implemented, the corresponding special effect can be generated according to the content, rhythm, emotion and other characteristics of the chorus segment. For example, the special effect can control the lighting changes of the speaker to adapt to the musical atmosphere conveyed by the chorus segment.
[0037] In the disclosed embodiment, a corresponding special effect control strategy is generated based on the chorus segment, and different audio special effects can be generated according to the chorus characteristics of different music, which can enhance the richness of the audio special effects and thus improve the user experience.
[0038] In the disclosed embodiment, not only the chorus segment can be identified, but also the main song, blank silence, a cappella, etc. of the song structure can be identified to identify the complete structure of the audio. Among them, the part of the lyrics text except the chorus can be identified as the main song, the part with almost no fluctuation according to the waveform of the audio can be identified as the blank silence part, and the part with voice but no background music can be identified as the a cappella part. During implementation, a special effect strategy can be generated based on this structure.
[0039] Among them, during implementation, in order to quickly and accurately identify the chorus segment in the target audio, such as Figure 2 The following are shown:
[0040] S201, constructing a first self-similarity matrix based on the similarities between multiple sentences in the lyrics text of the target audio.
[0041] During implementation, the lyrics text can be segmented based on the timestamp to obtain multiple sentences of the lyrics text. For example, two adjacent timestamps can be understood as one sentence. Thus, the segmentation points of each sentence in the lyrics text are obtained. Then, a self-similarity matrix is constructed based on the multiple sentences obtained by segmentation. For example, if each line of lyrics in the lyrics text corresponds to the timestamp of the audio playback progress, during implementation, multiple sentences can be segmented based on the timestamps of the following examples:
[0042] [00:00:00]Song title
[0043] [00:02:21]Word
[0044] [00:04:42]Qu
[0045] [00:05:42] Once in front of my eyes
[0046] [00:06:42]But it disappeared
[0047] …
[0048] The first self-similarity matrix in the disclosed embodiment, i.e., the text self-similarity matrix (Text Self-Similarity Matrix), generally refers to a matrix representation used to measure the similarity between various parts in a text in text analysis or natural language processing. Multiple diagonally continuous elements of this matrix can reflect the similarity between the segments corresponding to these elements.
[0049] To build the first self-similarity matrix of the lyrics text, each of the n lyrics needs to be calculated for text similarity with all the remaining n-1 lyrics. For example, the text similarity can be calculated by using edit distance, cosine similarity, Euclidean distance, Jaccard (Jaccard Index) similarity coefficient, or a semantic-based neural network model. This can construct an n×n matrix containing the similarity between any two lyrics.
[0050] The first self-similarity matrix can reflect the degree of similarity between sentences in the lyrics, that is, whether the sentences tend to be consistent in text content.
[0051] S202, based on the first self-similarity matrix, searching for multiple first paragraphs that appear repeatedly in the lyrics text.
[0052] During implementation, multiple first paragraphs that appear repeatedly in the lyrics text can be queried based on the following method:
[0053] Step A1, based on the length of the lyrics text, determine the shortest length and the maximum length of the repeated paragraphs.
[0054] For example, if the target audio contains n lyrics, the length of the repetition may be [2, n / 2], where the shortest length is 2 and the maximum length is half of the entire lyrics.
[0055] Step A2, according to the principle that the segment length is within the range from the shortest length to the maximum length, searching for repeated segments under different segment lengths from the first self-similarity matrix to obtain a plurality of repeated preliminary selected segments.
[0056] During implementation, the traversal method is used to find repeated paragraphs of different lengths. Sentences of length 2, 3, ... to n / 2 are traversed in sequence: for the segment length L (2≤L≤n / 2), a certain i~i+L segment is selected starting from the i-th sentence, and then the remaining text is searched for repeated segments. If so, this segment is a repeated candidate segment. During implementation, based on the traversal method, repeated traversal is avoided as much as possible. For example, for each i value, all segments of length L starting from the i value can be traversed. For example, when the i value is 1, it means that starting from the first sentence, segments of length 2 to length n / 2 are traversed. In this way, every possible segment length at each i value has been traversed. Then when traversing to each i value, in order to avoid repetition, the traversal can be started from the subsequent i+L+1 sentence.
[0057] For example, if there are 10 lines of lyrics, and the segment length is two lines, that is, the L value is 2, then find out whether there is a paragraph that repeats the paragraph from the remaining lyrics. Similarly, when the segment length is three lines, that is, the L value is 3, it means that the total length of the segment including the i-th line of lyrics is 3. Traverse the remaining lyrics to find out whether there is a paragraph that repeats the paragraph. Similarly, find multiple paragraphs that repeat in different lengths as preliminary paragraphs. During the traversal process, the judgment criterion for the similarity between the i~i+L segment and the k~k+L segment can be, when the values of each element in the first self-similarity matrix are continuous space values, confirm whether the text similarity of the corresponding sentences in the i~i+L segment and the k~k+L segment is higher than the preset value. If so, the two segments are repeated. Alternatively, in the case where the values of each element in the first self-similarity matrix binarize the value range of the text similarity, it is identified whether the text similarities of the corresponding sentences in the i~i+L segment and the k~k+L segment are both 1. If so, it is determined that the two are repeated.
[0058] Since there may be multiple repeated segments under each segment length, they need to be merged and repeated in step A3, that is, the sub-repeats with dependencies are removed.
[0059] Step A3: performing a deduplication operation on the multiple initially selected paragraphs to obtain multiple first paragraphs.
[0060] Through the deduplication operation, the longest and most repeated preliminary paragraphs are selected. For example, {ABC, ABC} is less repeated than {ABC, ABC, ABC}, or {AB, AB, AB} is shorter than {ABC, ABC, ABC}. The short and less repeated segments need to be deduplicated to obtain multiple first paragraphs. That is, the first paragraph finally obtained in this example is {ABC, ABC, ABC}.
[0061] In the disclosed embodiment, by determining the range of repeated paragraphs based on the length of the lyrics text, and then searching for repeated paragraphs of different lengths, it is possible to find potentially repeated paragraphs more comprehensively. Then, by performing a deduplication operation, some unnecessary repetitions can be removed, making the first paragraph more representative and accurate, thereby helping to find the chorus segment more accurately and improving the screening effect and quality.
[0062] S203, determining a chorus segment based on the multiple first paragraphs.
[0063] In a possible implementation manner, the first paragraph that is repeated the most may be selected as the chorus segment, or the longest first paragraph may be selected as the chorus segment.
[0064] In addition, as the core part of the audio, the chorus segment is easy to resonate with the listeners and is the most touching. Therefore, a machine learning model that can predict the high pitch and pleasantness of the music segment can be used to score multiple first paragraphs separately, and then the first paragraph with the highest score can be selected as the chorus segment.
[0065] Of course, in order to further improve the accuracy of screening out the chorus segment, the following text of the disclosed embodiment will continue to introduce how to use audio to assist in screening out the chorus segment.
[0066] To sum up, in the embodiments of the present disclosure, by constructing a first self-similarity matrix, the similarity between sentences in the lyrics text can be quantified, and the chorus part of the song, that is, the repeated part, can be more effectively identified, thereby providing more accurate chorus information for subsequent changes in audio special effects.
[0067] In some embodiments, in most cases, the lyrics of the chorus segment of the audio are basically repeated. However, in some special cases, the chorus segment of the song will change slightly, resulting in the chorus segment not necessarily being reproduced in an identical manner. For example, sometimes the corresponding two lines of lyrics are completely different, that is, the text similarity may approach 0. In this case, the candidate chorus segment obtained by constructing the first self-similarity matrix based on the similarity between multiple sentences in the lyric text of the target audio may miss or misselect the chorus segment. However, after a lot of research and comparison, it was found that for repeated chorus segments, even if the text content is not strictly consistent, the audio parts of the corresponding two sentences have a high degree of similarity. For this reason, the disclosed embodiments propose to combine the listening experience of the audio to further screen out the accurate chorus segment. Its specific implementation is as follows: Figure 3 As shown, based on multiple first paragraphs, determining the chorus segment can also be implemented as follows:
[0068] S301, using an audio self-similarity matrix constructed by audio similarities between audio segments corresponding to a plurality of sentences, identifying breakpoints in a first self-similarity matrix. The breakpoints indicate repeated segments that have different text contents and are suspected to be chorus parts.
[0069] Among them, the breakpoint, as the name implies, means that a complete segment is divided into at least two segments. Due to the inconsistency of the lyrics content, it may lead to complete repeated segments that cannot be identified simply based on the first self-similarity matrix. Therefore, it is necessary to identify the breakpoint to patch up different first paragraphs and obtain complete repeated segments. In implementation, the breakpoint can also be understood as a chorus segment that indicates that there is a significant difference in the text content but the audio similarity meets the preset conditions.
[0070] In implementation, the audio self-similarity matrix can be constructed based on the following method:
[0071] Step B1, dividing the target audio into multiple audio segments according to the segmentation points of multiple sentences.
[0072] According to the segmentation points of the provided multiple sentences, the target audio is cut and divided into multiple audio segments. In implementation, as described above, the segmentation point can be determined based on the starting point of the timestamp of each lyric in the song.
[0073] Step B2: extracting audio features of multiple audio clips.
[0074] Audio features are the auditory sense of audio. Audio features can capture the acoustic characteristics of audio signals, such as pitch, rhythm, timbre, etc. When implementing, audio features such as chroma (which converts audio signals into a series of 12-dimensional vectors) can be used. Neural network models can also be used to extract audio features of audio clips.
[0075] Step B3: Determine the similarity between the audio features of the multiple audio clips as the audio similarity.
[0076] Step B4: construct an audio self-similarity matrix based on audio similarity.
[0077] The audio self-similarity matrix is a matrix that represents the audio similarity within an audio sequence, and can reflect the similarity between various parts of the audio sequence. The structure is the same as the first self-similarity matrix, and each element in the audio self-similarity matrix represents the similarity between the corresponding two audio segments.
[0078] In the disclosed embodiment, by extracting audio features of multiple audio clips to construct an audio self-similarity matrix, the structural characteristics of the audio can be fully understood, thereby improving the accuracy of chorus recognition and providing support for subsequent identification of breakpoints in the first self-similarity matrix through the audio self-similarity matrix.
[0079] In the embodiment of the present disclosure, based on the constructed audio self-similarity matrix, the breakpoints in the first self-similarity matrix are identified, which is specifically implemented as follows:
[0080] Step C1, based on the multiple first paragraphs, binarize the first self-similarity matrix to obtain a second self-similarity matrix, so that the values of the two sentences corresponding to the repeated paragraphs in the second self-similarity matrix are the first specified values.
[0081] It should be noted here that the second self-similarity matrix after binarization only includes sentences in repeated segments whose values are the first specified value (such as 1), and the values of sentences in non-repeated segments are still the second specified value (such as 0) even if the text similarity between sentences is high.
[0082] like Figure 4As shown, in the second self-similarity matrix, the similarity values of the two corresponding sentences in the repeated paragraph are set to the first specified value, for example, 1 indicates that these sentences are very similar and belong to the same repeated sentence. The values of other positions are set to the second specified value, for example, 0 indicates that the similarity between these sentences is low.
[0083] Step C2, using the audio self-similarity matrix, correcting the second self-similarity matrix so that the values corresponding to the two sentences whose audio similarities meet the preset conditions in the second self-similarity matrix are candidate values.
[0084] The audio self-similarity matrix is used to compare the similarity between audios and determine whether two sentences are similar. Based on the second self-similarity matrix, the second self-similarity matrix is corrected according to the judgment result of the audio self-similarity matrix. The correction process mainly corrects the potential similar sentences that are not identified in the first self-similarity matrix into suspected similar sentences. For example, if the second specified value originally represented as dissimilar in the second self-similarity matrix is judged to be a similar sentence through audio features, its element value is modified to a candidate value. Figure 4 In the modified second self-similarity matrix shown in , the candidate values of (i2, j1), (i2, j4), and (i3, j4) are changed from the original "0" to "1?". This indicates that the two sentences are suspected to be highly similar, and subsequent operations are performed to further identify whether they need to be marked as similar.
[0085] In view of the fact that the audio similarity values of different sentences in the same audio are not very different, but they can also reflect the similarity between different sentences, therefore, in the embodiment of the present disclosure, the preset conditions may include sorting the audio similarity from high to low, and the first N sentences in the audio similarity sorting, where N is a positive integer greater than 1. It can also be understood as adjusting the values of the "audio self-similarity matrix": the first N are adjusted to the first specified value, such as 1; the values outside the first N are adjusted to the second specified value, such as 0.
[0086] In the disclosed embodiment, by selecting the top N sentences with the highest similarity, it is possible to ensure that the selected sentences have a high similarity in audio, thereby more accurately identifying and correcting the breakpoints in the text self-similarity matrix. This method helps to reduce misrecognition due to text differences, especially when dealing with chorus variants or repeated paragraphs.
[0087] Step C3: searching for a target sequence in the diagonal direction of the modified second self-similarity matrix, where the target sequence is constructed by continuous first designated values and / or candidate values.
[0088] Step C4: determining the candidate values in the target sequence as the breakpoints in the first self-similarity matrix.
[0089] The target sequence is searched in the diagonal direction of the modified second self-similarity matrix. The target sequence is a sequence constructed by continuous first specified values and / or candidate values. Figure 4 As shown in the second self-similarity matrix after correction, the shaded part is the target sequence to be found, and the "1?" at the position (i3, j4) indicates that the position is a breakpoint.
[0090] In the disclosed embodiment, with the assistance of the audio self-similarity matrix, the structure of the audio can be more comprehensively understood, and the chorus part in the audio can be more accurately identified. In particular, when the chorus has changes in the text, the breakpoints can be accurately identified and corrected in the second self-similarity matrix, thereby improving the recognition accuracy of the chorus segment.
[0091] S302: repairing the plurality of first paragraphs based on the breakpoint, so that the first paragraphs before and after the breakpoint are merged into a new first paragraph, so as to obtain a plurality of second paragraphs.
[0092] The overall concept of the disclosed embodiment is to find the breakpoints to repair the potential cut complete segments, so as to improve the accuracy of chorus segment recognition. In implementation, this purpose can be achieved only from the operation and optimization of the matrix, which will be described in detail later.
[0093] It can be understood that the first paragraphs before and after the breakpoint should be connected into a complete paragraph. In the first self-similarity matrix constructed based on the lyrics text, due to the difference in text content, a breakpoint appears, which cuts the complete paragraph apart. Then, the embodiment of the present disclosure uses the audio self-similarity matrix to find such a breakpoint, so as to stitch together the multiple first paragraphs that were cut apart, and restore the original repeated complete first paragraph, thereby improving the fluency of the text and the compactness of the information.
[0094] by Figure 4 For example, for ease of operation, in the embodiment of the present disclosure, multiple first paragraphs can be repaired based on the breakpoint in the following manner, so that the first paragraphs before and after the breakpoint are merged into a new first paragraph:
[0095] Step D1, resetting the breakpoints in the second self-similar matrix to the first specified value, and setting the candidate values other than the breakpoints to the second specified value, so as to obtain a third self-similar matrix.
[0096] like Figure 4 As shown in the third self-similar matrix, the candidate value "1?" at the breakpoint position in the second self-similar matrix is reset to the first specified value "1". The candidate values other than the breakpoints are set to the second specified value, such as setting the candidate values "1?" at the positions (i2, j1) and (i2, j4) to the second specified value "0".
[0097] Step D2, based on the third self-similarity matrix, searching for multiple paragraphs that appear repeatedly in the lyrics text to obtain multiple second paragraphs.
[0098] After obtaining the third self-similarity matrix, based on the above method, the third self-similarity matrix is traversed in sequence to find multiple paragraphs that appear repeatedly in the lyrics text, and after deduplication, multiple second paragraphs are obtained. For details, please refer to the operations of step A1 to step A3, the difference is that the object processed here is the third self-similarity matrix.
[0099] In the disclosed embodiment, multiple paragraphs that appear repeatedly in the lyrics text are found through the third self-similarity matrix, and the first paragraphs before and after the breakpoint are merged into the second paragraph, so that the complete chorus segment can be further found, thereby improving the accuracy of chorus segment recognition.
[0100] S303, selecting a chorus segment from the plurality of second paragraphs.
[0101] In addition to the aforementioned method of selecting the longest or most repeated second paragraph as the chorus segment, a scoring method is also used to select the chorus segment.
[0102] Furthermore, due to the complexity of the lyrics text, the multiple second segments selected above may still cut two sentences with related meanings. For this reason, the present disclosure can use a large language model (LLM) to perform text analysis on the lyrics to determine the semantic relevance between the sentences in the lyrics. If the LLM believes that the two sentences are closely semantically connected, they should not be the segmentation points, and the chorus candidate can be intervened accordingly. Specifically, during implementation, the screening can be achieved based on the following methods:
[0103] Step E1, among multiple second paragraphs, filter out the second paragraphs containing the target segmentation point, and obtain the remaining second paragraphs as candidate choruses; the target segmentation point is the starting point or end point of the second paragraph, and the target segmentation point is the segmentation point in the middle of multiple semantically related sentences.
[0104] During implementation, a large language model can be used to analyze the semantic associations between different sentences. For each second segment, the analysis results of the large model can be used to determine whether the sentences before and after the segmentation point of the second segment are semantically related. If so, it means that the segmentation point is unreasonable, and the segmentation of this paragraph is also unreasonable and needs to be filtered out.
[0105] Step E2: if there are multiple candidate choruses, score the multiple candidate choruses.
[0106] Step E3, selecting the candidate chorus with the highest score as the chorus segment.
[0107] During implementation, a machine learning model that can predict the pitch and appeal of a musical segment can be used to score the candidates, and the segment with the highest score can be selected as the final chorus segment.
[0108] In the disclosed embodiment, by filtering out the second paragraph containing the target segmentation point, those paragraphs that may be semantically incoherent due to inappropriate segmentation can be excluded, thereby improving the accuracy of the candidate chorus. Scoring multiple candidate choruses can comprehensively consider the high pitch and pleasantness of the chorus segment to measure its suitability as a chorus, thereby more comprehensively evaluating the chorus segment. In this way, the disclosed embodiment can accurately identify the chorus segment and improve the quality and rationality of the chorus selection.
[0109] In summary, by constructing an audio self-similarity matrix and identifying breakpoints, we can accurately find repeated segments that have different text content and belong to the chorus. Repairing the first paragraph based on the breakpoints ensures the integrity and accuracy of the chorus segment and reduces possible errors and omissions in the chorus recognition process. After accurately identifying and extracting the chorus segment, it is easier to control the changes in the sound effects later.
[0110] In addition to the chorus segment, other factors can also be introduced in the embodiments of the present disclosure to comprehensively generate special effects. For example, the beat, as the length of the notes in each measure of the score, determines the rhythm and rhythm of the music through the specific combination rules of strong beats and weak beats, which can affect the structure and emotional expression of the audio and is one of the important components of music. In order to make the special effects of the audio flash accordingly according to the characteristics of the beat, thereby enhancing the special effects rhythm of the audio, the embodiments of the present disclosure identify the beat of the target audio; based on the beat and chorus segment of the target audio, a corresponding special effects control strategy is generated.
[0111] During implementation, deep learning technology can be used, especially recurrent neural network (RNN) and convolutional neural network (CNN), and their powerful performance in music information retrieval can be used to train a beat recognition model for analyzing the target audio to effectively process complex rhythms and beat patterns, and ultimately generate a beat data sequence described by start time, end time, and strength level.
[0112] Taking the lighting effect as an example, the obtained beat data sequence can be combined with the aforementioned chorus segment to configure the audio's special effect lighting to achieve a synchronized flashing effect.
[0113] In the disclosed embodiment, by identifying the beat of the target audio and generating a specific combination rule of strong beats and weak beats, the special effect changes of the audio can be further enriched.
[0114] In addition, each song has its own unique emotion, which can be perceived by users and cause emotional resonance. For different song emotions, configuring corresponding audio special effects changes can create a matching atmosphere and make users feel like they are in the scene. Therefore, in the disclosed embodiment, a large model can also be used to identify the emotional category of the target audio; based on the emotional category of the target audio, in conjunction with the beat and / or chorus segment, a corresponding special effect control strategy is generated.
[0115] For example, a large model is used to construct the prompt "You are a professional musician. Your task is to analyze the song information provided in the reference materials. Please select the most appropriate emotional label for the song." to identify the emotional category of the target audio, such as positive love, negative love, positive soothing, negative soothing, positive passion, negative passion, positive joy, family and children, red songs, etc.
[0116] The obtained emotional category of the target audio is combined with the aforementioned beat data sequence and the chorus segment to configure the audio's special effects lights to achieve a synchronized flashing effect.
[0117] In the disclosed embodiment, by using a large model to identify audio emotions and generate corresponding special effects control strategies, the richness of audio special effects can be improved, allowing users to experience visual effects that match the music emotions, thereby improving user satisfaction.
[0118] In order to improve the accuracy of emotion recognition, in the embodiment of the present disclosure, the prompt information of the large model includes at least one of the following information:
[0119] 1) Role positioning information of the large model:
[0120] For example, "You are a professional musician and your task is to analyze the song information provided in the reference materials. Please choose the most appropriate emotional label for the song."
[0121] 2) Notes:
[0122] For example, based on the different BPM (Beats Per Minute) value ranges, the range of unselectable emotion tags is limited. For example, if BPM> the preset threshold, the "soothing" tag is strictly avoided. BPM is calculated by the formula: 60 / beat duration, and the beat duration is calculated based on the identified beat sequence described above, and the calculation formula is: the sum of all beat durations / total number of beats.
[0123] The specific value range can be set according to experience or actual needs, and the embodiments of the present disclosure are not limited to this.
[0124] 3) Pay attention to the content of the song:
[0125] For example, if the song information of the target audio mentions a strong sense of rhythm, cheerful melody, and lightness, strictly avoid selecting the "soothing" label.
[0126] That is, a label that is inconsistent with the prompt of the target audio cannot be selected.
[0127] 4) Tag collection:
[0128] In order to improve the accuracy of emotion label prediction, different types of label candidate sets and decision criteria can be provided.
[0129] For example, the criteria for judging family children include: the song has obvious parent-child or children's themes, including children, children's voices, mom and dad, childlike fun and innocence, enlightenment education, family fun, parent-child, cartoon and animation characters, children's lullaby, funny and cute actions, etc.
[0130] 5) Output format requirements:
[0131] In the output format, the following contents can be provided, such as:
[0132] Melody analysis: Comprehensive BPM and song information related to the melody characteristics.
[0133] Comprehensive analysis: Analyze the theme and emotion of the target audio by combining melody analysis with lyrics, comments, and connotations in the material.
[0134] Tag name: Output the most appropriate tag name, Family and Children, Love Positive, Love Negative, Passion Positive, Passion Negative, Joy Positive, Red Songs, Grassland Songs, Religious Songs, choose one of the 9. Do not include any information other than the tag.
[0135] 6) Output examples can be provided.
[0136] In summary, in the disclosed embodiments, the target audio can be described from multiple dimensions through beat recognition, audio structure recognition (including verse, chorus, blank part, a cappella part), and emotion recognition. Thus, a special effect control strategy is generated. This can improve the user's listening experience and user experience in many aspects.
[0137] Based on the same technical concept, the embodiment of the present disclosure also provides an audio-based special effect generation device 500, such as Figure 5 As shown, including:
[0138] An identification module 501 is used to identify a chorus segment in a target audio;
[0139] The generation module 502 is used to generate a corresponding special effect control strategy based on the chorus segment.
[0140] In some embodiments, the identification module includes:
[0141] A construction unit, configured to construct a first self-similarity matrix based on similarities between a plurality of sentences in the lyrics text of the target audio;
[0142] A query unit, used for querying a plurality of first paragraphs that appear repeatedly in the lyrics text based on the first self-similarity matrix;
[0143] The determining unit is used to determine the chorus segment based on the multiple first paragraphs.
[0144] In some embodiments, the query unit is specifically configured to:
[0145] Based on the length of the lyrics text, determine the minimum and maximum lengths of the repeated paragraphs;
[0146] According to the principle that the length of the segment is within the range of the shortest length to the maximum length, the repeated segments under different segment lengths are searched from the first self-similarity matrix to obtain a plurality of repeated preliminary selected segments;
[0147] A deduplication operation is performed on the multiple preliminary selected paragraphs to obtain multiple first paragraphs.
[0148] In some embodiments, the determining unit comprises:
[0149] The recognition subunit is used to identify the breakpoints in the first self-similarity matrix using the audio similarity between the audio segments corresponding to the plurality of sentences; the breakpoints indicate that the text content is different and is a repeated segment that is suspected to be a chorus part;
[0150] A repair subunit, used for repairing the plurality of first paragraphs based on the breakpoint, so that the first paragraphs before and after the breakpoint are merged into a new first paragraph, so as to obtain a plurality of second paragraphs;
[0151] The screening subunit is used to screen out the chorus segments from the plurality of second paragraphs.
[0152] In some embodiments, the determining unit is further configured to construct an audio self-similarity matrix based on the following method, including:
[0153] Divide the target audio into multiple audio segments according to the segmentation points of multiple sentences;
[0154] extracting audio features of multiple audio clips;
[0155] Determining similarities between audio features of the plurality of audio clips as audio similarity;
[0156] Based on audio similarity, an audio self-similarity matrix is constructed.
[0157] In some embodiments, the identification subunit is specifically used to:
[0158] Based on the plurality of first paragraphs, binarizing the first self-similarity matrix to obtain a second self-similarity matrix, so that the values of two sentences corresponding to the repeated paragraphs in the second self-similarity matrix are the first specified values;
[0159] Using the audio self-similarity matrix, the second self-similarity matrix is modified so that the values corresponding to the two sentences whose audio similarities satisfy the preset conditions in the second self-similarity matrix are candidate values;
[0160] In the diagonal direction of the modified second self-similarity matrix, searching for a target sequence, where the target sequence is constructed by continuous first specified values and / or candidate values;
[0161] Candidate values in the target sequence are determined as breakpoints in the first self-similarity matrix.
[0162] In some embodiments, the repair subunit is specifically used for:
[0163] Resetting the breakpoints in the second self-similarity matrix to the first specified value, and setting the candidate values outside the breakpoints to the second specified value, to obtain a third self-similarity matrix;
[0164] Based on the third self-similarity matrix, multiple paragraphs that appear repeatedly in the lyrics text are found to obtain multiple second paragraphs.
[0165] In some embodiments, the preset condition includes sorting the audio similarity from high to low, and the top N sentences in the audio similarity sorting, where N is a positive integer greater than 1.
[0166] In some embodiments, the screening subunit is specifically used for:
[0167] Among the multiple second paragraphs, the second paragraphs containing the target segmentation point are screened out, and the remaining second paragraphs are used as candidate choruses; the target segmentation point is the starting point or the end point of the second paragraph, and the target segmentation point is the segmentation point in the middle of multiple semantically related sentences;
[0168] In the case of multiple candidate choruses, the multiple candidate choruses are scored;
[0169] The candidate chorus with the highest score is selected as the chorus segment.
[0170] In some embodiments, it also includes:
[0171] A beat acquisition module, used to identify the beat of the target audio;
[0172] The generation module is specifically used to generate corresponding special effect control strategies based on the beat and chorus segments of the target audio.
[0173] In some embodiments, it also includes:
[0174] The sentiment analysis module is used to identify the emotion category of the target audio using a large model;
[0175] The generation module is specifically used to generate corresponding special effect control strategies based on the emotion category, beat and chorus segment of the target audio.
[0176] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.
[0177] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0178] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0179] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.
[0180] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0181] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a special effect generation method for audio. For example, in some embodiments, the special effect generation method for audio may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the special effect generation method for audio described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured as a special effect generation method for audio by any other appropriate means (e.g., by means of firmware).
[0182] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0183] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0184] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0185] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0186] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0187] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0188] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0189] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for generating special effects based on audio, comprising: Identify the chorus segment in the target audio; Based on the chorus segment, a corresponding special effect control strategy is generated.
2. The method according to claim 1, wherein: The step of identifying the chorus segment in the target audio includes: Constructing a first self-similarity matrix based on the similarities between multiple sentences in the lyrics text of the target audio; Based on the first self-similarity matrix, querying a plurality of first paragraphs that appear repeatedly in the lyrics text; The chorus segment is determined based on the plurality of first paragraphs.
3. The method according to claim 2, wherein: The step of searching for a plurality of first paragraphs that appear repeatedly in the lyrics text based on the first self-similarity matrix includes: Based on the length of the lyrics text, determine the minimum length and maximum length of the repeated paragraphs; According to the principle that the segment length is within the range from the shortest length to the maximum length, searching for repeated segments under different segment lengths from the first self-similarity matrix to obtain a plurality of repeated preliminary selected segments; A deduplication operation is performed on the multiple initially selected paragraphs to obtain the multiple first paragraphs.
4. The method according to claim 2, wherein: The step of determining the chorus segment based on the plurality of first paragraphs comprises: An audio self-similarity matrix constructed by using the audio similarity between the audio segments corresponding to the plurality of sentences is used to identify breakpoints in the first self-similarity matrix; the breakpoints indicate that the text content is different and is a repeated segment that is suspected to be a chorus part; Repairing the plurality of first paragraphs based on the breakpoint, so that the first paragraphs before and after the breakpoint are merged into a new first paragraph, so as to obtain a plurality of second paragraphs; The chorus segment is selected from the plurality of second paragraphs.
5. The method according to claim 4, wherein: Constructing the audio self-similarity matrix includes: Dividing the target audio into multiple audio segments according to the segmentation points of the multiple sentences; Extracting audio features of the multiple audio clips; Determining similarities between the audio features of the multiple audio clips as audio similarity; Based on the audio similarity, the audio self-similarity matrix is constructed.
6. The method according to claim 4, wherein: The step of constructing an audio self-similarity matrix using the audio similarities between the audio segments corresponding to the plurality of sentences, and identifying breakpoints in the first self-similarity matrix, comprises: Based on the multiple first paragraphs, binarizing the first self-similarity matrix to obtain a second self-similarity matrix, so that the values of two sentences corresponding to the repeated paragraphs in the second self-similarity matrix are first specified values; Using the audio self-similarity matrix, modifying the second self-similarity matrix so that the values corresponding to the two sentences whose audio similarities satisfy a preset condition in the second self-similarity matrix are candidate values; Searching for a target sequence in the diagonal direction of the modified second self-similarity matrix, where the target sequence is constructed by continuous first designated values and / or candidate values; The candidate values in the target sequence are determined as breakpoints in the first self-similarity matrix.
7. The method according to claim 6, wherein: The repairing of the plurality of first paragraphs based on the breakpoint so that the first paragraphs before and after the breakpoint are merged into a new first paragraph to obtain a plurality of second paragraphs includes: Resetting the breakpoints in the second self-similar matrix to the first specified values, and setting the candidate values other than the breakpoints to the second specified values, so as to obtain a third self-similar matrix; Based on the third self-similarity matrix, multiple paragraphs that appear repeatedly in the lyrics text are searched to obtain the multiple second paragraphs.
8. The method according to claim 6, wherein: The preset conditions include sorting the audio similarity from high to low, and sorting the top N sentences by audio similarity, where N is a positive integer greater than 1.
9. The method according to claim 4, wherein: The step of selecting the chorus segment from the plurality of second paragraphs includes: Among the plurality of second paragraphs, the second paragraphs containing the target segmentation point are screened out, and the remaining second paragraphs are obtained as candidate choruses; the target segmentation point is the starting point or the end point of the second paragraph, and the target segmentation point is the segmentation point in the middle of a plurality of semantically related sentences; In the case where a plurality of candidate choruses are included, scoring the plurality of candidate choruses; The candidate chorus with the highest score is selected as the chorus segment.
10. The method according to any one of claims 1 to 9, further comprising: Identifying the beat of the target audio; The generating a corresponding special effect control strategy based on the chorus segment includes: Based on the beat of the target audio and the chorus segment, a corresponding special effect control strategy is generated.
11. The method according to claim 10, further comprising: Using a large model to identify the emotion category of the target audio; The generating a corresponding special effect control strategy based on the beat of the target audio and the chorus segment includes: Based on the emotion category, beat and chorus segment of the target audio, a corresponding special effect control strategy is generated.
12. An audio-based special effects generation device, comprising: A recognition module, used to identify the chorus segment in the target audio; A generation module is used to generate a corresponding special effect control strategy based on the chorus segment.
13. The device according to claim 12, wherein: The identification module comprises: A construction unit, configured to construct a first self-similarity matrix based on similarities between a plurality of sentences in the lyrics text of the target audio; A query unit, configured to query a plurality of first paragraphs that appear repeatedly in the lyrics text based on the first self-similarity matrix; A determination unit is used to determine the chorus segment based on the multiple first paragraphs.
14. The device according to claim 13, wherein: The query unit is specifically used for: Based on the length of the lyrics text, determine the minimum length and maximum length of the repeated paragraphs; According to the principle that the segment length is within the range from the shortest length to the maximum length, searching for repeated segments under different segment lengths from the first self-similarity matrix to obtain a plurality of repeated preliminary selected segments; A deduplication operation is performed on the multiple initially selected paragraphs to obtain the multiple first paragraphs.
15. The device according to claim 13, wherein: The determining unit comprises: An identification subunit is used to identify a breakpoint in the first self-similarity matrix using an audio self-similarity matrix constructed by using audio similarities between the audio segments corresponding to the plurality of sentences; the breakpoint indicates a repeated segment with a difference in text content and a suspected chorus part; a repair subunit, configured to repair the plurality of first paragraphs based on the breakpoint, so that the first paragraphs before and after the breakpoint are merged into a new first paragraph, so as to obtain a plurality of second paragraphs; The screening subunit is used to screen out the chorus segment from the multiple second paragraphs.
16. The device according to claim 15, wherein: The determining unit is further configured to construct the audio self-similarity matrix based on the following method, including: Dividing the target audio into multiple audio segments according to the segmentation points of the multiple sentences; Extracting audio features of the multiple audio clips; Determining similarities between the audio features of the multiple audio clips as audio similarity; Based on the audio similarity, the audio self-similarity matrix is constructed.
17. The device according to claim 15, wherein: The identification subunit is specifically used for: Based on the multiple first paragraphs, binarizing the first self-similarity matrix to obtain a second self-similarity matrix, so that the values of two sentences corresponding to the repeated paragraphs in the second self-similarity matrix are first specified values; Using the audio self-similarity matrix, modifying the second self-similarity matrix so that the values corresponding to the two sentences whose audio similarities satisfy a preset condition in the second self-similarity matrix are candidate values; Searching for a target sequence in the diagonal direction of the modified second self-similarity matrix, where the target sequence is constructed by continuous first designated values and / or candidate values; The candidate values in the target sequence are determined as breakpoints in the first self-similarity matrix.
18. The device according to claim 17, wherein: The repair subunit is specifically used for: Resetting the breakpoints in the second self-similar matrix to the first specified values, and setting the candidate values other than the breakpoints to the second specified values, so as to obtain a third self-similar matrix; Based on the third self-similarity matrix, multiple paragraphs that appear repeatedly in the lyrics text are searched to obtain the multiple second paragraphs.
19. The device according to claim 17, wherein: The preset conditions include sorting the audio similarity from high to low, and sorting the top N sentences by audio similarity, where N is a positive integer greater than 1.
20. The device according to claim 15, wherein: The screening subunit is specifically used for: Among the plurality of second paragraphs, the second paragraphs containing the target segmentation point are screened out, and the remaining second paragraphs are obtained as candidate choruses; the target segmentation point is the starting point or the end point of the second paragraph, and the target segmentation point is the segmentation point in the middle of a plurality of semantically related sentences; In the case where a plurality of candidate choruses are included, scoring the plurality of candidate choruses; The candidate chorus with the highest score is selected as the chorus segment.
21. The apparatus according to any one of claims 12 to 20, further comprising: A beat acquisition module, used to identify the beat of the target audio; The generation module is specifically used to generate a corresponding special effect control strategy based on the beat of the target audio and the chorus segment.
22. The apparatus according to claim 21, further comprising: An emotion analysis module, used to identify the emotion category of the target audio using a large model; The generation module is specifically used to generate a corresponding special effect control strategy based on the emotion category, beat and chorus segment of the target audio.
23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 11.
24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.
25. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 11.
