Audio-based interaction method and device, equipment and storage medium

Abstract

The invention relates to an audio-based interaction method and device, equipment and a storage medium, and relates to the technical field of multimedia. According to the method, the recognition result is upgraded from pure tone source judgment to deep understanding of complete information and intention of the audio, and the degree parameter is output to indicate the expression degree of each audio feature in the audio, so that the fine degree of the recognition result is improved; moreover, multiple audio feature templates corresponding to the multiple audio features and the degree parameter of each audio feature template are provided, and on the basis, a user can adjust the degree parameter of any audio feature of the audio and can select any audio feature template to be mixed with the multiple audio features of the audio to generate a new audio. Therefore, the user can finely adjust the audio features of the audio and can integrate other audio features, so that the audio meeting the personalized requirements of the user is generated, the user does not need to perform excessive operations, and the quality of the generated audio is improved on the basis of ensuring the generation efficiency.

Description

Technical Field

[0001] This disclosure relates to the field of multimedia technology, and in particular to an audio-based interactive method, apparatus, device, and storage medium. Background Technology

[0002] When a user hears an audio clip, they might be interested in its audio features and can search for similar audio in an audio library. Audio libraries typically store timbre templates. When searching in an audio library, the timbre of the audio clip is matched against these timbre templates to output the templates with high timbre similarity. This method often results in the retrieved audio differing significantly from the user's desired audio, thus reducing audio quality. Summary of the Invention

[0003] This disclosure provides an audio-based interaction method, apparatus, device, and storage medium. The technical solution of this disclosure is as follows.

[0004] According to one aspect of the embodiments of this disclosure, an audio-based interaction method is provided, the method comprising: In response to the recognition operation of the input audio, the recognition result of the audio is displayed. The recognition result includes multiple audio features of the audio in multiple dimensions and a degree parameter for each audio feature. The degree parameter is used to indicate the degree of performance of the audio feature in the audio. The multiple dimensions include a timbre dimension and a target dimension. The target dimension is a dimension other than the timbre dimension. Displays the multiple audio feature templates corresponding to the multiple audio features and the degree parameters of each audio feature template; In response to the adjustment operation on the audio, the adjusted target audio is displayed. The adjustment operation includes at least one of adjusting the degree parameter of at least one audio feature and selecting at least one audio feature template, the selected audio feature template being used to blend with the multiple audio features to generate the target audio.

[0005] In some embodiments, the plurality of audio feature templates includes at least one audio feature template from a first audio template, wherein the first audio template is an audio template in an audio library whose similarity to the audio satisfies a similarity condition, and the similarity is the similarity between the plurality of audio feature templates of the first audio template and the plurality of audio features of the audio.

[0006] In some embodiments, the recognition result further includes the at least one first audio template and the similarity corresponding to each first audio template; the method further includes: In response to a playback operation on any of the first audio templates, the first audio template is played.

[0007] In some embodiments, there are multiple first audio templates, and the method further includes at least one of the following: In response to a blending operation on at least a portion of the first audio templates, the blended audio is displayed; In response to the mixing operation of the target audio and at least a portion of the first audio template, the mixed audio is displayed.

[0008] In some embodiments, the identification result further includes a reference audio, wherein multiple audio features of the reference audio are the same as multiple audio features of the audio, and the target audio is the reference audio after adjusting the multiple audio features based on the adjustment operation.

[0009] In some embodiments, the method further includes at least one of the following: In response to a playback operation on the reference audio, the reference audio is played; In response to a playback operation on the target audio, the target audio is played.

[0010] In some embodiments, the target dimension includes at least one of pitch range dimension, emotion dimension, speech rate dimension, rhythm dimension, and accent dimension.

[0011] In some embodiments, the method further includes: Display a first audio interface, which includes multiple audio templates; In response to the selection operation of the second audio template among the plurality of audio templates, the degree parameters of each of the multiple audio feature templates of the second audio template are displayed; In response to an adjustment operation on the degree parameter of at least one audio feature template of the second audio template, the audio obtained by adjusting the audio feature template based on the adjusted degree parameter is displayed.

[0012] In some embodiments, the method further includes: The second audio interface is displayed, which includes multiple audio templates, each with multiple audio feature templates and their respective degree parameters. In response to a blending operation on at least two audio feature templates of at least two audio templates on the second audio interface, the blended audio is displayed.

[0013] In some embodiments, the method further includes: In response to the scaling operation of each of the at least two audio feature templates, the scaling of each of the at least two audio feature templates is displayed; The response to a blending operation on at least two audio feature templates from at least two audio templates on the second audio interface, displaying the blended audio, includes: In response to the mixing operation of the at least two audio feature templates, the audio obtained by mixing the at least two audio feature templates according to their respective proportions is displayed.

[0014] In some embodiments, the method further includes: The third audio interface is displayed, which includes multiple candidate audio feature templates and degree parameters of each of the multiple candidate audio feature templates; In response to a blending operation on at least two candidate audio feature templates on the third audio interface, the blended audio is displayed.

[0015] In some embodiments, the method further includes at least one of the following: In response to the publishing operation of the target audio, an audio template corresponding to the target audio is published in a virtual space, and the audio template in the virtual space is available for use by the target user; In response to a publishing operation for any audio feature in the target audio, an audio feature template is published in a virtual space. The audio feature template includes the audio feature and a degree parameter of the audio feature. The audio feature template in the virtual space is available for use by the target user.

[0016] In some embodiments, the method further includes: In response to a request to publish in a virtual space, request to publish information is displayed in the virtual space. The request to publish information includes an audio sample and adjustment information for the audio sample. The request to publish information is used to request an audio template obtained based on the audio sample and the adjustment information.

[0017] According to another aspect of the embodiments of this disclosure, an audio-based interactive device is provided, the device comprising: The recognition unit is configured to perform a recognition operation in response to input audio and display the recognition result of the audio. The recognition result includes multiple audio features of the audio in multiple dimensions and a degree parameter for each audio feature. The degree parameter is used to indicate the degree of performance of the audio feature in the audio. The multiple dimensions include a timbre dimension and a target dimension, and the target dimension is a dimension other than the timbre dimension. The display unit is configured to display multiple audio feature templates corresponding to the multiple audio features and the degree parameters of each audio feature template; An adjustment unit is configured to perform an adjustment operation in response to the audio, displaying the adjusted target audio, wherein the adjustment operation includes at least one of an adjustment operation of a degree parameter of at least one audio feature and a selection operation of at least one audio feature template, the selected audio feature template being used to blend with the multiple audio features to generate the target audio.

[0018] In some embodiments, the plurality of audio feature templates includes at least one audio feature template from a first audio template, wherein the first audio template is an audio template in an audio library whose similarity to the audio satisfies a similarity condition, and the similarity is the similarity between the plurality of audio feature templates of the first audio template and the plurality of audio features of the audio.

[0019] In some embodiments, the recognition result further includes the at least one first audio template and a similarity score corresponding to each first audio template; the apparatus further includes a playback unit configured to perform: In response to a playback operation on any of the first audio templates, the first audio template is played.

[0020] In some embodiments, there are multiple first audio templates, and the display unit is further configured to perform at least one of the following: In response to a blending operation on at least a portion of the first audio templates, the blended audio is displayed; In response to the mixing operation of the target audio and at least a portion of the first audio template, the mixed audio is displayed.

[0021] In some embodiments, the identification result further includes a reference audio, wherein multiple audio features of the reference audio are the same as multiple audio features of the audio, and the target audio is the reference audio after adjusting the multiple audio features based on the adjustment operation.

[0022] In some embodiments, the device further includes a playback module configured to perform at least one of the following: In response to a playback operation on the reference audio, the reference audio is played; In response to a playback operation on the target audio, the target audio is played.

[0023] In some embodiments, the target dimension includes at least one of pitch range dimension, emotion dimension, speech rate dimension, rhythm dimension, and accent dimension.

[0024] In some embodiments, the display unit is further configured to perform: Display a first audio interface, which includes multiple audio templates; In response to the selection operation of the second audio template among the plurality of audio templates, the degree parameters of each of the multiple audio feature templates of the second audio template are displayed; In response to an adjustment operation on the degree parameter of at least one audio feature template of the second audio template, the audio obtained by adjusting the audio feature template based on the adjusted degree parameter is displayed.

[0025] In some embodiments, the display unit is further configured to perform: The second audio interface is displayed, which includes multiple audio templates, each with multiple audio feature templates and their respective degree parameters. In response to a blending operation on at least two audio feature templates of at least two audio templates on the second audio interface, the blended audio is displayed.

[0026] In some embodiments, the display unit is further configured to perform: In response to the scaling operation of each of the at least two audio feature templates, the scaling of each of the at least two audio feature templates is displayed; In response to the mixing operation of the at least two audio feature templates, the audio obtained by mixing the at least two audio feature templates according to their respective proportions is displayed.

[0027] In some embodiments, the display unit is further configured to perform: The third audio interface is displayed, which includes multiple candidate audio feature templates and degree parameters of each of the multiple candidate audio feature templates; In response to a blending operation on at least two candidate audio feature templates on the third audio interface, the blended audio is displayed.

[0028] In some embodiments, the apparatus further includes a publishing unit configured to perform at least one of the following: In response to the publishing operation of the target audio, an audio template corresponding to the target audio is published in a virtual space, and the audio template in the virtual space is available for use by the target user; In response to a publishing operation for any audio feature in the target audio, an audio feature template is published in a virtual space. The audio feature template includes the audio feature and a degree parameter of the audio feature. The audio feature template in the virtual space is available for use by the target user.

[0029] In some embodiments, the display unit is further configured to perform: In response to a request to publish in a virtual space, request to publish information is displayed in the virtual space. The request to publish information includes an audio sample and adjustment information for the audio sample. The request to publish information is used to request an audio template obtained based on the audio sample and the adjustment information.

[0030] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the aforementioned audio-based interaction method.

[0031] According to another aspect of the present disclosure, a computer-readable storage medium is provided that, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the above-described audio-based interaction method.

[0032] According to another aspect of the present disclosure, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements the above-described audio-based interaction method.

[0033] This disclosure provides an audio-based interaction method. This method, through audio recognition operations, can identify not only audio features in the timbre dimension but also audio features in other dimensions. This upgrades the recognition result from simply judging the timbre source to a deeper understanding of the complete information and intent of the audio, improving recognition accuracy. Furthermore, it outputs degree parameters to indicate the degree of each audio feature's representation in the audio, further enhancing the precision of the recognition results. It also provides multiple audio feature templates corresponding to these multiple audio features and degree parameters for each audio feature template. Based on this, users can adjust the degree parameter of any audio feature and select any audio feature template to mix with multiple audio features to generate new audio. This allows users to fine-tune the audio features and incorporate other audio features, thereby generating audio that meets the user's personalized needs without requiring excessive user intervention, improving the quality of the generated audio while ensuring generation efficiency.

[0034] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0036] Figure 1 This is a schematic diagram illustrating an implementation environment according to an exemplary embodiment.

[0037] Figure 2 This is a flowchart illustrating an audio-based interaction method according to an exemplary embodiment.

[0038] Figure 3 This is a flowchart illustrating another audio-based interaction method according to an exemplary embodiment.

[0039] Figure 4 This is a schematic diagram illustrating an audio input interface according to an exemplary embodiment.

[0040] Figure 5 This is a schematic diagram illustrating an audio recognition process according to an exemplary embodiment.

[0041] Figure 6 This is a schematic diagram illustrating another audio recognition interface according to an exemplary embodiment.

[0042] Figure 7 This is a schematic diagram illustrating a first audio interface according to an exemplary embodiment.

[0043] Figure 8 This is a schematic diagram illustrating another first audio interface according to an exemplary embodiment.

[0044] Figure 9 This is a schematic diagram illustrating a virtual space according to an exemplary embodiment.

[0045] Figure 10 This is a block diagram illustrating an audio-based interactive device according to an exemplary embodiment.

[0046] Figure 11 This is a block diagram illustrating a terminal according to an exemplary embodiment. Detailed Implementation

[0047] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0048] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0049] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the audio data involved in this disclosure was obtained with full authorization.

[0050] The audio-based interaction method provided in this disclosure can be executed by an electronic device, which can be provided as at least one of a terminal and a server. Figure 1 This is a schematic diagram of an implementation environment provided in this embodiment of the disclosure. See also: Figure 1 The implementation environment includes: terminal 101 and server 102.

[0051] In this embodiment, a target application is installed on terminal 101, providing audio recognition and audio generation functions. After audio is input into the target application, it can identify multiple audio features across multiple dimensions, as well as degree parameters indicating the prevalence of each audio feature. It also provides multiple audio feature templates corresponding to these features and degree parameters for each template. After a user adjusts the degree parameters of any audio feature and / or selects any audio feature template, the target audio generated based on that operation can be displayed. The selected audio feature template can be mixed with the multiple audio features of the audio to generate the target audio. Server 102 is the backend server for the target application, providing backend services. Optionally, terminal 101 performs audio recognition and audio generation through server 102, or terminal 101 itself performs audio recognition and audio generation.

[0052] Terminal 101 can be at least one of the following devices: smartphone, smartwatch, desktop computer, laptop, virtual reality terminal, augmented reality terminal, wireless terminal, and laptop computer. Terminal 101 has communication capabilities and can access wired or wireless networks. Terminal 101 can refer to one of multiple terminals; those skilled in the art will understand that the number of terminals can be more or less. Server 102 can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed file system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some embodiments, server 102 and terminal 101 are directly or indirectly connected via wired or wireless communication; this disclosure does not limit this. Optionally, the number of servers 102 can be more or less; this disclosure does not limit this. Of course, server 102 can also include other functional servers to provide more comprehensive and diversified services. In this embodiment, server 102 undertakes the main computing work and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work and terminal 101 undertakes the main computing work; or, server 102 or terminal 101 can each undertake computing work independently, and this embodiment does not limit this.

[0053] Figure 2 This is a flowchart illustrating an audio-based interaction method according to an exemplary embodiment, such as... Figure 2 As shown, the method is executed by a terminal and includes at least one of the following steps.

[0054] In step S201, in response to the recognition operation of the input audio, the recognition result of the audio is displayed. The recognition result includes multiple audio features of the audio in multiple dimensions and the degree parameter of each audio feature. The degree parameter is used to indicate the degree of performance of the audio feature in the audio. The multiple dimensions include the timbre dimension and the target dimension. The target dimension is the dimension other than the timbre dimension.

[0055] The audio can be either audio containing speech or audio excluding speech; no specific limitation is made here.

[0056] The timbre dimension includes, but is not limited to, fundamental frequency stability characteristics, formant characteristics, and timbre texture characteristics, which are used to characterize one or more of the timbre features. Fundamental frequency stability (FOStability) affects the smoothness or vibrato of the timbre. Formant characteristics determine the timbre's texture and fullness. Timbre texture includes aspects such as the "magnetism," "granularity," and "clarity" of the sound.

[0057] The target dimension's audio features include, but are not limited to, at least one of the following: pitch range, emotional prosody, speech rate, rhythm, and accent. Pitch range controls the frequency span from the lowest to the highest note in the audio. Speech rate pattern controls the speed of speech. Emotional prosody adjusts the emotional expression in the audio. Accent / Dialect adds or subtracts accent features from a specific region or culture. Stress pattern, also known as prosody, adjusts the emphasis and pace of speech.

[0058] It should be noted that for each dimension of audio features, the audio may contain one or more of them. Taking the timbre dimension as an example, it may include one or more of the following: fundamental frequency stability features, formant features, and timbre texture features. Taking the emotion dimension as an example, it may include one or more of the following: pleasant features, gentle features, relaxed features, sad features, and angry features.

[0059] Optionally, the target dimension audio features may also include features such as gender and age, which are also features of the speech object in the audio.

[0060] In some embodiments, for at least some of the multiple audio features, feature explanation information or example audio may also be provided to help users understand the role of each audio feature.

[0061] The degree parameter can be a percentage. For example, if the degree parameter of an XX accent is 80%, it means that the accent exists in the audio and that the accent is prominent in the audio.

[0062] In step S202, the multiple audio feature templates corresponding to the multiple audio features and the degree parameters of each audio feature template are displayed.

[0063] For each audio feature, one or more corresponding audio feature templates can be displayed, without specific limitations. Each audio feature template corresponds to the same dimension as that audio feature. The audio feature template can be any audio feature template for that dimension, or an audio feature template whose similarity to the audio feature meets preset conditions, or a frequently used audio feature template for that dimension, or the most recently released audio feature template for that dimension, without specific limitations.

[0064] In other embodiments, these multiple audio feature templates may also be audio feature templates among audio templates whose similarity to the original audio meets a similarity condition. If each audio feature corresponds to one audio feature template, then these multiple audio feature templates belong to the same audio template. If each audio feature corresponds to multiple audio feature templates, then there are multiple audio templates whose similarity to the original audio meets a similarity condition, and these multiple audio feature templates belong to different audio templates.

[0065] In step S203, in response to the audio adjustment operation, the adjusted target audio is displayed. The adjustment operation includes at least one of the following: adjusting the degree parameter of at least one audio feature and selecting at least one audio feature template. The selected audio feature template is used to mix with multiple audio features to generate the target audio.

[0066] Optionally, if the level parameter is presented as a progress bar, the level parameter can be adjusted by adjusting the progress components on the progress bar, such as a slider. Alternatively, if the level parameter is presented as a numerical value, the level parameter can be adjusted by modifying that value; no specific limitation is made here.

[0067] If the adjustment operation only includes adjusting the degree parameter of at least one audio feature, then the target audio is generated based on the adjusted degree parameter of the at least one audio feature and the unadjusted degree parameter of the audio feature. If the adjustment operation only includes selecting an audio feature template, then the selected audio feature template is mixed with multiple audio features of the original audio to obtain the target audio. If the adjustment operation includes both adjusting the degree parameter of at least one audio feature and selecting an audio feature template, then the selected audio feature template, the adjusted audio feature, and the unadjusted audio feature are mixed to obtain the target audio.

[0068] In some embodiments, the degree parameter of the audio feature template can also be adjusted. The adjustment operation may further include adjusting the degree parameter of the selected audio feature template. The audio feature template with the adjusted degree parameter is then mixed with the audio features of the original audio to obtain the target audio.

[0069] In some embodiments, the generated target audio can be played by performing a playback operation, allowing for a preview of the target audio. Similarly, each audio feature template can also be played using a playback operation.

[0070] In this embodiment, the input audio may or may not include speech content. Taking the original audio as the input as an example, if the original audio includes speech content, the generated target audio also includes speech content. The speech content included in the target audio may fall into at least one of the following categories: (1) The speech content included in the target audio is the same as that included in the original audio. In this way, by adjusting the audio features in the original audio, the audio features in the original audio can be conveniently adjusted without changing its original speech content; (2) The speech content included in the target audio is different from that included in the original audio. The target audio is the audio template to be obtained, and the speech content in the target audio is a fixed speech template; (3) The target audio includes part of the speech content in the original audio, and the duration of the target audio is shorter than that of the original audio. The target audio is the audio template to be obtained.

[0071] If the original audio does not include voice content, the target audio may or may not include voice content, or it may include voice content from the voice template.

[0072] Optionally, when the generated target audio is used as an audio template, the duration of the generated audio in this embodiment of the disclosure is fixed, thereby ensuring the uniformity of the audio templates generated in the target application.

[0073] This disclosure provides an audio-based interaction method. This method, through audio recognition operations, can identify not only audio features in the timbre dimension but also audio features in other dimensions. This upgrades the recognition result from simply judging the timbre source to a deeper understanding of the complete information and intent of the audio, improving recognition accuracy. Furthermore, it outputs degree parameters to indicate the degree of each audio feature's representation in the audio, further enhancing the precision of the recognition results. It also provides multiple audio feature templates corresponding to these multiple audio features and degree parameters for each audio feature template. Based on this, users can adjust the degree parameter of any audio feature and select any audio feature template to mix with multiple audio features to generate new audio. This allows users to fine-tune the audio features and incorporate other audio features, thereby generating audio that meets the user's personalized needs without requiring excessive user intervention, improving the quality of the generated audio while ensuring generation efficiency.

[0074] The above Figure 2The diagram shown is merely the basic process of this disclosure. The following section, based on a specific implementation method, further elaborates on the solution provided in this disclosure. See also... Figure 3 , Figure 3 This is a flowchart illustrating another audio-based interaction method according to an exemplary embodiment, the method being performed by a terminal, the method including at least one of the following steps.

[0075] In step S301, in response to the recognition operation of the input audio, the recognition result of the audio is displayed. The recognition result includes multiple audio features of the audio in multiple dimensions and the degree parameter of each audio feature. The degree parameter is used to indicate the degree of performance of the audio feature in the audio. The multiple dimensions include the timbre dimension and the target dimension. The target dimension is the dimension other than the timbre dimension.

[0076] In this embodiment of the disclosure, audio can be input directly or by inputting video. The terminal then extracts the audio from the input video and performs recognition on the extracted audio.

[0077] In this embodiment, audio can be input in various ways. For example, audio can be input via file upload, link input, or real-time recording. For file upload, local audio or video files can be uploaded directly; audio files can be in MP3 or WAV format, and video files can be in MP4 or AVI format. For link input, audio or video links from other platforms or websites can be pasted. For real-time recording, the target application can provide a recording function.

[0078] Optionally, audio is input on an audio input interface, which includes entry controls for various input methods. For example, see... Figure 4 , Figure 4 This is a schematic diagram illustrating an audio input interface according to an exemplary embodiment. The audio input interface has entry controls 401 for three different input methods. Furthermore, each entry control for each input method displays a prompt indicating the input method.

[0079] In this embodiment, video or audio is input via a link, allowing the terminal to directly access the video or audio indicated by the link without requiring the user to manually download it before inputting it. This reduces the number of steps and improves interaction efficiency.

[0080] In some embodiments, audio recognition is triggered directly after audio is entered on the audio input interface. In other embodiments, audio recognition is triggered only in response to an audio recognition operation after audio is entered on the audio input interface, allowing users to further confirm the entered audio and avoid accidental touches.

[0081] In some embodiments, the terminal identifies the audio through a server to obtain the identification result. Optionally, the server also performs preprocessing such as noise reduction on the input audio to ensure the quality of the input audio.

[0082] Optionally, audio recognition can be achieved using artificial intelligence models to obtain multiple audio features and their intensity parameters. For example, for emotion-related feature recognition, an artificial intelligence model can be used to analyze features such as intonation, loudness, rhythm, and timbre roughness, as well as the speech content, to achieve recognition. For speech rate and rhythm-related feature recognition, an artificial intelligence model can be used to analyze syllable duration, pause intervals, and stress patterns in the audio to achieve recognition.

[0083] One or more artificial intelligence models can be used to identify multiple audio features, such as timbre recognition models, emotion recognition models, and prosody recognition models.

[0084] In some embodiments, in response to the recognition operation, the terminal displays the recognition progress before displaying the recognition result. This progress can be displayed as a progress bar, and may also display the content currently being recognized.

[0085] Optionally, if the output recognition result includes three tasks performed sequentially: audio preprocessing, audio feature recognition, and audio matching, then optionally, different statuses can be displayed for tasks that have been completed, are being processed, and have not been processed. For example, see... Figure 5 , Figure 5 This is a schematic diagram illustrating an audio recognition process according to an exemplary embodiment. The terminal displays the recognition progress 501, the content being recognized 502, and the execution status of multiple tasks 503.

[0086] In step S302, the multiple audio feature templates corresponding to the multiple audio features and the degree parameters of each audio feature template are displayed.

[0087] In some embodiments, the multiple audio feature templates include at least one audio feature template from a first audio template, wherein the first audio template is an audio template in an audio library whose similarity to the audio satisfies a similarity condition, and the similarity is the similarity between the multiple audio feature templates of the first audio template and the multiple audio features of the audio.

[0088] The terminal can display audio feature templates by displaying at least one of the following: the name, icon, or label of the audio feature template.

[0089] The similarity condition can be set as needed, such as a similarity greater than 90%. In this embodiment, the provided audio feature template is an audio template with a high similarity to the original audio. Thus, the similarity between the audio feature template and the original audio features is high, and a new high-quality audio can be obtained by mixing such an audio feature template with the original audio features with one click. This preserves the essential features of the original audio and improves the audio generation efficiency while ensuring the quality of the generated audio.

[0090] In step S303, in response to the audio adjustment operation, the adjusted target audio is displayed. The adjustment operation includes at least one of the following: adjusting the degree parameter of at least one audio feature and selecting at least one audio feature template. The selected audio feature template is used to mix with multiple audio features to generate the target audio.

[0091] In some embodiments, the identification result further includes at least one first audio template and the similarity corresponding to each first audio template, and the first audio template is played in response to a playback operation on any first audio template.

[0092] The terminal displays the first audio template, which can be achieved by displaying at least one of the following: the name, icon, and label of the first audio template. If the original audio includes voice content, the first audio template may also include voice content; if the original audio does not include voice content, the first audio template may not include voice content.

[0093] The similarity can be displayed using at least one of the following methods: a progress bar, a percentage, or a simplified spectrogram. The similarity threshold can be set as needed, such as 95%.

[0094] Specifically, the similarity between the original audio and the first audio template can be determined by identifying the similarity between each audio feature of the same dimension and the audio feature template in both the original audio and the first audio template. Then, the mean of multiple similarities corresponding to multiple audio features is determined, or a weighted sum of multiple similarities is performed to obtain the similarity between the original audio and the first audio template. The weight of the similarity corresponding to each audio feature can be set and changed as needed; for example, the weight of audio features in the timbre dimension can be greater than the weight of audio features in other dimensions, so that the searched audio is as consistent as possible with the original audio in timbre.

[0095] In some embodiments, the identification results may also include the similarity between each audio feature template and its corresponding audio feature, in order to improve the information transmission rate.

[0096] In this implementation, for the identified audio, the recognition results also include audio templates from an audio library with high similarity, increasing the richness of the recognition results. Since the audio templates in the audio library are directly usable audio and provide playback controls to preview them, users can directly use the audio template if they are satisfied with the preview results, without needing to generate other audio, thus improving the convenience and efficiency of audio acquisition.

[0097] In some embodiments, there are multiple first audio templates, that is, multiple audio templates with high similarity to the original audio are found. The implementation may further include: in response to a mixing operation on at least some of the multiple first audio templates, displaying the mixed audio.

[0098] It should be noted that for any audio and audio template in the embodiments of this disclosure, the target application provides a playback function, allowing users to listen to the recommended audio template and the generated audio.

[0099] In some embodiments, a one-click synthesis control is also provided, which allows for one-click synthesis of the plurality of first audio templates. In other embodiments, multiple audio combinations are provided, and triggering any one of the audio combinations mixes the multiple audio templates in that audio combination to obtain a new audio.

[0100] In this implementation, since any first audio template can be listened to, and the audio features of these first audio templates are highly similar to the audio features of the original audio, the audio obtained by mixing these first audio templates is highly similar to the original audio. Thus, one-click mixing can produce new high-quality audio, which improves audio generation efficiency while ensuring the quality of the generated audio.

[0101] In some embodiments, the identification result further includes multiple audio feature templates for each first audio template and the similarity between each audio feature template and the corresponding audio feature in the original audio. Optionally, with the prompting of this information, the following implementation may also be included: in response to a mixing operation of at least two audio feature templates from at least two of the multiple first audio templates, the mixed audio is displayed.

[0102] In some embodiments, a ratio can be set for each audio feature template before mixing at least two audio feature templates, and the mixing can then be performed based on that ratio.

[0103] Optionally, the terminal also displays a scaling control for each audio feature template, allowing users to set the scaling ratio. This scaling control can be displayed as a progress bar or other formats. Taking a progress bar as an example, the progress bar includes a slider, which can be dragged to adjust the scaling ratio.

[0104] Taking the sound quality texture features of mixed audio template A and the accent features of audio template B as examples, the proportions of these two audio features can be set to 70% and 30%, respectively.

[0105] In this embodiment, for each first audio template similar to the original audio, the degree parameters of its multiple audio features or its corresponding similarity are also displayed. With the prompts of degree parameters or similarity, the user can select a suitable audio feature template from different first audio templates to mix and obtain audio that meets the user's needs, thereby improving the personalization of the audio.

[0106] In some embodiments, before performing a mixing operation on at least a portion of the first audio template, it is also possible to obtain the respective proportions of the at least a portion of the first audio template in response to a proportion setting operation on each of the at least a portion of the first audio template; and in response to the mixing operation on the at least a portion of the first audio template, to display the audio obtained by mixing the at least a portion of the first audio template based on the respective proportions of the at least a portion of the first audio template.

[0107] In some embodiments, the generated target audio can also be mixed with a first audio template to obtain new audio. Optionally, in response to the mixing operation of the target audio and at least a portion of the first audio template, the mixed audio is displayed.

[0108] In this embodiment, the generated target audio can be further mixed with audio templates that are highly similar to the original audio to obtain new audio, which further improves the flexibility of audio generation.

[0109] The target audio and the first audio template can be mixed evenly or mixed after setting a ratio; no specific limitation is made here.

[0110] In some embodiments, the identification result further includes multiple audio feature templates of each first audio template and the similarity between each audio feature template and the corresponding audio feature in the original audio. In this case, the following implementation is also included: in response to the mixing operation of the target audio and at least some of the audio feature templates in at least some of the first audio templates, the mixed audio is displayed.

[0111] It should be noted that for the audio feature template used for mixing in the first audio template, you can directly replace the audio feature in the target audio with the audio feature template, or you can mix the audio feature template with the corresponding audio feature in the target audio. No specific limitation is made here.

[0112] In some embodiments, the identification result also includes reference audio, the reference audio having the same multiple audio features as the original audio, and the target audio being the reference audio after adjusting the multiple audio features based on the adjustment operation.

[0113] The reference audio has the same multiple audio features as the original audio; that is, the reference audio is obtained based on the multiple audio features of the original audio. Optionally, the reference audio is obtained by audio cloning the original audio. By processing the original audio, a reference audio with the multiple audio features of the original audio can be obtained.

[0114] Audio cloning can be achieved through end-to-end text-to-speech models such as Tacotron (Text-to-Speech) and VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech), or through techniques such as Zero-shot Zero-Learning or Few-shot Learning. No specific limitations are specified here.

[0115] Among these features, many audio characteristics can be identified not only through artificial intelligence models but also through audio processing algorithms (such as FFT and LPC) combined with preset rules for feature extraction. Cloning is then achieved through splicing, which involves pre-recording a large database of acoustic units (such as phonemes and syllables) and then searching for and splicing the most matching acoustic units from the input audio.

[0116] Optionally, if the original audio includes speech content, the reference audio may also include speech content. This speech content may be the same as or different from the original speech content. If the reference audio is generated as an audio template, the speech content in the reference audio is generally the speech content in the speech template.

[0117] Optionally, in response to a playback operation on the reference audio, the terminal plays the reference audio. In this embodiment, reference audio is generated based on the audio features of the original audio and provided to the user, allowing the user to perceive the audio features in the original audio by listening to the reference audio. Furthermore, by adjusting the degree parameter of any audio feature, the corresponding audio feature of the reference audio can be directly adjusted without having to generate it from scratch, thus improving audio generation efficiency.

[0118] In some embodiments, the identification results include a first audio template and a reference audio. For example, see [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram illustrating an audio recognition interface according to an exemplary embodiment. The interface displays a first audio template 601, a similarity score 602 corresponding to the first audio template, a reference audio (i.e., cloned audio) and its playback control 603, and feature information 604 of the reference audio. The feature information includes multiple audio features of the reference audio and their degree parameters. It should be noted that since multiple audio features of the reference audio are the same as multiple audio features of the original audio, these multiple audio features are also the multiple audio features of the recognized original audio.

[0119] In other embodiments, the identification result includes one of a first audio template and a reference audio. When the identification result includes one of the first audio template and the reference audio, the mechanism for determining which audio to include is as follows: if the first audio template is included in the audio library, the identification result includes the first audio template. If the first audio template is not included in the audio library, the identification result includes the reference audio. This way, when the audio library includes an audio template with high similarity to the original audio, prioritizing the use of audio simulations from the audio library avoids the cloning process, saves resources, and improves efficiency.

[0120] In the above embodiments, the function of providing audio recognition and generating new audio based on the recognized audio in the target application has been described. In other embodiments, the target application may directly provide users with audio templates from an audio library. Users can directly use these audio templates or generate new audio based on these audio templates. The process is described in detail below.

[0121] In step S304, a first audio interface is displayed, which includes multiple audio templates.

[0122] In this embodiment of the disclosure, the example of displaying the recognition result on an audio recognition interface is used for explanation. Optionally, it is possible to directly jump from the audio recognition interface to the first audio interface, such as when the audio recognition interface displays an entry control for the first audio interface, or when jumping to the first audio interface in response to a preset operation on the audio recognition interface.

[0123] In this embodiment, the multiple audio templates can be audio recommended by the first audio interface, or audio templates found in the audio library based on a search operation. If the first audio interface provides a search box, audio searches can be performed by entering keywords or voice in the search box. Furthermore, audio searches can be performed based on information such as tags, popularity, publisher, and audio characteristics.

[0124] The terminal displays each audio template by displaying at least one of the following: the name, icon, and label of the audio template.

[0125] Optionally, the first audio interface also includes playback controls for each of the multiple audio templates. In response to the triggering operation of the playback control of any audio template, the terminal can also play the audio template, allowing the user to preview it.

[0126] For example, see Figure 7 , Figure 7 This is a schematic diagram illustrating a first audio interface according to an exemplary embodiment. The first audio interface displays multiple audio templates 701.

[0127] In step S305, in response to the selection operation of the second audio template among multiple audio templates, the degree parameters of each of the multiple audio feature templates of the second audio template are displayed.

[0128] The selection operation for any second audio template can be a trigger operation for that second audio template, and is not specifically limited here. Multiple audio feature templates are audio feature templates of the second audio template in multiple dimensions.

[0129] In step S306, in response to the adjustment operation of the degree parameter of at least one audio feature template of the second audio template, the audio obtained by adjusting the audio feature template based on the adjusted degree parameter is displayed.

[0130] Specifically, the system can respond to a playback operation on the adjusted second audio template to play the adjusted audio. Optionally, after generating the new audio, a prompt message can be displayed to indicate that the adjustment is complete.

[0131] In other embodiments, the mixed audio can also be displayed in response to a mixing operation of at least two audio templates among a plurality of audio templates. Furthermore, the ratios of the at least two audio templates can be set, and the at least two audio templates can be mixed based on their respective ratios.

[0132] Optionally, for the mixed audio, the audio templates used to obtain the audio and the proportion of each audio template are also displayed.

[0133] Optionally, the mixed audio can also be regenerated and saved, such as by displaying regeneration and save controls.

[0134] Optionally, for the regenerated audio, at least one of its name, label, multiple audio features, and degree parameters of each of the multiple audio features is also displayed.

[0135] For example, see Figure 8 , Figure 8 This is a schematic diagram of a first audio interface according to an exemplary embodiment. After mixing multiple audio templates to obtain a new audio 801, the diagram also displays the multiple audio templates used to obtain the audio 802 and the ratio 803 of each audio template.

[0136] In some embodiments, multiple audio feature templates and level parameters of each of the multiple audio templates can also be viewed. Optionally, the terminal displays a second audio interface, which includes multiple audio feature templates and level parameters of each of the multiple audio feature templates; in response to a mixing operation of at least two audio feature templates from at least two audio templates on the second audio interface, the mixed audio is displayed.

[0137] The user can jump directly from the audio recognition interface to the second audio interface, or from the first audio interface to the second audio interface. The multiple audio templates in the second audio interface can be the same as or different from the multiple audio templates in the first audio interface. Taking a jump from the first audio interface to the second audio interface, where multiple audio templates in the second audio interface are the same as multiple audio templates in the first audio interface, as an example, the user can respond to an expansion operation on the first audio interface to jump to the second audio interface. The second audio interface displays the multiple audio feature templates and their respective degree parameters for each audio feature template.

[0138] In this embodiment, for each of the multiple audio feature templates, the audio feature templates of different audio templates can be freely mixed, further ensuring the diversity of audio generation and meeting the personalized needs of users.

[0139] Among them, for the mixed multiple audio feature templates, they can be mixed uniformly or mixed according to a certain ratio. Taking the mixing according to a ratio as an example, the ratio can be set in the following ways: in response to the ratio setting operation of at least two audio feature templates, the ratio of each of the at least two audio feature templates is displayed; correspondingly, the process of displaying the mixed audio in response to the mixing operation of at least two audio feature templates in at least two audio templates on the second audio interface includes the following implementation: in response to the mixing operation of at least two audio feature templates, the audio obtained by mixing at least two audio feature templates according to the ratio of each of the at least two audio feature templates is displayed.

[0140] In this embodiment, when mixing audio feature templates, the mixing ratio of each audio feature template can be controlled, further improving the flexibility of audio generation and thus obtaining more personalized audio to meet user needs.

[0141] In this embodiment, not only can audio be obtained through audio recognition, but also any audio template can be freely selected from multiple audio templates provided by the target application as the base audio. By adjusting its audio feature template, new audio can be obtained, thereby obtaining new audio that meets the user's personalized needs. This improves the generation efficiency and further enhances the flexibility and convenience of audio generation.

[0142] In the above embodiments, the example of multiple audio feature templates depending on the existence of an audio template is used for illustration. In other embodiments, multiple audio feature templates exist independently without depending on the audio template. This embodiment includes the following steps 307-308.

[0143] In step S307, a third audio interface is displayed, which includes multiple candidate audio feature templates and degree parameters for each candidate audio feature template.

[0144] Among these, multiple candidate audio feature templates can exist independently of audio templates, meaning they are independent candidate audio feature templates. These multiple candidate audio feature templates can be audio feature templates recommended to users, or they can be audio feature templates displayed to users through search.

[0145] In step S308, in response to the mixing operation of at least two candidate audio feature templates on the third audio interface, the mixed audio is displayed.

[0146] In some embodiments, the first audio interface, the second audio interface, the third audio interface, and the audio recognition interface can all jump between each other. Since the first audio interface, the second audio interface, and the third audio interface are all interfaces used to provide audio templates in the target application, optionally, the first audio interface, the second audio interface, and the third audio interface can be sub-interfaces under the same interface label, such as "Audio Community".

[0147] It should be noted that any audio obtained in the embodiments of this disclosure can be listened to, used, or adjusted, such as by mixing with other audio, mixing with other audio feature templates, or adjusting the degree parameters of the audio feature template; no specific limitations are made here. Furthermore, any audio obtained in the embodiments of this disclosure can display its name, icon, tag, and other information. For mixed audio, the audio template used can also be displayed, and further, the mixing ratio can also be displayed.

[0148] In this embodiment, a third audio interface also provides multiple audio feature templates that do not depend on audio templates. This allows users to freely select audio feature templates to mix and create new audio, just like building blocks. This further improves the flexibility of audio generation and better meets users' personalized audio needs.

[0149] Optionally, since the generated audio can be adjusted, the terminal can also save each generated audio, that is, save multiple versions of audio, so that users can review and compare the effects of different versions of audio during the audio generation process.

[0150] In this embodiment of the disclosure, when generating new audio through a mixing method, the mixing can be achieved through a mixing algorithm. The mixing algorithm includes, but is not limited to, interpolation of the audio feature space and fusion of audio features. For example, audio features can be mapped and reconstructed using artificial intelligence models such as GAN (Generative Adversarial Network) and VAE (Variational Auto Encoder), and the large artificial intelligence model generates audio based on these audio feature conditions.

[0151] In the embodiments of this disclosure, the audio generated by any of the above implementation methods can be saved, and users can use this audio, such as for dubbing. In other embodiments, the audio generated by any of the above implementation methods and any audio feature in the audio can be published as a template within the target application, and the published audio template and audio feature template can be used by other users. This process involves at least one of the following steps 308-309.

[0152] In step S309, in response to the publishing operation of the target audio, the audio template corresponding to the target audio is published in the virtual space, and the audio template in the virtual space is allowed to be used by the target user.

[0153] The target user can be any user, or a user with access to audio templates in the virtual space. The method of obtaining access can be set and changed as needed, such as by paying virtual resources to obtain access, or by publishing audio templates or audio feature templates. No specific restrictions are imposed here.

[0154] Specifically, when the target audio does not include speech content, its corresponding audio template can be the audio itself or an audio segment extracted from the target audio. When the target audio includes speech content, the corresponding audio template can be an audio template in which the speech content in the target audio is replaced with the speech content in the speech template. Furthermore, the audio template corresponding to the target audio includes multiple audio features in the target audio and their respective degree parameters.

[0155] In this embodiment, the example of publishing the audio template corresponding to the target audio to a virtual space is used for illustration. However, any audio generated in this disclosure can have its corresponding audio template published to a virtual space.

[0156] The audio templates published in the virtual space can be stored in an audio library for later use. For example, users can find the audio template from the audio library through audio search or audio recognition. The target application can also proactively recommend the audio template to the user, without specific limitations. Accordingly, the audio template can be displayed on the aforementioned first audio interface, second audio interface, or audio recognition interface.

[0157] The virtual space can be a specific interface within the target application, where users can publish audio templates corresponding to the target audio to that interface, such as the audio community interface within the target application.

[0158] In step S310, in response to the publishing operation of any audio feature in the target audio, an audio feature template is published in the virtual space. The audio feature template includes the audio feature and the degree parameter of the audio feature. The audio feature template in the virtual space is available for use by the target user.

[0159] The audio feature templates published in the virtual space can be stored in an audio library for later use. For example, users can retrieve the audio feature template from the audio library through search or audio recognition. Target applications can also proactively recommend the audio feature template to users; no specific limitations are made here. Accordingly, the audio feature template can be displayed on the aforementioned first audio interface, second audio interface, third audio interface, or audio recognition interface.

[0160] In this embodiment, not only can audio tailored to individual user needs be generated within the target application, but the corresponding audio template can also be shared in a virtual space for other users. This allows for the continuous enrichment and generation of diverse audio templates within the target application, promoting communication and audio generation among users and collectively enriching the audio library. Furthermore, sharing can be based on individual audio features, enabling other users to select and combine different audio feature templates from the audio library when generating new audio, further enhancing the flexibility and convenience of audio generation.

[0161] In this embodiment of the disclosure, users are allowed not only to publish complete audio files, but also to independently extract and publish audio features from the audio. For example, a user can save accent features or emotional features from an audio file as an independent template and then publish it. In this way, when other users generate new audio files, they can select different templates from multiple audio feature templates and combine them like building blocks, such as an XX accent template + a gentle female base template + a broadcast rhythm template.

[0162] When users publish audio templates or audio feature templates, they can name the audio templates and audio feature templates, or the target application can actively name them. They can also add tags to the audio templates and audio feature templates, or the target application can actively add tags. There are no specific restrictions here.

[0163] In some embodiments, the community management module stores and manages user-uploaded audio templates and audio feature templates. This data can be in JSON (a data format) or other structured data formats. The community management module provides front-end calls and user interaction functions. Furthermore, it ensures the quality and compliance of audio content through permission management and review mechanisms.

[0164] In some embodiments, the target application may also request an audio template, a process that includes the following steps.

[0165] In step S311, in response to the request publishing operation in the virtual space, request publishing information is displayed in the virtual space. The request publishing information includes an audio sample and adjustment information for the audio sample. The audio publishing information is used to request the acquisition of an audio template based on the audio sample and the adjustment information.

[0166] The adjustment information is used to indicate how to adjust the audio samples to obtain a new audio template.

[0167] In some embodiments, a request posting control is displayed in the virtual space, through which the request posting operation is implemented.

[0168] For audio samples in the request posting information, a playback control is also displayed. In response to triggering the playback control, the audio sample can be played.

[0169] The request to publish information may also include comment controls, reply controls, and audio generation controls. The audio generation control is used to generate new audio based on the audio sample. The process of generating audio based on the audio sample is the same as any implementation in steps 301-303, and will not be repeated here. Optionally, in response to a trigger operation on the audio generation control, the terminal displays the recognition result of the audio sample, and then generates new audio based on the recognition result.

[0170] Optionally, in response to a reply operation based on the generated audio template, the generated audio template is displayed in the associated area of ​​the request posting information in the virtual space, and the audio template can be listened to and obtained by the user who posted the request and other users.

[0171] In this embodiment, the example of requesting the release information to include an audio sample is used for illustration. In other embodiments, the request release information does not include an audio sample, but only includes description information of the audio features in the audio template to be generated. That is, the request release information is used to request the acquisition of an audio template with the audio features corresponding to the description information.

[0172] For example, see Figure 9 , Figure 9 This is a schematic diagram illustrating a virtual space according to an exemplary embodiment. The virtual space displays user-posted request information 901.

[0173] In this embodiment of the disclosure, it is also possible to publish a request in the virtual space to obtain an audio template. This not only increases the probability that users can obtain an audio template that meets their needs, but also facilitates interaction and cooperation between users within the target application, further improving the flexibility of audio acquisition. This is conducive to promoting the generation of audio templates within the target application and enriching the audio library.

[0174] It should be noted that the sequence numbers of steps 301-311 above are merely for ease of execution and are not intended to restrict the execution order among them. For example, steps 301-303, 304-306, and 307-308 are execution steps corresponding to one embodiment, and there is no strict execution order among them. After any embodiment is completed, step 309 or 310 can be executed. Step 311 can be executed independently of the above embodiments, or it can be executed after any embodiment is executed.

[0175] This disclosure provides AI-based audio recognition, hybrid generation, and audio sharing functions. The method intelligently identifies the multi-dimensional audio features of input audio and, based on the recognition results, recommends similar audio templates from an audio library or generates matching audio using AI cloning technology. Furthermore, a visual interactive interface is provided, allowing users to finely and proportionally mix various audio features to generate highly customized composite audio. Moreover, the method also establishes an audio sharing mechanism, supporting users to share composable audio feature templates and complete audio templates, forming a user-co-created and shared audio generation system. This greatly enhances users' ability to acquire, create, and share unique audio, enabling personalized content creation.

[0176] The method provided in this disclosure eliminates the need for users to know the name of the audio file; they only need to upload or paste the audio. The system can then intelligently identify audio features in terms of timbre and other multi-dimensional audio features, and recommend or generate highly matching audio templates. This solves the difficulties of traditional audio search, reducing the time users spend searching for audio templates with specific audio features by an average of more than 50%, and greatly improving the efficiency and accuracy of audio search.

[0177] Furthermore, this disclosure provides a visual mixing interface for multi-dimensional audio features, allowing users to combine various audio feature templates like a color palette to generate unique audio, greatly expanding the creative boundaries. User satisfaction with the generated audio can be increased by more than 30%, and audio conflicts or homogenization problems are significantly reduced, achieving highly personalized and customized audio generation and breaking through the limitations of audio libraries. Moreover, this disclosure provides a visual and intuitive mixing interface and a one-click intelligent mixing function, enabling even users without any audio synthesis background to easily try audio creation. Preset mixing combinations and templates further lower the barrier to entry, facilitating greater user participation in audio creation. Furthermore, the multi-dimensional feature recognition functions for emotion, speech rate, rhythm, etc., provided in this disclosure make the generated audio not only similar to the original audio but also more expressive. By finely adjusting multi-dimensional audio features such as emotion and accent, the generated audio can better serve the emotional expression and narrative needs of the content, improving user experience and content appeal, and enriching audio expressiveness and realism. Furthermore, the embodiments of this disclosure greatly promote collaboration and knowledge sharing among users through the mechanism of sharing and composable audio feature templates, as well as the audio request function. User-generated content can feed back into the target application, forming a self-enriching and continuously evolving audio library, ensuring the long-term vitality and innovation of audio resources within the target application.

[0178] In this embodiment, intelligent audio recognition and similarity recommendation solve the problem of finding audio by listening. Users do not need to input audio names or tags; they only need to upload or record audio clips, and the target application can automatically extract their audio features. The target application establishes an audio feature vector space through a deep learning model, performs high-dimensional feature matching between user input and audio templates in the audio library, and recommends audio templates with similar audio features, labeling the similarity. Furthermore, when no similar audio template exists in the audio library, approximate audio can be generated through AI audio cloning and transfer technology. This function breaks through the traditional tag-based retrieval method, enabling users to find their ideal audio more naturally and intuitively.

[0179] Furthermore, in this embodiment, unlike simple audio-dimensional feature recognition, multi-dimensional feature recognition and matching, such as emotion, speech rate, and rhythm, are introduced. For emotion-dimensional features, emotional characteristics, such as "happy," "sad," and "serious," are analyzed through audio features, and audio with different emotional styles is provided during recommendations. For speech rate and rhythm-dimensional features, the system can extract rhythmic features such as speech rate (words / second), pause patterns, and stress distribution to ensure that the recommended audio maintains a consistent rhythm. Moreover, when recommending similar audio, not only is the overall similarity given, but also the similarity for each dimension of audio features is given separately, allowing users to intuitively understand that "it is 80% similar in sound quality, 70% similar in speech rate, and 90% similar in emotion." This mechanism ensures that the recommendation results are more accurate and meet users' complex needs for audio performance.

[0180] Furthermore, this embodiment provides a visual audio mixing interface, on which audio is broken down into multiple dimensions of audio features, and the degree parameters are presented in a progress bar, slider, or graphical manner. Users can freely adjust the proportion of different audio features in each dimension, just like mixing colors on a palette, for example, "70% gentle female voice + 30% XX accent"; real-time listening and preview are provided, allowing users to immediately perceive the adjustment effect; "one-click intelligent mixing" is supported, quickly generating recommended combinations and lowering the operational threshold. This interactive mode greatly enhances the user's creative freedom, meeting the refined needs of professional users while also helping beginners get started easily. In addition, this embodiment allows users to save custom-generated audio as an audio template and share it in a virtual space. Users can also extract audio features of a specific dimension of the audio, such as XX accent features or broadcast rhythm features, and save them as an audio feature template for sharing.

[0181] This disclosure provides a function capable of intelligently identifying audio features from input audio or video, and recommending similar audio templates in an audio library or generating matching audio using AI cloning technology, thereby solving the problem of users searching for audio by sound. Furthermore, this disclosure provides a flexible audio mixing and generation mechanism, allowing users to finely and visually mix multi-dimensional audio features to create personalized custom audio, breaking through the limitations of existing audio libraries. Moreover, this disclosure provides an intuitive and easy-to-use interactive interface and intelligent mixing function, enabling non-audio professionals to easily participate in audio creation, and providing preset templates to inspire creative ideas. Furthermore, this disclosure establishes an audio sharing and reuse mechanism, allowing users to share their created audio feature templates and audio, thereby promoting communication and collaboration among users and enriching the audio library together.

[0182] Figure 10This is a block diagram illustrating an audio-based interactive device according to an exemplary embodiment. (Refer to...) Figure 10 The device includes: The recognition unit 1001 is configured to perform a recognition operation in response to the input audio and display the recognition result of the audio. The recognition result includes multiple audio features of the audio in multiple dimensions and a degree parameter for each audio feature. The degree parameter is used to indicate the degree of performance of the audio feature in the audio. The multiple dimensions include a timbre dimension and a target dimension. The target dimension is a dimension other than the timbre dimension. The display unit 1002 is configured to display multiple audio feature templates corresponding to multiple audio features and the degree parameters of each audio feature template; The adjustment unit 1003 is configured to perform an adjustment operation in response to an audio input, displaying the adjusted target audio. The adjustment operation includes at least one of an adjustment operation of the degree parameter of at least one audio feature and a selection operation of at least one audio feature template, wherein the selected audio feature template is used to mix with multiple audio features to generate the target audio.

[0183] In some embodiments, the multiple audio feature templates include at least one audio feature template from a first audio template, wherein the first audio template is an audio template in an audio library whose similarity to the audio satisfies a similarity condition, and the similarity is the similarity between the multiple audio feature templates of the first audio template and the multiple audio features of the audio.

[0184] In some embodiments, the recognition result further includes at least one first audio template and a similarity score corresponding to each first audio template; the apparatus further includes a playback unit configured to perform: In response to a playback operation on any of the first audio templates, the first audio template is played.

[0185] In some embodiments, there are multiple first audio templates, and the display unit 1002 is further configured to perform at least one of the following: In response to a blending operation on at least a portion of the first audio templates, the blended audio is displayed; In response to a mixing operation on the target audio and at least a portion of the first audio template, the resulting mixed audio is displayed.

[0186] In some embodiments, the identification result may also include reference audio, the reference audio having the same multiple audio features as the audio, and the target audio being the reference audio after adjusting the multiple audio features based on the adjustment operation.

[0187] In some embodiments, the device further includes a playback module configured to perform at least one of the following: In response to a playback operation on the reference audio, play the reference audio; In response to a playback operation on the target audio, play the target audio.

[0188] In some embodiments, the target dimension includes at least one of pitch range dimension, emotion dimension, speech rate dimension, rhythm dimension, and accent dimension.

[0189] In some embodiments, the display unit 1002 is further configured to perform: Display the first audio interface, which includes multiple audio templates; In response to the selection operation of the second audio template among multiple audio templates, the degree parameters of each of the multiple audio feature templates of the second audio template are displayed; In response to an adjustment operation on the degree parameter of at least one audio feature template of the second audio template, the audio obtained by adjusting the audio feature template based on the adjusted degree parameter is displayed.

[0190] In some embodiments, the display unit 1002 is further configured to perform: The second audio interface is displayed, which includes multiple audio templates, each with multiple audio feature templates and their respective degree parameters. In response to a blending operation on at least two audio feature templates of at least two audio templates on a second audio interface, the blended audio is displayed.

[0191] In some embodiments, the display unit 1002 is further configured to perform: In response to an operation that sets the scale of each of the at least two audio feature templates, display the scale of each of the at least two audio feature templates; In response to a blending operation on at least two audio feature templates, display the audio obtained by blending at least two audio feature templates based on the respective proportions of the at least two audio feature templates.

[0192] In some embodiments, the display unit 1002 is further configured to perform: The third audio interface is displayed, which includes multiple candidate audio feature templates and their respective degree parameters. In response to a blending operation on at least two candidate audio feature templates on a third audio interface, the blended audio is displayed.

[0193] In some embodiments, the apparatus further includes a publishing unit configured to perform at least one of the following: In response to the publishing operation of the target audio, the corresponding audio template is published in the virtual space, and the audio template in the virtual space can be used by the target user; In response to a publishing operation for any audio feature in the target audio, an audio feature template is published in the virtual space. The audio feature template includes the audio feature and the degree parameter of the audio feature. The audio feature template in the virtual space is available for use by the target user.

[0194] In some embodiments, the display unit 1002 is further configured to perform: In response to a request posting operation in the virtual space, the request posting information is displayed in the virtual space. The request posting information includes an audio sample and adjustment information for the audio sample. The request posting information is used to request the acquisition of an audio template based on the audio sample and the adjustment information.

[0195] This disclosure provides an audio-based interactive device that, through audio recognition operations, can identify not only audio features in the timbre dimension but also audio features in other dimensions. This upgrades the recognition result from simply judging the timbre source to a deeper understanding of the complete information and intent of the audio, improving the accuracy of recognition. Furthermore, it outputs degree parameters to indicate the degree of each audio feature's representation in the audio, further enhancing the precision of the recognition results. It also provides multiple audio feature templates corresponding to these multiple audio features and degree parameters for each template. Based on this, users can adjust the degree parameters of any audio feature and select any audio feature template to mix with multiple audio features to generate new audio. This allows users to fine-tune the audio features and incorporate other audio features, thereby generating audio that meets their personalized needs without requiring excessive user intervention, improving the quality of the generated audio while maintaining efficiency.

[0196] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0197] In some embodiments, the electronic device is provided as a terminal. Figure 11 A structural block diagram of a terminal 1100 provided in an exemplary embodiment of this disclosure is shown. The terminal 1100 may be a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 1100 may also be referred to as a user device, portable terminal, laptop terminal, desktop terminal, or other names.

[0198] Typically, terminal 1100 includes a processor 1101 and a memory 1102.

[0199] Processor 1101 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1101 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1101 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1101 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1101 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0200] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1102 are used to store at least one program code, which is executed by the processor 1101 to implement the audio-based interactive method provided in the method embodiments of this disclosure.

[0201] In some embodiments, the terminal 1100 may also optionally include a peripheral device interface 1103 and at least one peripheral device. The processor 1101, memory 1102, and peripheral device interface 1103 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1103 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1104, a display screen 1105, a camera assembly 1106, an audio circuit 1107, and a power supply 1108.

[0202] Peripheral device interface 1103 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1101 and memory 1102. In some embodiments, processor 1101, memory 1102 and peripheral device interface 1103 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1101, memory 1102 and peripheral device interface 1103 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0203] The radio frequency (RF) circuit 1104 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1104 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1104 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1104 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1104 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1104 may also include circuitry related to NFC (Near Field Communication), which is not limited in this disclosure.

[0204] Display screen 1105 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1105 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1101 for processing. In this case, display screen 1105 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1105, which serves as the front panel of terminal 1100; in other embodiments, there may be at least two display screens, respectively disposed on different surfaces of terminal 1100 or in a folded design; in still other embodiments, display screen 1105 may be a flexible display screen, disposed on a curved or folded surface of terminal 1100. Furthermore, display screen 1105 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1105 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0205] The camera assembly 1106 is used to acquire images or videos. Optionally, the camera assembly 1106 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1106 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0206] The audio circuit 1107 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1101 for processing, or input to the radio frequency circuit 1104 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1100. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1101 or the radio frequency circuit 1104 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1107 may also include a headphone jack.

[0207] Power supply 1108 is used to power the various components in terminal 1100. Power supply 1108 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1108 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0208] Those skilled in the art will understand that Figure 11 The structure shown does not constitute a limitation on terminal 1100 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0209] In an exemplary embodiment, a computer-readable storage medium is also provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the above-described audio-based interaction method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0210] In an exemplary embodiment, a computer program product is also provided, the computer program product including a computer program that, when executed by a processor, implements the above-described audio-based interaction method.

[0211] In some embodiments, the computer program product involved in this disclosure may be deployed on an electronic device for execution, or on multiple electronic devices located in one location, or on multiple electronic devices distributed in multiple locations and interconnected through a communication network. The multiple electronic devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0212] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims. All the above-described optional technical solutions can be combined in any way to form optional embodiments of this application, and will not be elaborated upon here.

[0213] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An audio-based interaction method, characterized in that, The method comprises: in response to an identification operation on input audio, displaying an identification result of the audio, the identification result comprising a plurality of audio features of the audio in a plurality of dimensions and a degree parameter of each audio feature, the degree parameter being used to indicate a degree of performance of the audio feature in the audio, the plurality of dimensions comprising a timbre dimension and a target dimension, the target dimension being a dimension other than the timbre dimension; displaying a plurality of audio feature templates corresponding to the plurality of audio features and a degree parameter of each audio feature template; in response to an adjustment operation on the audio, displaying a target audio obtained by adjustment, the adjustment operation comprising at least one of an adjustment operation on a degree parameter of at least one audio feature and a selection operation on at least one audio feature template, the selected audio feature template being used to mix with the plurality of audio features to generate the target audio.

2. The audio-based interaction method of claim 1, wherein, The plurality of audio feature templates comprises at least one audio feature template in a first audio template, the first audio template being an audio template in an audio library that satisfies a similarity condition with the audio, the similarity being a similarity between the plurality of audio feature templates of the first audio template and the plurality of audio features of the audio.

3. The audio-based interaction method of claim 2, wherein, The identification result further comprises the at least one first audio template and a respective similarity of each first audio template; the method further comprises: in response to a playback operation on any first audio template, playing the first audio template.

4. The audio-based interaction method of claim 3, wherein, The first audio template is a plurality, and the method further comprises at least one of: in response to a mixing operation on at least part of the plurality of first audio templates, displaying audio obtained by mixing; in response to a mixing operation on the target audio and at least part of the first audio templates, displaying audio obtained by mixing.

5. The audio-based interaction method of claim 1, wherein, The identification result further comprises a reference audio, the plurality of audio features of the reference audio being the same as the plurality of audio features of the audio, and the target audio being the reference audio after the plurality of audio features are adjusted based on the adjustment operation.

6. The audio-based interaction method of claim 5, wherein, The method further comprises at least one of: in response to a playback operation on the reference audio, playing the reference audio; in response to a playback operation on the target audio, playing the target audio.

7. The audio-based interaction method of claim 1, wherein, The target dimension comprises at least one of a pitch range dimension, an emotion dimension, a speech rate dimension, a rhythm dimension, and an accent dimension.

8. The audio-based interaction method of claim 1, wherein, The method further comprises: displaying a first audio interface, the first audio interface comprising a plurality of audio templates; in response to a selection operation on a second audio template in the plurality of audio templates, displaying a respective degree parameter of a plurality of audio feature templates of the second audio template; in response to an adjustment operation on a degree parameter of at least one audio feature template of the second audio template, displaying an audio obtained by adjusting the audio feature template based on the adjusted degree parameter.

9. The audio-based interaction method of claim 1, wherein, The method further comprises: displaying a second audio interface, the second audio interface comprising a plurality of audio feature templates of a plurality of audio templates and a respective degree parameter of the plurality of audio feature templates; In response to a mixing operation on at least two audio feature templates in the second audio interface, display the mixed audio.

10. The audio-based interaction method of claim 9, wherein, The method further includes: In response to a proportion setting operation on the at least two audio feature templates, display the proportions of the at least two audio feature templates respectively; The response to the mixing operation on the at least two audio feature templates in the second audio interface includes: In response to the mixing operation on the at least two audio feature templates, display the audio mixed based on the proportions of the at least two audio feature templates respectively.

11. The audio-based interaction method of claim 1, wherein, The method further includes: Display a third audio interface, the third audio interface including a plurality of candidate audio feature templates and a degree parameter of each of the plurality of candidate audio feature templates; In response to a mixing operation on at least two candidate audio feature templates in the third audio interface, display the mixed audio.

12. The audio-based interaction method of claim 1, wherein, The method further includes at least one of: In response to a publishing operation on the target audio, publish the audio template corresponding to the target audio in a virtual space, the audio template in the virtual space being available for use by a target user; In response to a publishing operation on any audio feature in the target audio, publish an audio feature template in a virtual space, the audio feature template including the audio feature and a degree parameter of the audio feature, the audio feature template in the virtual space being available for use by a target user.

13. The audio-based interaction method of claim 1, wherein, The method further includes: In response to a request publishing operation in a virtual space, display request publishing information in the virtual space, the request publishing information including an audio sample and adjustment information of the audio sample, the request publishing information being used to request an audio template based on the audio sample and the adjustment information.

14. An audio-based interactive device, comprising: The apparatus includes: An identification unit configured to perform, in response to an identification operation on an input audio, display an identification result of the audio, the identification result including a plurality of audio features of the audio in a plurality of dimensions and a degree parameter of each of the plurality of audio features, the degree parameter being used to indicate a performance degree of the audio feature in the audio, the plurality of dimensions including a timbre dimension and a target dimension, the target dimension being a dimension other than the timbre dimension; A display unit configured to perform display a plurality of audio feature templates corresponding to the plurality of audio features and a degree parameter of each of the plurality of audio feature templates; An adjustment unit configured to perform, in response to an adjustment operation on the audio, display a target audio obtained by adjustment, the adjustment operation including at least one of an adjustment operation on a degree parameter of at least one audio feature and a selection operation on at least one audio feature template, the selected audio feature template being used to mix with the plurality of audio features to generate the target audio.

15. An electronic device, comprising: Include: A processor; A memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the audio-based interaction method according to any one of claims 1 to 13.

16. A computer readable storage medium characterized by: When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the audio-based interaction method of any one of claims 1 to 13.

17. A computer program product, characterised in that, The computer program product comprises a computer program which, when executed by a processor, implements the audio-based interaction method of any one of claims 1 to 13.

Images