An audio processing method, electronic device and program product

Abstract

Embodiments of the present disclosure provide an audio processing method, an electronic device and a program product. The audio processing method comprises: obtaining an initial audio stream; the initial audio stream comprising an audio stream corresponding to near-end sound and / or an audio stream corresponding to far-end sound; performing audio feature recognition on the initial audio stream to separate at least one audio; obtaining a preference selection for the at least one audio; confirming a target audio from the at least one audio based on the preference selection; and outputting a target sound corresponding to the target audio. Through the embodiment scheme, the audio can be processed according to the user preference, and the personalized output of the audio is realized.

Description

Technical Field

[0001] This disclosure relates to the field of audio processing, and in particular to an audio processing method, electronic device, and program product. Background Technology

[0002] When users make voice calls using electronic devices such as mobile phones and tablets, ensuring voice quality has always been an important technical challenge. Traditional audio processing algorithms have matured, but they have also encountered bottlenecks. Deep learning-based audio processing methods have become a development trend in recent years. While gradually solving the bottlenecks of traditional algorithms, current audio processing is still not personalized enough. Summary of the Invention

[0003] This disclosure provides an audio processing method, an electronic device, and a program product.

[0004] In a first aspect, embodiments of this disclosure provide an audio processing method, comprising:

[0005] Obtain the initial audio stream; the initial audio stream includes the audio stream corresponding to the near-end sound and / or the audio stream corresponding to the far-end sound.

[0006] The initial audio stream is subjected to audio feature recognition to separate at least one audio element;

[0007] Obtain the preference selection for the at least one audio;

[0008] Based on the preference selection, the target audio is confirmed from the at least one audio;

[0009] Output the target sound corresponding to the target audio.

[0010] Secondly, embodiments of this disclosure also provide an electronic device, including:

[0011] One or more processors;

[0012] A memory having stored one or more programs that, when executed by one or more processors, enable the one or more processors to implement the audio processing method.

[0013] One or more input / output (I / O) interfaces are connected between the processor and the memory and configured to enable information exchange between the processor and the memory.

[0014] Thirdly, this disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the audio processing method.

[0015] This embodiment of the scheme obtains an initial audio stream; the initial audio stream includes an audio stream corresponding to near-end sounds and / or an audio stream corresponding to far-end sounds; audio feature recognition is performed on the initial audio stream to separate at least one audio; a preference selection is obtained for the at least one audio; a target audio is confirmed from the at least one audio based on the preference selection; and the target sound corresponding to the target audio is output. Through this embodiment, audio can be processed according to user preferences, achieving personalized audio output. Attached Figure Description

[0016] In the accompanying drawings of the embodiments disclosed herein:

[0017] Figure 1 This is a schematic flowchart of an audio processing method provided in an embodiment of the present disclosure;

[0018] Figure 2 A schematic diagram illustrating an embodiment of a complex call scenario provided in this disclosure;

[0019] Figure 3 A schematic diagram of a first scheme for obtaining a preference selection for at least one audio file, provided in an embodiment of this disclosure;

[0020] Figure 4 A schematic diagram of an audio stream manifest embodiment provided by this disclosure;

[0021] Figure 5 A schematic diagram of enhanced and weakened sound vectors provided in an embodiment of this disclosure;

[0022] Figure 6 A schematic diagram of a second scheme for obtaining a preference selection for at least one audio file, provided in an embodiment of this disclosure;

[0023] Figure 7 This is a schematic flowchart of a method for selecting the audio corresponding to a target sound source from the audio corresponding to a sound source in a preference database, as provided in an embodiment of this disclosure.

[0024] Figure 8 This is a schematic flowchart of a method for outputting a target sound corresponding to a target audio, provided in an embodiment of this disclosure.

[0025] Figure 9 This is a schematic block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions of this disclosure, the communication-sensing data processing method and computer-readable storage medium provided in the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0027] The present disclosure will be described more fully below with reference to the accompanying drawings; however, the embodiments shown may be embodied in different forms, and the present disclosure should not be construed as limited to the embodiments set forth below. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will enable those skilled in the art to fully understand the scope of the disclosure.

[0028] The accompanying drawings of the embodiments disclosed herein are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the detailed embodiments to explain this disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the description of the detailed embodiments with reference to the accompanying drawings.

[0029] This disclosure may be described with reference to plan and / or cross-sectional views using the ideal schematic diagrams of this disclosure. Therefore, the example illustrations may be modified according to manufacturing techniques and / or tolerances.

[0030] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0031] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the disclosure. The term "and / or" as used in this disclosure includes any and all combinations of one or more of the associated enumerated entries. The singular forms "a" and "the" as used in this disclosure are also intended to include the plural forms, unless the context clearly indicates otherwise. The terms "comprising," "made of," etc., as used in this disclosure specify the presence of the stated feature, integral, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0032] Unless otherwise specified, all terms used in this disclosure (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined in this disclosure.

[0033] Users can make voice calls using electronic devices such as mobile phones and tablets. Ensuring voice quality has always been an important technical challenge. Traditional audio processing algorithms have matured, but they have also encountered bottlenecks. Deep learning-based audio processing methods have been a development trend in recent years. After gradually solving the bottlenecks of traditional algorithms, the pursuit will be for higher quality voice. Each user's usage environment and preferences are different. We hope to provide high-quality voice even in complex environments and allow each user to process voice according to their own preferences.

[0034] In complex scenarios where multiple speakers (the desired sound) and multiple interfering speakers (the unwanted sound) coexist, if the duration and energy of the interfering speakers exceed those of the user's sound, the sound processing model will mistakenly identify the interfering speakers as the target sound. For example, in environments like concerts or sports stadiums, surrounding sounds (singing, cheering, broadcasting, shouting, etc.) are constantly present, and their duration and energy may exceed those of the user, leading to misjudgment. Similarly, in situations like subways, cafeterias, or restaurants where other people are nearby, if these people are chatting loudly, it can also lead to misjudgment, especially when the user is not speaking while those around them are talking, their voices will have a longer duration and higher energy than the user's. There are also scenarios with multiple speakers, such as when I'm on the phone with my parents, both of them are speaking into the same phone. Regardless of the ratio of their voice duration and energy, both of their voices are what I want to hear. However, traditional solutions would amplify one sound and suppress another, which does not meet the user's actual needs. For example, the voices of certain special groups may be unclear due to illness, physical disabilities, or other reasons. If there is no data for these voices during the training phase of the sound processing model, they might be treated as noise and eliminated by the model. Traditionally, if a user is dissatisfied with the experience, they need to provide feedback to the manufacturer. The manufacturer then fine-tunes the sound processing model for that user's specific environment and preferences before releasing a new model. This process is time-consuming, costly, and the new model affects all users. In these complex environments where specific problems require specific analysis, only users truly understand what kind of sound they want and don't want. Therefore, it's necessary to give users control over the use of the sound processing model, allowing them to make their own choices.

[0035] This disclosure provides a user-preference-based intelligent voice processing method, which can acquire an initial audio stream; the initial audio stream includes an audio stream corresponding to near-end sounds and / or an audio stream corresponding to far-end sounds; perform audio feature recognition on the initial audio stream to separate at least one audio; acquire a preference selection for the at least one audio; confirm a target audio from the at least one audio based on the preference selection; and output the target sound corresponding to the target audio. This embodiment enables audio processing according to user preferences, achieving personalized audio output.

[0036] The solutions disclosed herein can be applied to any terminal with audio processing capabilities, including but not limited to mobile phones, tablets, personal computers, smart wearable devices, in-vehicle devices, and home appliances. The application scenarios of the solutions disclosed herein can include, but are not limited to, call scenarios and human-computer interaction scenarios.

[0037] The embodiments of this disclosure will be described in detail below.

[0038] This disclosure provides an audio processing method, such as... Figure 1 As shown, it may include steps S11-S15:

[0039] S11. Obtain the initial audio stream.

[0040] In this embodiment of the disclosure, the initial audio stream may include, but is not limited to, the voice stream collected by the terminal. For example, during a voice or video call between the terminal user and others, the terminal receives a voice stream from the terminal user, other people and / or things around the terminal user; or, during an interaction between the terminal user and the terminal, the terminal receives a voice stream from the terminal user, other people and / or things around the terminal user.

[0041] In this embodiment of the disclosure, the initial audio stream may include an audio stream corresponding to near-end sound or an audio stream corresponding to far-end sound. Near-end sound is the sound emitted by a person or object on the near-end device side, and far-end sound is the sound emitted by a person or object on the far-end device side.

[0042] In the embodiments disclosed herein, such as Figure 2 The diagram illustrates a complex call scenario. A first user terminal A1 (acting as a near-end device for collecting near-end audio) has near-end users (users on the near-end device side) and far-end users (users on the far-end device side). Near-end users include a first near-end user A, and far-end users include a first far-end user C. The first near-end user A establishes a call with the first user terminal A2 (acting as a far-end device for collecting far-end audio and transmitting it to the near-end device) of the first far-end user C through the first user terminal A1. Near-end users also include a second near-end user B. The near-end users are surrounded by persons C and D; that is, the interference sound from the near-end users comes from persons C and D, also referred to as interference persons C and D. Far-end users also include a second far-end user D. The far-end users are surrounded by persons E and F; that is, the interference sound from the far-end users comes from persons E and F, also referred to as interference persons E and F. For the first user terminal A1, the received audio is divided into near-end audio and far-end audio, with the echo being treated as a special type of near-end audio.

[0043] In this embodiment of the disclosure, when a near-end user speaks, the near-end sound may include the voice of a first near-end user A, the voice of a second near-end user B, and the voices of interfering persons C and D. Accordingly, the target audio stream is a mixed audio stream in which the voices of the first near-end user A, the voice of the second near-end user B, and the voices of interfering persons C and D overlap.

[0044] In this embodiment of the disclosure, when a remote user speaks, the remote sound may include the voice of the first remote user C, the voice of the second remote user D, and the voices of the interfering persons E and F. Accordingly, the target audio stream is a mixed audio stream in which the voices of the first remote user C, the second remote user D, and the interfering persons E and F overlap.

[0045] In this embodiment of the disclosure, the solution can be applied to a near-end device.

[0046] S12. Perform audio feature recognition on the initial audio stream to separate at least one audio segment.

[0047] In this embodiment of the disclosure, an audio feature can be identified from the initial audio stream using a preset, trained sound separation model to separate at least one audio segment.

[0048] In this embodiment of the disclosure, the sound separation model can separate at least one audio from a segment of audio where various sounds overlap, based on different audio features (e.g., different sound sources, such as different speakers), and each audio can correspond to an independent audio stream.

[0049] In this embodiment of the disclosure, for example, in the aforementioned embodiment, when a near-end user speaks, the voices of the first near-end user A, the second near-end user B, the interfering persons C and D, and the echo of the far-end sound may all be collected by the microphone of the first user terminal A1 (such as a mobile phone) and sent into the first user terminal A1 to form an initial audio stream, which is then sent to the sound separation module. The sound separation model can separate the near-end sound into at least one audio stream according to the audio characteristics of different speakers (i.e., different sound sources, such as A, B, C, D, and the echo). For example, they can be independent audio streams containing the voice of A, B, C, D, and the echo, respectively.

[0050] In this embodiment of the disclosure, for example, in the aforementioned embodiment, when a remote user speaks, the voices of the first remote user C, the second remote user D, and the interfering persons E and F may all be collected by the second user terminal A2 (such as a mobile phone) and transmitted to the first user terminal A1 (such as a mobile phone) to form an initial audio stream sent to the sound separation module. The sound separation model can separate the near-end sound into at least one audio stream according to the audio characteristics of different speakers (i.e., different sound sources, such as C, D, E, and F). For example, it can be an independent audio stream containing the voice of C, an independent audio stream containing the voice of D, an independent audio stream containing the voice of E, and an independent audio stream containing the voice of F.

[0051] In this embodiment of the disclosure, for example, if an initial audio stream contains the voice of person M, the voice of person N, and the sound of ocean waves, the sound separation model can separate the initial audio stream into at least one audio stream, such as three independent audio streams, which can be an independent audio stream containing the voice of person M, an independent audio stream containing the voice of person N, and an independent audio stream containing the sound of ocean waves.

[0052] In the embodiments of this disclosure, in different application scenarios, the target audio stream received by the terminal may not necessarily consist entirely of overlapping sounds, and the initial audio stream may also contain only the sound of a single sound source.

[0053] In this embodiment of the disclosure, each separated audio audio can be converted into a sound feature vector, thereby representing the audio features of each audio audio in a vector space.

[0054] In this embodiment of the disclosure, an audio file can be converted into a sound vector using a pre-trained sound feature vector model.

[0055] In this embodiment of the disclosure, the dimension of the sound vector is generally 200-500. Here, a dimension of 2 can be used to facilitate display and explanation on a Cartesian coordinate system (a two-dimensional rectangular coordinate system).

[0056] S13. Obtain preference selection for at least one audio.

[0057] In this embodiment of the disclosure, preference selection can be obtained directly based on preference selection of at least one audio, or the sound factors of the at least one audio can be processed first to determine an initial target sound based on the processing result. If the user is satisfied with the initial target sound, the initial target sound can be directly output. If the user is not satisfied with the initial target sound, preference selection for at least one audio can be obtained again.

[0058] In this embodiment of the disclosure, the method for determining the initial target sound is described below.

[0059] In this embodiment of the disclosure, before obtaining a preference selection for at least one audio file, the method may further include:

[0060] Detect at least one audio element;

[0061] The audio that meets the sound element requirements is used as the initial target sound.

[0062] In this embodiment of the disclosure, the sound element may include, but is not limited to, sound duration and / or sound energy.

[0063] In this embodiment of the disclosure, the sound duration and sound energy of each audio can be compared, and the audio with the longest sound duration and / or the highest sound energy can be selected as the target audio and sent to the remote terminal, while the other audio can be discarded.

[0064] In this embodiment of the disclosure, if the user is not satisfied with the default processing effect described above, a scheme to obtain a preference selection for at least one audio file can be executed.

[0065] In the embodiments of this disclosure, obtaining preference selection for at least one audio may include, but is not limited to, obtaining preference selection based on an audio stream list, and / or obtaining preference selection based on a preset preference database.

[0066] In the embodiments of this disclosure, the scheme for obtaining preference selection based on audio stream lists will be described in detail below.

[0067] In the embodiments disclosed herein, such as Figure 3 As shown, obtaining a preference selection for at least one audio file may include steps S21-S22:

[0068] S21. In response to the triggering of preset display conditions, display an audio stream list generated based on part or all of the audio in at least one audio file.

[0069] In this embodiment of the disclosure, the audio stream list may be implemented in ways including but not limited to lists, options, etc.

[0070] In this disclosed embodiment, the display conditions may include, but are not limited to:

[0071] A preset request was received; or,

[0072] At least one audio segment can be separated into multiple segments; or,

[0073] Confirm that the terminal has entered the preset environment.

[0074] In this embodiment of the disclosure, the preset request may include, but is not limited to, a request to retrieve a playlist. The user can actively trigger a preset button to send a request to retrieve a playlist, thereby actively retrieving / displaying the audio stream playlist; or, if it is determined that there are multiple separated audio streams, the audio stream playlist can be automatically retrieved / displayed; or, the preset environment may include, but is not limited to, the terminal entering a noisy environment (e.g., a meeting room, KTV, etc.). The preset environment can be determined by the user triggering a preset environment selection box, or by detecting a preset operation gesture. After confirming that the terminal has entered the preset environment, the audio stream playlist can be automatically retrieved / displayed.

[0075] In this disclosure embodiment, displaying an audio stream list generated based on part or all of the audio in at least one audio file may include:

[0076] Identify the audio features of each audio element in this part or all of the audio;

[0077] In response to identifying the audio features of any first audio among the various audios, determine the identifier corresponding to the first audio;

[0078] The first audio stream is displayed in the audio stream list, and this identifier indicates the audio characteristics of the first audio stream.

[0079] In this embodiment of the disclosure, the audio stream list may include some or all of the audio from at least one audio file, and display the salient features of each audio file to facilitate users to quickly understand the audio content and make a selection.

[0080] In this embodiment of the disclosure, for example, the identifier corresponding to the audio can be displayed, and each identifier can indicate the sound characteristics corresponding to the respective independent audio stream.

[0081] In the embodiments disclosed herein, such as Figure 4 The diagram shown is a schematic of an audio stream list embodiment. The audio stream list can include audio stream lists containing near-end sounds or audio stream lists containing far-end sounds.

[0082] In this embodiment of the disclosure, for example, for an audio stream list embodiment of near-end sound, the audio stream list can be used to display independent audio streams of the terminal user of the near-end device (e.g., "I"), independent audio streams of ocean wave sounds, and independent audio streams of some interference sounds (e.g., the recorded interference sound "bought"). These independent audio streams are displayed in the audio stream list with corresponding identifiers. For example, if an independent audio stream belongs to a user with a registered voiceprint within the terminal, the user's name (e.g., "I", "Xiaoming", "Xiaoli", etc.) can be displayed as the identifier. For unregistered voiceprints, the sound characteristics of the independent audio stream can be used as the identifier; for example, "ocean wave sound" can be used to identify the sound characteristic of the independent audio stream. If a voiceprint is not registered and no special sound characteristics are identified, the sound content text of the sound can be used as the identifier; for example, for the voice of an unknown person saying "bought," the sound content text "bought" can be directly used as the identifier of the corresponding independent audio stream. Therefore, words such as "I," "ocean wave sound," and "bought" can be displayed as identifiers of the responding independent video streams in the audio stream list corresponding to the near-end sound.

[0083] In this embodiment of the disclosure, for example, for an audio stream list embodiment of remote sound, the audio stream list can be used to display independent audio streams of the terminal user's father, mother, and some interference sounds (e.g., the recorded interference sound "news"). These independent audio streams are displayed in the audio stream list with corresponding identifiers. For example, if an independent audio stream belongs to a user with a registered voiceprint within the terminal, the user's name (e.g., father, mother, etc.) can be displayed as the identifier. For unregistered voiceprints, the sound characteristics of the independent audio stream can be used as the identifier. If a voiceprint is not registered and no special sound characteristics are identified, the sound content text of the sound can be used as the identifier. For example, for the voice "news" of an unknown person, the sound content text "news" can be directly used as the identifier of the corresponding independent audio stream. Therefore, words such as "father," "news," and "mother" can be displayed as identifiers of the corresponding independent video streams in the audio stream list corresponding to the remote sound.

[0084] In this embodiment of the disclosure, a corresponding selection box can be set after each audio identifier so that the end user can select the desired audio from these audios based on the selection box check according to preferences and needs.

[0085] In this embodiment of the disclosure, a selection box may not be set in the audio stream list, and the end user can directly touch the corresponding icon to select the audio corresponding to the icon.

[0086] In this embodiment of the disclosure, the audio stream list can distinguish between near-end sound and far-end sound. That is, it can provide an audio stream list corresponding to the near-end sound (which can be simply referred to as the near-end noise reduction list) when the near-end user is speaking, and provide an audio stream list corresponding to the far-end sound (which can be simply referred to as the far-end noise reduction list) when the far-end user is speaking. Alternatively, it can provide audio stream lists corresponding to both near-end sound and far-end sound simultaneously when both the near-end user and the far-end user are speaking. It can also provide an audio stream list corresponding to either the near-end sound or the far-end sound based on the near-end user's choice.

[0087] In this disclosure embodiment, displaying an audio stream list generated based on part or all of the audio in at least one audio file may further include:

[0088] Identify the audio features of each audio element in this part or all of the audio;

[0089] In response to the failure to identify the audio characteristics of any second audio among the various audio streams, the second audio is displayed as an unknown audio in the audio stream list, or the second audio is not displayed in the audio stream list.

[0090] In this embodiment of the disclosure, if a significant audio feature cannot be parsed, the audio identifier (that indicates the audio feature) may not be displayed, and the audio may be displayed as unknown audio (e.g., Figure 4 (The third audio shown in the video).

[0091] S22. In response to receiving a selection operation on the audio stream list, obtain the preference selection.

[0092] In this embodiment of the disclosure, obtaining preference selection in response to receiving a selection operation on an audio stream list may include:

[0093] Detect the selection operation of the identifier of audio in the audio stream list;

[0094] In response to detecting that any first identifier in the audio stream list is selected, the audio corresponding to the selected first identifier is selected as the preference selection.

[0095] In this embodiment of the disclosure, based on the audio identifiers displayed in the audio stream list, the user can select any one or more identifiers as preference identifiers, and the terminal can select the audio corresponding to the selected preference identifier as the preference selection.

[0096] In this embodiment of the disclosure, for example, for an audio stream list with selection boxes, the user can check the box corresponding to the identifier (such as a first identifier) ​​of the audio they want to send, and cross out the corresponding selection box for the audio they do not want to send. For identifiers that are neither checked nor crossed out, the terminal can process them in a default manner (e.g., may include, but is not limited to, deleting the corresponding audio). The terminal can use the selected (e.g., checked) audio as a preference selection based on the user's selection results.

[0097] In this embodiment of the disclosure, for example, for an audio stream playlist without a selection box, the user can select by directly touching the corresponding icon. The touched icon can be considered as the user's selected preference icon, and the untouched icon can be considered as the user's unselected non-preference icon. The terminal can use the selected (e.g., touched) audio as the preferred selection based on the user's selection result.

[0098] In this embodiment of the disclosure, for example, for an audio stream playlist without a selection box, the audio stream playlist can also have a selection area. Users can move selected preference markers into the selection area by dragging or clicking. The terminal can use the markers in the selection area as preference markers and can select the audio (e.g., within the selection area) as the preferred selection based on the user's choice. In another embodiment, the audio stream playlist can also have a delete area, a pending area, etc. Users can move markers in the audio stream playlist to the delete area and / or the pending area according to their preferences or needs. A marker being moved to the delete area indicates that the corresponding audio can be deleted, and a marker being moved to the pending area indicates that the corresponding audio is in a pending state or can be processed by default (e.g., including but not limited to deletion).

[0099] In this embodiment of the disclosure, the method may further include:

[0100] Convert the independent audio stream corresponding to the first identifier selected in the audio stream list into a first sound vector;

[0101] Vector enhancement is applied to the first sound vector;

[0102] The sound source corresponding to the first identifier is stored in a preset preference database, and the enhanced first sound vector is used to indicate the audio characteristics of the preferred sound source.

[0103] And / or,

[0104] Convert the audio streams that are not selected into a second sound vector;

[0105] Vector attenuation is applied to the second sound vector;

[0106] The audio sources that are not selected are stored in a preset preference database, and the audio characteristics of the unselected audio sources are indicated by a weakened second sound vector.

[0107] In this embodiment of the disclosure, vector enhancement of the first sound vector may include: multiplying the first sound vector by a first preset parameter;

[0108] Vector attenuation of the second sound vector may include dividing the second sound vector by a first preset parameter.

[0109] In this embodiment of the disclosure, since the first identifier is a preference identifier selected by the user, the first sound vector B1 can be enhanced.

[0110] In this embodiment of the disclosure, vector enhancement of the first sound vector B1 may include multiplying the first sound vector by a first preset parameter; the first preset parameter k is a positive integer greater than 1.

[0111] In this embodiment of the disclosure, the enhanced first sound vector B1 can be represented as: kB1=B1+.

[0112] In this embodiment of the disclosure, since the unselected identifier is a user-selected non-preference identifier, the second sound vector B2 can be attenuated.

[0113] In this embodiment of the disclosure, vector reduction of the second sound vector B2 may include dividing the second sound vector by a second preset parameter; the second preset parameter is a positive integer greater than 1, and the second preset parameter may be the same as or different from the first preset parameter.

[0114] In this embodiment of the disclosure, the weakened second sound vector B2 can be expressed as: B2 / k = B2-.

[0115] In this embodiment of the disclosure, the vector length of the enhanced sound vector is increased, and the vector length of the weakened sound vector is shortened. Therefore, the user's preference level can be determined based on the vector length of the sound vector. The longer the vector length, the higher the user's preference level for the sound source corresponding to the corresponding vector; the shorter the vector length, the lower the user's preference level for the sound source corresponding to the corresponding vector.

[0116] In this embodiment of the disclosure, the schematic diagrams of the enhanced and weakened sound vectors are as follows: Figure 5 As shown.

[0117] In this embodiment of the disclosure, if a user makes a preference selection for a certain audio multiple times (e.g., 3 times) (e.g., checking the box multiple times), the corresponding sound vector can be multiplied by the first preset parameter k multiple times; if a user makes a non-preference selection for a certain audio multiple times (e.g., crossing the box multiple times), the corresponding sound vector can be divided by the first preset parameter k multiple times. The sound vector calculated in this way can represent both the audio features and the user's degree of preference for it.

[0118] In this embodiment of the disclosure, the enhanced and weakened sound vectors can be saved to the preference database. The user's preferred sound source can be directly obtained from the preference database next time, avoiding the need for the user to select from the audio stream list every time.

[0119] In the embodiments disclosed herein, detailed implementation schemes for determining preference selection based on a preference database are described below.

[0120] In the embodiments disclosed herein, such as Figure 6 As shown, obtaining a preference selection for at least one audio file may include steps S31-S32:

[0121] S31. Determine the preferred target sound source based on the sound vector length corresponding to the sound source in the preset preference database.

[0122] In this embodiment of the disclosure, a preset preference database can pre-store a variety of sound sources and their corresponding audio features. The audio features can be represented by the sound vector length, and the selection can be based on these audio features when selecting a sound source preferred by the user.

[0123] In this embodiment of the disclosure, for example, the sound vector lengths of the stored sound sources can be compared, and the sound source with the longest sound vector length can be selected as the target sound source preferred by the user.

[0124] S32. Based on the similarity between the sound vector of the target sound source and the sound vector of the separated at least one audio source, select the audio corresponding to the target sound source from the audio corresponding to the sound source in the preference database as the preference selection.

[0125] In this embodiment of the disclosure, based on the target sound source selected by the above steps, it is possible to select from the separated audio whether there is audio that matches the target sound source. If there is, it is selected as a preferred choice.

[0126] In the embodiments disclosed herein, such as Figure 7 As shown, based on the similarity between the sound vector of the target sound source and the sound vector of the separated at least one audio source, the audio corresponding to the target sound source is selected from the audio corresponding to the sound source in the preference database as the preference selection, which may include steps S41-S44:

[0127] S41. Convert the separated audio into a third sound vector.

[0128] In this embodiment of the disclosure, the separated audio can be converted into corresponding sound vectors by a preset sound feature vector model and marked as the third sound vector B3.

[0129] S42. Determine the fourth sound vector corresponding to the audio features of the target sound source from the preference database.

[0130] In this embodiment of the disclosure, the fourth sound vector that identifies the target sound source from the preference database can be labeled as the fourth sound vector B4.

[0131] S43. Determine the similarity between the fourth sound vector and the third sound vector.

[0132] In this embodiment of the disclosure, the similarity calculation method between the fourth sound vector B4 and the third sound vector B3 may include, but is not limited to, cosine similarity calculation.

[0133] In this embodiment of the disclosure, the cosine similarity cosθ1 between the fourth sound vector B4 and the third sound vector B3 can be calculated using the following formula:

[0134]

[0135] S44. In response to the fact that the similarity between the fourth sound vector and the third sound vector corresponding to the fourth audio in the audio is greater than a preset similarity threshold, the fourth audio is confirmed as the preferred selection.

[0136] In this embodiment of the disclosure, the range of cosine similarity cosθ1 is [-1,1]. The higher the value of cosine similarity cosθ1, the more similar the audio features of the two audios are, and the greater the probability that they belong to the same sound or the same speaker (i.e. the same sound source).

[0137] In this embodiment, the similarity threshold can be predefined according to requirements, and no specific numerical value is limited here. For example, the similarity threshold can be set to 0.5, and if the cosine similarity cosθ1 is greater than 0.5, the two audio streams are considered to belong to the same sound source. In different application scenarios, the similarity threshold can be adjusted according to the actual situation.

[0138] In this embodiment of the disclosure, based on the above similarity judgment, if the features of the fourth sound vector B4 are similar to those of the third sound vector B3, the fourth audio can be confirmed as the preferred choice.

[0139] In this embodiment of the disclosure, if a preference selection cannot be matched from the separated audio through a preset user preference database, the preference selection can be determined again through the audio stream list.

[0140] S14. Based on preference selection, confirm the target audio from at least one audio.

[0141] In this embodiment of the disclosure, at least one target audio can be selected from at least one audio based on this preference.

[0142] In this embodiment of the disclosure, when a preference selection is determined, if the determined preference selection is an audio, then the audio is used as the target audio; if the determined preference selection is multiple audios, then all of the multiple audios can be used as the target audio, or one or more audios can be selected from the multiple audios as the target audio, or one or more audios can be selected as the target audio according to the frequency magnitude based on the historical selection frequency of the sound source corresponding to each audio in the multiple audios.

[0143] S15, Output the target sound corresponding to the target audio.

[0144] In this embodiment of the disclosure, before outputting the target sound, it can be first confirmed whether the target audio is the audio stream emitted by the sound source corresponding to the separated audio stream, so as to increase the accuracy of the output target sound.

[0145] In the embodiments disclosed herein, such as Figure 8 As shown, outputting the target sound corresponding to the target audio may include steps S51-S55:

[0146] S51. Convert the separated audio into a third sound vector.

[0147] In this embodiment of the disclosure, the separated audio can be converted into corresponding sound vectors by a preset sound feature vector model and marked as the third sound vector B3.

[0148] S52. Convert the target audio into a fifth sound vector.

[0149] In this embodiment of the disclosure, the target audio can also be converted into a corresponding sound vector through a preset sound feature vector model and marked as the fifth sound vector B5.

[0150] S53. Determine the similarity between the fifth sound vector and the third sound vector.

[0151] In this embodiment of the disclosure, the method for calculating the similarity between the fifth sound vector B5 and the third sound vector B3 may include, but is not limited to, cosine similarity calculation.

[0152] In this embodiment of the disclosure, the cosine similarity cosθ2 between the fifth sound vector B5 and the third sound vector B3 can be calculated using the following formula:

[0153]

[0154] S54. In response to the fact that the similarity between the fifth sound vector and the third sound vector corresponding to the third audio in the audio is greater than a preset similarity threshold, it is confirmed that the target audio and the third audio belong to the same sound source audio stream.

[0155] In this embodiment of the disclosure, the range of cosine similarity cosθ2 is [-1,1]. The higher the value of cosine similarity cosθ2, the more similar the audio stream features of the two audios are, and the greater the probability that they belong to the same sound or the same speaker (i.e., the same sound source).

[0156] In this embodiment, the similarity threshold can be predefined according to requirements, and no specific numerical value is limited here. For example, the similarity threshold can be set to 0.5, and two audio files are considered to belong to the same sound source if the cosine similarity (cosθ²) value is greater than 0.5. In different application scenarios, the similarity threshold can be adjusted according to the actual situation.

[0157] In this embodiment of the disclosure, based on the above similarity judgment, if the features of the fifth sound vector B5 and the third sound vector B3 are similar, it can be confirmed that the target audio and the third audio belong to the audio stream emitted by the same sound source.

[0158] S55. Take the sound in the third audio as the target sound and output it.

[0159] In this embodiment of the disclosure, by using the above similarity judgment, it can be determined that the target audio is indeed the audio stream collected by the terminal device, which ensures the accuracy of the sound output. Therefore, the sound in the third audio can be used as the target sound and output.

[0160] In this embodiment of the disclosure, before outputting the sound in the third audio as the target sound, the method may further include: reconfirming that the third audio is indeed the preferred selection.

[0161] In this disclosed embodiment, it is confirmed again that the third audio is indeed a preferred selection, including:

[0162] Calculate the length of the first vector of the third sound vector corresponding to the third audio;

[0163] Calculate the length of the second vector of the fifth sound vector;

[0164] Calculate the first ratio of the length of the second vector to the length of the first vector;

[0165] In response to the first ratio being greater than a preset first ratio threshold, it is confirmed that the third audio is indeed the preferred selection.

[0166] In this embodiment of the disclosure, if it is confirmed that the third audio is indeed the preferred selection, then it can be determined that the user does indeed prefer the sound source corresponding to the third audio.

[0167] In this embodiment of the disclosure, the method may further include: in response to a first ratio being less than or equal to a preset first ratio threshold, confirming that the third audio is not a preferred selection, that is, determining that the user does not prefer the sound source corresponding to the third audio.

[0168] In this embodiment of the disclosure, the first ratio of the length of the second vector to the length of the first vector can be calculated using the following formula:

[0169] In this embodiment of the disclosure, the value of the first ratio is α or 1 / α (α is a positive integer greater than 1), and the threshold of the first ratio can be set to, but is not limited to, 1. Typically, the value of α is k.

[0170] In this embodiment of the disclosure, if the value of the first ratio is greater than 1, that is, the value of the first ratio is α, it can be determined that the second vector length is the enhanced vector length, indicating that the user does indeed prefer the sound source corresponding to the third audio. Therefore, the audio processing module can directly send the third audio or increase the gain of the third audio before sending it.

[0171] In this embodiment of the disclosure, if the value of the first ratio is less than 1, that is, the value of the first ratio is 1 / α, it means that the user does not prefer the sound source corresponding to the third audio and the user does not want to send the sound of the third audio. At this time, the third audio can be discarded or the gain of the third audio can be reduced before sending. The shorter the length of the first vector, the greater the gain reduction. When the length of the first vector is short to a certain extent (for example, 1 / α is less than 0.2), the third audio is discarded.

[0172] In this embodiment of the disclosure, by selecting user preferences, the voices of the first near-end user A and the second near-end user B can be accurately transmitted in the aforementioned embodiments, eliminating the voices of interfering users C and D as well as the echo of far-end voices, thus achieving the effect of near-end noise reduction.

[0173] In this embodiment of the disclosure, the method may further include:

[0174] When an unknown audio stream is triggered in response to an unknown audio stream being displayed in the audio stream list, play the unknown audio.

[0175] In this embodiment of the disclosure, if an unknown audio stream exists in the audio stream list, the sound of the unknown audio can be played when the unknown audio is detected to be triggered. The user can determine whether the sound of the unknown audio needs to be output based on the playback content.

[0176] In this embodiment of the disclosure, when playing unknown audio, a portion of the unknown audio can be played, or all of the unknown audio can be played. During playback, a portion or all of the sound content of the unknown audio can be displayed. For example, only a portion of the key sound content of the unknown audio can be displayed.

[0177] In this embodiment of the disclosure, the method may further include: after identifying audio in the audio stream list, in response to the identified audio being triggered, playing the sound corresponding to the triggered identified audio, which also allows the user to determine whether the sound of the audio needs to be output based on the playback content.

[0178] In this embodiment of the disclosure, each audio file can be input into a pre-trained audio processing module for processing, so that the audio processing module can determine the target sound based on the independent audio file and output it.

[0179] In this disclosed embodiment, AI (Artificial Intelligence) is a known technology and discipline that simulates human intelligence. It enables computer systems to perceive, understand, learn, reason, and make decisions by simulating, expanding, and extending human intelligent capabilities. Large-scale models refer to the techniques used in the field of AI for training and inference using large-scale neural network models. All the models mentioned in the foregoing solutions can be implemented using large-scale models.

[0180] In this embodiment of the disclosure, the large model may include, but is not limited to, the Wav2Vec model. The Wav2Vec model is trained by a self-supervised learning objective and can automatically learn the implicit speech feature representation from unlabeled speech and directly extract speech features from the original waveform.

[0181] In this embodiment of the disclosure, in addition to the aforementioned call scenario, this solution can also be used in other scenarios, such as human-computer interaction scenarios. For example, when a user chats with an AI voice assistant, if the user says "recite a Tang poem," the AI ​​voice assistant will most likely recite "Quiet Night Thoughts." If the user listens to it too many times and doesn't want to listen anymore or doesn't like the poem, they have to say "change one" each time, which is quite troublesome. Moreover, after changing several poems, it may return to "Quiet Night Thoughts." After the current chat ends, the above process will be repeated in the next chat, resulting in a poor user experience. According to the solution of this embodiment of the disclosure, when the user doesn't like listening to "Quiet Night Thoughts," they can say "I don't like 'Quiet Night Thoughts.'" After the AI ​​voice assistant understands the meaning of this sentence through a large model, it can convert the text or audio stream corresponding to "Quiet Night Thoughts" into a vector, shorten the length of the vector, and save it to the user preference database. The next time the AI ​​voice assistant generates "Quiet Night Thoughts," it can add a prompt during the thinking process asking "Does the user like this poem?" It will then search the user preference database. If a similar vector is found and its length has shortened, it means the user doesn't like the poem. In this way, it can generate another Tang poem. This avoids the user hearing content they don't like and avoids the hassle of repeatedly saying "Change to another poem." This makes the AI ​​voice assistant increasingly "understand" the user and improves the user experience.

[0182] This disclosure embodiment includes at least the following advantages:

[0183] 1. The solution implemented in this application can process sound according to user preferences, making the final audio effect more in line with each user's different preferences and improving the user experience.

[0184] 2. The solution of this application embodiment can perform near-end noise reduction on received near-end sound and far-end noise reduction on received far-end sound, so that even if the other party's terminal voice processing capability is relatively weak, it can still be processed well in the terminal of the party, thus ensuring the overall call effect.

[0185] This disclosure also provides an electronic device 100, such as... Figure 9 As shown, it includes:

[0186] One or more processors 101;

[0187] The memory 102 stores one or more programs that, when executed by the one or more processors, cause the one or more processors 101 to implement the audio processing method.

[0188] One or more input / output (I / O) interfaces 103 are connected between the processor 101 and the memory 102 and configured to enable information interaction between the processor 101 and the memory 102.

[0189] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the audio processing method described above.

[0190] This disclosure also provides a computer program product, which includes a computer program that, when executed by a processor, implements the audio processing method.

[0191] In the embodiments disclosed herein, any of the foregoing audio processing methods are applicable to the embodiments of the electronic device, storage medium, and program product, and will not be described in detail here.

[0192] Those skilled in the art will understand that all or some of the functional modules / units disclosed above can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0193] In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be executed by several physical components working together.

[0194] Some or all of the physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technique for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only optical disc (CD-ROM), digital versatile disc (DVD) or other optical disc storage; magnetic cartridges, magnetic tapes, disk storage or other magnetic storage; and any other media that can be used to store desired information and can be accessed by a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0195] This disclosure has disclosed exemplary embodiments, and although specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. An audio processing method, comprising: Get the initial audio stream; The initial audio stream includes the audio stream corresponding to the near-end sound and / or the audio stream corresponding to the far-end sound; The initial audio stream is subjected to audio feature recognition to separate at least one audio element; Obtain the preference selection for the at least one audio; Based on the preference selection, the target audio is confirmed from the at least one audio; Output the target sound corresponding to the target audio.

2. The audio processing method according to claim 1, wherein, The step of obtaining preference selections for the at least one audio file includes: In response to the triggering of preset display conditions, an audio stream list generated based on some or all of the audio from the at least one audio source is displayed; In response to receiving a selection operation on the audio stream list, the preference selection is obtained.

3. The audio processing method according to claim 2, wherein, The display conditions include: A preset request was received; or, The separated at least one audio signal can be multiple; or, Confirm that the terminal has entered the preset environment.

4. The audio processing method according to claim 2, wherein, The display of the audio stream list generated based on part or all of the audio from the at least one audio source includes: Identify the audio features of each audio element in the partial or complete audio; In response to identifying the audio features of any first audio among the various audios, an identifier corresponding to the first audio is determined; The identifier corresponding to the first audio is displayed in the audio stream list, and the identifier can indicate the audio characteristics corresponding to the first audio.

5. The audio processing method according to claim 4, further comprising: The audio corresponding to the first identifier selected in the audio stream list is converted into a first sound vector; Vector enhancement is applied to the first sound vector; The sound source corresponding to the first identifier is stored in a preset preference database, and the enhanced first sound vector is used to indicate the audio features of the preferred sound source. And / or, Convert the unselected audio in the audio stream list into a second sound vector; Vector attenuation is applied to the second sound vector; The sound sources of the unselected audio are stored in the preference database, and the attenuated third sound vector is used to indicate the audio features of the unselected sound sources.

6. The audio processing method according to claim 2, wherein, The display of the audio stream list generated based on part or all of the audio from the at least one audio source includes: Identify the audio features of each audio element in the partial or complete audio; In response to the failure to identify any audio characteristics of the second audio among the various audio streams, the second audio is displayed as an unknown audio in the audio stream list.

7. The audio processing method according to claim 6, wherein, The method further includes: In response to the unknown audio displayed in the audio stream list being triggered, the unknown audio is played.

8. The audio processing method according to claim 1, wherein, The step of obtaining preference selections for the at least one audio file includes: The preferred target sound source is determined based on the length of the sound vector corresponding to the sound source in the preset preference database. Based on the similarity between the sound vector of the target sound source and the sound vector of the at least one separated audio, the audio corresponding to the target sound source is selected from the audio corresponding to the sound source in the preference database as the preference selection.

9. The audio processing method according to claim 1, wherein, The step of outputting the target sound corresponding to the target audio includes: The separated audio at least one audio value is converted into a third sound vector; The target audio is converted into a fifth sound vector; Determine the similarity between the fifth sound vector and the third sound vector; In response to the fact that the similarity between the fifth sound vector and the third sound vector corresponding to the third audio in the at least one audio is greater than a preset similarity threshold, it is confirmed that the target audio and the third audio belong to the same audio stream. The sound of the third audio is used as the target sound and output.

10. An electronic device, comprising: One or more processors; A memory having stored one or more programs thereon, which, when executed by one or more processors, cause the one or more processors to implement the audio processing method according to any one of claims 1-9; One or more input / output (I / O) interfaces are connected between the processor and the memory and configured to enable information exchange between the processor and the memory.

11. A computer program product comprising a computer program that, when executed by a processor, implements the audio processing method according to any one of claims 1-9.

Images