Method and system for modifying audio content for listener

Abstract

A method for modifying audio content for a listener is disclosed. The method includes determining an explicit emotion value defining an audio object emotion for each audio object. The method includes determining component factors representing one or more basic emotions in the explicit emotion values for each audio object. The method includes calculating a probability that a listener is associated with each of one or more basic emotions. The method includes calculating a priority value associated with each audio object based on a probability that the listener is associated with each of the one or more basic emotions represented in the component factors of each audio object and the component factors of each audio object. The method also includes generating a list. The method further includes modifying the audio content by adjusting a gain associated with at least one of the plurality of audio objects in the list.

Description

Technical Field

[0001] The present disclosure relates generally to modifying audio content, and particularly to modifying audio content based on listener preferences. Background Art

[0002] Sometimes when viewing content, users prefer to hear some parts of the audio at a higher volume, while other parts are heard at a lower volume. In addition, in the available content, users may like or dislike certain media objects.

[0003] Currently, a large number of multimedia devices (such as TVs and sound bars) are using object-based media transmission and rendering technology. Compared with channel-based systems, object-based media communication provides greater flexibility. For each multimedia scene, audio objects and video objects can be analyzed and encoded in a special way to provide a better user experience.

[0004] In addition, there are some technologies in the market that can be used to manage audio for a better user experience. These technologies include source separation and emotion-based processing.

[0005] Source separation is the technique of separating audio into its individual components. There are many existing techniques for source separation that mainly work based on the UNET architecture model.

[0006] Emotion-based processing greatly improves the personalization of the technology by making the features more emotion-oriented. Existing mature solutions for emotion detection via audio and video (combined or separate) exist using CNNs, which utilize objective audio / video features to detect the emotions contained therein.

[0007] However, there are some limitations, such as the personalization of audio objects is not prominent. Existing solutions in the field mainly focus on modifying the entire audio part. As an example: in content specific to children, content marked as adult is completely muted or frames are completely removed. There is no technology that automatically enhances, reduces or mutes specific audio objects considering the user's emotional profile.

[0008] The prior art does not include methods for detecting the listener's preferences and modifying the audio based on the preferences. In addition, the prior art does not include methods for selectively modifying some parts of the audio for their effects on the audio.

[0009] A solution is needed to overcome the above disadvantages. Summary of the invention

[0010] Technical Solution

[0011] This summary is provided to introduce in a brief form some of the concepts that are further described in the detailed description of the present disclosure. This summary is not intended to identify key or essential inventive concepts of the claimed subject matter, nor is it intended to determine the scope of the claimed subject matter. In accordance with the purposes of the present disclosure, the present disclosure as embodied and broadly described herein describes methods and systems for modifying audio content for a listener.

[0012] According to some example embodiments of the inventive concept, a method for modifying audio content for a listener is disclosed. The method includes determining an explicit emotion value defining an audio object emotion for each of a plurality of audio objects associated with the audio content. The method includes determining a component factor representing one or more basic emotions in the explicit emotion value of each audio object in a plurality of basic emotions. The method includes calculating a probability that a listener is associated with each of the one or more basic emotions represented in the component factor. The method includes calculating a priority value associated with each audio object based on the probability that a listener is associated with each of the one or more basic emotions represented in the component factor of each audio object and the component factor of each audio object. The method also includes generating a list of a plurality of audio objects arranged in a specified order with respect to the priority value associated with each of the plurality of audio objects. The method also includes modifying the audio content by adjusting a gain associated with at least one of the plurality of audio objects in the list.

[0013] According to some example embodiments of the inventive concept, a system for modifying audio content is disclosed. The system includes: an explicit emotion value determination engine configured to determine an explicit emotion value defining an audio object emotion of each audio object in a plurality of audio objects associated with the audio content. The system includes: an adaptation component factor determination engine configured to determine a component factor representing one or more basic emotions in the explicit emotion value of each audio object in a plurality of basic emotions. The system includes: an audio object modification engine configured to calculate a probability that a listener is associated with each of the one or more basic emotions represented in the component factor. The audio object modification engine is configured to calculate a priority value associated with each audio object based on the probability that the listener is associated with each of the one or more basic emotions represented in the component factor of each audio object and the component factor of each audio object. The audio object modification engine is also configured to generate a list including a plurality of audio objects arranged in a specified order with respect to the priority value associated with each of the plurality of audio objects. The audio object modification engine is also configured to modify the audio content by adjusting a gain associated with at least one audio object in the plurality of audio objects in the list.

[0014] These aspects and advantages will become more clearly understood from the following detailed description and claims taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A flow chart describing a method for modifying audio content according to an embodiment of the present disclosure is shown;

[0016] Figure 2 A schematic block diagram of a system for modifying audio content according to an embodiment of the present disclosure is shown;

[0017] Figure 3 An operational flow chart describing a process for modifying audio content according to an embodiment of the present disclosure is shown;

[0018] Figure 4 An architectural diagram describing a method for modifying audio content according to an embodiment of the present disclosure is shown;

[0019] Figure 5a An operational flow chart describing a process for generating a plurality of audio objects according to an embodiment of the present disclosure is shown;

[0020] Figure 5b A diagram depicting a U-Net source separation model according to an embodiment of the present disclosure is shown;

[0021] Figure 5c shows a graphical representation of memory usage by a U-Net source separation model according to an embodiment of the present disclosure;

[0022] Figure 5d A diagram is shown describing a plurality of audio objects in generated audio content according to an embodiment of the present disclosure;

[0023] Figure 6a An operational flow chart describing a process for determining emotion levels associated with a plurality of audio objects according to an embodiment of the present disclosure is shown;

[0024] Figure 6b A diagram depicting determination of emotion levels associated with a plurality of audio objects according to an embodiment of the present disclosure is shown;

[0025] Figure 7a An operational flow chart describing a process for determining an explicit emotion value associated with each audio object of audio content according to an embodiment of the present disclosure is shown;

[0026] Figure 7b shows a common scale according to an embodiment of the present disclosure;

[0027] Figure 7cA common scale according to an embodiment of the present disclosure is shown, wherein audio object emotions are mapped to a fixed preset range on the common scale;

[0028] Figure 7d A diagram describing determination of an explicit emotion value according to an embodiment of the present disclosure is shown;

[0029] Figure 8a An operational flow chart describing a process for determining component factors according to an embodiment of the present disclosure is shown;

[0030] Figure 8b shows a nuclear scale according to an embodiment of the present disclosure;

[0031] Figure 8c shows a modified nuclear scale based on feedback from a listener according to an embodiment of the present disclosure;

[0032] Figure 8d An embodiment of a nuclear scale describing the location of explicit emotions on the nuclear scale is shown according to an embodiment of the present disclosure;

[0033] Figure 8e shows a diagram depicting component factors as output based on listener feedback and explicit emotion values ​​for each audio object according to an embodiment of the present disclosure;

[0034] Figure 8f A graphical representation depicting the height of at least one adapted emotion kernel according to an embodiment of the present disclosure is shown.

[0035] Figure 9a An operational flow chart describing a process for audio object prioritization and gain adjustment according to an embodiment of the present disclosure is shown;

[0036] Figure 9b A diagram depicting audio object prioritization and gain adjustment for generating modified audio content according to an embodiment of the present disclosure is shown;

[0037] Fig.10 An architectural diagram showing a method for modifying audio content including another number of basic emotions according to an embodiment of the present disclosure;

[0038] Fig.11 A use case diagram describing a scenario for modifying audio content by enhancing a singer's voice according to an embodiment of the present disclosure is shown;

[0039] Fig.12 A use case diagram is shown describing a scenario in which a listener cannot modify audio content according to the prior art and a scenario in which a listener modifies audio content according to an embodiment of the present disclosure;

[0040] Fig.13A use case diagram is shown describing a scenario in which a listener modifies audio content by managing one or more audio objects according to an embodiment of the present disclosure;

[0041] Fig.14 A use case diagram is shown describing a scenario in which a listener controls one or more audio objects of audio content according to an embodiment of the present disclosure;

[0042] Fig.15 A use case diagram describing a scenario in which a listener enhances a human voice and suppresses BGM from audio content according to an embodiment of the present disclosure is shown;

[0043] Fig.16 A use case diagram is shown describing a scenario of enhancement of a music portion in audio content according to an embodiment of the present disclosure;

[0044] Fig.17 A use case diagram is shown that describes a scenario in which audio content can be personalized based on emotions associated with the audio content according to an embodiment of the present disclosure;

[0045] Fig.18 A use case diagram is shown describing a scenario of automatic enhancement of vocals / beats in audio content according to an embodiment of the present disclosure.

[0046] Furthermore, those skilled in the art will appreciate that the elements in the drawings are illustrated for simplicity and may not necessarily be drawn to scale. For example, a flow chart illustrates a system in terms of the most important steps involved to help improve understanding of the various aspects of the present invention. Furthermore, with respect to the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may only show those specific details relevant to understanding the embodiments of the present invention so as not to obscure the drawings with details that would be readily apparent to one of ordinary skill in the art having the benefit of the description herein. DETAILED DESCRIPTION

[0047] To promote an understanding of the principles of the invention, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the embodiments. It will be understood, however, that no limitation of the scope of the invention is intended thereby and such changes and further modifications in the illustrated systems and such further applications of the principles of the invention as illustrated therein are deemed to be within the ordinary scope of those skilled in the art to which the invention relates.

[0048] Those skilled in the art will understand that both the foregoing general description and the following detailed description are illustrative of the present invention and are not restrictive of the present invention.

[0049] References throughout this specification to "on one hand," "on the other hand," or similar language mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the phrases "in an embodiment," "in another embodiment," and similar language appearing throughout this specification may, but do not necessarily, all refer to the same embodiment.

[0050] The terms "includes", "comprising" or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process or system that includes a list of steps includes not only those steps but may also include other steps not expressly listed or inherent to such process or system. Similarly, without more constraints, one or more devices or subsystems or elements or structures or components following "includes" does not exclude the presence of other devices or other subsystems or other elements or other structures or other components or additional devices or additional subsystems or additional elements or additional structures or additional components.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention belongs.The systems, systems, and examples provided herein are illustrative only and not limiting.

[0052] Embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0053] Figure 1 A flow chart describing a method for modifying audio content according to an embodiment of the present disclosure is shown. Figure 1 In an embodiment, the audio content may be modified based on one or more preferences of a listener listening to the audio content. Examples of audio content may include, but are not limited to, songs, speeches, narration, and live broadcasts of events. In an embodiment, the audio content may be acquired from a video for modification. In an embodiment, the modification of the audio content may include an effect that enhances or reduces at least one aspect of the audio content. In an embodiment, at least one aspect may include background speech, a tune played with the audio content, background noise, etc.

[0054] According to an embodiment of the present disclosure, at step 102 , method 100 includes determining an explicit emotion value defining an audio object emotion for each audio object of a plurality of audio objects associated with audio content.

[0055] Further, at step 104 , the method 100 includes determining component factors representing one or more basic emotions in the explicit emotion value of each audio object in the plurality of basic emotions.

[0056] At step 106 , method 100 includes calculating a probability that the listener is associated with each of the one or more basic emotions represented in the component factors.

[0057] At step 108 , the method 100 proceeds to calculate a priority value associated with each audio object based on the component factor of each audio object and a probability that the listener associates with each of the one or more basic emotions represented in the component factor of each audio object.

[0058] At step 110 , the method 100 proceeds to generate a list including a plurality of audio objects arranged in a specified order with respect to a priority value associated with each of the plurality of audio objects.

[0059] At step 112 , method 100 includes modifying the audio content by adjusting a gain associated with at least one audio object of the plurality of audio objects in the list.

[0060] Figure 2 2 is a schematic block diagram of a system 202 for modifying audio content according to an embodiment of the present disclosure. Figure 2 In an embodiment, system 202 may be incorporated into a user equipment (UE). Examples of UE may include, but are not limited to, televisions (TVs), laptop computers, tablet computers, smart phones, and personal computers (PCs). Examples of audio content may include, but are not limited to, live broadcasts of songs, speeches, narrations, and events. In an embodiment, audio content may be obtained from a video for modification. In an embodiment, the modification may be based on separating the audio content into a plurality of audio objects and changing the amplitude of at least one audio object in the audio content. In an embodiment, changing the amplitude may include adjusting a gain associated with at least one audio object. In an embodiment, adjusting the gain may result in one or more of: reducing the amplitude of at least one audio object, increasing the amplitude of at least one audio object, and removing at least one audio object from the audio content. In an embodiment, the modification may be based on one or more preferences of the listener of the audio.

[0061] System 202 may include a processor 204, a memory 206, data 208, a module 210, a resource 212, a display unit 214, a receiving engine 216, an audio object recognition engine 218, an emotion level determination engine 220, an explicit emotion value determination engine 222, an adaptation component factor determination engine 224, and an audio object modification engine 226.

[0062] In an embodiment, the processor 204, the memory 206, the data 208, the module 210, the resource 212, the display unit 214, the receiving engine 216, the audio object recognition engine 218, the emotion level determination engine 220, the explicit emotion value determination engine 222, the adaptation component factor determination engine 224 and the audio object modification engine 226 can be electrically and / or physically connected to each other.

[0063] As will be appreciated, the system 202 may be understood as one or more of hardware, software, logic-based programs, configurable hardware, and the like. In an example, the processor 204 may be a single processing unit or multiple units, all of which may include multiple computing units. The processor 204 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, processor cores, multi-core processors, multiprocessors, state machines, logic circuits, application specific integrated circuits, field programmable gate arrays, and / or any device that manipulates signals based on operational instructions. Among other capabilities, the processor 204 may be configured to retrieve and / or execute computer-readable instructions and / or data stored in the memory 206.

[0064] In an example, the memory 206 may include any non-transitory computer-readable medium known in the art, including, for example, volatile memory (such as static random access memory (SRAM) and / or dynamic random access memory (DRAM)) and / or non-volatile memory (such as read-only memory (ROM), erasable programmable ROM (EPROM), flash memory, hard disk, optical disk, and / or magnetic tape). The memory 206 may include data 208. Among other aspects, the data 208 is used as a repository for storing data processed, received, and generated by one or more of the processor 204, the memory 206, the data 208, the module 210, the resource 212, the display unit 214, the receiving engine 216, the audio object recognition engine 218, the emotion level determination engine 220, the explicit emotion value determination engine 222, the adaptation component factor determination engine 224, and the audio object modification engine 226.

[0065] Among other things, the module 210 may include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement data types. The module 210 may also be implemented as a signal processor, a state machine, a logic circuit, and / or any other device or component that manipulates signals based on operational instructions.

[0066] In addition, module 210 can be implemented in hardware as instructions executed by at least one processing unit (e.g., processor 204) or by a combination thereof. The processing unit can be a general-purpose processor that executes instructions to cause the general-purpose processor to perform operations, or the processing unit can be dedicated to performing the desired functions. In another aspect of the present disclosure, module 210 can be machine-readable instructions (software) that, when executed by a processor / processing unit, can perform any of the described functions.

[0067] In some example embodiments, module 210 may be machine-readable instructions (software) that, when executed by processor 204 / processing unit, perform any of the described functions.

[0068] The resources 212 may be physical and / or virtual components of the system 202 that provide inherent capabilities and / or contribute to the performance of the system 202. Examples of the resources 212 may include, but are not limited to, memory (e.g., memory 206), a power supply unit (e.g., a battery), a display unit (e.g., display unit 214), etc. In addition to the processor 204 and the memory 206, the resources 212 may also include a power supply unit / battery unit, a network unit, etc.

[0069] The display unit 214 may display various types of information (e.g., media content, multimedia data, text data, etc.) to the system 202. The display unit 214 may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, a plasma cell display, an electronic ink array display, an electronic paper display, a flexible LCD, a flexible electrochromic display, and / or a flexible electrowetting display.

[0070] In an example, the receiving engine 216, the audio object recognition engine 218, the emotion level determination engine 220, the explicit emotion value determination engine 222, the adaptation component factor determination engine 224, and the audio object modification engine 226 include, among other things, routines, programs, objects, components, data structures, etc. that perform specific tasks or implement data types. The receiving engine 216, the audio object recognition engine 218, the emotion level determination engine 220, the explicit emotion value determination engine 222, the adaptation component factor determination engine 224, and the audio object modification engine 226 can also be implemented as a signal processor, a state machine, a logic circuit, and / or any other device or component that manipulates a signal based on an operating instruction. In addition, the receiving engine 216, the audio object recognition engine 218, the emotion level determination engine 220, the explicit emotion value determination engine 222, the adaptation component factor determination engine 224, and the audio object modification engine 226 can be implemented in hardware, instructions executed by a processing unit, or a combination thereof. The processing unit may include a computer, a processor (such as the processor 204), a state machine, a logic array, or any other suitable device capable of processing instructions. The processing unit may be a general purpose processor which executes instructions to cause the general purpose processor to perform the required tasks, or the processing unit may be dedicated to performing the required functions.

[0071] In an embodiment, the receiving engine 216 may be configured to receive audio content. In an embodiment, the receiving engine 216 may be configured to receive audio content as input. In an embodiment, the receiving engine 216 may be configured to receive a video and obtain audio content from the video by processing the video.

[0072] Continuing with the above embodiment, the audio object recognition engine 218 can be configured to separate the audio content into multiple audio objects. In an embodiment, the audio object recognition engine 218 can be configured to separate the audio content by preprocessing the input to generate preprocessed audio content. In addition, when preprocessing, the audio object recognition engine 218 can be configured to feed the preprocessed audio content to a U-Net-based source separation model to generate multiple outputs. Next, the audio object recognition engine 218 can be configured to perform post-processing on the multiple outputs to generate multiple audio objects associated with the audio content from the audio content.

[0073] In response to the audio object recognition engine 218 separating the audio content into a plurality of audio objects, the emotion level determination engine 220 may be configured to determine an audio object emotion level associated with each of the plurality of audio objects. Examples of audio object emotions may include, but are not limited to, admiration, awe, appreciation, amusement, anger, anxiety, awe, embarrassment, boredom, calm, confusion, desire, disgust, and empathy pain. In an embodiment, the emotion level determination engine 220 determining the audio object emotion level of each audio object may include determining one or more audio features associated with each audio object.

[0074] In an embodiment, the one or more audio features may include a fundamental frequency associated with each audio object, a time-varying characteristic of the frequency, a root mean square (RMS) value associated with the amplitude, and a speech speed. In response to determining the one or more audio features, the emotion level determination engine 220 may be configured to determine an emotion probability value associated with each audio object based on the one or more audio features. Continuing with the above embodiment, the emotion level determination engine 220 may be configured to determine an audio object emotion level associated with each audio object based on the emotion probability value.

[0075] Continuing with the above embodiment, after separating the audio content into multiple audio objects and determining the audio object emotion level of each of the multiple audio objects, the explicit emotion value determination engine 222 can be configured to determine the explicit emotion value of each audio object. In an embodiment, the explicit emotion value can define the audio object emotion level of each of the multiple audio objects related to the audio content. In an embodiment, in order to determine the explicit emotion value, the explicit emotion value determination engine 222 can be configured to map the audio object emotion level of each audio object on a common scale to determine the range of the audio object emotion in each audio object. In an embodiment, the common scale can include multiple basic emotions. An example of the number of basic emotions can be 5, and the basic emotions can include extremely sad emotions, sad emotions, neutral emotions, happy emotions, and extremely happy emotions. In an embodiment, the number of basic emotions may not be limited thereto, and the number of basic emotions may vary between 4 and 27. In addition, the common scale may be one of a pleasure scale and an arousal scale including multiple basic emotions.

[0076] To this end, it is understood that in response to determining the range, the explicit emotion value determination engine 222 can be configured to determine a bias for the audio object emotion level of each audio object. In an embodiment, the bias can be the minimum value of the range determined as above. In addition, the explicit emotion value determination engine 222 can be configured to add the audio object emotion level associated with each audio object mapped on the common scale to the bias to determine the explicit emotion value of each audio object.

[0077] Continuing with the above embodiment, after determining the explicit emotion value of each audio object related to the audio content, the adaptation component factor determination engine 224 can be configured to determine the component factors of one or more basic emotions represented in the explicit emotion value of each audio object. In an embodiment, the one or more basic emotions can be in a plurality of basic emotions.

[0078] For this reason, it is understood that in order to determine the component factors representing one or more basic emotions, the adaptation component factor determination engine 224 can be configured to map the explicit emotion value of each audio object on the nuclear scale. In an embodiment, the nuclear scale can include multiple adaptive emotion cores representing multiple basic emotions. In an embodiment, the component factors representing one or more basic emotions can be based on the contribution of one or more basic emotions represented by one or more adaptive emotion cores in the explicit emotion value of each audio object. In an embodiment, the contribution of one or more basic emotions can be determined based on the position of the explicit emotion value of each audio object on one or more adaptive emotion cores when mapping.

[0079] Subsequently in an embodiment, the adaptation component factor determination engine 224 may be configured to adjust the size of at least one of the multiple adaptation emotion cores. In an embodiment, the size may be adjusted based on multiple feedback parameters related to the listener. Examples of multiple feedback parameters may include, but are not limited to, visual feedback, sensor feedback, previous feedback, and manual feedback related to the listener. In an embodiment, the adaptation component factor determination engine 224 may be configured to obtain multiple feedback parameters from at least one of the memory 206 or the listener in real time. In an embodiment, multiple feedback parameters may be pre-stored in the memory 206. In an embodiment, an interface on the UE may be presented to the listener to share multiple feedback parameters with the system 202.

[0080] For this reason, it is understood that after obtaining multiple feedback parameters, the adaptation component factor determination engine 224 can be configured to adjust the size of at least one adaptation emotion core. In an embodiment, resizing can include increasing or decreasing one or more parameters associated with the shape of at least one adaptation emotion core. Examples of one or more parameters can include, but are not limited to, the slope, height, length, width, radius, and angle of at least one adaptation emotion core.

[0081] In an embodiment, in the case where at least one adapted emotion core is determined to be a trapezoidal shape, adjusting the size may include increasing or decreasing one or more of the slope and height of at least one adapted emotion core. In an embodiment, in the case where at least one adapted emotion core is determined to be a rectangular shape, adjusting the size may include increasing or decreasing one or more of the length and width of at least one adapted emotion core. In an embodiment, in the case where at least one adapted emotion core is determined to be a circular shape, adjusting the size may include increasing or decreasing the radius of at least one adapted emotion core. In an embodiment, in the case where at least one adapted emotion core is determined to be a triangular shape, adjusting the size may include increasing or decreasing one of the angle and height of at least one adapted emotion core.

[0082] In an embodiment, increasing the size of at least one adapted emotion core may indicate that the listener's emotion is similar to at least one basic emotion represented by the at least one adapted emotion core. In addition, in an embodiment, decreasing the size of at least one adapted emotion core may indicate that the listener's emotion is not similar to at least one basic emotion represented by the at least one adapted emotion core.

[0083] After determining the component factors representing one or more basic emotions, the audio object modification engine 226 can be configured to calculate the probability that the listener is associated with each of the one or more basic emotions represented in the component factors. In an embodiment, the probability can be calculated by the audio object modification engine 226 based on one of the following items: a plurality of feedback parameters related to the listener; and a ratio of the area of ​​one or more adapted emotion cores corresponding to each basic emotion represented in the component factors to the total area of ​​multiple adapted emotion cores for the multiple basic emotions.

[0084] Continuing with the above embodiment, after calculating the probability, the audio object modification engine 226 can be configured to calculate the priority value associated with each audio object. In an embodiment, the priority value can be determined based on the probability that the listener is associated with each of the one or more basic emotions represented in the component factor of each audio object and the component factor representing the one or more basic emotions. In an embodiment, the audio object modification engine 226 can be configured to calculate the priority value by performing a weighted summation of the probability that the listener is associated with each basic emotion represented in the component factor and the component factor representing the one or more basic emotions. Next, the audio object modification engine 226 can be configured to generate a list of multiple audio objects arranged in a specified order with respect to the priority value associated with each of the multiple audio objects.

[0085] Continuing with the above embodiment, the audio object modification engine 226 may be configured to modify the audio content by adjusting a gain associated with at least one of the plurality of audio objects in the list. In an embodiment, the audio object modification engine 226 may be configured to perform one or more of a plurality of steps. In an embodiment, the plurality of steps may include:

[0086] A gain of 1 is assigned to the audio object corresponding to the highest priority value in the list, and a gain of 0 is assigned to another audio object corresponding to the lowest priority value in the list. In an embodiment, assigning a gain of 0 may indicate removing the another audio object from the audio content.

[0087] A non-zero gain is assigned to the audio object corresponding to the lowest priority value, and a gain of 1 is assigned to the audio object corresponding to the highest priority value. In an embodiment, assigning a non-zero gain may indicate the effect of changing the audio object. In an embodiment, assigning a gain of a value less than 1 and greater than 0 may have the effect of making the audio object sound smaller.

[0088] Continuing with the above embodiment, after performing one or more of the multiple steps, the audio object modification engine 226 can be configured to calculate the gain associated with one or more audio objects in the list except the audio object with the highest priority value and another audio object with the lowest priority value. In an embodiment, the gain associated with the one or more audio objects can be calculated based on the gain associated with the audio object with a priority value higher than the priority value of the one or more audio objects and the gain associated with the audio object with a priority value lower than the priority value of the one or more audio objects. In an embodiment, the audio object modification engine 226 can be configured to distribute the gain of the audio objects between the audio object with the highest priority value and the audio object with the lowest priority value in the list in order of priority to a value between the highest priority value and the lowest priority value. Next, the audio object modification engine 226 can be configured to perform a weighted summation of the gain associated with each audio object in the list to modify the audio content. After modifying at least one audio object, the audio object modification engine 226 can be configured to combine a plurality of modified audio objects to generate modified audio content.

[0089] Figure 3 An operational flow chart describing a process for modifying audio content according to an embodiment of the present disclosure is shown. Figure 3 , examples of audio content may include, but are not limited to, songs, speeches, narration, and live broadcasts of events. In an embodiment, the audio content may be obtained from a video for modification. In an embodiment, the modification may be based on separating the audio content into multiple audio objects and changing the amplitude of at least one audio object in the audio content. In an embodiment, changing the amplitude may include adjusting a gain associated with at least one audio object. In an embodiment, adjusting the gain may result in one or more of: reducing the amplitude of at least one audio object, increasing the amplitude of at least one audio object, and removing at least one audio object from the audio content. In an embodiment, the modification may be based on one or more preferences of a listener of the audio.

[0090] Continuing with the above example, at step 302 , the process may include receiving audio content as input.

[0091] At step 304, the process may include performing audio object recognition for the audio content. The audio object recognition may include separating the audio content into "N" audio objects using a "lite" source separation technique. In an embodiment, the "lite" source separation technique may refer to a source separation technique supported by the UE. In an embodiment, the process may include identifying audio emitting objects in the audio / video content. In an embodiment, the audio emitting objects of a particular audio / video content may be a person, a drum, and a car horn. In an embodiment, the "N" audio objects may be, for example, Figure 1 and Figure 2 In an embodiment, the separation may be performed as in Figure 2 The audio object recognition engine 218 mentioned in the above is executed. In an embodiment, the "lite" source separation technology can be used to separate the audio content to identify the various audio objects present in the input, such as human voice, background music, etc.

[0092] At step 306, the process may include performing an emotion level determination to determine an audio object emotion level and an audio object emotion associated with each of the plurality of audio objects. In an embodiment, the audio object emotion may also be referred to interchangeably as emotion. In an embodiment, the emotion level may be determined using an audio object emotion level such as in Figure 2 The emotion level determination engine 220 mentioned in the embodiment determines the audio object emotion level. In an embodiment, each audio object may include the required information as follows: (person, funny, 7); (drum, cheerful, 5); (car horn, angry, 2). In an embodiment, the audio object emotion level may be a factor between 0 and 10, where 10 represents the extreme degree of the audio object emotion included in the object.

[0093] At step 308, the process may include performing explicit emotion value determination to determine an explicit emotion value associated with each audio object by remapping the audio object emotion level associated with each of the plurality of audio objects to a common scale of the plurality of basic emotions by adding a fixed bias. The value of the emotion in the common scale is referred to as the explicit emotion value. In an embodiment, the explicit emotion value may be determined by, for example, Figure 2 The explicit emotion value determination engine 222 mentioned in the embodiment determines the explicit emotion value. In an embodiment, the basic emotions may include an extremely sad emotion, a sad emotion, a neutral emotion, a happy emotion, and an extremely happy emotion. The audio object emotion level of each audio object can be mapped to an explicit emotion value by adding a bias value to the audio object emotion level of each audio object.

[0094] At step 309, the process may include performing an adapted component factor determination. At step 309, the process may include steps 310 and 311. At step 310, the process performs a component factor determination to determine a component of each basic emotion in the audio object emotion by using a plurality of adapted emotion kernels. In an embodiment, the components may be as in Figure 1 and Figure 2 In an embodiment, the components may be based on the shape associated with each of the plurality of adapted emotion kernels and the explicit emotion value of each audio object. In step 311, the process performs emotion kernel adaptation. In step 311, the plurality of adapted emotion kernels may be modified based on feedback from the listener. In an embodiment, for yes / no based feedback from the listener, one or more parameters of at least one adapted emotion kernel are increased or decreased by a small positive constant (Δ). In an embodiment, the components may be based on the shape associated with each of the plurality of adapted emotion kernels and the explicit emotion value of each audio object. In step 312, the process performs emotion kernel adaptation. In step 313, the plurality of adapted emotion kernels may be modified based on feedback from the listener. In an embodiment, for yes / no based feedback from the listener, one or more parameters of at least one adapted emotion kernel are increased or decreased by a small positive constant (Δ). In an embodiment, the components may be based on the shape associated with each of the plurality of adapted emotion kernels and the explicit emotion value of each audio object. In step 314, the process performs emotion kernel adaptation. In step 315, the process performs emotion kernel adaptation. Figure 2 The adaptation component factor determination engine 224 mentioned in determines the component factors. In an embodiment, the component factors determined for each audio object emotion may be expressed as factors of a plurality of basic emotion components.

[0095] At step 312, the process may include performing audio object modification. At step 312, the process may include steps 313 and 314. At step 313, the process performs audio object prioritization associated with multiple audio objects. The audio object prioritization may include determining a probability that a listener likes to view a particular emotion, and determining a priority value associated with each audio object based on the probability regarding the listener's preference for each audio object. To determine the priority value, the component factors may be used as weights for the probability that the listener likes to view a particular emotion. The weighted sum of such probabilities may determine the priority value of a particular audio object among the multiple audio objects. In an embodiment, the priority value of a particular audio object among the multiple audio objects may be determined by, for example, Figure 2 The audio object modification engine 226 mentioned in determines the priority value of each audio object.

[0096] At step 314, the process may include performing gain adjustment to adjust the gain associated with each audio object after calculating the priority value. In an embodiment, the gain of each audio object may be adjusted to attenuate, remove, or enhance a particular audio object. In an embodiment, a particular audio object may be, for example, Figure 1 and Figure 2 In an embodiment, the gain may be adjusted by the audio object modification engine 226.

[0097] At step 316 , after the gains are adjusted, the process may include combining the plurality of audio objects and outputting the audio content with the adjusted gains to the listener.

[0098] In step 318, the process may include obtaining feedback from the listener to adapt the core shape of multiple adaptive emotion cores. Feedback based on supervision may be used to determine the listener's preference profile for a specific emotion. In an embodiment, multiple other feedback parameters may also be used to determine the listener's preference profile. Multiple other feedback parameters may include visual feedback, previous feedback, sensor feedback, and manual feedback. Feedback may be used to adjust the size of multiple adaptive emotion cores, and may also be used to update the probability that the listener likes a specific emotion. In an embodiment, feedback may be obtained by component factor determination engine 224. In an embodiment, resizing may include increasing or decreasing one or more parameters associated with the shape of multiple adaptive emotion cores. In an embodiment, one or more parameters may include slope, height, length, width, radius, and angle of multiple adaptive emotion cores.

[0099] In an embodiment, in the case where it is determined that the multiple adaptive emotion cores are in a trapezoidal shape, adjusting the size may include increasing or decreasing one or more of the slope and height of the multiple adaptive emotion cores. In an embodiment, in the case where it is determined that the multiple adaptive emotion cores are in a rectangular shape, adjusting the size may include increasing or decreasing one or more of the length and width of the multiple adaptive emotion cores. In an embodiment, in the case where it is determined that the multiple adaptive emotion cores are in a circular shape, adjusting the size may include increasing or decreasing the radius of the multiple adaptive emotion cores. In an embodiment, in the case where it is determined that the multiple adaptive emotion cores are in a triangular shape, adjusting the size may include increasing or decreasing one of the angle and height of the multiple adaptive emotion cores.

[0100] Figure 4 An architectural diagram describing a method for modifying audio content according to an embodiment of the present disclosure is shown. Figure 4 In an embodiment, the audio content may be modified based on the preferences of the listener listening to the audio content. In an embodiment, the architecture diagram may include Figure 2 The audio object recognition engine 218, the emotion level determination engine 220, the explicit emotion value determination engine 222, the adaptation component factor determination engine 224, and the audio object modification engine 226 mentioned therein. In addition, the architecture diagram may include a media device 416. In an embodiment, the media device 416 may include a display 418, a user interaction module 420, a camera 422, a memory 424, an operating system 426, one or more applications 428, and one or more input / output interfaces 430. In an embodiment, the memory 424 may be as in Figure 2 The memory 206 mentioned in the embodiment. In an embodiment, the method may be performed by the system 202 that deploys the system components of the architecture diagram.

[0101] Continuing with the above embodiment, the audio object recognition engine 218 can be configured to pre-process the input audio. In an embodiment, the input audio can be the audio content received for modification. In addition, the audio object recognition engine 218 can be configured to perform pre-processing and source separation on the audio content using a pre-trained model. In addition, the audio object recognition engine 218 can be configured to perform post-processing on the output audio generated when the source is separated. After post-processing, the audio object recognition engine 218 can be configured to generate a final output. In an embodiment, the final audio can be the source-separated audio separated into multiple audio objects. In an embodiment, the audio object recognition engine 218 can include an audio processor 402 for performing pre-processing and post-processing. In addition, the audio object recognition engine 218 can include a source separator 404 for performing source separation.

[0102] Subsequently, the emotion level determination engine 220 may be configured to determine an audio object emotion and an audio object emotion level associated with each of the plurality of audio objects. Next, the explicit emotion value determination engine 222 may be configured to map the audio object emotions to a common scale based on a predefined mapping value of a fixed set of audio object emotion values ​​and audio object emotions.

[0103] To this end, it is understood that the adaptation component factor determination engine 224 can be configured to determine the component factors of the basic human emotions in the emotions of the identified audio object. In an embodiment, the basic human emotions can be as shown in Figure 2 The adaptation component factor determination engine 224 may need to adapt the adaptation emotion core according to the listener's emotional response. In an embodiment, the adaptation emotion core may be as shown in Figure 2 In an embodiment, the adaptation component factor determination engine 224 may include a component factor determiner 406 for determining component factors and an emotion core adapter 408 for adapting multiple adaptation emotion cores based on the emotional response of the listener received from the user feedback module 410.

[0104] Continuing with the above embodiment, the audio object modification engine 226 can be configured to determine a priority value for each audio object based on the component factors and the shape of the adapted emotion kernel. In addition, the audio object modification engine 226 can be configured to adjust the gain associated with each audio object in an order of priority associated with each of the plurality of audio objects to enhance or weaken the effect of at least one audio object. In an embodiment, the audio object modification engine 226 can include an audio object priority engine 412 for determining a priority value and a gain adjuster 414 for performing gain adjustment.

[0105] Figure 5aAn operational flow chart 500a describing a process for generating a plurality of audio objects according to an embodiment of the present disclosure is shown. Figure 5a In an embodiment, the audio content may be input as in Figure 2 The system 202 mentioned in the above is received, and the audio content can be separated into multiple audio objects as output. In an embodiment, the output as multiple audio objects can be referred to as "a i ”, 1≤i≤N, “N” can be the number of audio objects.

[0106] In an embodiment, the audio content may be provided by Figure 2 The audio object recognition engine 218 mentioned in the above is separated into multiple audio objects. Examples of multiple audio objects may include but are not limited to human voice, background, music, etc. In an embodiment, the audio object recognition engine 218 may be configured to perform source separation on the audio content and generate N source-separated audio outputs a i , 1≤i≤N. In an embodiment, the value "N" may depend on the model used to perform source separation to generate multiple audio objects from audio content. In an embodiment, the present disclosure may utilize a modified version of a U-Net source separation model. In an embodiment, the "U-Net" source separation model may be a "Spleeter" model.

[0107] In an embodiment, the process may include step 502a. In step 502a, the process may include performing pre-processing of the audio content in response to receiving the audio content as input. In an embodiment, the pre-processing may include:

[0108] Get the STFT of the input audio content.

[0109] Transpose, zero padding and interleaving (to obtain the expected shape) of the input vector of audio content are performed.

[0110] Next, the process may include step 504a. At step 504a, the process may include proceeding to the following steps: feeding the preprocessed audio content to a U-Net source separation model U-Net source separation model to generate an output based on the preprocessed audio content.

[0111] Continuing with the above embodiment, the process may include steps 5-6a. At step 506a, the process may include performing post-processing on the output generated by the U-Net source separation model to generate a new output. In an embodiment, the post-processing may include:

[0112] Spectrogram reconstruction, i.e., taking the interleaved output to construct a 2D vector.

[0113] Output the inverse STFT of the audio block.

[0114] In an embodiment, the new output generated may be PCM data related to the source separated audio content. In an embodiment, the length of the audio content may depend on the minimum length of the input required for processing by the model. In an embodiment, the minimum range may be on the order of seconds.

[0115] Figure 5b A diagram illustrating a U-Net source separation model according to an embodiment of the present disclosure is shown. Figure 5b In an embodiment, a commonly and widely used source separation model may be a U-Net source separation model that works based on a UNET architecture using audio features such as spectrograms. As mentioned above, the value of N may depend on the number of stems present in the source separation model. In an embodiment, the U-Net source separation model may be configured to generate a minimum of 2 to a maximum of 5 source separation outputs. In an embodiment, the present disclosure adopts a modified version of the U-Net source separation model. In an embodiment, the U-Net source separation model may be a modified model as used in the present disclosure. In an embodiment, the modification may include removing one or more unsupported layers / nodes, and processing one or more unsupported layers / nodes separately outside a tensorflow lite model as part of preprocessing. In an embodiment, the modification may also include removing layers involving unsupported operators from a Spleeter Tensorflow model, and performing such steps externally using ordinary mathematical operations. In addition, the conversion of a slice model to Tensorflow lite may be performed using only built-in TensorFlow-Lite operators.

[0116] Figure 5c FIG. 2 shows a graphical representation of the use of memory 206 by a U-Net source separation model according to an embodiment of the present disclosure. Figure 5c , in an embodiment, the x-axis may be the call time and the y-axis may be the memory usage. In an embodiment, the memory usage may be a function of the model size, remaining substantially constant after the U-Net source separation model and the interpreter are loaded. Furthermore, the call time may be of the order of the length of the input audio content. In an embodiment, "n" seconds of input may require a call time less than or equal to "n" seconds.

[0117] Figure 5d FIG. 5 is a diagram illustrating a method for generating a plurality of audio objects in audio content 501a according to an embodiment of the present disclosure. Figure 5d In an embodiment, the audio content 501a may include two source-separated audio objects, such as human voice (a1) and background music (BGM) (a2). In an embodiment, the two source-separated audio objects may be outputs of a dual-stem (N=2) model.

[0118] Figure 6a An operational flow chart 600a is shown describing a process for determining audio object emotion levels associated with a plurality of audio objects according to an embodiment of the present disclosure. Figure 6a In an embodiment, a plurality of audio objects may be generated by separating audio content. In an embodiment, an audio object emotion level of each audio object may be determined to further determine explicit emotion values ​​associated with the plurality of audio objects. In an embodiment, the audio object emotion level of each audio object may be determined to further determine explicit emotion values ​​associated with the plurality of audio objects. Figure 2 The audio object emotion level determination engine 220 mentioned in determines the audio object emotion level of each audio object.

[0119] In an embodiment, the audio object emotion level determination engine 220 may receive a plurality of audio objects as input. In an embodiment, the number of audio objects may be "N". In addition, the audio object emotion level determination engine 220 may be configured to determine the audio object emotion level of each audio object as output. In an embodiment, the audio object emotion level determination engine 220 may also be configured to determine the emotions present in the source separated audio content. In an embodiment, the source separated audio content may be a plurality of audio objects. In an embodiment, the audio content in an audio object may be referred to as a i , the emotion of each audio object and the audio object emotion level can be referred to as e i and v i .

[0120] According to an embodiment of the present disclosure, the process may include step 601a. ​​In step 601a, the process may be configured to determine multiple audio features associated with the audio content. In an embodiment, the multiple audio features may be one of a fundamental frequency, a time-varying characteristic of the fundamental frequency, a root mean square (RMS) value of the amplitude, a speech speed, and the like. Next, the process may include step 602a. In step 602a, the process may include determining the audio emotion probabilities of multiple audio objects using an emotion probability determination audio model. In an embodiment, the emotion probability determination audio model may include one or more statistical models, such as a hidden Markov model, preconfigured using learning audio data or video data. In an embodiment, the audio emotion probability may be an audio object emotion level v representing the extreme degree of the emotion of the audio object. i Examples of audio object emotions may include, but are not limited to, admiration, awe, appreciation, amusement, anger, anxiety, awe, embarrassment, boredom, calm, confusion, desire, disgust, and empathy for pain, sadness, neutral emotions, and happy emotions. In an embodiment, the process may include step 603a. At step 603a, the process may be configured to determine the audio object emotion level.

[0121] Figure 6bshows a diagram depicting the determination of an audio object emotion level associated with a plurality of audio objects according to an embodiment of the present disclosure. Referring to Figure 6b , in an embodiment, the audio object emotion level determination engine 220 may receive a plurality of audio objects such as a human voice (a1) and BGM (a2). Additionally, an output may be generated that describes the emotion and the degree of extremity of the emotion represented associated with the plurality of audio objects, where the audio object emotion level is (0 < v < 1, 0 - moderate, 1 - extreme). In an embodiment, the output for the emotion and audio object emotion level of the human voice may be an excited emotion and an audio object emotion value of 0.9. In an embodiment, the output for the emotion and audio object emotion level of the BGM may be a happy emotion and an audio object emotion value of 0.2.

[0122] Figure 7a shows an operation flowchart 700a of a process for determining an explicit emotion value associated with each audio object of audio content according to an embodiment of the present disclosure. Referring to Figure 7a , in an embodiment, as Figure 1 described, the explicit emotion value may define the audio object emotion of each audio object among the plurality of audio objects associated with the audio content. In an embodiment, the explicit emotion value of each audio object may be determined by an explicit emotion value determination engine 222 as mentioned in Figure 2 . In an embodiment, the explicit emotion value of each audio object may be determined based on the mapping of the audio object emotion level associated with each audio object on a common scale. In an embodiment, the common scale may be one of a pleasure scale and an arousal scale. In an embodiment, the common scale may include a plurality of basic emotions. Examples of the plurality of basic emotions may include, but are not limited to, an extremely sad emotion, a sad emotion, a neutral emotion, a happy emotion, and an extremely happy emotion.

[0123] Continuing the above embodiment, the explicit emotion value determination engine 222 may be configured to receive the audio object emotion level v i associated with each of the plurality of audio objects as input, and determine the explicit emotion value c iAs output. In an embodiment, the explicit emotion value determination engine 222 can be configured to re-quantize each audio object of each pair and the audio object emotion level associated with each audio object to a common scale including multiple basic emotions, so that the absolute position of each audio object can be determined on the common scale. In an embodiment, the hedonic scale can be used with multiple basic emotions (such as extremely sad emotions, sad emotions, neutral emotions, happy emotions, and extremely happy emotions). In an embodiment, the purpose of the explicit emotion value determination engine 222 can be to find the position of a given emotion-value pair in the common scale of 0 to 50. The numbers 0 to 50 may be just a representation of the range of a particular emotion on the common scale.

[0124] Continuing with the above embodiment, the process may include step 702a. At step 702a, the process may include determining, by the explicit emotion value determination engine 222, a bias β corresponding to the audio object emotion level of each audio object received as input. i In an embodiment, determining the bias may be based on maintaining a list of several but limited emotions (which may be outputs of emotion level determinations, such as fear, anger, awe, excitement, calm, etc.). In an embodiment, determining the bias may also include mapping each emotion of the audio object onto a range on a common scale such that:

[0125] Each emotion i The range R that must be mapped onto a common scale of emotions i :[a i ,b i ), where a i ,b i ∈[0,50], and b i -a i ≥10.

[0126] The mapping may be fixed knowledge and may be considered as a predetermined constant.

[0127] In response to determining the range, the offset can be calculated as:

[0128] β i =a i , where a i It's emotion i The mapping range R i :[a i ,b i ) is the minimum value of .

[0129] Continuing with the above embodiment, the process may include step 704a. At step 704a, the process may include determining the explicit emotion value based on Equation 1:

[0130] c i =βi +v i ×10,

[0131] Among them, the explicit value is the emotion e i A measure of the position on a common scale of emotions, containing the audio object a i The sentiment value v included in i .

[0132] In an embodiment, the explicit emotion value may be a remapping of the audio object emotion level of a single audio object in the plurality of audio objects to a common scale. In an embodiment, the explicit emotion value may be used to quantify the plurality of audio emotions into factors of at least one basic human emotion, and quantify the priority by considering the listener's emotion preference. In an embodiment, the at least one basic human emotion may be among the plurality of basic emotions.

[0133] Figure 7b A common scale according to an embodiment of the present disclosure is shown. Figure 7b In an embodiment, the common scale may be a common scale having a plurality of basic emotions such as extremely sad emotions, sad emotions, neutral emotions, happy emotions, and extremely happy emotions. In an embodiment, the plurality of basic emotions may include, for example, Figure 7b The range described in (0 to 50).

[0134] Figure 7c A common scale according to an embodiment of the present disclosure is shown, wherein audio object emotions are mapped to a fixed preset range on the common scale. Figure 7c In an embodiment, audio object emotions (such as fear, anger, awe, excitement, calmness, etc.) are mapped to fixed preset ranges on a common scale.

[0135] Figure 7d A diagram is shown describing the determination of an explicit emotion value according to an embodiment of the present disclosure. Figure 7d In an embodiment, the explicit emotion value determination engine 222 may receive a plurality of audio objects such as vocals and BGM, emotions associated with each audio object, and audio object emotion levels. The emotion and emotion value of the vocals may be an excited emotion and an audio object emotion value of 0.9, and the emotion and emotion value of the BGM may be a happy emotion and an audio object emotion value of 0.2. In addition, the bias determined for each audio object may be 38 and 30. Next, an explicit emotion value may be generated for each audio object, such that the explicit emotion value of the vocals may be 47, and the explicit emotion value of the BGM may be 32.

[0136] Figure 8a An operational flow chart 800a describing a process for determining component factors according to an embodiment of the present disclosure is shown. Figure 8bA nuclear scale according to an embodiment of the present disclosure is shown. Figure 8c A modified nuclear scale based on feedback from listeners is shown in accordance with an embodiment of the present disclosure. Figure 8d An embodiment of a nuclear scale depicting the location of explicit emotions on the nuclear scale is shown according to an embodiment of the present disclosure. Figure 8e A diagram depicting component factors as output based on listener feedback and an explicit emotion value of each audio object according to an embodiment of the present disclosure is shown. Figure 8f A graphical representation depicting the height of at least one adapted emotion kernel according to an embodiment of the present disclosure is shown. Figure 8a , Figure 8b , Figure 8c , Figure 8d , Figure 8e , Figure 8f In an embodiment, as in Figure 1 As described in , the component factor may represent one or more basic emotions in a plurality of basic emotions in the explicit emotion value of each audio object. In an embodiment, the component factor may be determined in the audio object emotion associated with each of the plurality of audio objects of the audio content. In an embodiment, the component factor may be determined by Figure 2 The adaptive component factor determination engine 224 mentioned in is determined. In an embodiment, the component factor may be determined based on a plurality of adaptive emotion kernels existing on the kernel scale and the explicit emotion value of each audio object.

[0137] In an embodiment, the process may include step 802a. At step 802a, the process may include determining the component factor by the adaptation component factor determination engine 224. The adaptation component factor determination engine 224 may be configured to receive as input (a) an explicit emotion value for each audio object in the plurality of audio objects. i ,c i ), 1≤i≤N. In response, the adaptation component factor determination engine 224 may be configured to determine the component factor (a i ,(Ej i ,Cj i )), 1≤i≤N, 1≤j≤M. In an embodiment, "N" may be the number of audio objects, and "M" may be the number of basic emotions. In an embodiment, "j" may be one or more basic emotions among a plurality of basic emotions "M" in the explicit emotion value of each audio object. In an embodiment, the number of basic emotions "M" may be 5. In an embodiment, the number of basic emotions "M" may vary between 4 and 27.

[0138] In an embodiment, mathematically, the component factor Cj i can be defined as i The explicit sentiment value c iThe basic emotion Ej among the multiple basic emotions in i The percentage component, where 1≤i≤N, 1≤j≤M.

[0139] In order to determine the component factors, multiple adapted emotion cores may be required. In an embodiment, each emotion core may be a representation shape of the listener's preference for each basic emotion represented by each of the multiple adapted emotion cores. In an embodiment, the size of the multiple adapted emotion cores represents an illustrative measure of the listener's preference for one of the multiple basic emotions (such as, E1: extremely sad, E2: sad, E3: neutral, E4: happy and E5: extremely happy). In an embodiment, E1, E2, E3, E4 and E5 may be multiple adapted emotion cores.

[0140] In an embodiment, the process may include step 804a. In step 804a, the process may include adjusting the size of the multiple adapted emotion cores. In an embodiment, adjusting the size of the multiple adapted emotion cores may include increasing or decreasing one or more parameters associated with the shapes of the multiple adapted emotion cores. In an embodiment, the one or more parameters may include the slope, height, length, width, radius, and angle of the multiple adapted emotion cores.

[0141] In an embodiment, in the case where it is determined that the multiple adapted emotion cores are trapezoidal in shape, adjusting the size may include increasing or decreasing one or more of the slope and height of the multiple adapted emotion cores. In an embodiment, in the case where it is determined that the multiple adapted emotion cores are rectangular in shape, adjusting the size may include increasing or decreasing one or more of the length and width of the multiple adapted emotion cores. In an embodiment, in the case where it is determined that the multiple adapted emotion cores are circular in shape, adjusting the size may include increasing or decreasing the radius of the multiple adapted emotion cores. In an embodiment, in the case where it is determined that the multiple adapted emotion cores are triangular in shape, adjusting the size may include increasing or decreasing one of the angle and height of the multiple adapted emotion cores.

[0142] In an embodiment, the process may include step 806a. In step 806a, the process may include changing the size of each adapted emotion core according to feedback from the listener. In an embodiment, the shape of each adapted emotion core in the plurality of adapted emotion cores may be adapted by changing the size of each adapted emotion core according to the interests of the listener. Feedback from the listener as positive feedback or negative feedback may be obtained using one or more of the following known means:

[0143] Visual feedback: Assuming that visual data of the listener watching the content is available, feedback can be obtained from the listener's facial expressions.

[0144] Sensor Feedback: Based on the data levels available with the processor (such as data from the listener's smartwatch), the data can be used to infer the current automatic listener's response as positive or negative feedback.

[0145] Prior feedback: Knowledge such as past volume control behaviors corresponding to past audio emotions can be recorded and used as prior knowledge to automatically understand the listener's expected behavior at the current time.

[0146] Manual feedback: In addition to the above automatic way of obtaining feedback, another way may be to manually ask the listener for feedback on liking a specific emotion. In an embodiment, the specific emotion may be among multiple audio object emotions.

[0147] Next, once positive or negative feedback is received from the listener for a specific basic emotion among the plurality of basic emotions, it may be necessary to adapt the shape of one or more of the plurality of adapted emotion cores according to the listener's interest.

[0148] The adaptation steps may include:

[0149] Start with a base shape of a plurality of adapted emotion cores for each listener. In an embodiment, the base shape may be a default shape for each of the plurality of adapted emotion cores.

[0150] Based on positive or negative feedback from listeners recorded by any of the mentioned means, one or more adapted emotion kernels may be adapted as follows:

[0151] m + =m + +Δ,

[0152] m - =m - -Δ,

[0153] c=c+Δ

[0154] in,

[0155] like Figure 8f Described, m + is the positive slope of the kernel, m - is the negative slope of the kernel, c is the height of the kernel, and ε is a small constant.

[0156] In an embodiment, one or more adapted emotion cores may be shaped in the form of a trapezoid.

[0157] In an embodiment, if a listener does not like to hear extreme sadness / sadness emotions, based on the listener's feedback, one or more adapted emotion cores may be adjusted.

[0158] Next, in response to adapting one or more adapted emotion kernels based on the feedback, a component factor can be determined based on the explicit emotion values ​​ci of the plurality of audio objects ai by locating the explicit emotion value on the kernel scale for each of the plurality of audio objects. Based on the position of the explicit emotion value of each audio object on the kernel scale, a percentage contribution of the plurality of basic emotions can be determined.

[0159] Reference Figure 8b In an embodiment, the kernel scale may include a plurality of adapted emotion kernels representing a plurality of basic emotions. In an embodiment, as shown, a basic shape, also referred to as a default shape of the adapted emotion kernel scale, may be a bias for each of the plurality of basic emotions with respect to a constant listener.

[0160] Reference Figure 8c In an embodiment, if the listener does not like to hear extreme sadness or sorrowful emotions, based on the feedback, the Figure 8c Update multiple adapted emotion cores as described in .

[0161] Reference Figure 8d , based on the explicit sentiment value c i , the percentage contribution of one or more basic emotions can be found. In an embodiment, for audio object a i The explicit sentiment value c i The percentage contributions of multiple basic emotions Ej:1≤j≤M are shown in Table 2 below:

[0162] Ej mood Cj: Contribution factor E1 Extremely sad 0 E2 sad 0 E3 neutral 0.3(30％) E4 hapiness 0.7(70％) E5 Extremely happy 0

[0163] Table 2 describes the contributions of multiple basic emotions in the component factors.

[0164] In an embodiment, multiple adapted emotion kernels play an important role in determining component factors by symbolizing that "emotions are subjective in nature", so that "what may be sad for one person may not be so sad for another person". By maintaining a basic number of adapted emotion kernels and adapting based on feedback from listeners, it can be easily taken into account that audio object emotions are subjective.

[0165] Reference Figure 8e In an embodiment, through the component factor determination engine 224, the input may be vocals = 47 and background music (BGM) = 32, and the output may indicate that the vocals contain 100% E5 (extremely happy) and the BGM contains 30% E3 (neutral) and 70% E4 (happy).

[0166] Reference Figure 8f In an embodiment, the shape of the adapted emotion core may be adapted by changing the size of the adapted emotion core according to the interests of the listener. +” is the positive slope of the kernel, “m - ” is the negative slope of the kernel, and “c” is the height of the kernel.

[0167] Figure 9a An operational flow chart 900a is shown describing a process for audio object prioritization and gain adjustment according to an embodiment of the present disclosure. Figure 9b A diagram is shown describing audio object prioritization and gain adjustment for generating modified audio content according to an embodiment of the present disclosure. Figure 9a and Figure 9b In an embodiment, audio object prioritization may include, for example, Figure 1 , calculating the probability of the listener being associated with each of one or more basic emotions represented in the component factors in a plurality of basic emotions and calculating the priority value associated with each of the plurality of audio objects based on the probability. In addition, the gain adjustment may include adjusting the gain associated with at least one audio object in the plurality of audio objects for modifying the audio content. In an embodiment, after the gain adjustment is performed on the at least one audio object, the plurality of modified audio objects may be combined to generate modified audio content for the listener. In an embodiment, the audio object priority sorting and gain adjustment may be performed as described in Figure 2 The audio object modification engine 226 mentioned in is executed.

[0168] Continuing with the above example, the audio object modification engine 226 may be configured to modify the audio object as in Figure 2 The component factor determination engine 224 mentioned in the above receives the component factors (representing a plurality of basic emotions (a i ,(Ej i ,Cj i )), 1≤i≤N,1≤j≤M) as input. In an embodiment, "N" may be the number of audio objects, and "M" may be the number of basic emotions. In an embodiment, "j" may be one or more basic emotions among the multiple basic emotions "M" represented in the component factor. In an embodiment, the number of basic emotions "M" may be 5. In an embodiment, the number of basic emotions "M" may vary between 4 and 27. In addition, in response to receiving the input, the audio object modification engine 226 may be configured to generate a prioritized list of multiple audio objects. A i ,1≤i≤N.

[0169] In an embodiment, the process may include step 902a. At step 902a, the process may include determining, by the audio object modification engine 226, a priority of a particular audio object among the plurality of audio objects based on the listener's preference information.

[0170] In an embodiment, each audio object a may be determined as follows i Priority value:

[0171] Determine a probability that the listener is associated with each of the one or more basic movements in the plurality of basic emotions. The probability that the listener is associated with each of the one or more basic movements may be determined based solely on feedback from the listener. Alternatively, the probability may be determined based on Equation 2 using information contained in the plurality of adapted emotion kernels:

[0172]

[0173] Among them, Ej is one of the basic emotions E1, E2, E3, E4, and E5 explained above.

[0174] Determine the priority value: Once the listener associates with each of the one or more basic emotions, the probability p j If we know that, we can determine each audio object a based on equation 3 i Priority Value:

[0175] Among them, Cj i is an audio object a i Component factors of basic emotion Ej.

[0176] Once the priority value of each audio object is determined, the audio object modification engine 226 may be configured to sort the plurality of audio objects in order of priority as follows:

[0177] A i =a k ,

[0178] Make a k In the audio object {a k -{A 1 ,..A i-1}},1≤i,k≤N has the maximum priority value.

[0179] In addition, to perform gain adjustment, the audio object modification engine 226 may be configured to receive a prioritized audio object list A. i (1≤i≤N) as input and generates modified audio content as output.

[0180] In an embodiment, the audio object modification engine 226 may be configured to provide appropriate gains to the prioritized audio objects based on the priority values ​​so as to remove / attenuate or enhance specific audio objects, and may also be configured to mix the gain-adjusted audio objects to generate a final modified audio output. In an embodiment, the final modified audio output may be the modified audio content.

[0181] In an embodiment, the process may include step 904a. In step 904a, the process may include selecting the audio object list A after a given priority order. i Make A i The priority is greater than A i+1 In the case of a priority of (1≤i≤N), the gain is adjusted by the listener through the audio object modification engine 226 according to the preset preferences set by the listener. In an embodiment, the preset preferences may include options to ask whether to completely remove certain parts of the audio or to enhance or reduce the effect. In an embodiment, the gain adjustment must be handled appropriately for the following scenarios:

[0182] By setting the gain to 0 (G N = 0) is assigned to the lowest priority audio object and the gain 1 (G 1 =1) is assigned to the highest priority audio object to completely remove a portion of the audio content. In an embodiment, the lowest priority audio object may be an audio object with the lowest priority value in the prioritized audio object list, and the highest priority audio object may be an audio object with the highest priority value in the prioritized audio object list.

[0183] By assigning a non-zero gain to the lowest priority audio object (G N >0) and assigns a gain of 1 to the highest priority audio object (G 1 >1) to increase or decrease the effect.

[0184] Continuing with the above embodiment, once multiple audio objects A N and A 1 G N and G 1 , the remaining audio objects A can be determined based on Equation 4 i Gain:

[0185]

[0186] Furthermore, once the gain is determined, the modified output can be simply mixed based on Equation 5:

[0187]

[0188] Reference Figure 9bIn an embodiment, the audio object modification engine 226 may receive a plurality of audio objects such as human voice and BGM, and generate a list of audio objects based on priority values. In addition, based on the priority values, modified audio content may be generated. In an embodiment, the human voice may represent an extremely happy emotion, and the component factor may be 0.9, and the BGM may represent a happy emotion with a component factor of 0.3 and a neutral emotion with a component factor of 0.7.

[0189] Fig.10 1002 is a diagram showing a method for modifying audio content including a metaprocessor according to an embodiment of the present disclosure. In an embodiment, the diagram 1002 may be as shown in FIG. Figure 4 In addition, the number of basic emotions on the common scale utilized by the explicit emotion value determination engine 220 can vary between 4 and 27. In another embodiment, the adaptation component factor determination engine 224 can be configured to use feedback from the listener, such that feedback can be generated based on reinforcement learning.

[0190] In addition, in an embodiment, the audio object recognition engine 218 can use an audio object-based codec to utilize metadata 1000 related to audio object information such as Dolby Atmos, rather than performing source separation of the audio content. In an embodiment, a metadata processor can be deployed to process and find information for each audio object information directly based on the input metadata.

[0191] In an embodiment, the plurality of adapted emotion cores may have a plurality of shapes. In an embodiment, the plurality of shapes may include a trapezoidal shape, a triangular shape, a circular shape, a rectangular shape, etc. A trial and error method may be used to change / initialize the plurality of shapes according to the most suitable shape. The shapes may also be adjusted using feedback from listeners based on reinforcement learning.

[0192] Fig.11 A use case diagram is shown that describes a scenario for modifying audio content by enhancing a singer's voice according to an embodiment of the present disclosure. Figure 5d , Figure 6b , Figure 7d , Figure 8e , Figure 9b and Fig.11 , Fig.11 A scenario for modifying audio content to enhance a presenter's voice according to the present disclosure is shown.

[0193] Fig.12A use case diagram 1200a is shown describing a scenario where a listener cannot modify audio content according to the prior art. The listener may not like loud audio or audio associated with anger / rage, and may have to manually lower the volume of the television (TV) playing the audio. However, when the volume is lowered, the listener may not be able to hear the reporter clearly, or if the listener increases the volume, the background noise of the people in the TV may become louder. Fig.12 A use case diagram 1200b is shown describing a scenario in which a listener modifies audio content according to an embodiment of the present disclosure. In an embodiment, since the smart TV can understand the listener's preferences, the listener can avoid lowering the volume of a specific audio object (such as one or more people shouting).

[0194] Fig.13 A use case diagram 1300 is shown describing a scenario in which a listener modifies audio content by managing one or more audio objects according to an embodiment of the present disclosure. In an embodiment, one or more audio objects may be associated with audio object emotions such as anger and shouting. In an embodiment, a listener may not like audio that includes anger and shouting, and may be able to reduce the impact of anger and shouting in the audio content. In an embodiment, the audio content may be a live recording of one or more protesters, making it difficult for the listener to hear a reporter's report relative to the one or more protesters.

[0195] Fig.14 A use case diagram 1400 is shown describing a scenario in which a listener controls one or more audio objects of audio content according to an embodiment of the present disclosure. In an embodiment, one or more audio objects may represent audio object emotions, such as happy, calm, harsh, and noise. In an embodiment, the listener may be exercising, and the effect of audio objects associated with a calm audio object emotion among the one or more audio objects may be increased by removing audio objects representing harsh and noise emotions.

[0196] Fig.15 A use case diagram 1500 is shown describing a scenario in which a listener enhances a human voice and suppresses BGM from audio content according to an embodiment of the present disclosure. In an embodiment, the listener may suffer from a hearing disease and use a hearing aid so that the listener feels that the audio signal is loud but unclear. In an embodiment, the system disclosed in the present disclosure can be configured to understand the listener's annoyance or disinterest in the loud sound that makes the listener unclear, thereby automatically suppressing unwanted audio objects.

[0197] Fig.16A use case diagram 1600 is shown describing a scenario of enhancement of a music portion in audio content according to an embodiment of the present disclosure. In an embodiment, the music portion may be preferred by the listener, and the system disclosed in the present disclosure may be configured to detect the preference based on previous experience and enhance the music portion preferred by the listener.

[0198] Fig.17 A use case diagram 1700 is shown that describes a scenario in which audio content can be personalized based on emotions associated with the audio content according to an embodiment of the present disclosure. In an embodiment, the audio content can be a song. In an embodiment, the songs can be classified based on emotions contained in the lyrics, BGM, and other factors associated with the songs. In addition, the system disclosed in the present disclosure can be configured to classify the songs by calculating the priority using a personalized emotion kernel method.

[0199] Fig.18 A use case diagram 1800 is shown describing a scenario of automatic enhancement of vocals / beats in audio content according to an embodiment of the present disclosure. In an embodiment, enhancement may be performed by the system disclosed in the present disclosure based on the listener's preferences while the listener is dancing. In an embodiment, the system may be configured to enhance a portion of the audio content that the listener may enjoy and the vocals / beats in a certain portion of the audio content without having to manually enhance the dance moves of the listener dancing while listening to the audio content.

[0200] Although specific language has been used to describe the present disclosure, any limitation caused by it is not intended. As will be clear to those skilled in the art, various work modifications can be made to the method to implement the inventive concept taught herein. The accompanying drawings and the foregoing description give examples of embodiments. Those skilled in the art will appreciate that one or more of the elements described can be well combined into a single functional element. Alternatively, some elements can be separated into multiple functional elements. Elements from one embodiment can be added to another embodiment. For example, the order of the process described herein can be changed and is not limited to the manner described herein.

[0201] In addition, the actions of any flow chart need not be implemented in the order shown; nor do they necessarily need to perform all actions. In addition, those actions that are independent of other actions can be performed in parallel with other actions. The scope of the embodiments is in no way limited by these specific examples. Many variations are possible, such as differences in structure, size, and use of materials, whether or not explicitly given in the specification. The scope of the embodiments is at least as broad as the scope given by the appended claims.

[0202] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, the benefits, advantages, solutions to problems, and any components that may cause any benefit, advantage, or solution to occur or become more apparent should not be construed as a critical, required, or essential feature or component of any or all claims.

Claims

1. A method for modifying audio content, the method comprising: determining an explicit emotion value defining an audio object emotion for each audio object of a plurality of audio objects associated with the audio content (102); determining component factors (104) representing one or more basic emotions in the explicit emotion values ​​of each audio object in a plurality of basic emotions; calculating a probability (106) that the listener is associated with each of the one or more basic emotions represented in the component factors; calculating a priority value associated with each audio object based on the probability that the listener is associated with each of the one or more basic emotions represented in the component factor of each audio object and the component factor of each audio object (108); generating a list (110) comprising the plurality of audio objects arranged in a specified order with respect to the priority value associated with each of the plurality of audio objects; and The audio content is modified by adjusting a gain associated with at least one audio object of the plurality of audio objects in the list (112).

2. The method according to claim 1, further comprising: The modified audio content is generated by combining a plurality of modified audio objects.

3. The method according to claim 1, wherein: The step of determining the explicit emotion value of each audio object comprises: mapping the audio object emotion level of each audio object onto a common scale to determine a range of audio object emotions in each audio object, wherein the common scale includes the plurality of basic emotions; determining a bias for the audio object emotion level for each audio object, wherein the bias is a minimum value of the range; and The audio object emotion level associated with each audio object mapped on the common scale is added to the bias to determine the explicit emotion value for each audio object.

4. The method according to claim 1, wherein: The common scale is one of a pleasure scale and an arousal scale.

5. The method according to claim 1, wherein: The steps of determining the component factors include: The explicit emotion value of each audio object is mapped onto a kernel scale comprising a plurality of adapted emotion kernels representing the plurality of basic emotions, wherein the component factors are based on contributions of the one or more basic emotions of the plurality of basic emotions represented by the one or more adapted emotion kernels in the explicit emotion value of each audio object.

6. The method according to claim 5, further comprising: obtaining, in real time, a plurality of feedback parameters associated with the listener from at least one of a memory or the listener; The size of at least one of the plurality of adapted emotion kernels is adjusted based on the plurality of feedback parameters.

7. The method according to claim 5, wherein: The contribution of the one or more basic emotions is determined based on the position of the explicit emotion value of each audio object on the one or more adapted emotion kernels when mapping.

8. The method according to claim 1, wherein: Calculating the probability associated with each basic emotion represented in the component factors is based on at least one of the following: a plurality of feedback parameters associated with the listener stored in a memory; or A ratio of the area of ​​one or more adapted emotion cores corresponding to each basic emotion represented in the component factors to the total area of ​​the plurality of adapted emotion cores of the plurality of basic emotions.

9. The method according to claim 6 or 8, wherein: The plurality of feedback parameters includes at least one of visual feedback, sensor feedback, previous feedback, and manual feedback associated with the listener.

10. The method according to claim 1, wherein: The steps of calculating the priority value associated with each audio object include: A weighted summation of the probability of the listener being associated with each basic emotion represented in the component factors and the component factors representing the one or more basic emotions is performed.

11. The method according to claim 1, wherein: The step of modifying the audio content by adjusting the at least one parameter comprises: Do one or more of the following: assigning a gain of 1 to an audio object in the list corresponding to a highest priority value, and assigning a gain of 0 to another audio object in the list corresponding to a lowest priority value, wherein assigning the gain of 0 indicates removing the another audio object from the audio content; and Assigning a non-zero gain to an audio object corresponding to a lowest priority value and assigning a gain of 1 to an audio object corresponding to a highest priority value, wherein assigning a non-zero gain indicates that an effect of the audio object is changed; calculating gains associated with the one or more audio objects based on gains associated with audio objects having priority values ​​higher than one or more audio objects in the list other than an audio object having a highest priority value and another audio object having a lowest priority value and gains associated with audio objects having priority values ​​lower than the one or more audio objects; and A weighted summation of the gains associated with each audio object in the list is performed to modify the audio content.

12. The method according to claim 1, further comprising: receiving the audio content as input; Separating the audio content into the plurality of audio objects; as well as An audio object emotion level associated with each audio object is determined.

13. The method according to claim 12, wherein: The step of separating the audio content into the plurality of audio objects comprises: preprocessing the input to generate preprocessed audio content; and feeding the preprocessed audio content to a source separation model to generate an output; and The output is post-processed to generate from the audio content the plurality of audio objects associated with the audio content.

14. The method according to claim 1 or 12, wherein: The audio object emotion level associated with each audio object is determined based on the following steps: determining one or more audio features associated with each audio object, wherein the one or more audio features include at least one of a fundamental frequency associated with each audio object, a time-varying characteristic of the frequency, a root mean square (RMS) value associated with the amplitude, or a speech speed; determining an emotion probability value associated with each audio object based on the one or more audio features; and The audio object emotion level associated with each audio object is determined based on the emotion probability value.

15. A system (202) for modifying audio content for a listener, the system (202) comprising: an explicit emotion value determination engine (222) configured to determine an explicit emotion value defining an audio object emotion for each audio object of a plurality of audio objects associated with the audio content; an adaptation component factor determination engine (224) configured to determine component factors representing one or more basic emotions in the explicit emotion values ​​of each audio object in a plurality of basic emotions; as well as The audio object modification engine (226) is configured to: calculating a probability that the listener is associated with each of the one or more basic emotions represented in the component factors; calculating a priority value associated with each audio object based on the probability that the listener is associated with each of the one or more basic emotions represented in the component factor of each audio object and the component factor of each audio object; generating a list comprising the plurality of audio objects arranged in a specified order with respect to the priority value associated with each of the plurality of audio objects; as well as The audio content is modified by adjusting a gain associated with at least one audio object of the plurality of audio objects in the list.

Images