Digital audio emotional response filter

By using devices and methods in social networks, receiving and processing voice signals to reduce specific emotions, the excessive emotional problems that users do not want to feel are solved, and emotional control of voice playback is achieved.

CN119013725APending Publication Date: 2024-11-22SONY INTERACTIVE ENTERTAINMENT LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380034168.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-20
Filing Date
2023-04-14
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

In social networks, users may not want to feel excessive emotions or specific types of emotions in voice playback by another user, but prior art is difficult to effectively filter or reduce these emotions.

Method used

A device and method are designed to receive voice signals from computerized communication sessions through computer storage devices and processors, determine emotions based on signals, and change voice playback through signal processing or robot voice, etc. to reduce the occurrence of specific emotions. .

Benefits of technology

It realizes the partial or complete reduction of the occurrence of specific emotions in voice playback, allowing users to extract specific emotions from one party's voice but not the other party's emotions, and improves users' ability to control voice content.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119013725A_ABST
    Figure CN119013725A_ABST
Patent Text Reader

Abstract

The primary user has the ability to selectively digital signal process the incoming voice chat audio application (206) to prevent feeling over emotional communications. When the user begins to shout, the incoming audio level may be standardized, or when anger / aggressive tones are used to generate (208) robotic speech or toneless speech, the tones may be cancelled.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates generally to digital audio emotionally responsive filters. Background Art

[0002] As recognized herein, in a social networking scenario, a primary user may not wish to experience excessive emotion in another user's speech playback in the network, or may not wish to experience a particular type of emotion, but may still detect what an emotional user is saying. Summary of the invention

[0003] An apparatus includes at least one computer storage device that is not a transient signal and includes instructions executable by at least one processor to receive an incoming signal from a computerized communication session. The signal represents speech of at least a first participant in the computerized communication session. The instructions are executable to determine at least a first emotion based at least in part on the signal; and alter playback of the speech to reduce the first emotion that occurs during playback.

[0004] In some embodiments, the instructions may be executable to determine at least a second emotion based at least in part on the signal; and not modify the playback of the speech to reduce the second emotion appearing during playback. In such instances, the instructions may be executable to receive input that specifies the first emotion as an emotion to be modified during playback and the second emotion as an emotion not to be modified during playback. This means that a user may extract a particular emotion from the speech of one participant, but may not wish to extract another emotion from the speech of the participant.

[0005] In an example embodiment, the instructions may be executable to receive an incoming signal from the computerized communication session. The signal represents the voice of at least a second participant in the computerized communication session. The instructions may be executable to determine at least the first emotion based at least in part on the signal; and not to modify the playback of the second participant's voice having the first emotion when the second participant's voice is played back. This means that a user may extract a particular emotion from the voice of one participant, but may not want to extract the same emotion from the voice of another participant. Thus, the instructions may be executable to receive an input that specifies the first participant whose emotion should be modified and the second participant whose emotion should not be modified.

[0006] In a non-limiting example, the instructions are executable to alter playback of the speech of the first participant to reduce the first emotion present upon playback at least in part by playing the speech after filtering the speech through signal processing circuitry.

[0007] In other non-limiting examples, the instructions may be executable to alter a playback of the speech, at least in part by playing utterances spoken by the first participant in a robotic voice, to reduce the first emotion present during the playback.

[0008] In another aspect, a method includes receiving a user-specified first emotion as an emotion to be reduced upon playback; and altering a speech signal in a computerized communication system to reduce the first emotion in the speech signal upon playback of the speech signal.

[0009] In another aspect, a device includes at least one microphone; at least one transceiver; and at least one processor programmed with instructions to receive input specifying a first emotion to be reduced in outgoing speech received from the microphone prior to transmitting the speech through the transceiver. The instructions are executable to, in response to detecting the first emotion in the outgoing speech, alter the speech to reduce the first emotion therein, thereby revealing altered speech; and transmit the altered speech through the transceiver.

[0010] The details of the present application as to its structure and operation may be best understood with reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram of an exemplary system in accordance with the principles of the present invention;

[0012] Figure 2 Example logic for mitigating excessive emotion in incoming voice chat is shown in example flow chart format consistent with the principles of the invention;

[0013] Figure 3 Example logic for mitigating excessive emotion in outgoing voice chat is shown in example flow chart format consistent with the principles of the invention;

[0014] Figure 4 Example screenshot showing sentiment filtering selection for incoming voice chat;

[0015] Figure 5 Example screenshot showing sentiment filtering selection for outgoing voice chat;

[0016] Figure 6 and 7 Example screenshots showing additional details for sentiment filtering; and

[0017] Figure 8 and 9Example logic is shown in example flow chart format for employing one or more machine learning (ML) models consistent with the principles of the invention. DETAILED DESCRIPTION

[0018] The present disclosure generally relates to a computer ecosystem including aspects of a network of consumer electronics (CE) devices, such as, but not limited to, a computer gaming network. The system herein may include server and client components, which may be connected via a network so that data may be exchanged between the client and server components. The client components may include one or more computing devices, including a gaming console (e.g., a Sony ) or game consoles manufactured by Microsoft or Nintendo or other manufacturers; virtual reality (VR) headsets; augmented reality (AR) headsets; portable televisions (e.g., smart TVs, TVs with Internet capabilities); portable computers, such as laptops and tablet computers; and other mobile devices, including smartphones and additional examples discussed below. These client devices can operate using various operating environments. For example, some of the client computers can use, for example, a Linux operating system; an operating system from Microsoft; or a Unix operating system; or an operating system produced by Apple, Inc. or Google; or a Berkeley Software Distribution or Berkeley Standard Distribution (BSD) OS, including descendants of BSD. These operating environments can be used to execute one or more browsing programs (such as browsers made by Microsoft or Google or Mozilla) or other browser programs that can access websites hosted by Internet servers discussed below. Moreover, an operating environment according to the principles of the present invention can be used to execute one or more computer game programs.

[0019] A server and / or gateway may be used, which may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, the client and server may be connected via a local intranet or a virtual private network. The server or controller may be connected via a game console (such as Sony personal computer, etc.) to instantiate.

[0020] Information may be exchanged between the client and the server over the network. To this end and for security, the server and / or client may include firewalls, load balancers, temporary storage devices and proxies, and other network infrastructure for reliability and security. One or more servers may form an apparatus for implementing a method of providing a secure community (such as an online social networking site or a player network) to network members.

[0021] The processor may be a single chip processor or a multi-chip processor that may execute logic with the aid of various lines such as address lines, data lines, and control lines, as well as registers and shift registers.

[0022] The components included in one embodiment may be used in other embodiments in any appropriate combination. For example, any of the various components described herein and / or depicted in the accompanying drawings may be combined, interchanged, or excluded from other embodiments.

[0023] “A system having at least one of A, B, and C” (similarly, “a system having at least one of A, B, or C” and “a system having at least one of A, B, C”) includes systems having A alone, B alone, C alone, A and B together, A and C together, B and C together, and / or A, B and C together.

[0024] refer to Figure 1 , an example system 10 is shown, which may include one or more of the example devices mentioned above and further described below in accordance with the principles of the present invention. The first of the example devices included in the system 10 is a consumer electronics (CE) device, such as an audio video device (AVD) 12, such as, but not limited to, an Internet-enabled TV with a TV tuner (equivalently, a set-top box that controls the TV). Alternatively, the AVD 12 may also be a computerized Internet-enabled ("smart") phone, a tablet computer, a notebook computer, a head-mounted device (HMD) and / or a headset (such as smart glasses or VR headsets), another wearable computerized device, a computerized Internet-enabled music player, a computerized Internet-enabled headset, a computerized Internet-enabled implantable device (such as an implantable skin device), etc. In any case, it should be understood that the AVD 12 is configured to perform the principles of the present invention (e.g., communicate with other CE devices to perform the principles of the present invention, execute the logic described herein, and perform any other functions and / or operations described herein).

[0025] Thus, to follow such principles, AVD 12 may be built from some or all of the components shown. For example, AVD 12 may include one or more touch-enabled displays 14, which may be implemented by a high-definition or ultra-high-definition "4K" or higher flat screen. Touch-enabled displays 14 may include, for example, a capacitive or resistive touch sensing layer with an electrode grid for touch sensing consistent with the principles of the present invention.

[0026] AVD 12 can also include one or more speakers 16 for outputting audio according to the principles of the present invention, and at least one additional input device 18, such as an audio receiver / microphone, for inputting audible commands to AVD 12 to control AVD 12. Example AVD 12 can also include one or more network interfaces 20 for communicating through at least one network 22 such as the Internet, WAN, LAN, etc. under the control of one or more processors 24. Therefore, interface 20 can be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12 to perform the principles of the present invention, including other elements of AVD 12 described herein, such as controlling display 14 to present images thereon and receive input therefrom. In addition, it should be noted that network interface 20 can be a wired or wireless modem or router, or other suitable interfaces, such as a wireless telephone transceiver or a Wi-Fi transceiver as mentioned above, etc.

[0027] In addition to the foregoing, the AVD 12 may also include one or more input and / or output ports 26 (e.g., a high-definition multimedia interface (HDMI) port or a universal serial bus (USB) port) to physically connect to another CE device and / or a headphone port to connect headphones to the AVD 12 for presenting audio from the AVD 12 to the user through the headphones. For example, the input port 26 may be connected to a cable TV or satellite source 26a of audio-video content via a wire or wirelessly. Thus, the source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, the source 26a may be a game console or optical disc player containing content. The source 26a, when implemented as a game console, may include some or all of the components described below with respect to the CE device 48.

[0028] AVD 12 may further include one or more computer memory / computer readable storage media 28, such as a disk-based or solid-state storage device that is not a transient signal, which storage device is embodied in the chassis of the AVD as a stand-alone device in some cases, or as a personal video recording device (PVR) or video disc player inside or outside the chassis of the AVD for playing back AV programs, or as a removable storage medium or server described below. Also, in some embodiments, AVD 12 may include a location or positioning receiver, such as but not limited to a cell phone receiver, a GPS receiver and / or an altimeter 30, which is configured to receive geographic location information from a satellite or cell phone base station and provide the information to the processor 24 and / or in conjunction with the processor 24 to determine the altitude at which the AVD 12 is placed. Component 30 may also be implemented by an inertial measurement unit (IMU), which typically includes a combination of motion sensors such as accelerometers, gyroscopes, and magnetometers to determine the position and orientation of the AVD 12 in three dimensions; or by an event-based sensor such as an event detection sensor (EDS) that outputs a binary indication of a change in the direction of a parameter.

[0029] Continuing the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras such as webcams, event-based sensors, and / or cameras integrated into AVD 12 and controlled by processor 24 to collect pictures / images and / or videos according to the principles of the present invention. AVD 12 may also include a Bluetooth transceiver 34 and other near field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technology, respectively. An example NFC element may be a radio frequency identification (RFID) element.

[0030] Still further, the AVD 12 may include one or more auxiliary sensors 38 (e.g., pressure sensors, motion sensors (such as accelerometers, gyroscopes, registers) or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, gesture sensors (e.g., for sensing gesture commands)) that provide input to the processor 24. For example, one or more of the auxiliary sensors 38 may include one or more pressure sensors that form a layer of the touch-enabled display 14 itself, and may be, but are not limited to, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic pressure sensors, etc.

[0031] The AVD 12 may also include an air TV broadcast port 40 for receiving OTA TV broadcasts, thereby providing input to the processor 24. In addition to the foregoing, it should be noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering the AVD 12, which may be a kinetic energy harvester that can convert kinetic energy into electricity to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 may also be included. One or more tactile / vibration generators 47 may be provided for generating tactile signals that can be felt by a person holding the device or in contact with the device. The tactile generator 47 can thus vibrate all or part of the AVD 12 using an electric motor connected to an eccentric and / or unbalanced weight via a rotatable shaft of the motor so that the shaft can rotate under the control of the motor (which in turn can be controlled by a processor such as processor 24) to produce vibrations of various frequencies and / or amplitudes in various directions as well as force simulation.

[0032] In addition to the AVD 12, the system 10 may also include one or more other CE device types. In one example, the first CE device 48 may be a computer game console that can be used to send computer game audio and video to the AVD 12 via commands sent directly to the AVD 12 and / or through a server described below, while the second CE device 50 may include components similar to the first CE device 48. In the example shown, the second CE device 50 may be configured as a computer game controller controlled by a player or a head-mounted display (HMD) worn by a player. The HMD may include a heads-up transparent or opaque display for presenting AR / MR content or VR content, respectively.

[0033] In the example shown, only two CE devices are shown, and it should be understood that fewer or more devices may be used. The devices herein may implement some or all of the components shown for AVD 12 and / or CE devices. Any of the components shown in the following figures may incorporate some or all of the components shown in the case of AVD 12.

[0034] Referring now to the at least one server 52 described above, the at least one server includes at least one server processor 54, at least one tangible computer-readable storage medium 56 (such as a disk-based or solid-state storage device), and at least one network interface 58, which allows communication with other illustrated devices over the network 22 under the control of the server processor 54, and can actually facilitate communication between the server and client devices in accordance with the principles of the present invention. It should be noted that the network interface 58 can be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as a wireless telephone transceiver.

[0035] Thus, in some embodiments, server 52 may be an Internet server or an entire server "cluster" and may include and perform "cloud" functionality such that the devices of system 10 may access a "cloud" environment via server 52 in an example embodiment for, for example, a networked gaming application. Alternatively, server 52 may be implemented by one or more gaming consoles or other computers in the same room or nearby as the other devices shown.

[0036] The components shown in the following figures may include some or all of the components shown herein.Any user interface (UI) described herein may be merged and / or extended, and UI elements may be mixed and matched between UIs.

[0037] The present invention principle can adopt various machine learning models, including deep learning models. The machine learning model consistent with the present invention principle can use various algorithms trained in various ways, including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning and other forms of learning. Examples of such algorithms that can be implemented by computer circuit systems include one or more neural networks, such as convolutional neural networks (CNNs), recursive neural networks (RNNs) and a class of RNNs called long short-term memory (LSTM) networks. Support vector machines (SVMs) and Bayesian networks can also be considered as examples of machine learning models. In addition to the types of networks described above, the models herein can also be implemented by classifiers.

[0038] As understood herein, performing machine learning may thus involve accessing training data and then training a model based on the training data so that the model can process further data to make inferences. An artificial neural network / artificial intelligence model trained by machine learning may thus include an input layer, an output layer, and a plurality of hidden layers between the input layer and the output layer, the hidden layers being configured and weighted to make inferences about appropriate outputs.

[0039] Reference now Figure 2Beginning at block 200, an incoming voice signal from, for example, a voice chat computerized communication session may be received at a user terminal, such as may be instantiated by any of the apparatus described herein. Continuing to decision diamond 202, it is determined whether the voice signal reveals an emotion that satisfies a threshold, and if not, then at block 204, the voice is played on a speaker.

[0040] On the other hand, if the emotion in the speech is assessed to meet the threshold, then the logic flows from decision diamond 202 to block 206 to filter or otherwise modify the speech signal to reduce the emotion (if not completely eliminate the emotion), thereby playing the modified version at block 208.

[0041] Although Figure 2 involves detecting emotions in incoming speech, but Figure 3 It involves allowing users to process their voice to prevent excessive emotions in outgoing chats. Starting from decision diamond 300, it is determined whether the user has enabled self-filtering, if not, the process ends at state 302.

[0042] When self-filtering is enabled, the logic moves to block 304 to pick up the user's voice through a microphone (e.g., a computer microphone of a computer terminal such as any device used for voice chat herein). Continuing to decision diamond 306, it is determined whether the voice signal leaks an emotion that meets the threshold, and if not, the voice is transmitted as is to the recipient device from a communication interface (e.g., any interface described herein) at block 308.

[0043] On the other hand, if the emotion in the speech is assessed to meet the threshold, then the logic flows from decision diamond 306 to block 310 to filter or otherwise modify the speech signal to reduce the emotion (if not completely eliminate the emotion), thereby transmitting the modified version at block 312.

[0044] Figure 4 A UI is shown that may be presented on display 400 (e.g., any display herein) for allowing a user to identify Figure 2 A prompt 402 may be selected to enable speech filtering of incoming speech to reduce, if not eliminate, a particular type of emotion. A list 404 of emotions may be presented from which the user may select one or more emotions to filter.

[0045] Figure 5 A UI is shown that may be presented on display 400 (e.g., any display herein) for allowing a user to identify Figure 35. A prompt 500 may be selected to enable speech filtering of the user's outgoing speech to reduce, if not eliminate, a particular type of emotion. A list 502 of emotions may be presented from which the user may select one or more emotions to filter. As shown, the list 502 may include the emotions themselves as well as speech attributes such as "loudness," the selection of which will cause the outgoing speech signal to be modulated in volume to remain below a threshold volume.

[0046] Figure 6 A UI is shown that can be presented on a display 400 (e.g., any display herein) for allowing a user to identify how speech is filtered to reduce emotion. A prompt 600 may be presented to the user to select a filtering technique. In the example shown, the user may select at 602 to play the speech that triggers the emotion threshold in a generally pitchless robotic voice. The user may also select at 604 to reduce the volume of the emotional speech only. The user may also select at 606 to play the speech in its received timbre, but where the speech is flat or otherwise played with reduced emotion. Generative adversarial networks (GANs) are an example technology for such speech filtering.

[0047] It should be noted that in response to an emotion triggering a threshold, the system may, by default, perform speech-to-text on the emotional speech and reproduce the text in a robotic voice.

[0048] Figure 7 A UI is shown that can be presented on display 400 (e.g., any display herein) for allowing a user to identify specific users to filter if their speech contains an emotion that meets a threshold. The UI includes a prompt 700 for the user to identify other users to filter, followed by a list 702 of individual users plus an "all" selector to filter the emotion of all users' speech.

[0049] Figure 8 Techniques are shown for training one or more machine learning (ML) models, such as one or more CNNs, to detect specific emotions in speech and when those emotions meet thresholds for employing the filtering / altering techniques described herein. Beginning at block 800, a training set of recorded human speech is input to one or more ML models. In addition to the speech recordings, the speech is live annotated and input to the ML model to train the model. The live may include specific emotions annotated for each speech, and an indication of whether the specific speech with the annotated live emotion is considered to meet the filtering threshold. At block 802, the ML model is trained based on the training set input at block 800.

[0050] Fig. 9The indication is then at block 900, the speech signal (incoming or outgoing, as the case may be) is input to the trained ML model. Continuing to block 902, an indication is received back from the output of the ML model indicating whether the speech should be altered in accordance with the disclosure herein. At block 904, the speech is processed in accordance with the indication, i.e., if the indication is that the speech contains an effect of an emotion that satisfies a threshold, then the speech is altered. At block 906, the speech is played (or transmitted, as the case may be).

[0051] Although the UI herein is shown as enabling a user to define specific emotions, users, and filtering techniques, it should be understood that all incoming speech may be automatically filtered for predetermined emotions using predetermined techniques, for example, thereby alleviating the selection burden on the user.

[0052] While particular embodiments are herein shown and described in detail, it is to be understood that the subject matter encompassed by the present invention is limited only by the claims.

Claims

1. A device comprising: At least one computer storage device that is not a transitory signal and that includes instructions executable by at least one processor to: receiving an incoming signal from a computerized communications session, the signal representing speech of at least a first participant in the computerized communications session; determining at least a first emotion based at least in part on the signal; and The playback of the speech is altered to reduce the first emotion present during the playback.

2. The apparatus of claim 1 , wherein the instructions are executable to: Based at least in part on the signal, determining at least a second emotion; and The playback of the speech is not altered to reduce the second emotion that occurs during the playback.

3. The apparatus of claim 2, wherein the instructions are executable to: An input is received that specifies the first emotion as an emotion to be altered upon playback and the second emotion as an emotion not to be altered upon playback.

4. The apparatus of claim 1 , wherein the instructions are executable to: receiving an incoming signal from the computerized communications session, the signal representing speech of at least a second participant in the computerized communications session; Based at least in part on the signal, determining at least the first emotion; and The playback of the second participant's speech having the first emotion is not altered when the second participant's speech is played back.

5. The apparatus of claim 4, wherein the instructions are executable to: An input is received, the input specifying the first participant whose emotion should be altered and the second participant whose emotion should not be altered.

6. The apparatus of claim 4, wherein the instructions are executable to: The playback of the speech of the first participant is altered at least in part by playing the speech after filtering the speech through a signal processing circuit to reduce the first emotion present during the playback.

7. The apparatus of claim 4, wherein the instructions are executable to: A playback of the speech is altered, at least in part, by playing utterances spoken by the first participant in a robotic voice to reduce the first emotion present during the playback.

8. The apparatus of claim 1, comprising the at least one processor programmed with the instructions.

9. A method comprising: receiving a first emotion specified by a user as an emotion to be reduced during playback; as well as A speech signal in a computerized communication system is altered to reduce the first emotion in the speech signal when the speech signal is played back.

10. The method of claim 9, wherein the speech signal is received by a receiver, and the method comprises: The speech signal is altered after receiving the speech signal and before playing back the speech signal to reduce the first emotion.

11. The method of claim 9, comprising altering the speech signal to reduce the first emotion prior to transmitting the speech signal.

12. The method according to claim 9, comprising: Identify at least a second emotion; as well as The playback of the speech is not altered to reduce the second emotion that occurs during the playback.

13. The method of claim 12, wherein the instructions are executable to: An input is received that specifies the first emotion as an emotion to be altered upon playback and the second emotion as an emotion not to be altered upon playback.

14. The method according to claim 9, comprising: receiving an incoming signal from a computerized communications session, the signal representing speech of at least a second participant in the computerized communications session; determining at least the first emotion based at least in part on the signal; as well as The playback of the second participant's speech having the first emotion is not altered when the second participant's speech is played back.

15. The method according to claim 14, comprising: An input is received, the input specifying the first participant whose emotion should be altered and the second participant whose emotion should not be altered.

16. The method according to claim 9, comprising: The speech signal is altered at least in part by filtering the speech signal with a signal processing circuit.

17. The method according to claim 9, comprising: The speech signal is altered at least in part by playing the utterance in a robotic voice.

18. An apparatus comprising: At least one microphone; at least one transceiver; as well as At least one processor programmed with instructions to: receiving an input specifying a first emotion to be reduced in outgoing utterances received from the microphone prior to transmitting the utterances through the transceiver; in response to detecting the first emotion in outgoing utterance, altering the utterance to reduce the first emotion therein, thereby revealing altered utterance; and The altered utterance is transmitted through the transceiver.

19. The apparatus of claim 18, wherein the instructions are executable to: In response to detecting a second emotion in the outgoing utterance, the utterance is not altered prior to transmitting the utterance to reduce the second emotion therein.

20. The apparatus of claim 18, wherein the instructions are executable to: The outgoing utterance is altered by transmitting a robotic version of the outgoing utterance.