Method and device for managing retention

Abstract

A method and apparatus for managing hold are disclosed. When a voice communication session is in a hold state, the session is automatically monitored to determine when the session is no longer in a hold state. When it is determined that the session is no longer in a hold state, a user interface output is rendered that is perceptible to a calling user who initiated the session and indicates that the hold state of the session has stopped. In some implementations, an audio stream of the session can be monitored to determine a candidate end of the hold state based on processing of the audio stream. In response, a response solicitation signal is injected into an outgoing portion of the audio. The audio stream can be further monitored for a response to the response solicitation signal (if any). The response (if any) can be processed to determine whether the end of the hold state is an actual end of the hold state.

Description

[0001] Description of the case

[0002] This application is a divisional application. The application number of the original application is 201880094616.6, the application date is June 28, 2018, and the name of the invention is “Method and device for management and maintenance”. Technical Field

[0003] The present application relates to methods and apparatus for managing retention. Background Art

[0004] Humans can participate in voice communication sessions (e.g., phone calls) using various client devices. When an individual (referred to herein as a "caller" or "user") calls a specific number and no personnel are currently available to answer the call, many organizations can put the caller on hold. The hold state indicates that the caller is waiting to interact with a living person (also referred to herein as a "user"). Music is often played for the user while the user is on hold. In addition, the music may be interrupted by various human recorded voices that can provide additional information, such as information about the organization the user is calling (e.g., the organization's website, the organization's normal business hours, etc.). In addition, an automated voice can update the user with an estimated remaining wait time to indicate how long the user will continue to be on hold.

[0005] When a call is on hold, the caller must closely monitor the call to determine when a second user, such as a service representative, becomes active in the call. For example, when the music on hold switches to a human voice, the caller must determine whether the voice they are hearing is a pre-recorded voice or a living service representative. In order to enable close monitoring of the call on hold initiated via the client device, the caller can turn up the call volume, place the audio output of the call in a hands-free phone mode, and / or repeatedly activate the screen of the client device when the call is on hold (to check to ensure that the call is still active and on hold). Those and / or other monitoring activities of the caller on hold may increase the power consumption of the client device. For example, such activities may increase the power consumption of the mobile phone used for the call, which may cause the battery of the mobile phone to be consumed quickly. In addition, those and / or other monitoring activities on hold may require the caller to perform a large amount of input on the client device, such as input for increasing the volume, activating the hands-free phone mode, and / or activating the screen. Summary of the invention

[0006] Implementations described herein relate to automatic monitoring of a voice communication session while the session is on hold to determine when the session is no longer on hold. When it is determined that the session is no longer on hold, a user interface output that is perceptible to a calling user initiating the session is rendered, the user interface output indicating that the hold state of the session has stopped. In various implementations, a client on hold (e.g., operating at least in part on a client device initiating the voice communication session) can be used to monitor at least an incoming portion of an audio stream of the session to determine when the session is no longer on hold. In some of those various implementations, the client on hold determines a candidate end of the hold state based on processing of the audio stream. The candidate end of the hold state can be based on detecting the occurrence of one or more events in the audio stream. As some non-limiting examples, candidate ends of a hold state can be based on detecting transitions in the audio stream (e.g., any transition or transition from “music on hold” to human speech), detecting any human speech (e.g., using voice activity detection), detecting new human speech (e.g., using speaker diarization), detecting occurrences of certain words and / or phrases (e.g., “hello,” “hi,” and / or the name of the calling user), and / or other events.

[0007] In certain versions of those various implementations, in response to detecting a candidate end of the hold state, the client on hold causes a response solicitation signal to be injected into the outgoing portion of the audio stream (so that it can be "heard" by the called party). The response solicitation signal can be a recorded human voice speaking one or more words or a synthetically generated voice speaking one or more words. The one or more words can be, for example, "Hello", "Are you there", "Hi, are you on the line", etc. The client on hold can further monitor the response to the response solicitation signal (if any) and determine whether the response indicates that the candidate end of the hold state is the actual end of the hold state. If so, the client on hold can render a user interface output that is perceptible to the calling user who initiated the session and indicates that the hold state of the session has stopped (i.e., the voice communication session is no longer on hold). If not, the client on hold can continue to monitor for another occurrence of a candidate end of the hold state. In some implementations, the client on hold determines whether the response indicates a candidate end of the hold state is an actual end of the hold state by: determining a likelihood that the response is human speech; converting the response to text (e.g., using a speech-to-text processor) and determining whether the text is responsive to a response solicitation signal; determining that the response is not prerecorded speech (e.g., includes speech characteristics that are different from speech characteristics of prerecorded speech for the voice communication session); and / or, based on one or more other criteria. The client on hold may optionally choose to use a trained machine learning model to determine the likelihood that the response is human speech.

[0008] In these and other ways, the client on hold can monitor the incoming portion of the audio stream of the session on hold and dynamically determine when to provide a response solicitation signal. In addition, the client on hold can utilize the response to the response solicitation signal (if any) when determining whether the hold state of the session has stopped. These actions performed by the client on hold can be performed without any intervention from the calling user and without requiring the client device to audibly render the audio stream of the voice communication session. In addition, as described herein, in various implementations, the client on hold can be automatically initiated (without any user input) or with minimal user input (e.g., by clicking a graphical element or a single spoken command).

[0009] The voice communication session can utilize various protocols and / or infrastructures, such as any of the Voice over Internet Protocol (VOIP), the Public Switched Telephone Network (PSTN), a Private Branch Exchange (PBX), various video and / or audio conferencing services. In various implementations, the voice communication session is between the client device of the calling user (initiating the voice communication session) and one or more devices of the called party. The voice communication session enables two-way audio communication between the calling user and the called party. The voice communication session can be a direct peer-to-peer session between the client device of the calling user and the device of the called party, and / or can be routed through various servers, networks, and / or other resources. The voice communication session can occur between various devices. For example, the voice communication session can be between: the client device of the calling user (e.g., a mobile phone, a stand-alone interactive speaker, a tablet computer, a laptop computer) and the landline phone of the called party; the client device of the calling user and the client device of the called party; the client device of the calling user and the PBX of the called party; and so on.

[0010] In some implementations described herein, a client on hold that operates at least partially on a client device may be initiated in response to the client device detecting that a voice communication session (initiated by the client device) has been placed on hold. A client device such as a mobile phone may examine the audio stream of a voice communication session and determine that the session is on hold in various ways. As an example, the client device may determine that the session is on hold based on detecting music such as typical "music on hold" in an incoming portion of the audio stream. For example, the incoming portion of the audio stream may be processed and compared to a list of music known to be on hold (e.g., the audio characteristics of the audio stream may be compared to the audio characteristics of music known to be on hold) to determine whether the incoming portion of the audio stream is typical music on hold. Such a list may be stored locally on the client device and / or on a remote server to which the client device may be connected via a network (e.g., a cellular network). Additionally or alternatively, the incoming portion of the audio stream may be processed and compared to a list of voices known to be on hold. As another example, the client device may determine that the session is on hold based on detecting any music in the incoming portion of the audio stream. As yet another example, a client device may additionally or alternatively determine that a session is on hold based on comparing the dialed number of the session to a list of phone numbers that are known to put callers on hold. For example, if a user calls "Hypothetical Utility Company," the client device may store the phone number associated with "Hypothetical Utility Company" as a number that typically puts callers on hold before the user can speak to a live representative. Further, the list of phone numbers that are known to put callers on hold may have a corresponding list of music that is known to be on hold and / or voice that is known to be on hold used by the number. Additionally or alternatively, a user may provide to a client device phone numbers that typically put the user on hold. With the user's permission, these user-provided phone numbers may be shared across client devices and may be added to the list of numbers that typically put people on hold on other client devices.

[0011] In some implementations, the user can indicate to the client device that they have been placed on hold. In some versions of those implementations, the client device can detect that the user may be on hold and provide a user interface output (e.g., an optional graphical element and / or an audible prompt) that prompts the user whether to initiate a client on hold. If the user responds with an affirmative user interface input (e.g., a selection of an optional graphical element and / or a spoken affirmative input), the client on hold can be initiated. In some other versions of those implementations, the user can initiate a client on hold without the client device detecting that the user may be on hold and / or without the client device prompting the user. For example, the user can provide a spoken command (e.g., "Assistant, initiate on hold monitoring") to initiate a client on hold and / or can select an optional graphical element, the presence of which does not depend on determining that the user may be on hold. In many implementations, the client device can monitor the audio stream of the entire voice communication session and detect whether the user has been placed on hold at a point other than the start of the session. For example, the user can interact with a representative who places the user on hold when transferring the session to a second representative. Note that in various implementations, a client on hold may operate in the background to detect when a voice communication session has been placed on hold, and may be "initiated" (e.g., transitioned to an "active" state), where it then performs other aspects of the present disclosure (e.g., detects when the voice communication session is no longer on hold).

[0012] When the client on hold is initiated, the client on hold can monitor at least an incoming portion of the audio stream of the voice communication session to determine when the voice communication session is no longer on hold. When the session is no longer on hold, the calling user can interact with a live person such as a company representative, a receptionist at a doctor's office, etc. The monitoring of the audio stream of the voice communication session using the client on hold can be performed without direct interaction from the user (e.g., the user need not listen to the session while the session is on hold).

[0013] In some implementations, the client on hold can determine when to change the music on hold to a human voice. This human voice can sometimes be a recording of a person, so the client on hold will determine whether a recording is being played or whether a living person has joined the session. In various implementations, the client on hold can ask a question to the detected voice (referred to herein as a "response solicitation signal") and see if the voice responds to the question. For example, when a human voice is detected in the audio signal of the session, the client on hold can ask "Are you there?" and see if the voice responds to the question. The appropriate response to the question initiated by the client on hold indicates that the hold is over and the second person has joined the session. In other implementations, the question is ignored, and the client on hold can determine that the second person has not yet joined the session. For example, if the client on hold sends "Is anyone there?" as an input to the audio signal and does not receive a response (for example, on the contrary, the music on hold continues to play), it can indicate that the voice is a recording and the session is still on hold.

[0014] In some implementations, a "candidate end of a hold event" can be used to determine when a hold may end. In many implementations, the candidate end of the hold event can initiate the client on hold to send a response solicitation signal on the audio channel of the session to see if the voice is human. The candidate end of the hold event can be detected in a variety of ways. For example, the client device can detect when the music stops playing and / or when the person starts talking. Various audio fingerprint processes including discrete Fourier transform (DFT) can be used to determine the change from music to people speaking. DFT can monitor blocks of the session on hold and determine when enough changes from a block are detected compared to the previous block (for example, a block when the music stops playing is detected and a change from music to human voice is detected in an additional block). In various implementations, one or more machine learning models can be trained and used to determine when the session on hold changes from audio to human voice.

[0015] In many implementations, the threshold for determining when to ask a question via an audio signal (sometimes referred to as a "response solicitation signal") is low, and since asking questions takes little computing resources, a client on hold will often ask questions (without fault if a human is not currently on the other end of the conversation). In some of those implementations, a first machine learning model may be used to detect candidate ends of a hold event and determine when to ask questions as input to the audio signal. Determining whether a response is detected may require more computing resources, and in various implementations, a second machine learning model (in addition to the first machine learning model) may determine whether a person has responded to the question. The second machine learning model for detecting whether a person has responded to a response solicitation signal may be stored locally on the client device and / or stored externally to the client device, such as on one or more remote computing systems, often referred to as the "cloud". In some implementations, a client on hold may use a single machine learning model to combine processing of all parts of a session on hold. In some of those implementations, a machine learning model may be used to process an audio stream and provide an output indicating the likelihood that the session is on hold. In some versions of those implementations, a first, more easily satisfied likelihood threshold may be used to determine a candidate end of a hold state, while a second, more difficult to satisfy likelihood threshold may be used to determine the actual end of the hold state.

[0016] In various implementations, one or more machine learning models may utilize an audio stream as input, and one or more models may generate various outputs, including a determination that a voice communication session has been placed on hold, a determination that the hold has potentially ended and a response solicitation signal should be sent as input to the audio stream of the voice communication session, and / or a determination that the voice communication session on hold has ended and a response solicitation signal does not have to be sent. In some implementations, a single machine learning model may perform all audio stream analysis for a client on hold. In other implementations, the outputs of different machine learning models may be provided to a client on hold. Additionally or alternatively, in some implementations, portions of a client on hold may provide input to and / or receive output from one or more machine learning models, while portions of the client on hold do not interact with any machine learning model.

[0017] Additionally or alternatively, some voice communication sessions can verbally indicate the estimated remaining hold time when they are on hold. In many implementations, the client on hold can determine the estimated remaining hold time by analyzing the natural language within the audio stream of the voice communication session, and can indicate the estimated remaining hold time to the user. In some such implementations, the estimated remaining hold time can be rendered to the user as a dialog box on a client device with a display screen, such as a push pop-up message, which expresses "Your on hold call with "Hypothetical Water Company" has provided an updated remaining estimated hold of 10 minutes (your call on hold with "Hypothetical Water Company" has provided an updated remaining estimated hold of 10 minutes). "This message can appear on the client device in a variety of ways, including as part of the client on hold, as a new pop-up window on the screen, as a text message, etc. In addition, the client device can additionally or alternatively use one or more speakers associated with the client device to render this information to the user as a verbal indication. In some implementations, the client on hold can learn the average time that the user spends on hold with a known number, and supply the average hold time (e.g., with a countdown) to the user when a more specific estimate is unknown. When using an audio stream as input, a machine learning model associated with a client on hold can learn when an estimated remaining hold length has been indicated in the audio stream and / or learn an estimated hold time for a known number.

[0018] The machine learning model may include a feedforward neural network, a recurrent neural network (RNN), a convolutional neural network (CNN), etc. The machine learning model may be trained using a supervised training data set with labeled outputs corresponding to given inputs. In some implementations, a labeled audio stream of a set of previously recorded voice communication conversations on hold may be used as a training set for the machine learning model.

[0019] In various implementations, the client on hold can utilize speaker segmentation, which can partition the audio stream of the conversation to detect a single voice. Speaker segmentation is the process of dividing the input audio stream into homogeneous segments according to the speaker identity. It answers the question of "who is speaking when" in a multi-speaker environment. For example, speaker segmentation can be used to identify that the first segment of the input audio stream is attributable to the first human speaker (without specifically identifying who the first human speaker is), the second segment of the input audio stream is attributable to a different second human speaker (without specifically identifying who the first human speaker is), the third segment of the input audio stream is attributable to the first human speaker, and so on. When a specific voice is detected, the client on hold can query the voice to see if the client on hold receives a response. If the voice does not respond to the question of the client on hold (e.g., "Hello, are you there?"), the client on hold can determine that the recognized voice is a recording, not an indication of the end of the hold. The client on hold can learn the specific voice as a recording of the voice, and if the voice is heard again during the period when the voice communication session is on hold, the voice will be ignored. For example, speech characteristics of a particular voice and / or words spoken by a particular voice may be identified, and future occurrences of these speech characteristics and / or words in a voice communication session may be ignored. In other words, many times when a person is on hold, the recording played for the user will be a loop that includes music interrupted by the same recording (or one of multiple recordings). If a voice communication session is on hold and loops back to the same identified voice, the voice identified as a recording in the held recording loop will be ignored (i.e., the same voice will not be prompted again with a question). In some such implementations, the recorded voice may be shared across many client devices as a known voice recording.

[0020] In some implementations, the content detected on the audio signal will be a strong indicator that a human user is on the call who does not need to respond to the solicitation signal. For example, if the client on hold detects one of a list of keywords and / or phrases such as the caller's first name, the caller's last name, the caller's full name, etc., the client on hold can determine that a living human user is on the call without asking any questions through the audio stream of the voice communication session. Additionally or alternatively, service representatives often follow scripts when interacting with users. The client on hold can monitor the typical scripted greetings from service representatives of a particular company to identify the end of the hold without sending questions through the audio stream. For example, suppose a user calls "Hypothetical Utility Company" at a particular number. The client on hold can learn the scripted responses that the service representatives at "Hypothetical Utility Company" use when the service representatives answer the voice communication session. In other words, the client on hold may learn that the service representative at "Hypothetical Utility Company" begins a voice communication session with the user after the hold ends with a scripted message such as "Hello, my name is [service representative's name] and I work with Hypothetical Utility Company. How may I help you today?" Detecting the scripted message may trigger the end of the client on hold without further querying the voice and seeing if it is a live second user.

[0021] Once the client on hold detects the end of the hold, in various implementations, the client on hold can send a scripted message to the second user who is now also active in the session. For example, the client on hold can send a message expressing "Hello, I represent Jane Doe. I am notifying her and she will be here momentarily. (Hello, I represent Jane Doe. I am notifying her and she will be here soon.)". This message helps to keep the second user on the call when notifying the end of the hold of the user who initiated the session. Additionally or alternatively, the voice communication session can be switched to another client instead of switching back to the user to interact with the session. In some such implementations, another client can use known information about the user and / or information about a specific voice communication session provided by the user to another client to interact with the voice communication session. For example, a user can provide another client with information about when they want to book dinner at "Hypothetical Fancy Restaurant", and another client can interact with a human user with an attached life to make a dinner reservation for the user.

[0022] In many implementations, when the client on hold determines that the hold state has ended (that is, the hold ends and the human is on the call), the user who initiated the session is notified. In some implementations, the user can choose how they want to be notified while or near the client on hold. In other implementations, the user can choose how they want to be notified as a setting in the client on hold. The client device itself can be used to notify the user. For example, the client device can notify the user by causing the client device to render a ringtone, causing the client device to vibrate, causing the client device to provide a spoken output (for example, "you are no longer on hold"), etc. For example, the client device can vibrate when the hold ends, and the user can press a button on the client device to start interacting with the session.

[0023] Additionally or alternatively, the client on hold can notify the user through one or more other client devices and / or peripheral devices (e.g., Internet of Things (IoT) devices) that are shared on the same network and / or form part of the same coordinated "ecosystem" of the client device controlled by the user. The client on hold can be aware of other devices on the network through the device topology. For example, if the client on hold knows that the user is in a room with smart lights, the user can choose to be notified by changing the state of the smart lights (e.g., flashing lights on and off, dimming lights, increasing light intensity, changing light color, etc.). As another example, a user who is engaged with a display screen such as a smart TV can choose to be notified by a message that appears on the smart TV display screen. In other words, the user can watch TV while the session is on hold, and can be notified by the client on hold via its TV that the hold is over, so the user can re-enter the session. As another example, a voice communication session can be conducted via a mobile phone, and notifications can be rendered via one or more smart speakers and / or other client devices. In various implementations, the client device for the voice communication session can be a mobile phone. Alternative client devices can be used for voice communication sessions. For example, a client device for a voice communication session may include a dedicated automated assistant device (e.g., a smart speaker and / or other dedicated assistant device) with the ability to conduct a voice communication session for a user.

[0024] Implementations disclosed herein can enhance the usability of client devices by reducing the time that client devices interact with voice communication sessions on hold. Computing resources can be saved by running client processing on hold in the background of computing devices instead of client devices interacting completely with voice communication sessions on hold. For example, many users will output voice communication sessions on hold through speakers associated with client devices. Compared with outputting sessions on speakers, background monitoring of voice communication sessions requires client devices to perform less computing processing. Additionally or alternatively, when compared with outputting voice communication sessions on hold through one or more speakers associated with client devices (which may also include the output of audio streams that users can hear when client devices are close to his or her ears and the audio streams of voice communication sessions on hold output by external speakers), executing the holding process in the background of client devices can save the battery life of client devices.

[0025] The above is provided as an overview of various implementations disclosed herein. Additional details regarding those various implementations as well as additional implementations are provided herein.

[0026] In some implementations, a method implemented by one or more processors is provided, the method including detecting that a voice communication session is in a hold state. The voice communication session is initiated by a client device of a calling user, and detecting that the voice communication session is in a hold state is based at least in part on an audio stream of the voice communication session. The method also includes initiating a client on hold on the client device. Initiating the client on hold is during the voice communication session and is based on detecting that the voice communication session is in a hold state. The method also includes monitoring the audio stream of the voice communication session for a candidate end of the hold state using the client on hold. Monitoring the audio stream of the voice communication session occurs without requiring direct interaction from the calling user. The method also includes detecting a candidate end of the hold state based on the monitoring. The method also includes: in response to detecting the candidate end of the hold state: sending a response solicitation signal from the client device as an input to the audio stream of the voice communication session; monitoring the audio stream of the voice communication session for a response to the response solicitation signal; and determining that a response to the response solicitation signal indicates that the candidate end of the hold state is an actual end of the hold state. The actual end of the hold state indicates that a human user is available to interact with the calling user in the voice communication session. The method also includes causing a user interface output to be rendered in response to determining the actual end of the hold state. The user interface output is perceptible to the calling user and indicates the actual end of the hold state.

[0027] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0028] In some implementations, detecting a candidate end of the hold state includes detecting a human voice speaking in an audio stream of the voice communication session.

[0029] In some implementations, the client device is a mobile phone or a stand-alone interactive speaker.

[0030] In some implementations, initiating the client on hold is responsive to a user interface input provided by a calling user at a client device. In some versions of those implementations, the method further comprises: in response to detecting that the voice communication session is on hold: rendering at the client device a suggestion for initiating the client on hold. In those versions, the user interface input provided by the calling user is an affirmative user interface input, the affirmative user interface input being provided in response to rendering the suggestion at the client device.

[0031] In some implementations, placing the voice communication session on hold is automatically initiated by the client device in response to detecting that the voice communication session is on hold.

[0032] In some implementations, detecting that the voice communication session is on hold includes detecting music in an audio stream of the voice communication session and optionally determining that the music is included in a list of music known to be on hold.

[0033] In some implementations, detecting that the voice communication session is on hold is further based on determining that a telephone number associated with the voice communication session is on a list of telephone numbers that are known to place callers on hold.

[0034] In some implementations, detecting a candidate end of the hold state includes determining at least a threshold change in the audio stream using an audio fingerprint.

[0035] In some implementations, determining that a response to a response solicitation signal indicates a candidate end of a hold state is the actual end of the hold state includes: processing the response using at least one machine learning model to generate at least one prediction output; and based on the at least one prediction output, determining that the candidate end of the hold state is the actual end of the hold state. In some versions of those implementations, at least one prediction output includes predicted text for the response, and determining that the candidate end of the hold state is the actual end of the hold state based on the prediction output includes determining that the text is in response to the response solicitation signal. In some additional or alternative versions of these implementations, at least one prediction output includes a prediction of whether the response is human speech, and determining that the candidate end of the hold state is the actual end of the hold state based on the prediction output includes determining that the prediction of whether the response is human speech indicates that the response is human speech.

[0036] In some implementations, the method further includes, after determining that the response to the response solicitation signal indicates that the candidate end of the hold state is the actual end of the hold state: sending a hold end message from the client device as input to the audio stream of the voice communication session. The hold end message is audible to a human user and indicates that the calling user is returning to the voice communication session. In some of those implementations, the method further includes: after determining that the response to the response solicitation signal indicates that the candidate end of the hold state is the actual end of the hold state, ending the client on hold on the client device.

[0037] In some implementations, a user interface output indicating the actual end of the hold state is rendered via the client device, an additional client device linked to the client device, and / or a peripheral device (eg, a networked light).

[0038] In some implementations, the method further includes identifying one or more pre-recorded voice characteristics of a pre-recorded human voice associated with a telephone number (or other unique identifier) ​​associated with the voice communication session. In some versions of those implementations, determining that a response to the response solicitation signal indicates a candidate end of the hold state is an actual end of the hold state includes: determining one or more response voice characteristics of the response; and, determining that the one or more response voice characteristics are different from the one or more pre-recorded voice characteristics.

[0039] In some implementations, a method implemented by one or more processors is provided, the method comprising receiving a user interface input provided via a client device. The user interface input is provided by a calling user when a voice communication session is in a hold state. The voice communication session is initiated by the client device, and the called party controls the hold state of the voice communication session. The method also includes: in response to receiving the user interface input: monitoring a candidate end of the hold state from audio generated by the called party during the voice communication session. The method also includes detecting the candidate end of the hold state based on the monitoring. The method also includes: in response to detecting the candidate end of the hold state: sending an audible output by the client device to be included in the voice communication session. The audible output includes a recorded human voice speaking one or more words or a synthetically generated voice speaking one or more words. The method also includes: monitoring the audio generated by the called party after the audible output; and determining that the audio generated by the called party after the audible output meets one or more criteria, the criteria indicating that the candidate end of the hold state is the actual end of the hold state. The actual end of the hold state indicates that a human user is available to interact with the calling user in the voice communication session. The method also includes causing the user interface output to be rendered in response to determining the actual end of the hold state. The user interface output is perceptible to the calling user and indicates the actual end of the hold state.

[0040] These and other implementations of the technology may optionally include one or more of the following features.

[0041] In some implementations, determining that audio generated by the called party following the audible output meets one or more criteria includes: generating text by performing speech-to-text conversion on the audio generated by the called party following the audible output; and determining that the text is in response to one or more words in the audible output.

[0042] In some implementations, the user interface input is an affirmative response to a graphical and / or audible suggestion rendered by the client device, wherein the suggestion is a suggestion to initiate the client on hold to monitor the hold end state. In some of those implementations, the suggestion is rendered by the client device in response to detecting that the call is on hold based on audio generated by the called party during the voice communication session.

[0043] In some implementations, a method implemented by a client device that initiates a voice communication session is provided, the method comprising: while the voice communication session is in a hold state: monitoring an audio stream of the voice communication session for the occurrence of human voice speaking in the audio stream; in response to detecting the occurrence of human voice during monitoring: sending a response solicitation signal as an input to the audio stream; monitoring the audio stream for a response to the response solicitation signal; determining whether the response to the response solicitation signal is a human response in response to the solicitation signal; and when it is determined that the response is a human response in response to the response solicitation signal: rendering a user interface output that enables the calling user to perceive and indicates that the hold end state is present.

[0044] In addition, some implementations include one or more processors of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in an associated memory, and wherein the instructions are configured to cause any of the foregoing methods to be performed. Some implementations also include one or more non-transitory computer-readable storage media storing computer instructions executable by the one or more processors to perform any of the foregoing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a block diagram illustrating an example environment in which various implementations may be implemented.

[0046] Figure 2 is a diagram illustrating an example interaction between a client device and a voice communication session.

[0047] Figure 3 is a diagram illustrating another example interaction between a client device and a voice communication session.

[0048] Figure 4 is a diagram illustrating another example interaction between a client device and a voice communication session.

[0049] Figure 5 is a flow chart illustrating an example process according to implementations disclosed herein.

[0050] Figure 6 is a block diagram illustrating an example architecture of a computing device. DETAILED DESCRIPTION

[0051] Figure 1 An example environment 100 is shown in which various implementations may be implemented. The example environment 100 includes one or more client devices 102. For the sake of brevity and simplicity, the term "on-hold client" as used herein to "serve" a particular user may generally refer to a combination of an on-hold client 104 operated by a user on a client device 102 and one or more cloud-based on-hold components (not shown).

[0052] The client device 102 may include, for example, one or more of a desktop computing device, a laptop computing device, a touch-sensitive computing device (e.g., a computing device that can receive input via touch from a user), a mobile phone computing device, a computing device of a user's vehicle (e.g., an in-vehicle communication system), a standalone interactive speaker, a smart device such as a smart TV, a projector, and / or a wearable device of a user including a computing device (e.g., a watch of a user with a computing device, glasses of a user with a computing device, a virtual or augmented reality computing device, etc.). Additional and / or alternative computing devices may be provided.

[0053] In some implementations, the client 104 on hold can participate in a conversation session in response to a user interface input, even when the user interface input is not explicitly related to the client 104 on hold. For example, the client 104 on hold can examine the content of the audio stream of the voice communication session and / or the content of the user interface input and participate in the conversation session. For example, in response to certain words appearing in the audio stream of the voice communication session and / or based on other prompts in the user interface input, the client on hold can participate in the conversation session. In many implementations, the client 104 on hold can utilize voice recognition to convert the speech from the user into text, and respond to the text accordingly, for example, by providing search results, general information and / or taking one or more response actions (e.g., initiating hold detection, etc.).

[0054] Each client device 102 can execute a corresponding instance of the client 104 being maintained. In various implementations, one or more aspects of the client 104 being maintained can be implemented outside the client device 102. For example, one or more components of the client 104 being maintained can be implemented on one or more computing systems (collectively referred to as "cloud" computing systems) that are communicatively coupled to the client device 102 via one or more local area networks and / or wide area networks (e.g., the Internet). Each client computing device 102 may include one or more memories for storing data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The operations performed by one or more computing devices 102 and / or the client 104 being maintained can be distributed over multiple computer systems. The client 104 being maintained can be implemented as a computer program running on one or more computers running in one or more locations, for example, coupled to each other via a network.

[0055] In many implementations, the client 104 on hold may include a corresponding voice capture / text-to-speech conversion (“TTS”) / speech-to-text conversion (“STT”) module 106, a natural language processor 108, an audio stream monitor 110, a hold detection module 112, and other components.

[0056] The client 104 on hold may include the corresponding voice capture / TTS / STT module 106 described above. In other implementations, one or more aspects of the voice capture / TTS / STT module 106 may be implemented separately from the client 104 on hold. Each voice capture / TTS / STT module 106 may be configured to perform one or more functions: for example, to capture the user's voice via a microphone (not shown) integrated in the client device 102; to convert the captured audio to text (and / or other representations or embeddings); and / or to convert text to speech. For example, in some implementations, because the client device 102 may be limited in computing resources (e.g., processor cycles, memory, battery, etc.), the voice capture / TTS / STT module 106 local to each client device 102 may be configured to convert a limited number of different spoken phrases, particularly phrases that invoke the client 104 on hold, into text (or other forms, such as lower dimensional embeddings). Other voice input may be sent to a cloud-based on-hold client component (not shown), which may include a cloud-based TTS module and / or a cloud-based STT module.

[0057] The natural language processor 108 of the client 104 on hold processes the natural language input generated by the user through the client device 102, and may generate annotation output for use by one or more components of the client 104 on hold. For example, the natural language processor 108 may process the natural language free-form input generated by the user through one or more user interface input devices of the client device 102. The generated annotation output includes one or more annotations of the natural language input, and optionally one or more (e.g., all) of the terms of the natural language input.

[0058] In some implementations, the natural language processor 108 is configured to recognize and annotate various types of grammatical information in the natural language input. For example, the natural language processor 108 may include a part-of-speech tagger that is configured to annotate terms with their grammatical roles. Also, for example, in some implementations, the natural language processor 108 may additionally and / or alternatively include a dependency parser (not shown) that is configured to determine syntactic relationships between terms in the natural language input.

[0059] In some implementations, the natural language processor 108 may additionally and / or alternatively include an entity tagger (not shown) configured to annotate entity references in one or more segments, such as references to people (e.g., including literary characters, celebrities, public figures, etc.), organizations, and locations (real and fictional), etc. The entity tagger of the natural language processor 108 may annotate references to entities at a higher level of granularity (e.g., to enable identification of all references to a class of entities such as people) and / or at a lower level of granularity (e.g., to enable identification of all references to a specific entity such as a specific person). The entity tagger may rely on the content of the natural language input to resolve specific entities and / or may optionally communicate with a knowledge graph or other entity database to resolve specific entities.

[0060] In some implementations, the natural language processor 108 may additionally and / or alternatively include a coreference resolver (not shown) configured to group or "cluster" references to the same entity based on one or more contextual cues. For example, a coreference resolver may be used to resolve the term "there" to "Hypothetical Café" in the natural language input "I liked Hypothetical Cafe last time we ate there".

[0061] In many implementations, one or more components of the natural language processor 108 may rely on annotations from one or more other components of the natural language processor 108. For example, in some implementations, the entity tagger described may rely on annotations from a coreference resolver and / or a dependency resolver when annotating all mentions of a particular entity. Likewise, for example, in some implementations, a coreference resolver may rely on annotations from a dependency resolver when clustering references to the same entity. In many implementations, when processing a particular natural language input, one or more components of the natural language processor 108 may determine one or more annotations using relevant previous inputs and / or other relevant data outside of the particular natural language input.

[0062] In many implementations, the client 104 on hold can interact with the voice communication session on hold without any required interaction from the user of the session. In some additional or alternative implementations, the client 104 on hold can initiate a process on hold, terminate a process on hold, notify the user that the voice communication session is no longer on hold, and / or transfer the voice communication session no longer on hold to an additional client on the client device 102.

[0063] In many implementations, the client device 102 and / or the client 104 on hold can use the audio stream monitor 110 to monitor the incoming and / or outgoing portions of the audio stream of the voice communication session. For example, the incoming portion of the audio stream can include the audio portion that the caller hears after the voice communication session is conducted (e.g., another human voice, music, etc.). Similarly, the outgoing portion of the audio stream of the voice communication session can include what the caller says to another caller via the audio stream and / or other signals provided by the client on hold (such as a solicitation response query asking whether another person is on the call). In some such implementations, the client device 102 can use the audio stream monitor 110 to detect when the voice communication session has been placed on hold, and pass the voice communication session on hold to the client 104 on hold. Additionally or alternatively, the client 104 on hold can monitor the audio stream of the voice communication session and the client 104 on hold itself can determine when to place the voice communication session on hold. Signals in the audio stream detected by the audio stream monitor 110 indicating that the voice communication session has been placed on hold may include detecting music known to be on hold, detecting any music (since users are unlikely to play songs to each other over the voice communication session), transitions from human speech to music, transitions from music to human speech, and the like.

[0064] The hold detection module 112 may use the determinations made by the audio stream monitor 110 regarding the audio stream of the voice communication session to determine when the voice communication session has been placed on hold, the voice communication session is no longer on hold, the predicted remaining wait time, etc. The hold detection module 112 may provide an indication to the user of the client device 102 when the session is no longer on hold, as well as pass the voice communication session to an additional client on the client device 102 to interact with the voice communication session (which may or may not require further interaction from the user).

[0065] Additionally or alternatively, the user may indicate to the client device 102 through the user interface that the voice communication session has been placed on hold and that the user wishes to initiate the process of placing the session on hold using the client 104 on hold. The hold detection module 112 may place the session on hold when it receives an affirmative indication from the user through the user interface in the client device 102 that the session has been placed on hold by: recommending that the session has been placed on hold and the user responds in an affirmative manner to initiate the process of placing the session on hold, and / or the user directly indicates through the user interface in the client device 102 that the session has been placed on hold to initiate the process of placing the session on hold using the hold detection module 112. In other implementations, the hold detection module 112 may automatically initiate the process of placing the session on hold when it detects a session on hold.

[0066] In many implementations, the hold detection module 112 may additionally or alternatively determine when a session is no longer on hold. In many implementations, a user may indicate how the user wishes to be notified when the process of being on hold ends. For example, a user may wish to receive a voice communication session on a mobile computing system that indicates that it is from a number on hold. Additionally or alternatively, when the end of the hold is detected, the user may request that a connected smart device such as a smart light within the same ecosystem of the client device 102 respond in some way. For example, a smart light on the same network as the client device 102 may be instructed to flash on and off, dim the intensity, increase the intensity, change color, etc. to indicate the end of the hold of the voice communication session. Additionally or alternatively, a user watching a smart TV may request that a notification appear on the TV when the end of the hold is detected.

[0067] Figure 2 , 3 4 and 5 respectively show the client (such as Figure 1 The interaction between the client 104) and the voice communication session on hold is shown. Figure 2 An image 200 is shown that includes a client 202 on hold interacting with a voice communication session 206 that is still on hold. In response to detecting a potential (also referred to as a "candidate") end of the voice communication session on hold, the client 202 on hold may send a response solicitation signal over the audio stream of the voice communication session 206 to determine whether an additional live user has become active in the session. In many implementations, the client on hold may determine a text phrase (e.g., "Are you there") to send as the response solicitation signal. In some such implementations, a text-to-speech module (similar to Figure 1 The illustrated speech capture / TTS / STT module 106) can convert text phrases into speech to provide as input to an audio stream.

[0068] In various implementations, a potential end of a voice communication session being on hold may be detected by the client 202 on hold detecting any of a variety of signals in the audio stream of the voice communication session, including changes in music, changes in music to human speech (potentially recorded speech as well as live speech), signals detected by various signal processing techniques such as discrete Fourier transforms, outputs of neural network models, etc. Human speech may be analyzed as a signal, and additionally or alternatively, a speech-to-text module (similar to Figure 1 The voice capture / TTS / STT module 106 shown in the figure can convert human speech into text. The language spoken in the text in the audio stream can also be processed by a natural language processor (such as Figure 1The natural language processor 108 shown in the figure is used to analyze the human voice detected in the audio stream to determine the meaning of the content spoken by the human voice. The output of the natural language processor can be further used to determine the potential end of the voice communication session. Additionally or alternatively, the output of the natural language processor can be used to determine that a living human user has entered the session. For example, the output of the natural language processor can provide input to one or more neural network models.

[0069] In some implementations, a neural network model may learn to recognize one or more "voices" to be ignored in a voice communication session. The voices may include one or more individual speakers, background music, background noise, etc. For example, one or more neural network models may include a recurrent neural network (RNN). The RNN may include at least one memory layer, such as a long short-term memory (LSTM) layer. The memory layer includes one or more memory cells to which inputs may be applied sequentially, and in each iteration of the applied inputs, the memory cells may be used to calculate new hidden states based on the inputs of that iteration and based on the current hidden states (which may be based on the inputs of the previous iteration). In some implementations, a model may be used to generate speaker segmentation results for any of various lengths of audio clips. As an example, an audio stream of a voice communication session may be divided into one or more data frames. Each data frame may be a portion of an audio signal, such as a 25 millisecond or other duration portion. Frame features (or the frames themselves) may be applied sequentially as input to a trained speaker segmentation model to generate a series of outputs, each output including a corresponding probability for each of N invariant speaker labels. For example, the frame features of audio frame 1 may initially be applied as input to generate N probabilities, where each of the N probabilities corresponds to one of the N speaker labels; next, the frame features of audio data frame 2 may be applied as input to generate N probabilities, where each of the N probabilities is for a corresponding one of the N speaker labels; etc. It should be noted that although the N probabilities generated for audio data frame 2 are specific to audio data frame 2, they will depend on the processing of audio data frame 1 because the model can be an RNN model.

[0070] Additionally or alternatively, the N probabilities may indicate whether a session has been placed on hold, whether a session is still on hold, and / or whether a potential hold end signal has been detected. In many implementations, an estimated remaining hold time may be determined for a voice communication session (by knowledge of a typical hold length for a particular called number that the client on hold has and / or the estimated remaining hold time as indicated in the audio stream of the voice communication session). According to many implementations, the estimated remaining hold time may be an additional input to a machine learning model, wherein the shorter the remaining estimated hold time, the more likely the machine learning model is to output a hold end.

[0071] In other implementations, the client on hold can use the knowledge of the potential remaining hold time to increase and / or decrease its threshold for sending a response solicitation signal (with or without using one or more machine learning models). For example, if the voice communication session is predicted to continue to be on hold for 20 minutes, the client on hold can have a higher threshold for sending a response solicitation signal. Similarly, a voice communication session that is predicted to last only a few minutes (e.g., 3 minutes) can have a lower threshold for sending a response solicitation signal.

[0072] Detection of a potential end of the voice communication session hold may cause the client 202 on hold to send a response solicitation signal through the audio stream of the voice communication session to determine whether an additional user has joined the voice communication session and the hold is ended. For example, the client 202 on hold may send a response solicitation signal 204, such as "Are you there". Additionally or alternatively, the response solicitation signal may be any of a variety of questions that prompt a response such as "Is anyone there", "Hello, are you there", "Am I still on hold", etc.

[0073] In many implementations, the response solicitation signal can prompt a predictable response to the additional living human user who has ended the voice communication session on hold. For example, the response to the response solicitation signal "Are you there" 204 can include "yes" and / or similar similar words or phrases indicating a positive response (e.g., "Yeah", "Yup" and phrases that can contain a positive response). Sending the response solicitation signal as input to the audio stream of the voice communication session can be computationally very inexpensive. Additionally or alternatively, it is unlikely that the recording (which can be played during the voice communication session on hold) is disrupted by repeatedly asking the same question, so the threshold for sending the response solicitation query may be low. In other words, since there are few negative effects (if any) from frequently sending the response solicitation signal, the client on hold according to many implementations will frequently send the response solicitation signal. In addition, if the client on hold fails to send the response solicitation signal when the response solicitation signal should be sent, the voice communication session may potentially terminate and require the user to start the process of being on hold with the phone number again.

[0074] In many implementations, when the hold is not ended, a response solicitation signal 204 can be sent over the audio stream of the voice communication session. When the response solicitation signal is sent and the voice communication session hold is not ended, the client 202 on hold will not detect a response 208 in the audio stream of the voice communication session 206.

[0075] In many implementations, the recorded voice may occur again while the voice communication session is on hold. In some such implementations, the recorded voice will not respond to a response request signal, and the client on hold can learn not to send a response request signal to the voice in the future. For example, while on hold, the phone number can play a recording that contains information about the called number (such as a website, business hours, etc.). This recording containing information about the number can be looped several times while the voice communication session is on hold. Once the client on hold determines that the voice does not respond to a response request signal, the client on hold can learn not to send additional response request signals to the particular voice. In many implementations, the client on hold can learn to ignore the voice using one or more of a variety of signals generated by a particular voice (e.g., a voice fingerprint), including the pitch of the voice, the recognition of the voice itself, and / or the particular sequence of words the voice is saying.

[0076] Figure 3 An image 300 is shown that includes a client 302 on hold interacting with a voice communication session 306. In many implementations, the client 302 on hold may send a response solicitation signal 304 such as "Is anyone there?" as input to the audio stream of the voice communication session. An STT module such as Figure 1 The voice capture / TTS / STT module 106 shown in FIG. 1 converts the text response solicitation signal provided by the client on hold into speech. For example, the client on hold may provide the text phrase "Is anyone there" as a response solicitation signal. The STT module may convert this phrase into spoken language, which may then be transmitted as input to the audio signal of the voice communication session. Figure 2 Determining when to send a response solicitation signal 304 is described. Image 300 also shows that the client on hold receives a response 308 "Yes, I am here" to the response solicitation signal, and can determine that the voice communication session is no longer on hold. Upon determining that the voice communication session is no longer on hold, the client on hold can convert the detected input into an audio stream and use the STT module ( Figure 1 The voice capture / TTS / STT module 106 shown in ) converts the input into text. In addition, a natural language processor (such as a natural language processor 108) can analyze the text response to the response request signal to provide the meaning of the text response.

[0077] As mentioned above about Figure 2As described, in many implementations, the question "Is anyone there?" 304 will typically elicit an affirmative response from the second user, such as "Yes, I am here". In other implementations, the response solicitation signal may be stated as typically elicits a negative response. For example, the question "Am I still on hold?" may elicit a negative response from the second user, such as "No, you are not on hold". In some implementations, when it is determined that the session is no longer on hold, the client on hold may utilize a typical response to a particular response solicitation signal used in part. In many implementations, once the client on hold 302 determines that the session is no longer on hold, the user conducting the voice communication session may be notified.

[0078] In some implementations, the user may be notified that the session is no longer on hold. For example, once the voice communication session is on hold, the mobile phone may ring and / or vibrate to simulate a new incoming session. Additionally or alternatively, a networked device near the user may be used as a notification that the voice communication session is on hold and has ended. For example, a user conducting a voice communication session may be near a smart light. The smart light may flash, dim, increase in intensity, change color, etc. to notify the user. Additionally or alternatively, a message may be pushed to the screen with which the user is interacting, including a mobile phone, a computing device, a television, etc. For example, a user watching a smart TV in the same device topology as the client device used to initiate the voice communication session may receive a notification on the TV when the session ends on hold. In various implementations, the user may choose how to be notified as a preference on hold. Additionally or alternatively, the user may choose how to be notified when the process of being on hold begins.

[0079] Figure 4 An image 400 is shown, which includes a client 402 on hold and a voice communication session 406. In many implementations, the client on hold can receive a very strong indication that the voice communication session is on hold. In some such implementations, the client on hold will not send a response solicitation signal, but can continue to inform the user that the session is no longer on hold. Human voice detected in the audio stream can be detected using the STT module ( Figure 1 The voice capture / TTS / STT module 106 shown in FIG. 1 converts the voice capture / TTS / STT module 106 into a text output, which can be provided to a natural language processor (such as in FIG. Figure 1The natural language processor 108 shown in FIG. 4 may include a natural language processor 108 to provide the meaning of the text to the client on hold. For example, the message 404 "Hello Ms. Jane Doe. My name is John Smith and I represent 'Hypothetical Utility Company'. How may I help you today?" may contain a strong indication that the voice communication session is no longer on hold. For example, detecting a user name (such as Jane Doe and / or Ms. Doe), detecting a phrase indicating an additional user name (such as "My name is John Smith"), and other phrases (such as "How may I help you today?") may all individually and / or in combination cause the client on hold to determine that the voice communication session on hold has ended without sending a response solicitation signal. In many implementations, when the client on hold determines that the voice communication session is no longer on hold, the user may be notified as previously described.

[0080] Figure 5 is a flowchart illustrating an example process 500 according to many implementations disclosed herein. For convenience, reference is made to the system description for performing the operation Figure 5 The system may include various components of various systems, such as one or more components of client device 102. In addition, although the operations of process 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, or added.

[0081] At block 502, the client device may optionally determine that the voice communication session is on hold. Figure 1 As described by the hold detection module 112 shown, the client device can determine that the voice communication session is on hold in a variety of ways, including detecting known held music, detecting any music, detecting a change from human voice to music, direct input from the user that the session has been placed on hold, determining that the called number is a known number that typically puts the user on hold, and any of a variety of signal processing techniques including discrete Fourier transforms, and determinations made by one or more machine learning models associated with the client on hold in the client device.

[0082] At block 504, the client device initiates a Figure 1 The described held clients 104 are held clients.

[0083] At block 506, the client on hold may monitor the incoming and / or outgoing portions of the audio stream of the voice communication session on hold. In many implementations, the client on hold may monitor the incoming and / or outgoing portions of the audio stream of the voice communication session on hold in a manner similar to that described above. Figure 1 The audio stream monitor 110 is described in a manner to monitor an audio stream.

[0084] At block 508, the client on hold may determine when to send a response solicitation signal via the audio stream of the voice communication session. Figure 2 Various ways in which a client on hold may determine to send a response solicitation signal are described. In many implementations, a client on hold may send one or more response solicitation signals and continue to send the response solicitation signals until the voice communication session is no longer on hold and / or the client on hold receives an indication from the user to end the hold process (e.g., the user is tired of waiting on hold and wishes to end the hold process and call the phone number later). In other implementations, a client on hold may not send a response solicitation signal. For example, a strong indicator may be detected that indicates that the session is no longer on hold (as described above with reference to FIG. 1 ). Figure 4 As described above), and the client on hold can determine that the voice communication session is no longer on hold without sending a response solicitation signal.

[0085] At box 510, the client on hold can determine that the voice communication session is no longer on hold. In various implementations, this determination can be made based on the response to the response solicitation signal received. In other implementations, the strength of the information monitored by the audio stream can be used to make this determination, which is strong enough to indicate that the voice communication session is no longer on hold without sending a response solicitation signal. Additionally or alternatively, the client on hold can send one or more response solicitation signals (that are not responded to), and then receive such a strong indication that the voice communication session is no longer on hold without sending additional response solicitation signals.

[0086] At block 512, the client on hold notifies the user that the voice communication session is no longer on hold. Figure 1 Various ways in which a client on hold can notify a user that a voice communication session is on hold are described. Additionally or alternatively, the client on hold can transfer the voice communication session to another client associated with a client device to handle the voice communication session on behalf of the user. For example, once the client on hold determines that the voice communication session is no longer on hold, the client on hold can transfer the voice communication session to the second client, which can interact with additional people on the voice communication session on behalf of the user.

[0087] Figure 66 is a block diagram of an example computer system 610. The computer system 610 generally includes at least one processor 614 that communicates with a number of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624, for example, including a memory 625 and a file storage subsystem 626, a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow a user to interact with the computer system 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computer systems.

[0088] The user interface input devices 622 may include a keyboard, a pointing device such as a mouse, trackball, touch pad or graphics tablet, a scanner, a touch screen incorporated into a display, an audio input device such as a voice recognition system, a microphone, and / or other types of input devices. In general, the use of the term "input device" is intended to include all possible types of devices and ways of inputting information into the computer system 610 or a communication network.

[0089] User interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display such as an audio output device. The display subsystem may include a cathode ray tube (CRT), a flat panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide a non-visual display such as via an audio output device. Typically, the use of the term "output device" is intended to include devices and modes of all possible types of information output from the computer system 610 to a user or another machine or computer system.

[0090] Storage subsystem 624 stores programming and data structures that provide the functionality of some or all of the modules described herein. For example, storage subsystem 624 may include an executable Figure 1 selected aspects of the client device shown, Figure 5 The process 500 shown in , any operations discussed herein, and / or the logic of any other devices or applications discussed herein.

[0091] These software modules are typically executed by the processor 614 alone or in combination with other processors. The memory 625 used in the storage subsystem 624 may include multiple memories, including a main random access memory (RAM) 630 for storing instructions and data during program execution and a read-only memory (ROM) 632 in which fixed instructions are stored. The file storage subsystem 626 may provide persistent storage for program and data files, and may include a hard drive, a floppy disk drive and associated removable media, a CD-ROM drive, an optical drive, or a removable media cartridge. Modules that implement the functionality of certain implementations may be stored in the storage subsystem 624 by the file storage subsystem 626, or in other machines accessible to the processor 614.

[0092] The bus subsystem 612 provides a mechanism for the various components and subsystems of the computer system 610 to communicate with each other as intended. Although the bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple busses.

[0093] Computer system 610 can be of various types, including a workstation, server, computing cluster, blade server, server farm, or any other data processing system or computing device. Due to the ever-changing nature of computers and networks, Figure 6 The description of the computer system 610 depicted in FIG. 6 is intended only as a specific example to illustrate some implementations. Figure 6 Many other configurations of computer system 610 may have more or fewer components than the one shown.

[0094] In situations where the systems described herein collect personal information about users (or "participants" as they are often referred to herein) or may make use of personal information, users may be provided with the opportunity to control whether a program or feature collects information about the user (e.g., information about the user's social network, social actions or activities, occupation, the user's preferences, or the user's current geographic location), or to control whether and / or how content that may be more relevant to the user is received from content servers. Similarly, certain data may be processed in one or more ways prior to storage or use so as to remove personally identifiable information. For example, the identity of a user may be processed so that no personally identifiable information about the user can be determined, or where geographic location information is available, the user's geographic location may be generalized (e.g., to a city, zip code, or state level) so that the user's specific geographic location cannot be determined. Thus, users may have control over how information about the user is collected and / or used.

[0095] Although several implementations have been described and illustrated herein, a variety of other means and / or structures for performing functions and / or obtaining results and / or one or more advantages described herein may be utilized, and each of these variations and / or modifications is considered to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary, and the actual parameters, dimensions, materials, and / or configurations will depend on the specific application or applications for which the teachings are used. Using only routine experiments, those skilled in the art will recognize or be able to determine many equivalents of the specific implementations described herein. Therefore, it should be understood that the aforementioned implementations are given only by way of example, and within the scope of the attached claims and their equivalents, implementations may be practiced in a manner different from that specifically described and claimed. The implementation of the present disclosure is directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, if such features, systems, articles, materials, kits, and / or methods are not mutually contradictory, any combination of two or more such features, systems, articles, materials, kits, and / or methods is included within the scope of the present disclosure.

Claims

1. A method implemented by one or more processors, the method comprising: identifying a telephone number associated with a voice communication session initiated by a client device of a calling user; determining whether the identified telephone number is associated with an entity known to have placed the calling user on hold; In response to determining that the identified telephone number is associated with the entity known to have placed the calling user on hold: identifying speech in an audio stream of the voice communication session; determining whether the speech in the audio stream is known to be on hold based on one or more previous voice communication sessions with the identified telephone number; and In response to determining that the speech in the audio stream is the known speech on hold, it is determined that the voice communication session is on hold.

2. The method of claim 1 , wherein determining whether the identified telephone number is associated with an entity known to have placed the calling user on hold comprises: comparing the identified calling telephone number to a global list of telephone numbers known to have placed the calling user on hold; and A determination is made that the identified telephone number is contained in the global list of telephone numbers known to have placed a calling user on hold.

3. The method of claim 1 , wherein determining whether the identified telephone number is associated with the entity known to have placed the calling user on hold comprises: comparing the identified telephone number to a personal list of telephone numbers known to have previously placed the calling party on hold; and A determination is made that the identified telephone number is contained in the personal list of telephone numbers known to have previously placed the calling party on hold.

4. The method according to claim 1, further comprising: In response to determining that the voice communication session is in the hold state, initiating a client on hold on the client device, wherein initiating the client on hold is during the voice communication session and is based on determining that the voice communication session is in the hold state.

5. The method according to claim 4, further comprising: monitoring an audio stream of the voice communication session for a candidate end of the hold state using the client on hold, wherein monitoring the audio stream of the voice communication session occurs without direct interaction from the calling user; and In response to detecting a candidate end of the hold state, a response solicitation signal is sent from the client device as input to an audio stream of the voice communication session.

6. The method according to claim 5, further comprising: monitoring an audio stream of the voice communication session for a response to the response solicitation signal; and determining that the response to the solicitation signal indicates that the candidate end of hold is an actual end of the hold state, wherein the actual end of the hold state indicates that a human user is available to interact with the calling user in the voice communication session; and A user interface output is caused to be rendered in response to determining the actual end of the hold state, wherein the user interface output is perceptible to the calling user and the user interface output indicates the actual end of the hold state.

7. The method according to claim 4, further comprising: monitoring, using the client on hold, an audio stream of the voice communication session for actual termination of the hold state, wherein monitoring the audio stream of the voice communication session occurs without direct interaction from the calling user; and A user interface output is caused to be rendered in response to detecting the actual end of the hold state, wherein the user interface output is perceptible to the calling user and indicates the actual end of the hold state.

8. A method implemented by a client device initiating a voice communication session, the method comprising: When the voice communication session is in a held state: monitoring the audio stream of the voice communication session for the presence of human speech speaking in the audio stream; In response to detecting an occurrence of the human voice during the monitoring period: Sending a response request signal as input of the audio stream; monitoring the audio stream for a response to the response solicitation signal; determining whether the response to the response solicitation signal is the human response in response to the response solicitation signal based on determining whether the human response corresponds to at least one predicted response to the response solicitation signal; When it is determined that the response is a human response in response to the response solicitation signal: causing a user interface output to be rendered, wherein the user interface output is perceptible to the calling user and indicates an actual end of the hold state, wherein the user interface output comprises an audio output, a vibration output, and a visual output; and When it is determined that the response is a human response that is not in response to the response solicitation signal: Identify one or more speech characteristics of the voice speaking the response and / or one or more words spoken by the voice for use in determining not to send a response solicitation signal when one or more future occurrences of the one or more speech characteristics and / or the one or more words are detected.

9. The method according to claim 8, wherein: The client device is a mobile phone or a stand-alone interactive speaker.

10. The method according to claim 8, wherein: The hold state is automatically initiated by the client device in response to detecting that the voice communication session is in the hold state.

11. A non-transitory computer-readable medium configured to store instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: identifying a telephone number associated with a voice communication session initiated by a client device of a calling user; determining whether the identified telephone number is associated with an entity known to have placed the calling user on hold; In response to determining that the identified telephone number is associated with the entity known to have placed the calling user on hold: identifying speech in an audio stream of the voice communication session; determining whether the speech in the audio stream is known to be on hold based on one or more previous voice communication sessions with the identified telephone number; and In response to determining that the speech in the audio stream is known to be on hold, it is determined that the voice communication session is on hold.

12. The non-transitory computer-readable medium of claim 11, wherein determining whether the identified telephone number is associated with an entity known to have placed the calling user on hold comprises: comparing the identified calling telephone number to a global list of telephone numbers known to have placed the calling user on hold; and A determination is made that the identified telephone number is contained in the global list of telephone numbers known to have placed a calling user on hold.

13. The non-transitory computer-readable medium of claim 11, wherein determining whether the identified telephone number is associated with the entity known to have placed the calling user on hold comprises: comparing the identified telephone number to a personal list of telephone numbers known to have previously placed the calling party on hold; and A determination is made that the identified telephone number is contained in the personal list of telephone numbers known to have previously placed the calling party on hold.

14. The non-transitory computer readable medium of claim 11, wherein: The operations also include: In response to determining that the voice communication session is in the hold state, initiating a client on hold on the client device, wherein initiating the client on hold is during the voice communication session and is based on determining that the voice communication session is in the hold state.

15. The non-transitory computer readable medium of claim 14, wherein: The operations also include: monitoring an audio stream of the voice communication session for a candidate end of the hold state using the client on hold, wherein monitoring the audio stream of the voice communication session occurs without direct interaction from the calling user; and In response to detecting a candidate end of the hold state, a response solicitation signal is sent from the client device as input to an audio stream of the voice communication session.

16. The non-transitory computer readable medium of claim 15, wherein: The operations further include: monitoring an audio stream of the voice communication session for a response to the response solicitation signal; and determining that the response to the response solicitation signal indicates that the candidate end of hold is an actual end of the hold state, wherein the actual end of the hold state indicates that a human user is available to interact with the calling user in the voice communication session; and A user interface output is caused to be rendered in response to determining the actual end of the hold state, wherein the user interface output is perceptible to the calling user and the user interface output indicates the actual end of the hold state.

17. The non-transitory computer readable medium of claim 14, wherein: The operations also include: monitoring an audio stream of the voice communication session for actual termination of the hold state using the client on hold, wherein monitoring the audio stream of the voice communication session occurs without direct interaction from the calling user; and A user interface output is caused to be rendered in response to detecting the actual end of the hold state, wherein the user interface output is perceptible to the calling user and indicates the actual end of the hold state.

18. A method implemented by one or more processors of a client device, the method comprising: receiving a user interface input provided via the client device, the user interface input being provided by a calling party while a voice communication session is on hold, wherein the voice communication session is initiated by the client device and wherein the called party controls the hold state; In response to receiving the user interface input: monitoring audio generated by the called party during the voice communication session for a candidate end of the hold state; detecting a candidate end of the hold state based on the monitoring; In response to detecting a candidate end of the hold state: sending, by the client device, audible output for inclusion in the voice communication session, wherein the audible output comprises recorded human speech saying the one or more words or synthetically generated speech saying the one or more words; monitoring audio generated by the called party subsequent to the audible output; determining that the audio generated by the called party subsequent to the audible output satisfies one or more criteria indicating that the candidate end of the hold state is an actual end of the hold state, wherein the actual end of the hold state indicates that a human user is available to interact with the calling party in the voice communication session, wherein in response to determining that the audio generated by the called party subsequent to the audible output does not satisfy the one or more criteria: recognizing one or more speech characteristics of a voice speaking the audio and / or one or more words spoken by the voice for use in determining not to send an audible output upon detecting one or more future occurrences of the one or more speech characteristics and / or the one or more words; and A user interface output is caused to be rendered in response to determining the actual end of the hold state, wherein the user interface output is perceptible to the calling user and the user interface output indicates the actual end of the hold state.

19. The method of claim 18, wherein determining that audio generated by the called party subsequent to the audible output satisfies one or more criteria comprises: generating text by performing speech-to-text conversion of the audio generated by the called party subsequent to the audible output; Determining the text is responsive to the one or more words in the audible output.

20. A method implemented by a client device initiating a voice communication session, the method comprising: When the voice communication session is in a held state: monitoring an audio stream of the voice communication session for the presence of human speech speaking in the audio stream; In response to detecting an occurrence of the human voice during the monitoring period: Sending a response request signal as input of the audio stream; monitoring the audio stream for a response to the response solicitation signal; determining whether a response to the response solicitation signal is a human response in response to the response solicitation signal; and When it is determined that the response is the human response in response to the response solicitation signal: causing a user interface output to be rendered, the user interface output being perceptible to the calling user and indicating that the hold state is ended, and When it is determined that the response is a human response that is not in response to the response solicitation signal: Identify one or more speech characteristics of the voice speaking the response and / or one or more words spoken by the voice for use in determining not to send a response solicitation signal when one or more future occurrences of the one or more speech characteristics and / or the one or more words are detected.

Images