Automatic call classification and screening

Abstract

Embodiments described herein relate to methods, systems, and computer-readable media for automatically answering a call. In some implementations, a method includes receiving, at a client device, a call from a calling device. The method also includes determining whether the call matches an auto-answer criterion based on the identifier related to the call, and responsive to determining that the call matches the auto-answer criterion, answering the call without user input and without a user prompt to the client device. The method further includes generating a call embedding of the call based on the received audio of the call, comparing the call embedding to a junk embedding to determine whether the call is a junk call, and terminating the call in response to determining that the call is a junk call.

Description

[0001] This application is a divisional application of the following application. The original international filing date was November 18, 2020. International application number of the original application: PCT / US2020 / 060965 The original application's national application number was 202080075899.7. Original invention title: Automatic call classification and screening Cross-referencing

[0002] This application claims the benefit of U.S. Provisional Application 62 / 937,769 entitled “Automatic Spam Call Detection”, filed November 19, 2019, which is incorporated herein by reference in its entirety. Background Technology

[0003] Spam calls, including robocalls, are a large and growing problem. Users in the United States receive over 4 billion robocalls every month. Many spammers spoof or deceive their numbers, limiting the effectiveness of number-based anti-spam tools, such as lists of numbers associated with spammers.

[0004] The background description provided herein is for the purpose of generally presenting the context of this disclosure. The work of the inventors of this invention is described in this background section, as well as aspects of the description that may not have been prior art at the time of filing, without express or implied acknowledgment as prior art to this disclosure. Summary of the Invention

[0005] In some embodiments, a computer-implemented method includes receiving a plurality of audio records, wherein each audio record corresponds to a corresponding call of a plurality of calls and metadata for each call of the plurality of calls; generating a corresponding embedding for each call of the plurality of calls based on the corresponding audio records using a trained machine learning model; grouping the plurality of calls into a plurality of clusters based on the corresponding embeddings; and storing a plurality of tuples in a database, each tuple including a specific audio record of the plurality of audio records, associated metadata, the embedding of the specific audio record, and a cluster identifier of the specific audio record.

[0006] In some implementations, the computer-implemented method further includes obtaining corresponding text copies of the plurality of audio recordings; and storing the corresponding text copies in corresponding tuples, wherein the corresponding embedding for generating each of the plurality of calls is also based on the text copy of the call.

[0007] In some embodiments, the computer-implemented method further includes, for each of the plurality of clusters: determining a representative embedding of the cluster; determining a call count matching the cluster; determining a spam call count of the cluster; and calculating a score for the cluster based on the call count or one or more of the spam call count. In some embodiments, the method further includes determining a subset of the plurality of clusters, wherein the score of each cluster in the subset satisfies a score threshold; and sending the representative embedding of each cluster in the subset to a client device. In some embodiments, the method further includes receiving data indicating one or more of the current country or home country of the client device. In some embodiments, determining the subset further includes selecting the subset based on the received data, wherein clusters comprising calls having metadata that does not match the current country or the home country are excluded from the subset. In some embodiments, determining the representative embedding of the cluster includes one of: selecting an embedding corresponding to a first audio record in the cluster as the representative embedding; selecting the average of a plurality of embeddings corresponding to calls in the cluster as the representative embedding; and selecting a specific embedding of the plurality of embeddings that is closest to the average of the plurality of embeddings.

[0008] In some implementations, the embedding of each call within each of the plurality of clusters is within a threshold edit distance from the embeddings of other calls in that cluster. In some implementations, the method includes receiving performance parameters; and updating the machine learning model based on the performance parameters.

[0009] In some implementations, a computer-implemented method for automatically answering calls includes: receiving a call from a calling device at a client device; determining, by the client device, whether the call matches an automatic answering criterion based on an identifier associated with the call; in response to determining that the call matches the automatic answering criterion, answering the call by the client device without user input and without prompting the user to the client device; and after answering the call, generating a call embedding of the call by the client device based on the received audio of the call using a trained machine learning model; comparing the call embedding with a spam embedding by the client device to determine whether the call is a spam call; and terminating the call in response to determining that the call is a spam call.

[0010] In some embodiments, the method further includes generating and storing a text copy of the call by the client device. In some embodiments, generating the call embedding is also based on the text copy of the call.

[0011] In some implementations, the method includes prompting the user of the client device in response to determining that the call is not a spam call, wherein prompting the user includes causing the client device to ring and providing a text copy of the audio of the received call.

[0012] In some implementations, responding to the call includes: establishing a connection with the calling device; and sending audio from the client device to the calling device, wherein the audio is generated by the client device without user input.

[0013] In some implementations, determining that the call matches an auto-answering criterion is based on at least one of the following: determining that the identifier associated with the call matches a spam caller list; determining that the identifier associated with the call is a spoofed number; determining that the identifier associated with the call is a private number; determining that the identifier associated with the call and the identifier associated with the client device meet a similarity threshold; determining that the identifier associated with the call is not in the contact list of the user of the client device; or determining that the identifier associated with the call is not in the call log of the client device.

[0014] In some implementations, determining that the call does not match the automatic answering criteria is based on at least one of the following: determining that the identifier associated with the call is in the contact list of the user on the client device; determining that the identifier associated with the call is in the call log of the client device; determining that the identifier associated with the call indicates that the call is an emergency call; or determining that an emergency call was made from the client device within a threshold from the current time.

[0015] In some implementations, the user of the client device is prompted in response to determining that the call does not match the automatic answering criteria.

[0016] In some implementations, the garbage embedding is stored locally on the client device, and the method further includes receiving the garbage embedding from the server and storing the garbage embedding locally on the client device.

[0017] Some implementations include a computing device that includes a processor and a memory coupled to the processor, on which instructions are stored, which, when executed by the processor, cause the processor to perform any of the methods described herein.

[0018] Some implementations include a non-transitory computer-readable storage medium having instructions stored thereon that, when executed by the processor, cause the processor to perform any of the methods described herein. Attached Figure Description

[0019] Figure 1 This is a block diagram of an example network environment used for one or more implementations described herein.

[0020] Figure 2 This is a block diagram illustrating an example method for generating garbage embeddings according to some implementations.

[0021] Figure 3 This is a block diagram illustrating an example method for automatically answering calls and terminating spam calls according to some implementations.

[0022] Figure 4 This is an illustration of an example user interface according to some implementation methods.

[0023] Figure 5 This is an illustration of an example user interface according to some implementation methods.

[0024] Figure 6 This is a block diagram of an example device used in one or more embodiments described herein. Detailed Implementation

[0025] Some implementations described herein relate to methods, systems, and computer-readable media for generating models of bots and / or spam callers. In some implementations, call embeddings (spam embeddings) representing calls from bots and / or spam callers are generated. The described implementations use a trained machine learning model to generate spam embeddings (e.g., numerical representations) from audio recordings (e.g., a training corpus). The corpus may include audio recordings of calls or other audio recordings containing spam content. For example, spam embeddings are provided from a server to a client device that stores them in local storage.

[0026] Some implementations described herein relate to methods, systems, and computer-readable media for automatically answering calls and detecting spam calls. When a call is received at a client device, the client device determines whether the call meets automatic answering criteria. If the call meets the automatic answering criteria, the call is automatically answered (without disturbing the user). A call embedding is obtained based on the call content, such as audio and / or a text copy of the audio received from the caller. The call embedding is compared with a spam embedding to determine whether the call is spam. In some implementations, the comparison may be performed by an on-device model (e.g., a trained machine learning model) that analyzes the call and determines whether the call meets the automatic answering criteria. If the call is determined to be spam, the call is terminated; otherwise, the user is prompted about the call. Information about the call, such as a text copy, is provided to the user before the user interacts with the call.

[0027] The described technology can automatically detect and mitigate spam calls based on the content (text transcript) and / or the manner of speaking (call audio) without interrupting the user. Detection and mitigation are performed locally on the client device and do not require an active internet connection. The described technology can reduce the number of calls the user interacts with. The described technology can be implemented in call processing applications, virtual assistant applications, or other applications that execute locally on the client device receiving the call.

[0028] Figure 1 A block diagram of an exemplary network environment 100 is shown, which can be used in some of the embodiments described herein. In some embodiments, network environment 100 includes one or more server systems, for example, Figure 1 The example server system 102. For example, server system 102 may communicate with network 130. Server system 102 may include server device 104 and database 106 or other storage devices. In some implementations, server device 104 may provide clustering application 158.

[0029] Network environment 100 may also include one or more client devices, such as client devices 120, 122, 124, and 126, which can communicate with each other and / or communicate with server system 102 and / or second server system 140 via network 130. Network 130 can be any type of communication network, including one or more of the Internet, local area network (LAN), wireless network, switch or hub connection, etc.

[0030] For ease of explanation, Figure 1 A box is shown for server system 102, server device 104, and database 106, and four boxes are shown for client devices 120, 122, 124, and 126. Server boxes 102, 104, and 106 may represent multiple systems, server devices, and network databases, and these boxes may be provided in different configurations than those shown. For example, server system 102 may represent multiple server systems that can communicate with other server systems via network 130. In some implementations, server system 102 may include, for example, a cloud hosting server. In some examples, database 106 and / or other storage devices may be provided within a server system box, which is separate from server device 104 and can communicate with server device 104 and other server systems via network 130.

[0031] Furthermore, there can be any number of client devices. Each client device can be any type of electronic device capable of communication, such as a desktop computer, laptop computer, portable or mobile device, cellular phone, smartphone, tablet computer, television, set-top box or entertainment device, wearable device (e.g., display glasses or goggles, watch, headset, armband, jewelry, etc.), personal digital assistant (PDA), etc. Some client devices may also have a local database similar to database 106 or other storage. In some embodiments, network environment 100 may not have all the components shown and / or may have other components including other types of elements besides those described herein, or other types of elements in addition to those described herein.

[0032] In various implementations, end users U1, U2, U3, and U4 can communicate with server system 102 and / or with each other using corresponding client devices 120, 122, 124, and 126. In some examples, users U1, U2, U3, and U4 can interact with each other via applications running on the corresponding client devices and / or server system 102 and / or via network services implemented on server system 102, such as social networking services or other types of network services. For example, the corresponding client devices 120, 122, 124, and 126 can transmit data to and from one or more server systems, such as server system 102.

[0033] In some implementations, server system 102 may provide appropriate data to client devices, enabling each client device to receive or share communication content uploaded to server system 102. In some examples, users U1-U4 may interact via audio or video conferencing, audio, video, or text chat, or other communication modes or applications.

[0034] The network services implemented by server system 102 may include systems that allow users to perform various communications, form links and associations, upload and publish shared content, and / or perform other functions, such as images, text, video, audio, and other types of content. For example, client devices may display received data, such as content posts sent or streamed to client devices and originating from different client devices via the server and / or network services (or directly from different client devices), or content posts originating from the server system and / or network services.

[0035] In some implementations, any one of client devices 120, 122, 124, and / or 126 can provide one or more applications. For example, such as Figure 1As shown, client device 120 can provide call application 152. Client devices 122-126 can also provide similar applications. Call application 152 can be implemented using the hardware and / or software of client device 120. In different implementations, call application 152 can be a standalone client application, for example, executed on any of client devices 120-124. Call application 152 can provide various call-related functions, such as receiving calls, automatically answering calls, prompting the user, generating text copies, detecting spam calls, etc.

[0036] The user interfaces on client devices 120, 122, 124, and / or 126 enable the display of user content and other information, including images, videos, data, and other content, as well as communications, settings, notifications, and other data. This user interface can be displayed using software on the client devices, software on the server devices, and / or a combination of client software and server software running on server device 104. The user interface can be displayed by a display device on the client devices, such as a touchscreen or other display screen, a projector, etc.

[0037] Other implementations of the features described herein can be used with any type of system and / or service. For example, other network services (e.g., internet access) may be used instead of the social networking service, or other networking services may be used in addition to the social networking service. Any type of electronic device can utilize the features described herein. Some implementations may provide one or more of the features described herein on one or more client or server devices that are disconnected from or intermittently connected to a computer network.

[0038] Figure 2 This is a flowchart illustrating an example method 200 for generating garbage embeddings according to some implementations. In some implementations, for example, method 200 can be implemented as follows: Figure 1 The method is implemented on the server system 102 shown. In some embodiments, some or all of method 200 can be implemented on, for example, Figure 1The method is implemented on one or more client devices 120, 122, 124, or 126, one or more server devices, and / or both server and client devices. In the described example, the implemented system includes one or more digital processors or processing circuitry (“processors”), and one or more storage devices (e.g., database 106 or other memory). In some implementations, different components of one or more servers and / or clients may perform different blocks or other portions of method 200. In some examples, the first device is described as the block that performs method 200. Some implementations may have one or more blocks of method 200 performed by one or more other devices (e.g., other client devices or server devices), which may send results or data to the first device.

[0039] In some implementations, method 200 or a portion thereof may be automatically initiated by the system. In some implementations, the implementing system is the first device. For example, the method (or a portion thereof) may be executed periodically (e.g., once a day, once a week, once a month, etc.) or based on one or more specific events or conditions, such as a threshold number of spam reports received from a client device (e.g., indicating that spam calls should trigger a user alert), a predetermined time period that has expired since the last execution of method 200, and / or one or more other conditions that may occur, which may be specified in settings read by the method.

[0040] Method 200 may begin at box 202. At box 202, multiple audio records and metadata associated with audio recordings are received. In some embodiments, the audio records may be records of telephone calls or other audio / video calls with at least two participants. For example, the audio records may be obtained from a training database that includes records of calls and / or other audio obtained with the consent of all participants (e.g., call recipients and initiators) from those records. In some embodiments, the record may include only the caller portion of the audio. In some embodiments, in addition to the audio records, call-related metadata may also be received. In some embodiments, the metadata may include, for example, a caller identifier, such as a caller's phone number. In some embodiments, the metadata may include the caller's country information. Box 202 may be followed by box 204.

[0041] In box 204, a text copy of each audio recording is obtained. For example, a text copy can be obtained by transcribing the audio recording using speech-to-text technology. In some implementations, the text copy may be received along with the call recording. Box 204 may be followed by box 206.

[0042] In box 206, a corresponding embedding is generated for each of the multiple audio recordings. In some implementations, the audio may be converted to a byte stream and resampled, for example, converted to 8K, before generating the embedding. In some implementations, an audio spectrogram may be obtained. The embedding may be a low-dimensional representation of a sparse vector generated, for example, from an audio recording and / or a text copy. The generation of the embedding is made irreversible, for example, by not including enough data to reconstruct the audio recording and / or text copy. For example, the embedding may be generated using a trained machine learning model. Each embedding may be a digital representation generated by a trained machine learning model based on an audio recording and / or a text copy of the audio recording.

[0043] Embeds of calls that include similar audio (e.g., pre-recorded audio messages) may be similar to each other, while embeddings of calls that include different audio may be different from each other. The similarity between two embeddings can be indicated in a variety of ways included in the transformation space. For example, the similarity between two embeddings can be indicated by edit distance, such as the minimum number of operations required to transform one embedding to another, the Euclidean distance between the two embeddings, etc. In some implementations, the embeddings of two calls may be similar when the content spoken in the calls (e.g., as indicated by text copies of the two calls) is similar. For example, when the text copies of the two calls include similar text (e.g., “Hello! I am John. I am XYZ from bank, offering you a credit card…”), the embeddings of the calls may have greater similarity compared to when the text copies do not include similar text. In some implementations, for example, when the embeddings are generated based on audio recordings and text copies, the similarity between the embeddings of two calls can be based on both the audio and the text copies. Box 206 may be followed by box 208.

[0044] In box 208, calls can be grouped into multiple clusters. For example, grouping can be performed such that the embedding of each call within a cluster is within a threshold edit distance of the corresponding embeddings of other calls within the cluster. Grouping can be performed in various ways, including in the transformation space. Furthermore, grouping can be performed such that the corresponding embeddings of calls in other clusters are not within the threshold edit distance of calls within the cluster. Box 208 can be followed by box 210.

[0045] At box 210, multiple tuples can be stored in the database. Each tuple may include a specific audio record, associated metadata, an embedding of the specific audio record, and a cluster identifier of the cluster to which the specific audio record is grouped. In implementations that obtain a text copy of a specific audio record, the text copy may also be stored in a tuple. Box 210 may be followed by box 212.

[0046] In box 212, a representative embedding is determined for each of the multiple clusters. In some implementations, the embedding associated with the first audio record in the cluster, such as the audio record associated with the earliest timestamp in the metadata, can be selected as the representative embedding. In some implementations, the average of multiple embeddings corresponding to a call in the cluster can be selected as the representative embedding. In some implementations, a particular embedding among multiple embeddings corresponding to a call in the cluster closest to the average of the multiple embeddings (e.g., by edit distance, or other distance metric) can be selected as the representative embedding. Box 212 may be followed by box 214.

[0047] In box 214, a corresponding score can be calculated for each of the multiple clusters. In some implementations, the call count in a cluster can be determined, for example, by counting the number of tuples in a database that includes cluster identifiers. In some implementations, the number of calls in a cluster that have been identified as spam calls can be determined, for example, based on metadata. In some implementations, the cluster score can be based on the call count and / or the number of calls in the cluster that have been identified as spam calls. For example, a cluster with a large proportion (e.g., 75%) of spam calls may have a higher cluster score than a cluster with a smaller proportion (e.g., 20%) of spam calls. In another example, when two clusters have similar proportions of spam calls (e.g., 50%), the cluster score of the cluster with a larger call count (e.g., 10,000 calls) may be higher than the cluster score of the cluster with a smaller call count (e.g., 100 calls). Box 214 may be followed by box 216.

[0048] In box 216, a subset of multiple clusters (or a model based on a subset of multiple clusters) can be identified. For example, a subset can be identified as including those clusters among multiple clusters that are associated with scores that meet a score threshold. For example, a subset may include clusters that might represent spam calls.

[0049] Furthermore, representative clusters (embedded elements used for call modeling within the cluster) from a subset of clusters can be sent to one or more client devices. These representative embeddings can be used by the client devices to compare with call embeddings generated locally on the client devices (e.g., based on received calls) and to determine whether the received call is a spam call. For example, when the call embedding of a received call matches at least one representative embedding, the client device can determine that the received call is a spam call; the representative embedding indicates that the received call resembles features identified as spam calls, for example, matching those of a spam cluster.

[0050] In some implementations, the client device may be, for example, a mobile phone connected to a cellular phone network. In these implementations, data may be received from the client device indicating the client device's home country (e.g., country of registration) or current country (e.g., the location where the device is currently located). In these implementations, determining a subset of the cluster may further include selecting a subset of the cluster based on the received data. For example, a subset of the cluster may be selected such that clusters including calls with metadata that do not match the current country or home country are excluded from that subset. Selecting a subset in this way can provide the benefit of sending only such representative embeddings to the client device because it is appropriate for the client device's location. Following box 216 may be box 202, where audio recordings and metadata may be received for additional calls.

[0051] In some implementations, performance parameters related to spam detection can be obtained, such as precision (e.g., the percentage of calls detected as spam that are actually spam), recall (e.g., the percentage of calls detected as spam as a percentage of total spam calls), latency for determining whether a call is spam, etc. Based on performance parameters, one or more clusters (e.g., low-precision clusters) can be removed from a subset. Furthermore, the machine learning model can be updated, for example, by retraining and regenerating the spam embeddings after retraining. For example, the model can be trained to generate smaller embeddings in response to high latency in determining spam call detection. Training the model can include updating the weights of one or more nodes in one or more layers of the model. The spam embeddings can also be updated based on additional training data when additional records not previously used to generate the spam embeddings become available. In some implementations, based on performance parameters, a model based on one or more clusters can be improved, for example, by removing low-precision clusters.

[0052] The various boxes of method 200 can be combined, divided into multiple boxes, or executed in parallel. For example, boxes 202 and 204 can be combined. In another example, boxes 212 and 214 can be executed in parallel. In some implementations, the boxes can be executed in different orders. For example, box 210 can be executed before box 208, or boxes 208 and 210 can be executed in parallel.

[0053] Method 200 or portions thereof may be repeated any number of times with additional input. In some implementations, for example, when method 200 is repeated upon receiving an additional record, the clusters identified from previous iterations of method 200 may be updated in box 208. Furthermore, upon receiving an additional record that matches a cluster, the representative embedding of one or more clusters may be updated (via execution box 212) and / or the cluster score may be updated (via execution box 214).

[0054] Although the previous reference Figure 2 The discussion involves call embedding, but any type of representation can be generated based on call audio and / or call text copies to identify different types of calls, such as regular or genuine calls, bot calls (e.g., in cases where the caller is an automated agent), spam calls (e.g., in cases where the caller does not want the caller), etc. This representation can then be used to perform clustering, allowing the clusters to model calls that are a subset of the cluster. Based on the dataset, any number of clusters can be generated. For example, each cluster can be associated with one or more types of bot or spam calls (e.g., a specific bot caller, a specific call topic), etc.

[0055] In some implementations, call representation can be used to indicate whether a call is automated based on whether the caller audio in the call includes human speech (received from the caller) or machine-generated speech (received from the caller). In these implementations, one or more machine learning models can be used to analyze the call audio, and these machine learning models are trained to distinguish between human speech and machine-generated speech.

[0056] Figure 3 This is a flowchart illustrating an example method for automatically answering calls and terminating spam calls according to some implementations. In some implementations, method 300 may be implemented on one or more client devices 120, 122, 124, or 126, such as... Figure 1 As shown. In the example, the implemented system includes one or more digital processors or processing circuits (“processors”) and one or more storage devices.

[0057] In some implementations, method 300 or a portion thereof may be automatically initiated by a client device. For example, method 300 may be automatically initiated upon receiving a call, such as a telephone call, a Voice over Internet Protocol (VoIP) call, a video call, etc. In some implementations, the system is a client device configured to receive telephone calls, VoIP calls, video calls, etc., such as a cellular phone or a wearable device (e.g., a smartwatch). In some implementations, the system is a client device, such as a tablet computer, laptop computer, desktop computer, or other device configured to receive VoIP calls, video calls, etc.

[0058] Method 300 is implemented using specific user permissions. For example, a call configuration user interface can be provided that allows the user to specify whether spam call detection / mitigation is enabled. The user is provided with information that enabling spam call detection requires enabling call logging and transcription, and is also provided with the option to completely disable spam call detection.

[0059] Method 300 may begin at block 302. At block 302, a call is received at the client device. The call may be a voice call, such as a telephone call, a VoIP call, such as one received via an over-the-top (OTT) calling application, a video call, etc. In some implementations, the call may be received by a calling application running on the client device. Block 302 may be followed by block 304.

[0060] In box 304, it is determined whether the call matches an auto-answer criterion. In some implementations, a call can be determined to match an auto-answer criterion when a call-related identifier (e.g., a phone number, a username associated with an OTT calling application, etc.) matches a spam caller list. The call-related identifier can be an identifier of the caller (calling device) from which the call was received. For example, a spam caller list can be stored locally on the client device. In some implementations, with user permission, a user's contact list stored locally on the client device can be used to determine whether a call matches an auto-answer criterion. If the user denies such permission, the contact list is not accessed, and any criteria requiring access to that contact list are skipped (not used in box 304).

[0061] In some implementations, when the identifier (telephone number) associated with a call is a spoofed number, it can be determined that the call matches an automatic answering standard. For example, the spoofed number could be a non-existent number, such as a number not assigned to a telephone user, or an invalid number, such as a number that does not conform to telephone numbering standards.

[0062] In some implementations, a call can be determined to match an auto-answering criterion when the identifier (telephone number) associated with the call is a private number. For example, if the call is a telephone call and no caller ID is received, or a caller ID indicating a private number is received, the call can be determined to be from a private number.

[0063] In some implementations, a call can be determined to match an automated answering standard when the identifier associated with the call (the caller's phone number) and the identifier associated with the client device (the called party's phone number) meet a similarity threshold. For example, a similarity threshold can be determined to be met when the first six digits (or other combinations of digits) of the calling and called phone numbers are the same.

[0064] In some implementations, a call can be determined to match an auto-answer criterion if the identifier associated with the call is not in the user's contact list on the client device, such as if it were stored locally on the client device. The contact list may include the user's personal contacts, such as personal contacts maintained through a personal account, and / or the user's business contacts, such as business contacts maintained through a business account, such as those stored in a corporate directory or address book.

[0065] In some implementations, when the identifier associated with the call is not in the call log of the client device, the call can be determined to match an auto-answer criterion. For example, such a call could be considered to be from a first-time caller or an unknown caller.

[0066] In different implementations, various combinations described above (as well as additional criteria, such as whether the phone number is within a certain range) can be used to determine whether a call matches the automatic answering criteria. In some implementations, the user of the client device can select the automatic answering criteria. For example, the user can specify any subset of the above as their automatic answering criteria. Different users can specify different automatic answering criteria. In some implementations, a user interface can be provided that allows the user to provide user input specifying the automatic answering criteria. The automatic answering criteria can be stored locally on the client device and / or synchronized with a server.

[0067] In some implementations, different auto-answer criteria can be used on the same client device based on context. For example, the auto-answer criteria may differ when the device is in Do Not Disturb (DND) mode or when the user is detected to be busy (e.g., based on user-permitted data, such as appointments). In another example, the auto-answer criteria may vary based on time of day, day of week, etc. For example, a strict auto-answer criteria may be used when the device is in DND mode, when the user is busy, or when it is nighttime (or a holiday), while a less strict auto-answer criteria may be used in other situations. In some implementations, the location of the client device when the call is received (e.g., home, work, abroad, etc.) can be determined if the user permits it, and the client device's location can be used to select the auto-answer criteria.

[0068] In some implementations, a call may be determined not to meet the auto-answer criteria when at least one of the following conditions is met: the identifier associated with the call is in the user's contact list on the client device; the identifier associated with the call is in the call log of the client device (e.g., the user recently made a call to or answered a call from the calling device associated with the identifier); the identifier associated with the call indicates that the call is an emergency call; or if an emergency call is made from the client device within a threshold from the current time (the time the call was received).

[0069] If the call matches the auto-answer criteria, then box 304 can be followed by box 306. Otherwise, box 304 can be followed by box 320.

[0070] In box 306, the call is answered automatically by the client device. For example, the call is answered without user input (e.g., selecting an answer button or similar input) and without prompting the user, such as without causing the client device to ring. In some implementations, answering the call may include establishing a connection between the client device and the calling device, and sending automatically generated audio from the client device (generated without user input or interaction). For example, the automatically generated audio may indicate that the call has been automatically answered and request further information from the caller. For example, the automatically generated audio may include “I am the virtual assistant of <username>; who are you?” or other messages indicating that the call has been automatically answered and / or requesting additional information from the caller. Such audio may be responded to by the caller, and further back-and-forth interaction may occur. For example, if the caller states “I am Hotel X, and I am confirming your 6 PM reservation,” a response such as “Thank you” may be provided. Box 306 may be followed by box 308.

[0071] In box 308, a text copy of the call can be generated. For example, the text copy may include a caller portion and an automatically generated portion. The call copy is generated locally on the client device using any suitable voice-to-text technology. In some implementations, the text copy may be stored locally on the client device in relation to the call. Box 308 may be followed by box 310.

[0072] In box 310, a call embedding can be generated (e.g., based on call content). In some implementations, a trained machine learning model (e.g., similar to the reference model) can be used. Figure 1 The described trained machine learning model generates call embeddings locally on the client device. In some implementations, the call embedding generation may be based on the audio of the received call. In some implementations, the call embedding generation may also be based on a text copy of the call. Box 310 may be followed by box 312.

[0073] In box 312, it is determined whether the call is spam, such as an unwanted call, a call made by a bot, etc. To make this determination, the call embedding is compared with a spam embedding, which is, for example, using a reference... Figure 2 The described method (e.g., on a server) generates and downloads the call embeddings from the server to the local storage of the client device, for example, when the client device has an active network connection. In some implementations, one or more spam embeddings may be generated locally on the client device, for example, based on previous calls marked as spam by the user. In some implementations, spam embeddings may include multiple embeddings, each representing a specific cluster. Comparing call embeddings may include performing a comparison between the call embedding and one or more of the multiple embeddings, for example, to determine whether the distance (e.g., Euclidean distance) between the call embedding and at least one of the multiple embeddings satisfies a distance threshold.

[0074] For example, if p denotes a calling embedding and q denotes one of the garbage embeddings, and each embedding has n dimensions, then the Euclidean distance between embeddings d(p,q) = d(q,p) is given by the following equation.

[0075]

[0076] In some implementations, a first threshold distance can be used for comparison. If the distance between the call embedding and the spam embedding is less than the first threshold distance, an embedding match is determined, and the call is detected as a spam call. In some implementations, a second threshold distance (greater than the first threshold distance) can be used. If the distance between the two embeddings is greater than or equal to the first threshold distance and less than the second threshold distance, and there are no other spam embeddings within the second threshold distance from the call embedding, an embedding match can be determined, and the call is detected as a spam call.

[0077] In some implementations, a single threshold distance can be used, and a call is determined to be a spam call if the distance between the call embedding and at least one of the multiple embeddings meets the distance threshold. If the call is determined to be a spam call, then box 312 may be followed by box 314. Otherwise, box 312 may be followed by box 320.

[0078] In box 314, the call is terminated. For example, terminating a call may include disconnecting the connection between the client device and the caller device. In some implementations, terminating a call may also include providing the calling device with an audio message, such as, "<username> does not wish to receive your calls. Please remove from your list," or other messages. Furthermore, when spam calls are detected, a spam warning may be displayed on the user interface of the client device. Box 314 may be followed by box 316.

[0079] In box 316, the call log on the client device is updated. In some implementations, the call log may be updated to add the call (e.g., caller identifier and call reception time) and a text copy of the call. In some implementations, the call status in the call log may be set to missed. For example, because a connection has been established between the calling device and the client device, the client device's operating system (or other call processing component) may detect the call as "answered"; however, because the call is automatically answered, the call status in the call log is updated to "missed," so that when the user checks the call log, only calls actively answered by the user have the status "answered," while unanswered calls and automatically answered calls have the status "missed." This status update can provide a consistent user experience, where a call is marked as answered only when the user actively contacts the call, such as when manually answering the call. Furthermore, providing a text copy of the call and / or saved audio (which can be played back by the user) in the call log allows the user to view the interaction with the caller of the automatically answered call. For example, the user can determine upon inspection that the call is not spam and can provide feedback indicating that classifying it as spam is incorrect. With the user's permission, this user feedback can be used as a performance parameter to update the machine learning model and / or garbage embeddings. Box 316 can be followed by box 302, where another call can be received.

[0080] If it is determined in box 304 that the call does not match the auto-answer criteria, or if it is determined in box 312 that the call is not a spam call, then box 320 is executed. At box 320, the user of the client device is prompted. If box 320 is executed immediately after box 304 (call does not match auto-answer criteria), the prompting user can include making the client device ring. If box 320 is executed immediately after box 312 (the call is automatically answered, but is not a spam call), the prompting user can include making the client device ring and providing a text copy of the received call audio. For example, a portion or summary of the text copy of the received audio can be provided; for instance, if the caller says, “This is Hotel X, and I’m confirming my 6 p.m. reservation,” the prompting user can include the displayed or spoken text, “A call from Hotel X to confirm a 6 p.m. reservation.” In another example, if the caller says, “Hello! This is Jane, I have an urgent question about your pipeline account,” the prompting user can include the displayed or spoken text, “An urgent call from Jane regarding a reservation.”

[0081] For example, salient features from the text copy, such as caller identity, call subject, date / time, etc., can be extracted and provided to the user. Furthermore, before the user joins the call, a message such as "Forwarding your call to <username>" can be provided to the caller. If the user allows, the text copy can be analyzed to provide additional information. For example, if the call is from a doctor's office and relates to an emergency (e.g., a leak in the home), the user can be given an indication that the call is important. In some implementations, the call's category (e.g., appointment, follow-up, social, etc.) or context (important / unimportant) can be determined and provided to the user. If the caller's audio is unclear or uses language with low transcription accuracy, an indication of this effect can be provided via the user interface, such as "Caller's audio is unclear" or "Foreign language," while prompting the user. Box 320 may be followed by box 302.

[0082] In some implementations, a second call can be received at the client device while the first call is being automatically answered (e.g., box 306). In this case, the second call can be automatically rejected. In some implementations, the user can be provided with options such as manually rejecting the second call or merging the second call with an ongoing call. In some implementations, if the user is on a call and receives a second call, method 300 can be used to handle the second call; however, unlike prompting the user (box 320), the caller can be provided with a message that the user is busy. In this case, the second call can be added to the call log as a missed call along with a related text copy.

[0083] Method 300 can be fully executed on the client device receiving the call. Furthermore, the technical advantage of the described method is that it does not require the client device to have an active internet connection, thus allowing spam call detection and mitigation even when the device is not connected to the internet. Additionally, since the method is executed locally, it does not consume network resources. Furthermore, user data is not sent to a server or other third-party devices. Therefore, the described technique can solve the problem of spam calls, including calls from bots, without disturbing the user and in a manner that does not require sharing user data. If the user permits, an indication can be sent to the server based on user feedback to classify the call as "spam" or "non-spam."

[0084] The various boxes in method 300 can be combined, divided into multiple boxes, or executed in parallel. In another example, boxes 306 and 308 can be combined or executed in parallel. Furthermore, different call answering criteria can be used based on the phone status (the phone is in DND mode, ringing off, or silent mode, etc.) or based on user-permitted contextual factors (e.g., time of day, whether the client device is stationary or in a vehicle, etc.).

[0085] In some implementations, the recording and text copy of the automated answering call can be stored locally on the client device and made available to the user for viewing. The user is given the option to delete the recording and copy.

[0086] If the client device is configured with one or more other devices, such as wirelessly coupled wearable devices, such as headphones, wristbands, watches, etc., or another device that shares the identifier of the client device, then in box 306, ensure that no user prompts are generated on any such device.

[0087] In some implementations, users may be provided with the option to indicate when a particular call has been misclassified. For example, a user may provide an indication that a particular call detected as spam or a bot call actually originates from a legitimate caller. In these implementations, if at least a threshold number of such user-provided indications are received and determined to be reliable, the cluster associated with such a call may be updated to indicate that the call is not spam or a bot call. In some implementations, the cluster may be updated periodically, and the updated representative embedding (or other representation) may be provided to client devices for call screening, call classification, detection of spam or bot callers, and / or automated call answering.

[0088] Figure 4 This is an illustration of an example user interface 400 according to some implementation methods. Figure 4 In the example shown, the client device receives a call from the phone number “(310) XXX-XXX”, which uses the reference here. Figure 3 The described technology is used to respond. The user interface 400 includes a text copy 402 (“The caller said…”) and a category 404 (“Appointment”). The user interface 400 provides the user of the client device with the option to answer or reject the call.

[0089] Figure 5 This is an illustration of an example user interface 500 according to some implementation methods. Figure 5 The example shown illustrates the user interface of the call log on a client device. In the call log, the most recent call 502 from “(415) XXX-XXXX” is displayed as “missed” and categorized as spam. A text copy of the call is also shown – “You’ve won a free vacation…”. While in Figure 5Not shown, but in some implementations, a user interface element may be included that allows the user to initiate playback of the call log in addition to or as an alternative to a text copy. The user interface allows the user to see spam calls being answered automatically and silently without disturbing the user. Another call 504 from “(310)-XXX-XXXX” is shown as “missed” along with a corresponding call copy (and / or log). Call 504 is automatically answered and determined not to be spam; however, the user did not answer the call, so it is classified as a “missed” call. A text copy of the automatically answered call is displayed. Other calls involving the user via the client device are also shown.

[0090] Figure 6 This is a block diagram of an example device 600 that can be used to implement one or more features described herein. In one example, device 600 can be used to implement a client device, for example, Figure 1 Any of the client devices 115 shown. Optionally, device 600 may implement a server device, such as server 104. In some embodiments, device 600 may be used to implement a client device, a server device, or both. Device 600 may be any suitable computer system, server, or other electronic or hardware device as described above.

[0091] One or more methods described herein can run in a standalone program that can run on any type of computing device, on a mobile application (“app”) running on a mobile computing device (e.g., mobile phone, smartphone, tablet, wearable device (wristwatch, armband, jewelry, headwear, virtual reality goggles or glasses, augmented reality goggles or glasses, head-mounted display, etc.), laptop, etc.).

[0092] In some embodiments, device 600 includes processor 602, memory 604, and input / output (I / O) interface 606. Processor 602 may be one or more processors and / or processing circuitry that execute program code and control the basic operations of device 600. "Processor" includes any suitable hardware system, mechanism, or component that processes data, signals, or other information. Processors may include systems with a general-purpose central processing unit (CPU) having one or more cores (e.g., in a single-core, dual-core, or multi-core configuration), multiple processing units (e.g., in a multiprocessor configuration), a graphics processing unit (GPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a complex programmable logic device (CPLD), dedicated circuitry for implementing functions, dedicated processors for implementing neural network-based processing, neural circuitry, processors optimized for matrix computations (e.g., matrix multiplication), or other systems. In some embodiments, processor 602 may include one or more coprocessors that implement neural network processing. In some implementations, processor 602 may be a processor that processes data to produce a probabilistic output; for example, the output produced by processor 602 may be inaccurate or may be accurate within a range from the expected output. Processing is not necessarily limited to a specific geographical location or has time constraints. For example, the processor may perform its functions in a “real-time,” “offline,” “batch processing mode,” etc. Parts of the processing may be performed at different times and in different locations by different (or the same) processing systems. The computer may be any processor communicating with memory.

[0093] Memory 604 is typically provided in device 600 for access by processor 602 and can be any suitable processor-readable storage medium, such as random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory, etc., adapted to store instructions executed by the processor and located separately from and / or integrated with processor 602. Memory 604 may store software operated by processor 602 on server device 600, including operating system 608, machine learning application 630, other applications 612, and application data 614. Other applications 612 may include applications such as call applications, data display engines, web hosting engines, image display engines, notification engines, social networking engines, etc. In some embodiments, machine learning application 630 and / or other applications 612 may include instructions that enable processor 602 to perform the functions described herein, for example, Figure 2 and Figure 3 Some or all of the methods.

[0094] Other applications 612 may include, for example, calling applications, media display applications, communication applications, web hosting engines or applications, mapping applications, media sharing applications, etc. One or more methods disclosed herein can operate in various environments and platforms, for example, as a standalone computer program that can run on any type of computing device, as a mobile application (“app”) running on a mobile computing device, etc.

[0095] In various implementations, the machine learning application may utilize a Bayesian classifier, support vector machine, neural network, or other learning techniques. In some implementations, the machine learning application 630 may include a trained model 634, an inference engine 636, and data 632. In some implementations, data 632 may include training data, such as data used to generate the training model 634. For example, training data may include any type of data, such as audio recordings, text copies, etc. When the trained model 634 is a model that generates embeddings, the training data may include audio recordings and associated metadata.

[0096] Training data can be obtained from any source, such as a data repository specifically tagged for training, or data licensed for use as training data for machine learning. In an implementation where one or more users are permitted to use their respective user data to train a machine learning model (e.g., trained model 634), the training data may include such user data.

[0097] In some implementations, training data may include synthetic data generated for training purposes, such as data not based on user input or activity in the context of training, for example, data generated from audio recordings of no human participants or other computer-generated audio recordings. In some implementations, machine learning application 630 excludes data 632. For example, in these implementations, a trained model 634 may be generated, for example, on different devices and provided as part of machine learning application 630. In various implementations, the trained model 634 may be provided as a data file including the model structure or form and associated weights. Inference engine 636 may read the data file of trained model 634 and implement a neural network with node connectivity, layers, and weights based on the model structure or form specified in the trained model 634.

[0098] In some implementations, the training model 634 may include one or more model forms or structures. For example, the model form or structure may include any type of neural network, such as a linear network, a deep neural network implementing multiple layers (e.g., "hidden layers" between input and output layers, where each layer is a linear network), a convolutional neural network (e.g., a network that segments or divides input data into multiple parts or pieces, processes each tile separately using one or more neural network layers, and aggregates the results of processing from each tile), a sequence-to-sequence neural network (e.g., a network that takes sequential data such as words in a sentence, frames in a video, etc., as input and produces a sequence of results as output), and so on. The model form or structure may specify the connectivity between various nodes and organize the nodes into layers.

[0099] For example, nodes in the first layer (e.g., the input layer) can receive data as input data 632 or application data 614. For example, when the trained model 634 generates embeddings, the input data may include audio recordings and / or copies of text generated from audio recordings. Subsequent intermediate layers may receive the outputs of nodes in the previous layer as input, according to the connectivity specified in the model form or structure. These layers may also be referred to as hidden layers or latent layers.

[0100] The final layer (e.g., the output layer) produces the output for the machine learning application. For example, the output could be an embedding of the input audio recording. In some implementations, the model form or structure also specifies the number and / or type of nodes in each layer.

[0101] In various implementations, the trained model 634 may include multiple nodes, which are arranged in each layer depending on the model structure or form. In some implementations, nodes may be computational nodes without memory, for example, configured to process an input unit to produce an output unit. Computations performed by a node may include, for example, multiplying each of the multiple node inputs by a weight to obtain a weighted sum, and adjusting the weighted sum using bias or intercept values ​​to produce a node output. In some implementations, the computation performed by a node may also include applying a step / activation function to the adjusted weighted sum. In some implementations, the step / activation function may be a nonlinear function. In various implementations, such computation may include operations such as matrix multiplication. In some implementations, computations performed by multiple nodes may be performed in parallel, for example, using the multi-processor cores of a multi-core processor, using individual processing units of a GPU, or dedicated neural circuits. In some implementations, nodes may include memory, for example, capable of storing and using one or more earlier inputs while processing subsequent inputs. For example, nodes with memory may include Long Short-Term Memory (LSTM) nodes. LSTM nodes can use memory to maintain their "state," allowing them to function like finite state machines (FSMs). Models with such nodes can be used to process sequential data, such as words in sentences or paragraphs, frames in videos, speech, or other audio.

[0102] In some implementations, training model 634 may include weights for each node. For example, the model may be started as multiple nodes, organized into layers as specified in the model's form or structure. During initialization, appropriate weights may be applied to the connections between each pair of nodes, each pair connected according to the model's form, such as nodes in consecutive layers of a neural network. For example, weights may be randomly assigned or initialized to appropriate default values. The model may then be trained, for example, using data 632 to produce results.

[0103] For example, training may include applying supervised learning techniques. In supervised learning, training data may include multiple inputs (audio recordings) and a corresponding expected output for each input (e.g., embeddings, spam / non-spam labels, etc.). Based on a comparison between the model's output and the expected output, the values ​​of the weights are automatically adjusted, for example, in a way that increases the probability that the model will produce the expected output when given similar inputs.

[0104] In some implementations, training may include applying unsupervised learning techniques. In unsupervised learning, only input data may be provided, and a model may be trained to distinguish between the data, for example, by clustering the input data into multiple groups, where each group includes input data that are similar in some way, such as having similar embeddings. For example, the model may be trained to distinguish between audio recordings, enabling it to differentiate between spam and genuine calls.

[0105] In some implementations, unsupervised learning can be used to generate knowledge representations, such as knowledge representations that can be used by machine learning application 630. For example, unsupervised learning can be used to generate the above-mentioned reference. Figure 2 and 3 The embedding used. In various embodiments, the trained model includes a set of weights corresponding to the model structure. In embodiments where data 632 is omitted, the machine learning application 630 may include a previously trained model 634, for example, by the developer of the machine learning application 630, by a third party, etc. In some embodiments, the trained model 634 may include a set of weights that are fixed, for example, downloaded from a server that provides the weights.

[0106] Machine learning application 630 also includes an inference engine 636. The inference engine 636 is configured to apply a trained model 634 to data such as application data 614 to provide inference. In some embodiments, the inference engine 636 may include software code to be executed by processor 602. In some embodiments, the inference engine 636 may specify a circuit configuration (e.g., for a programmable processor, for a field-programmable gate array (FPGA), etc.) that enables processor 602 to apply the trained model. In some embodiments, the inference engine 636 may include software instructions, hardware instructions, or a combination thereof. In some embodiments, the inference engine 636 may provide an application programming interface (API) that can be used by operating system 608 and / or other applications 612 to invoke the inference engine 636, for example, to apply the trained model 634 to application data 614 to produce inference. For example, inference for a garbage detection model may be the classification of received calls, for example, based on comparison with garbage embeddings.

[0107] Machine learning application 630 can provide several technical advantages. For example, when a trained model 634 is generated based on unsupervised learning, the inference engine 636 can apply the trained model 634 to generate knowledge representations (e.g., numerical representations) from input data, such as application data 614. For example, a model trained to generate embeddings can produce a representation of a call with a data size (e.g., 1KB) smaller than the input audio recording (e.g., 1MB). In some implementations, such representations can help reduce processing costs (e.g., computational costs, memory usage, etc.) to generate outputs (e.g., labels, classifications, etc.).

[0108] In some implementations, such a representation can be provided as input to different machine learning applications, which generate outputs from the output of inference engine 636. In some implementations, the knowledge representation generated by machine learning application 630 can be provided to different devices for further processing, for example, via a network. For example, using a reference... Figure 2 The described technique generates spam embeddings that can be provided to client devices for comparison and detection of spam calls, as referenced. Figure 3 As shown. In such an implementation, providing audio recordings that represent knowledge rather than spam calls can offer technical benefits, such as enabling faster data transmission at a reduced cost. In another example, a model trained to cluster audio recordings into documents can generate clusters from input audio recordings. The clusters can be adapted for further processing (e.g., determining whether a received call is spam) without requiring access to the original recordings, thus saving computational costs.

[0109] In some implementations, the machine learning application 630 may be implemented offline. In these implementations, a trained model 634 may be generated in a first stage and provided as part of the machine learning application 630. In some implementations, the machine learning application 630 may be implemented online. For example, in such an implementation, an application that invokes the machine learning application 630 (e.g., operating system 608, one or more other applications 612) may utilize the inference generated by the machine learning application 630, for example, to provide inference to a user, and may generate system logs (e.g., actions taken by the user based on the inference if permitted by the user; or the results of further processing if used as input for further processing). The system logs may be generated periodically, for example, hourly, monthly, quarterly, etc., and may be used, with user permission, to update the trained model 634, for example, to update the embeddings of the trained model 634.

[0110] In some implementations, the machine learning application 630 may be implemented in a manner that adapts to the specific configuration of the device 600 on which it executes the machine learning application 630. For example, the machine learning application 630 may determine a computation graph that utilizes available computing resources (e.g., processor 602). For example, the machine learning application 630 may determine that the processor 602 includes a GPU with a specific number of GPU cores (e.g., 1000) and implement an inference engine accordingly (e.g., as 1000 separate processes or threads).

[0111] In some implementations, machine learning application 630 may implement a whole trained model. For example, trained model 634 may include multiple trained models, each applicable to the same input data. In these implementations, machine learning application 630 may select a particular trained model, for example, based on available computing resources, the success rate of previous inferences, etc. In some implementations, machine learning application 630 may execute inference engine 636 to apply multiple trained models. In these implementations, machine learning application 630 may combine the outputs from applying the various models, for example, using a voting technique that scores the individual outputs from applying each trained model, or by selecting one or more specific outputs. Furthermore, in these implementations, machine learning application may apply a time threshold to apply the individual trained models (e.g., 0.5 ms) and utilize only those individual outputs available within the time threshold. Outputs not received within the time threshold may not be utilized, for example, and may be discarded. This approach may be suitable, for example, when a machine learning application is invoked, for example, by a specified time limit provided by operating system 608 or one or more applications 612, for example, to determine whether the call is spam.

[0112] In different implementations, the machine learning application 630 may produce different types of output. For example, the machine learning application 630 may provide representations or clusters (e.g., a numerical representation of the input data), labels (e.g., input data including images, documents, audio recordings, etc.), etc. In some implementations, the machine learning application 630 may produce output based on a format specified by the calling application (e.g., operating system 608, or one or more applications 612).

[0113] Any software in memory 604 may alternatively be stored in any other suitable storage location or computer-readable medium. Furthermore, memory 604 (and / or other connected storage devices) may store one or more messages, one or more taxonomy, electronic encyclopedia, dictionary, thesaurus, knowledge base, message data, grammar, user preferences, and / or other instructions and data used in the features described herein. Memory 604 and any other type of storage (disk, optical disc, magnetic tape, or other tangible media) may be considered "memory" or "storage device".

[0114] I / O interface 606 provides functionality that enables device 600 to interface with other systems and devices. Interface devices may be included as part of device 600 or may be separate and communicate with device 600. For example, network communication devices, storage devices (e.g., memory and / or database 106), and input / output devices may communicate via I / O interface 606. In some embodiments, the I / O interface may be connected to interface devices such as input devices (keyboard, pointing device, touchscreen, microphone, camera, scanner, sensor, etc.) and / or output devices (display device, speaker device, printer, motor, etc.). I / O interface 606 may also include a telephone interface, for example, for coupling device 600 to a cellular network or other telephone network.

[0115] Examples of interface devices that can be connected to I / O interface 606 may include one or more display devices 620, which can be used to display content, such as images, videos, and / or the user interface for output applications described herein. Display device 620 may be connected to device 600 via a local connection (e.g., a display bus) and / or via a network connection, and can be any suitable display device. Display device 620 may include any suitable display device, such as an LCD, LED, or plasma display, CRT, television, monitor, touchscreen, 3D display, or other visual display device. For example, display device 620 may be a flat panel display provided on a mobile device, multiple displays provided in goggles or head-mounted devices, or a monitor screen of a computer device.

[0116] For ease of explanation, Figure 6Each of the following blocks is shown: processor 602, memory 604, I / O interface 606, and software blocks 608, 612, and 630. These blocks may represent one or more processors or processing circuitry, operating systems, memory, I / O interfaces, applications, and / or software modules. In other embodiments, device 600 may not have all the components shown and / or may have other elements including other types of elements, other than or in addition to those shown herein. While some components are described as performing the blocks and operations as described in some embodiments herein, environment 100, device 600, any suitable component or combination of components of a similar system, or any suitable processor or processor associated with such a system may perform the described blocks and operations.

[0117] The methods described herein can be implemented by computer program instructions or code executable on a computer. For example, the code can be implemented by one or more digital processors (e.g., microprocessors or other processing circuitry) and can be stored on a computer program product, including non-transitory computer-readable media (e.g., storage media), such as magnetic, optical, electromagnetic, or semiconductor storage media, including semiconductor or solid-state memory, magnetic tape, removable computer disks, random access memory (RAM), read-only memory (ROM), flash memory, hard disks, optical disks, solid-state memory drives, etc. The program instructions can also be contained in and provided as electronic signals, for example, as software delivered as a service (SaaS) from a server (e.g., a distributed system and / or cloud computing system). Optionally, one or more methods can be implemented in hardware (logic gates, etc.) or in a combination of hardware and software. Example hardware can be a programmable processor (e.g., a field-programmable gate array (FPGA), a complex programmable logic device), a general-purpose processor, a graphics processor, an application-specific integrated circuit (ASIC), etc. One or more methods can be executed as part of or a component of an application running on a system, or as an application or software running in conjunction with other applications and an operating system.

[0118] Although the specification has been described with respect to specific embodiments, these embodiments are merely illustrative and not limiting. The concepts shown in the examples can be applied to other examples and embodiments.

[0119] Some implementations discussed herein may provide users with one or more opportunities to control whether information is collected, stored, used, and how information about them is collected, stored, and used, in cases where personal information about a user (e.g., user data, information about the user's social networks, the user's location and time at that location, the user's biometric information, the user's activities, and demographic information) is collected or used. That is, the systems and methods discussed herein collect, store, and / or use user personal information specifically when explicit authorization is received from the relevant user. For example, providing users with control over whether a program or function collects user information about that particular user or other users associated with that program or function. Presenting one or more options to each user for whom personal information is to be collected to allow control over the collection of information related to that user, thereby providing permission or authorization regarding whether information is collected and regarding which parts of the information are to be collected. For example, one or more such control options may be provided to users via a communication network. Furthermore, before storing or using certain data, it may be processed in one or more ways to remove personally identifiable information. As an example, a user's identity may be processed so that personally identifiable information cannot be determined. As another example, the geographic location of a user's device can be generalized to a larger area, making it impossible to pinpoint the user's specific location.

[0120] Note that, as is known to those skilled in the art, the functional blocks, operations, features, methods, devices, and systems described in this disclosure can be integrated or divided into different combinations of systems, devices, and functional blocks. Routines for a particular implementation can be implemented using any suitable programming language and programming techniques. Different programming techniques can be employed, such as procedural or object-oriented programming. Routines can be executed on a single processing device or multiple processors. Although steps, operations, or calculations may be presented in a specific order, this order can be changed in different specific implementations. In some implementations, multiple steps or operations shown as sequential in this specification can be performed simultaneously.

Claims

1. A method, characterized in that, include: The client device receives a call from the calling device; The client device determines whether the call meets predefined call processing standards based on an identifier associated with the call; In response to determining that the call meets the predefined call processing criteria, the client device answers the call without user input; During the call, the client device generates a transcription of at least a portion of the call audio from the call; and The transcription is presented by the client device via a user interface during the call.

2. The method according to claim 1, characterized in that, The client device answers the call without generating a prompt for the user of the client device.

3. The method according to claim 1, characterized in that, Generating the transcription includes: transcribing audio received from the calling device using a trained machine learning model that is executed locally on the client device.

4. The method according to claim 1, characterized in that, Also includes: In response to answering the call, the client device sends automatically generated audio data to the calling device, the automatically generated audio data being associated with audio indicating that the call has been automatically answered and requesting the caller to provide further information.

5. The method according to claim 1, characterized in that, This also includes after answering the call: The client device generates a call embedding of the call based on the received audio of the call using a trained machine learning model, wherein the call embedding is a digital representation of the call; The client device compares the call embedding with a spam embedding to determine whether the call is a spam call; and In response to determining that the call is a spam call, the call is terminated.

6. The method according to claim 1, characterized in that, Also includes: In response to receiving user input after presenting the transcription, the client device switches the call from automatic processing to active user participation during the call.

7. The method according to claim 1, characterized in that, The transcription is presented in response to determining that the call is not a spam call.

8. The method according to claim 1, characterized in that, Determining whether the call meets the predefined call processing criteria includes: determining whether the identifier does not exist in at least one of the contact list or call logs stored locally on the computing device.

9. A computing device, characterized in that, include: One or more processors; and One or more storage devices that store instructions, which, when executed by the one or more processors, cause the one or more processors to: Receive a call from the calling device; Based on the identifiers associated with the call, determine whether the call meets predefined call processing criteria; In response to determining that the call meets the predefined call processing criteria, the call is answered without user input; During the call, a transcription of at least a portion of the call audio is generated; and During the call, the transcription is presented via a user interface.

10. The computing device according to claim 9, characterized in that, The instructions that cause the one or more processors to generate the transcription include instructions that cause the one or more processors to use a trained machine learning model executed locally on the computing device to transcribe audio received from the calling device.

11. The computing device according to claim 9, characterized in that, The instructions also cause the one or more processors to send automatically generated audio data to the calling device in response to executing instructions for answering the call, the automatically generated audio data being associated with audio indicating that the call is being automatically answered and requesting the caller to provide further information.

12. The computing device according to claim 9, characterized in that, After the one or more processors execute the instructions to respond to the call, the instructions also cause the one or more processors to: Based on the received audio of the call, a call embedding is generated using a trained machine learning model, the call embedding being a digital representation of the call; The call embedding is compared with the spam embedding to determine whether the call is a spam call; as well as In response to determining that the call is a spam call, the call is terminated.

13. The computing device according to claim 9, characterized in that, The instructions also cause the one or more processors to respond to receiving user input during the call, changing the call from automatic processing to active user participation.

14. The computing device according to claim 9, characterized in that, The one or more processors execute the instructions to present the transcription in response to executing instructions that determine that the call is not a spam call.

15. The computing device according to claim 9, characterized in that, Instructions that cause the one or more processors to determine whether the call meets the predefined call processing criteria include instructions that cause the one or more processors to determine whether the identifier does not exist in at least one of the contact list or call logs stored locally on the computing device.

16. A non-transitory computer-readable storage medium encoded with instructions, characterized in that, When the instruction is executed by one or more processors of a computing device, the one or more processors cause the processors to: Receive a call from the calling device; Based on the identifiers associated with the call, determine whether the call meets predefined call processing criteria; In response to determining that the call meets the predefined call processing criteria, the call is answered without user input; During the call, a transcription of at least a portion of the call audio is generated; and During the call, the transcription is presented via a user interface.

17. The non-transitory computer-readable storage medium according to claim 16, characterized in that, The instructions that cause the one or more processors to generate the transcription include instructions that cause the one or more processors to transcribe audio received from the calling device using a trained machine learning model that is executed locally on the computing device.

18. The non-transitory computer-readable storage medium according to claim 16, characterized in that, The instruction also causes the one or more processors to send automatically generated audio data to the calling device in response to the instruction to answer the call, the automatically generated audio data being associated with audio indicating that the call is automatically answered and requesting the caller to provide further information.

19. The non-transitory computer-readable storage medium according to claim 16, characterized in that, After the one or more processors execute the instruction to acknowledge the call, the instruction further causes the one or more processors to: Based on the received audio of the call, a call embedding is generated using a trained machine learning model, wherein the call embedding is a digital representation of the call; The call embedding is compared with the spam embedding to determine whether the call is a spam call; and In response to determining that the call is a spam call, the call is terminated.

20. The non-transitory computer-readable storage medium according to claim 16, characterized in that, The instructions also cause the one or more processors to respond to receiving user input during the call, changing the call from automatic processing to active user participation.

21. A method, characterized in that, include: The computing system receives multiple audio recordings, each corresponding to a specific call among multiple calls, and metadata for each of the multiple calls; The computing system uses a trained machine learning model to generate a corresponding embedding for each of the multiple calls based on the corresponding audio recordings, and the corresponding embedding is a representation of the call; The computing system, based on the corresponding embedding, groups the multiple calls into multiple clusters; and The computing system stores the corresponding embedding of each of the plurality of calls in a database, associated with the metadata of the corresponding call in the plurality of calls.

22. The method according to claim 21, characterized in that, Also includes: The computing system obtains corresponding text copies of the multiple audio records; and The computing system stores the corresponding text copy in a corresponding data structure, wherein the generation of the corresponding embedding for each of the plurality of calls is also based on the corresponding text copy of the call.

23. The method according to claim 21, characterized in that, For each of the plurality of clusters, the following is also included: The computing system determines the representative embedding of the cluster; The computing system determines the count of calls that match the cluster; The computing system determines the spam call count of the cluster; and The computing system calculates the cluster's score based on one or more of the call count or the spam call count.

24. The method according to claim 23, characterized in that, Also includes: The computing system determines a subset of the plurality of clusters, wherein the score of each cluster in the subset of clusters satisfies a score threshold; as well as The computing system sends the representative embedding of each cluster in the subset to the client device.

25. The method according to claim 24, characterized in that, Also includes: The computing system receives data indicating one or more of the current country or home country of the client device, and wherein determining the subset further includes: The computing system selects the subset based on the received data, wherein clusters that include calls with metadata that do not match the current country or the home country are excluded from the subset.

26. The method according to claim 24, characterized in that, Determining the representative embedding of the cluster includes one of the following: The computing system selects the embedding corresponding to the first audio record in the cluster as the representative embedding; The computing system selects the average of multiple embeddings corresponding to calls in the cluster as the representative embedding; as well as The computing system selects the embedding among the plurality of embeddings that is closest to the average value of the plurality of embeddings.

27. The method according to claim 21, characterized in that, The embedding of each call within each of the plurality of clusters is within a threshold edit distance from the embedding of other calls in that cluster.

28. The method according to claim 21, characterized in that, Also includes: The computing system receives performance parameters; as well as The computing system updates the machine learning model based on the performance parameters.

29. A computing system, characterized in that, include: processor; and A memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 21-28.

30. A non-transitory computer-readable medium having instructions stored thereon, characterized in that, When the instruction is executed by the processor, it causes the processor to perform the method according to any one of claims 21-28.

Images