Provides protection against speech in audio signals

By introducing anti-noise into the audio signal, providing security defenses for phoneme, word and sentence levels, it solves the problems of automatic analysis interception and data privacy violations of audio signals, and realizes the security and privacy protection of voice communication.

CN114600187BActive Publication Date: 2024-11-26INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 11 Cites 0 Cited by

Patent Information

Application Number
CN202080072357.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2019-10-14
Filing Date
2020-10-02
Publication Date
2024-11-26
Estimated Expiration
2040-10-02

AI Technical Summary

Technical Problem

The prior art is difficult to effectively protect audio signals from interception and data privacy violations by automatic audio analysis, especially in voice communications, where there are risks such as intellectual property theft and industrial espionage.

Method used

By introducing antinoise into the audio signal, providing security defenses for phoneme, word and sentence levels, leveraging antinoise components and machine learning techniques, adjusting noise intensity for enhanced security, and adding antinoise to the audio acquisition device to prevent interception.

Benefits of technology

Effectively block automatic audio analysis, enhance data privacy and protection, ensure the security of voice communications, while maintaining human understanding, avoiding anti-noise-induced latency or quality impacts.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114600187B_ABST
    Figure CN114600187B_ABST
Patent Text Reader

Abstract

Embodiments for providing, by a processor, countermeasure protection of speech in an audio signal. Automated audio analysis of audio signals may provide security defenses for one or more audio devices using countermeasure noise.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates generally to computing systems and, more particularly, to various embodiments for providing adversarial protection of speech in audio signals. Background Art

[0002] In today's society, consumers, businessmen, educators, etc. communicate in real time, across long distances, and many times without boundaries or limits through various media. The emergence of computer and network technology makes it possible to improve the quality of life, while enhancing daily activities and simplifying information sharing. Computing systems may include the Internet of Things (IoT), which is the interconnection of computing devices scattered around the world using existing Internet infrastructure. That is, IoT is based on the idea that everyday objects (not only computers and computer networks) can be read, identified, located, addressed and controlled via IoT communication networks (e.g., self-organizing systems or the Internet). In other words, IoT can refer to uniquely identifiable devices and their virtual representations in Internet-like structures. With the tremendous progress and development of technology, the greater the demand for progress in these systems, the greater the advantage of efficiency and improvement. Summary of the invention

[0003] Various embodiments are provided for providing countermeasure protection for speech in audio signals by a processor. In one embodiment, by way of example only, a method is provided for providing countermeasure protection for speech in audio signals by a processor again. By using countermeasure noise, security defense can be provided for one or more audio devices to counter automatic audio analysis of audio signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] In order to readily understand the advantages of the present invention, a more particular description of the invention briefly described above will be presented by reference to specific embodiments shown in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope, additional features and details of the present invention will be described and explained through the use of the accompanying drawings, in which:

[0005] Figure 1 is a block diagram depicting an exemplary cloud computing node according to an embodiment of the present invention;

[0006] Figure 2 is an additional block diagram illustrating an exemplary cloud computing environment according to an embodiment of the present invention;

[0007] Figure 3 is an additional block diagram illustrating the abstract model layers according to an embodiment of the present invention;

[0008] Figure 4are additional block diagrams depicting various user hardware and cloud computing components functioning in accordance with aspects of the present invention;

[0009] Figure 5 is a diagram depicting additional exemplary operations for providing countermeasure protection of speech in an audio signal by a processor, in which aspects of the present invention may also be implemented;

[0010] Figure 6 is a diagram depicting additional exemplary operations for providing countermeasure protection of speech in an audio signal by a processor, in which aspects of the present invention may also be implemented;

[0011] Figure 7 is a flow chart depicting an exemplary method for providing countermeasure protection of speech in an audio signal by a processor, in which aspects of the present invention may also be implemented; and

[0012] Figure 8 is an additional flow chart depicting an exemplary method for providing countermeasure protection of speech in an audio signal by a processor, in which aspects of the present invention may also be implemented. DETAILED DESCRIPTION

[0013] Computing systems may include large-scale computing known as "cloud computing," in which resources may interact and / or be accessed via a communication system such as a computer network. Resources may be simulations and / or emulations of software renderings of computing devices, storage devices, applications, and / or other computer-related devices and / or services running on one or more computing devices, such as servers. For example, multiple servers may transmit and / or share information that may be expanded and / or shrunk across servers depending on the amount of processing power, storage space, and / or other computing resources needed to complete a requested task. The term "cloud" refers to the cloud-like appearance of a diagram of interconnectivity between computing devices, computer networks, and / or other computer-related devices interacting in such an arrangement.

[0014] In addition, the Internet of Things (IoT) is an emerging concept of computing devices that can be embedded in objects, especially appliances, and connected through a network. An IoT network may include one or more IoT devices or "smart devices", which are physical objects such as appliances with computing devices embedded in them. Many of these objects are independently operable devices, but they can also be paired with a control system or, alternatively, a distributed control system (such as a distributed control system running on a cloud computing environment).

[0015] Therefore, the massive increase in the use of IoT devices, mobile communication devices (e.g., smart phones, tablets, smart watches, or the computer itself, etc.) in various settings in computing systems, especially computing systems within cloud computing environments, provides users with various beneficial uses.

[0016] For example, many users use one or more applications on various computing systems, such as, for example, one or more applications on a mobile device with a microphone, for verbal communication with other users via telecommunications or computer networks. Some computing systems may use artificial intelligence ("AI") and enable automatic analysis of large amounts of audio signals. In particular, speech-to-text models allow spoken words to be transcribed into a text format that can be used for further processing, such as clustering, text mining, filtering by keywords, etc. A malicious adversary intercepting the transmission of an audio signal including speech or accessing an audio signal stored in a database may result in interception of data and violation of data privacy, which may result in negative consequences (such as theft of intellectual property or even industrial espionage) for the intended recipient or user of the communication.

[0017] Thus, the various embodiments described herein provide a novel solution for preserving and protecting audible data by providing adversarial protection against speech in audio signals. In one aspect, security defenses on one or more audio devices can be provided in the face of automatic audio analysis of adversarial audio signals by using adversarial noise, adversarial patches, or a combination thereof. In one aspect, a target or "boundary" area (e.g., an entire portion of a screen / display or one or more selected areas) can be selected to capture a digital or analog audio signal in order to provide adversarial noise to the digital or analog audio signal. The adversarial noise can be determined, calculated, and / or selected.

[0018] Adversarial noise can target different levels of misclassification, such as the phoneme level, word level, and / or sentence level. For example, at the phoneme level, adversarial noise can be directed to misclassify individual phonemes as silence or randomly distorted phonemes (which will result in an "incomprehensible" transcription). At the word level, adversarial noise can be directed to misclassify entire words (e.g., potentially distorting grammar). At the sentence level, adversarial noise can be directed to misclassify entire sentences.

[0019] The present invention can introduce adversarial noise at the word or sentence level, prohibit the interception of audio signals, and prevent the system from automatically detecting the adversarial noise, thereby enhancing data privacy and protection (e.g., the adversarial noise is an additional layer of protection).

[0020] Because adversarial noise may induce some delay in communication, a complete word / sentence may need to be transmitted (e.g., spoken or uttered) before the adversarial noise can be created and added to the audio signal. However, by introducing the adversarial noise at the phoneme level, only a very small delay is introduced (e.g., 10 milliseconds "ms" for a standard speech-to-text system).

[0021] The user may specify as input the strength of the adversarial defense against noise that should be applied to speech in the audio signal.

[0022] The adversarial noise strength may be adjusted (e.g., by a user using an input device and / or machine learning operations). The noise strength may be increased and / or decreased depending on the degree to which the noise strength affects the quality of the audio signal. For example, the noise strength may be adjusted to greater adversarial noise levels, which provide a higher degree of safety (because they cause more difficulty for a computer system to correctly classify the audio signal). For example, the adversarial noise strength may also be implemented as a portion of the entire audio recording in time, where the adversarial defense actively modifies the original signal (e.g., window length, acceptable delay, spacing frequency, etc.)

[0023] Likewise, increasing or decreasing the strength of the adversarial noise may affect the quality of the audio signal (e.g., a smaller / weaker level of adversarial noise), such as, for example, where the user or 3rd party entity does not notice (e.g., is not aware of) the audible noise. Alternatively, the noise strength may be adjusted to a level (e.g., a larger / stronger level) at which the user or 3rd party entity perceives artifacts but will still be able to understand the spoken content / communication of the audio.

[0024] In another aspect, the present invention may also play adversarial noise through an audio / communication device (e.g., a speaker) and record it along with speech through an audio capture device (e.g., a microphone) to avoid automatic analysis of audio signals intercepted by the attacked audio capture device.

[0025] The present invention may also report an estimate of the current defense strength (eg, similar to a battery state of charge, and may be an estimate of the effectiveness of the current defense strength).

[0026] Thus, the present invention secures and protects data (eg, personal / proprietary information) from automated analysis of audio signals inappropriately obtained through interception, eavesdropping, and / or unauthorized access to one or more databases (eg, audio databases).

[0027] The present invention may also report an estimate of the current defense strength (eg, similar to a battery state of charge, and may be an estimate of the effectiveness of the current defense strength).

[0028] On the other hand, the present invention can modify a digital or analog audio signal collected by an audio collection device (e.g., a microphone) by adding adversarial noise to the digital or analog audio signal. The adversarial noise may cause the speech-to-text model to misclassify the digital or analog audio signal. For example, the adversarial noise may cause the speech-to-text model to fail to detect any speech at all (e.g., the speech-to-text model will transcribe silence). The adversarial noise may cause the speech-to-text operation / model to transcribe random speech. The adversarial noise may cause the speech-to-text model to produce a transcription as a target (e.g., the noise may be designed to cause the speech of the word "patent" to be transcribed as "toaster"). The addition of adversarial noise prevents computer systems from automatically parsing and analyzing the spoken content in the audio signal. It should be noted that the adversarial noise applied to the digital or analog audio signal affects the automated processing of the audio signal, but the adversarial noise is low enough for an organism (e.g., a person) to still understand the speech and in most cases will not even notice any added noise.

[0029] It is understood that although the present disclosure includes detailed descriptions about cloud computing, the implementation of the teachings recorded herein is not limited to cloud computing environments. Instead, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0030] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal management effort or interaction with the provider of the service. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0031] Features are as follows:

[0032] On-demand self-service: Cloud consumers can unilaterally and automatically provision computing capabilities, such as server time and network storage, as needed without requiring manual interaction with the service provider.

[0033] Wide Area Network Access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

[0034] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. There is location independence in the sense that consumers typically do not control or know the exact location of the resources provided, but are able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0035] Rapid elasticity: In some cases, the ability to scale out quickly and in quickly can be provisioned quickly and elastically. To the consumer, the capacity available for provisioning often appears to be unlimited and can be purchased in any quantity at any time.

[0036] Metered Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the utilized services.

[0037] The service model is as follows:

[0038] Software as a Service (SaaS): The capability provided to the consumer is to use the provider's applications running on the cloud infrastructure. The applications are accessible from a variety of client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0039] Platform as a Service (PaaS): The capability provided to consumers is to deploy consumer-created or acquired applications onto cloud infrastructure, where the applications are created using programming languages ​​and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including networks, servers, operating systems, or storage, but has control over the deployed applications and possible configuration of the application hosting environment.

[0040] Infrastructure as a Service (IaaS): The capabilities provided to consumers are processing, storage, networking, and other basic computing resources on which consumers can deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and possibly limited control over selected networking components (e.g., host firewalls).

[0041] The deployment model is as follows:

[0042] Private Cloud: The cloud infrastructure is operated only for the organization. It can be managed by the organization or a third party and can exist inside or outside the building.

[0043] Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It can be managed by the organization or a third party and can exist on-premises or off-premises.

[0044] Public cloud: Cloud infrastructure is available to the general public or large industrial groups and is owned by an organization that sells cloud services.

[0045] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community or public) that remain a unique entity but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0046] The cloud computing environment is service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure consisting of a network of interconnected nodes.

[0047] Reference now Figure 1 , a schematic diagram of an example of a cloud computing node is shown. Cloud computing node 10 is only one example of a suitable cloud computing node and is not intended to impose any limitations on the scope of use or functionality of the embodiments of the present invention described herein. Regardless, cloud computing node 10 can be implemented and / or perform any of the functions set forth above.

[0048] In the cloud computing node 10, there is a computer system / server 12, which can operate with many other general or special computing system environments or configurations. Examples of well-known computing systems, environments and / or configurations suitable for use with the computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems or devices, etc.

[0049] Computer system / server 12 may be described in the general context of computer system executable instructions, such as program modules, executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer system / server 12 may be practiced in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network or IoT network. In a distributed cloud computing environment, program modules may be located in local and remote computer system storage media including memory storage devices.

[0050] like Figure 1As shown, the computer system / server 12 in the cloud computing node 10 is shown in the form of a general-purpose computing device. The components of the computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 to the processor 16.

[0051] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, these architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0052] Computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system / server 12, and it includes volatile and nonvolatile media, removable and non-removable media.

[0053] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, and commonly referred to as a "hard drive"). Although not shown, a disk drive for reading from and writing to a removable, non-volatile disk (e.g., a "floppy disk") may be provided, as well as an optical drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media. In this case, each may be connected to the bus 18 via one or more data media interfaces. As will be further depicted and described below, the system memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of an embodiment of the present invention.

[0054] A program / utility 40 having a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in system memory 28 by way of example and not limitation. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. Program modules 42 generally perform the functions and / or methods of embodiments of the present invention described herein.

[0055] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, etc.; one or more devices that enable a user to interact with the computer system / server 12; and / or any device that enables the computer system / server 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via an input / output (I / O) interface 22. However, the computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), an IoT network, and / or a public network (e.g., the Internet), via a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18. It should be understood that, although not shown, other hardware and / or software components may be used in conjunction with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0056] Reference now Figure 2 , depicts an illustrative cloud computing environment 50. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which a local computing device used by a cloud consumer can communicate, such as a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, and / or an automobile computer system 54N. The nodes 10 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above. This allows the cloud computing environment 50 to provide infrastructure, platform, and / or software as a service for which the cloud consumer does not need to maintain resources on a local computing device. It should be understood that Figure 2 The types of computing devices 54A-N shown in are intended for illustration only, and computing node 10 and cloud computing environment 50 may communicate with any type of computing device over any type of network and / or network addressable connection (eg, using a web browser).

[0057] Reference now Figure 3 , showing the cloud computing environment 50 ( Figure 2 ) provides a set of functional abstraction layers. It should be understood in advance that Figure 3 The components, layers, and functions shown in are intended to be illustrative only, and embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0058] The device layer 55 includes physical and / or virtual devices with embedded and / or independent electronics, sensors, actuators, and other objects to perform various tasks in the cloud computing environment 50. Each device in the device layer 55 combines networking capabilities with other functional abstraction layers so that information obtained from the device can be provided to the device, and / or information from other abstraction layers can be provided to the device. In one embodiment, the various devices including the device layer 55 can be incorporated into a network of entities collectively referred to as the "Internet of Things" (IoT). As will be understood by those of ordinary skill in the art, such a network of entities allows for the intercommunication, collection, and dissemination of data to achieve various purposes.

[0059] As shown, the device layer 55 includes sensors 52, actuators 53, a "learning" thermostat 56 with integrated processing, sensors and networked electronics, a camera 57, a controllable household outlet / receptacle 58, and a controllable electrical switch 59, as shown. Other possible devices may include, but are not limited to, various additional sensor devices, networked devices, electronic devices (such as remote control devices), additional actuator devices, so-called "smart" appliances (such as refrigerators or washing machines / dryers), and a wide variety of other possible interconnected objects.

[0060] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: host 61; server 62 based on RISC (Reduced Instruction Set Computer) architecture; server 63; blade server 64; storage device 65; and network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0061] Virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers 71; virtual storage 72; virtual networks 73, including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0062] In one example, the management layer 80 may provide the functionality described below. Resource provisioning 81 provides dynamic procurement of computing resources and other resources for performing tasks within a cloud computing environment. Metering and pricing 82 provides cost tracking when resources are utilized in a cloud computing environment, as well as billing or invoicing for consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides cloud computing resource allocation and management so that the required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-scheduling and procurement of cloud computing resources, where future demand is anticipated based on the SLA.

[0063] The workload layer 90 provides examples of functions that can utilize a cloud computing environment. Examples of workloads and functions that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom education delivery 93; data analysis processing 94; transaction processing 95; and in the context of the illustrated embodiment of the present invention, workloads and functions 96 are used to configure various settings for various computer-controlled devices to provide countermeasure protection in audio signals. In addition, the workloads and functions 96 for providing countermeasure protection in audio signals can include operations such as data analysis (including data collection and processing from various environmental sensors), image analysis, task and device analysis, and / or data analysis functions. Those of ordinary skill in the art will understand that the workloads and functions 96 for providing countermeasure protection in audio signals can also work in conjunction with other parts of various abstraction layers (such as hardware and software 60, virtualization 70, management 80 and other workloads 90 (e.g., such as data analysis processing 94)) to achieve the various purposes of the illustrated embodiment of the present invention.

[0064] Now turn to Figure 4 , a block diagram depicting exemplary functional components 400 of various mechanisms according to the illustrated embodiment is shown. In one aspect, Figure 1-3 Each of the devices, components, modules and / or functions described in the Figure 4 devices, components, modules and functions, and, Figure 1-3 One or more of the operations and steps may also be included in Figure 4 In one or more operations or actions, the computer system / server 12 is again shown, which can be combined with the intelligent countermeasure protection service 402.

[0065] In one aspect, computer system / server 12 can provide virtualized computing services (i.e., virtualized computing, virtualized storage, virtualized networking, etc.) to one or more computing devices, as described herein. More specifically, computer system / server 12 can provide virtualized computing, virtualized storage, virtualized networking, and other virtualized services executed on a hardware substrate.

[0066] In one aspect, the computer system / server 12 may provide a countermeasure protection service 402 (e.g., a smart countermeasure protection service 402) that may communicate and / or associate with one or more computing devices, such as, for example, a smart audio capture / generation device 430 (e.g., a hearing aid, a microphone, a speaker, a communication device, etc.) and / or a user device 420 (e.g., an Internet of Things "IoT" computing device or other type of audio generation or capture device, such as, for example, a camera, a television, a smart phone, a smart watch, a desktop computer, a laptop computer, a tablet, a smart watch, and / or another electronic device that may have one or more processors, memory, and / or wireless communication technology).

[0067] The intelligent countermeasure protection service 402, the intelligent audio collection / generation device 430 and / or the user device 420 can each communicate via one or more communication methods (such as a computing network, a wireless communication network or other network means for implementing communication (in Figure 4 In one aspect, the intelligent countermeasure protection service 402 may be installed locally on the intelligent audio acquisition / generation device 430 and / or the user device 420. Alternatively, the intelligent countermeasure protection service 402 may be located outside each of the intelligent audio acquisition / generation device 430 and / or the user device 420 (e.g., via a cloud computing server).

[0068] Intelligent countermeasure protection service 402 can be combined with processing unit 16 to perform various calculations, data processing and other functions according to various aspects of the present invention. A database 404 of preferences and interests (e.g., knowledge domains or ontologies) is shown, as well as a security defense component 406, a countermeasure noise component 408, a machine learning component 410, and / or a recording / transcription component 414.

[0069] The security defense component 406 can use counter-noise to provide security defense for one or more audio devices for automatic audio analysis of audio signals. The security defense component 406 associated with the counter-noise component 408 can apply the counter-noise to one or more display frames of one or more image display devices. In doing so, the counter-noise component 408 can determine or select the type of counter-noise to implement on the digital audio signal, the analog audio signal, or a combination thereof. The security defense component 406 associated with the counter-noise component 408 can then apply the counter-noise to communications in the digital audio signal, the analog audio signal, or a combination thereof (e.g., communication data at the phoneme level, word level, and / or sentence level, etc.).

[0070] In an additional aspect, the security defense component 406 associated with the machine learning component 410 can use machine learning operations to analyze audio signals. In one aspect, the audio signal can be transmitted and sent in real time from a user and / or from a recorded message located in an audio database (such as, for example, in the knowledge domain component 412).

[0071] The security defense component 406 associated with the machine learning component 410 can estimate the theoretical security defense strength against noise. The security defense component 406 associated with the machine learning component 410 can estimate the actual security defense strength against noise via a feedback loop operation. The security defense component 406 can adjust the degree of security defense strength against noise.

[0072] It should be noted that the theoretical defense strength is determined by estimating the level or degree of performance degradation as a function of the current noise level (e.g., how much the performance of the speech-to-text AI model will degrade). This estimate can be based on offline measurements. The actual defense strength determines the difference in the output of the speech-to-text AI model on the audio signal with and without adversarial noise. The actual defense strength may be computationally more expensive than the theoretical defense strength, on the other hand, it is a more accurate measure of the effectiveness of the current noise level.

[0073] The database of audio files, recordings and / or transcriptions may include and / or be associated with a knowledge domain 412, which may be an ontology of expressions, keywords, concepts representing a knowledge domain. For example, all recorded audio files may be classified, organized and / or saved / retrieved based on the type of communication within the audio file / recording. In one aspect, the term "domain" is a term intended to have its ordinary meaning. In addition, the term "domain" may include a system's professional field or a collection of materials, information, content and / or other resources related to a specific topic or multiple topics. A domain may refer to information related to any specific topic or combination of selected topics. The term ontology is also a term with its ordinary meaning. In one aspect, the term ontology in its broadest sense may include anything that can be modeled as an ontology, including but not limited to taxonomies, vocabularies, vocabulary, etc. For example, an ontology may include information or content related to a domain of interest or the content of a specific category or concept. The ontology may be continuously updated using information synchronized with a source, adding information from a source to the ontology as a model, an attribute of a model, or an association between models within an ontology.

[0074] Additionally, the knowledge domain 412 may include one or more external resources, such as, for example, links to one or more Internet domains, web pages, etc. For example, text data may be hyperlinked to a web page that may describe, explain, or provide additional information related to the audio data.

[0075] In another aspect, the security defense component 406 of the computer system / server 12 can work in conjunction with the processing unit 16 to implement various aspects of the present invention. For example, the security defense component 406 can undergo various data analysis functions to analyze audio data transmitted from one or more devices (such as, for example, the user device 420 and / or the smart audio acquisition / generation device 430).

[0076] The security defense component 406 can receive and analyze each physical attribute associated with the audio data. The security defense component 406 can cognitively receive and / or detect audio data for the user device 420 and / or the smart audio acquisition / generation device 430 (e.g., audio data that can be transcribed).

[0077] The security defense component 406, the anti-noise component 408, and / or the machine learning component 410 can access and monitor one or more audio data sources (such as, for example, user devices 420 and / or intelligent audio acquisition / generation devices 430 (e.g., radio, telephone, television, computer, etc.)) to detect audio data. For example, the user device 420 and / or the intelligent audio acquisition / generation device 430 can broadcast communications (e.g., voice commands) from the user 440. The security defense component 406, the anti-noise component 408, and / or the machine learning component 410 can monitor and access communications (e.g., voice commands) from one or more users.

[0078] As part of detecting audio data, the security defense component 406, the noise combating component 408, and / or the machine learning component 410 may identify from the audio data one or more sources of the audio data, contextual factors associated with the audio data, one or more characteristics of the audio data, one or more semantic phrases, sentences, words, or combinations thereof of a user.

[0079] The recording / transcription component 414 can also dynamically record and store audio data communications from the user 440. For example, machine learning can learn and record each communication / audio data created and / or directed to it by the user 440. The recording / transcription component 414 can then record the communication data (e.g., record, transcribe and / or save the audio transcription), extract one or more keywords / phrases, and / or use NLP to transcribe the speech to text form. Moreover, words, phrases and / or complete sentences related to the audio data made by other parties (e.g., all or part of the conversation) can be transcribed in text form based on NLP extraction operations (e.g., keyword extraction based on NLP). The text data can be relayed, sent, stored or further processed so that the same audio data (e.g., all or part of the conversation) can be heard or monitored by the counter protection service 402 at a later point in time.

[0080] Thus, the security defense component 406 associated with the counter-noise component 408 and / or the machine learning component 410 may modify all or part of the audio data (e.g., speech-to-text data). The security defense component 406 associated with the counter-noise component 408 and / or the machine learning component 410 may even send an alert to the user 440 indicating the application of the counter-noise. The user 440 may then adjust (e.g., increase and / or decrease) the intensity level of the counter-noise.

[0081] Now go to Figure 5 , graph 500 depicts an exemplary operation for providing adversarial protection of speech in an audio signal. That is, graph 500 depicts a schematic representation of an adversarial attack on an audio signal. In one aspect, Figure 1-4 Each of the devices, components, modules and / or functions described in the Figure 5 devices, components, modules and functions. In addition, Figure 1-4 One or more components, functions and / or features of may be present in Figure 4 For the sake of brevity, repeated descriptions of similar elements, components, modules, services, applications and / or functions used in other embodiments described herein are omitted.

[0082] In one aspect, the X-axis 501 of the graph 500 represents time, and the Y-axis 502 represents the amplitude or signal strength of the audio signal. An original audio signal 503 (e.g., benign and can be analog or digital) is represented as a solid line. An adversarial audio signal 504 is represented as a dashed line and is modified using adversarial noise.

[0083] When adversarial noise is applied, the output sentence 505 corresponds to the original audio signal 503, such as the original audio signal 503 obtained from a speech-to-text model (e.g., a recursive neural network, etc.), which takes the original audio signal 503 as input.

[0084] Output sentence 506 is created by using an audio signal while applying a target adversarial noise defense as an input to a speech-to-text model (e.g., a recursive neural network, etc.). This results in small subtle changes in output sentence 506 that are difficult for a person to detect. However, output sentence 507 is created by using an audio signal together with a non-targeted adversarial noise defense as an input to a speech-to-text model (e.g., a recursive neural network, etc.). This results in completely random changes to the input sentence. Therefore, output sentences 506 and 507 show the resulting output sentences based on the degree or level of adversarial noise intensity applied to adversarial audio signal 504.

[0085] Thus, the adversarial audio signal 504 is the result of modifying the original audio signal 503 , which may be captured by an audio capture device and may be digital or analog, by applying the adversarial noise.

[0086] Now go to Figure 6 , which depicts an exemplary attack vector operation 600 for providing countermeasure protection against speech in an audio signal. In one aspect, Figure 1-4 Each of the devices, components, modules and / or functions described in the Figure 6 devices, components, modules and functions. And, Figure 1-4 One or more components, functions and / or features of may be present in Figure 4 For the sake of brevity, repeated descriptions of similar elements, components, modules, services, applications and / or functions used in other embodiments described herein are omitted.

[0087] In operation, attack vector operations 600 include user 601 (e.g., a user / speaker identified as "Alice") recording an audio signal and desiring to send the audio signal and / or store the audio signal in a database. An audio acquisition / generation device 605 (e.g., a microphone) may utilize, for example, microphone / machine learning operations to record / transcribe an audio message provided by user 601, and may communicate, transmit, and / or send the created audio signal to an adversarial security defense component 604.

[0088] The adversarial safety defense component 604 may add the selected level of adversarial noise (e.g., adding adversarial noise at the phoneme level, word level, and / or sentence level) to the original audio signal transmitted by the user 601, with minimal or no change to the original audio signal represented (e.g., speech), and without affecting the understanding of the message by the recipient 602 (e.g., user "Bob"). As in box 606, the adversarial safety defense component 604 may communicate, transmit, store, and / or send the created adversarial protected audio signal (e.g., the audio signal with the applied adversarial noise) to the recipient 602. The adversarial safety defense component 604 may also estimate the current adversarial noise intensity level by analyzing the adversarial protected audio signal, and modify the adversarial protected audio signal before communicating, transmitting, storing, and / or sending the created adversarial protected audio signal. In one aspect, a feedback loop may be used via an audio acquisition / generation device 605 (e.g., a microphone) to help analyze and / or estimate the adversarial noise intensity level.

[0089] Recipient 602 (e.g., a user identified as "Bob") may receive the audio signal and may understand the message transmitted by user 601. Adversary 603 (e.g., adversarial listener "Eve") listens to messages transmitted and / or stored between user 601 and recipient 602 and analyzes the audio signal using a machine learning model.

[0090] An adversary 603 having access to the transmitted or stored adversarial protected audio signal attempts to apply automated analysis operations to attempt to classify the stolen adversarial protected audio signal. However, the adversarial noise applied to the adversarial protected audio signal interferes with the automated analysis and / or reduces the success rate of the automated analysis, thereby preserving and protecting the audio signal transmitted by the user 601.

[0091] Now turn to Figure 7 , depicts an additional method 700 for providing countermeasure protection in an audio signal by a processor, in which various aspects of the illustrated embodiments may be implemented. Function 700 may be implemented as a method executed as instructions on a machine, wherein the instructions are included on at least one computer-readable medium or a non-transitory machine-readable storage medium. Function 700 may begin in block 702.

[0092] As in block 704 , security defenses may be provided on one or more audio devices against automated audio analysis of audio signals by using countermeasure noise. Functionality 700 may end in block 706 .

[0093] Now turn to Figure 8 , depicts an additional method 800 for providing countermeasure protection in an audio signal by a processor, in which various aspects of the illustrated embodiments may be implemented. Function 800 may be implemented as a method executed as instructions on a machine, wherein the instructions are included on at least one computer-readable medium or a non-transitory machine-readable storage medium. Function 800 may begin in block 802.

[0094] The counter-noise may be selected, as in block 804. A level of counter-noise strength of the counter-noise may be applied to the audio signal, as in block 806. The audio signal may be modified using the counter-noise, as in block 808. Functionality 800 may end in block 810.

[0095] In one aspect, combining Figure 7 The operations of method 700 and / or 800 may include each of the following as well as at least one block of method 700 or 800 and / or as part thereof. The operations of method 700 and / or 800 may apply the countermeasure noise to the digital audio signal, the analog audio signal, or a combination thereof, and / or determine or select the type of countermeasure noise to implement on the digital audio signal, the analog audio signal, or a combination thereof in the communication.

[0096] The operations of methods 700 and / or 800 may use machine learning operations to analyze audio signals, where the audio signals are transmitted and sent in real time from a user or from a recorded message located in an audio database.

[0097] The operations of methods 700 and / or 800 may estimate the theoretical security defense strength against noise via a feedback loop operation, and / or estimate the actual security defense strength against noise. The operations of methods 700 and / or 800 may adjust the degree of security defense strength against noise.

[0098] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the present invention.

[0099] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a line.

[0100] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium in the corresponding computing / processing device.

[0101] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as "C" programming language or similar programming languages). The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In certain embodiments, in order to perform various aspects of the present invention, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can execute the computer-readable program instructions by utilizing the state information of the computer-readable program instructions to personalize the electronic circuit.

[0102] Various aspects of the present invention are described herein with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to embodiments of the present invention. It will be understood that each frame of the flow chart and / or block diagram and the combination of frames in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0103] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0104] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0105] Flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each frame in flow chart or block diagram can represent the module, segment or part of instruction, and it comprises one or more executable instructions for realizing the logical function of appointment.In some alternative embodiments, the function mentioned in the frame may not occur in the order mentioned in the figure.For example, two frames shown continuously can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, and this depends on the function involved.It will also be noted that the combination of each frame of block diagram and / or flow chart and the frame in block diagram and / or flow chart can be realized by the dedicated hardware-based system of performing specified function or action or performing the combination of special hardware and computer instruction.

Claims

1. A method for providing countermeasure protection in an audio signal by one or more processors, comprising: A security defense against automatic audio analysis of an audio signal including speech is provided on one or more audio devices by adding adversarial noise to an audio signal including speech, such that the adversarial noise distorts the speech to thwart automatic audio analysis, wherein the adversarial noise is added to misclassify the speech at one of a phoneme level, a word level, or a sentence level by the automatic audio analysis while maintaining the speech being understandable to a user.

2. The method of claim 1, further comprising applying the countermeasure noise to a digital audio signal, an analog audio signal, or a combination thereof.

3. The method of claim 1, further comprising determining or selecting a type of the countermeasure noise to be implemented on a digital audio signal, an analog audio signal, or a combination thereof in the communication.

4. The method of claim 1 , further comprising analyzing the audio signal using a machine learning operation, wherein: The audio signals are communicated and sent in real time from the user or from recorded messages located in an audio database. The method according to claim 1 , further comprising estimating the theoretical security defense strength against noise.

6. The method of claim 1 further comprising estimating the actual security defense strength against noise via a feedback loop operation.

7. The method according to claim 1 also includes adjusting the degree of strength of the security defense against noise.

8. A system for providing countermeasure protection in an audio signal, comprising: One or more computers having executable instructions that, when executed, cause the system to: A security defense against automatic audio analysis of an audio signal including speech is provided on one or more audio devices by adding adversarial noise to an audio signal including speech, such that the adversarial noise distorts the speech to thwart automatic audio analysis, wherein the adversarial noise is added to misclassify the speech at one of a phoneme level, a word level, or a sentence level by the automatic audio analysis while maintaining the speech being understandable to a user.

9. The system according to claim 8, wherein: The executable instructions apply the countermeasure noise to a digital audio signal, an analog audio signal, or a combination thereof.

10. The system according to claim 8, wherein: The executable instructions determine or select the type of countermeasure noise to implement on a digital audio signal, an analog audio signal, or a combination thereof in the communication.

11. The system according to claim 8, wherein: The executable instructions use machine learning operations to analyze the audio signals, wherein the audio signals are communicated and sent in real time from a user or from recorded messages located in an audio database.

12. The system according to claim 8, wherein: The executable instructions estimate the theoretical security defense strength against noise.

13. The system according to claim 8, wherein: The executable instructions operate via a feedback loop to estimate the actual security defense strength against noise.

14. The system according to claim 8, wherein: The executable instructions adjust the degree of strength of the security defense against noise.

15. A computer program product for providing countermeasure protection in an audio signal by one or more processors, the computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising: An executable portion that provides a security defense against automatic audio analysis of an audio signal on one or more audio devices by adding adversarial noise to an audio signal including speech, such that the adversarial noise distorts the speech to thwart automatic audio analysis, wherein the adversarial noise is added to misclassify the speech at one of a phoneme level, a word level, or a sentence level by the automatic audio analysis while maintaining the speech being understandable to a user.

16. The computer program product of claim 15, further comprising an executable portion for applying the countermeasure noise to a digital audio signal, an analog audio signal, or a combination thereof.

17. The computer program product of claim 15, further comprising an executable portion for determining or selecting a type of the countermeasure noise to implement on a digital audio signal, an analog audio signal, or a combination thereof in a communication.

18. The computer program product of claim 15, further comprising an executable portion for analyzing the audio signal using a machine learning operation, wherein: The audio signals are communicated and sent in real time from the user or from a recorded message located in an audio database.

19. The computer program product of claim 15, further comprising an executable portion, the executable portion: estimating the theoretical security strength of the anti-noise defense; or The actual security defense strength against noise is estimated via a feedback loop operation.

20. The computer program product of claim 15, further comprising an executable portion for adjusting a degree of strength of the security defenses against noise.

Citation Information

Patent Citations

  • Method, device, and program for voice masking

    CN104685560A

  • Eavesdropping-proof recording method and interference shielding device

    CN107295148A

  • Secure dynamic communication network and protocol

    CN107750441A

  • Interference system for preventing eavesdropping recording

    CN107911192A

  • Annotation data protection method, terminal device and system

    CN108364018A