Audio signal generation system and method

Abstract

The method for generating an audio signal may include obtaining a bone conduction audio signal and an air conduction audio signal. The method may further include obtaining a trained machine learning model that provides a mapping relationship between bone conduction data derived from a particular bone conduction audio signal and one or more equivalent air conduction data derived from a particular equivalent air conduction audio signal. The method may further include determining a target equivalent air conduction data set corresponding to the bone conduction audio signal using the trained machine learning model based on the bone conduction audio signal and the air conduction audio signal. The method may further include causing an audio signal output device to output a target audio signal representing a user voice based on the target equivalent air conduction data set.

Description

[Technical field]

[0001] The present application relates generally to the field of signal processing, and more particularly to a system and method for generating an audio signal based on a bone-conducted audio signal and an air-conducted audio signal. [Background technology]

[0002] With the popularization of electronic devices, communication between people becomes more and more convenient. When using electronic devices to communicate, a user can obtain an audio signal when the user speaks by a microphone (e.g., a bone conduction microphone or an air conduction microphone). The audio signal obtained by the microphone can represent the user's voice. However, it may be difficult to ensure that the audio signal obtained by the microphone is clear enough. For example, a bone conduction audio signal obtained by a bone conduction microphone may lose some important information. An air conduction audio signal obtained by an air conduction microphone may be noisy. Therefore, it is desirable to provide a system and method for generating an audio signal with less noise and higher fidelity. Summary of the Invention [Means for solving the problem]

[0003] According to an aspect of the present application, a system for audio signals may be provided. The system may include at least one storage medium and at least one processor in communication with the at least one storage medium. The at least one storage medium may include an instruction set. When executing the instruction set, the at least one processor may cause the system to perform one or more of the following operations: The system may acquire a bone conduction audio signal acquired by a bone conduction sensor and an air conduction audio signal acquired by an air conduction sensor. The bone conduction audio signal and the air conduction audio signal may represent a user's voice. The system may further acquire a trained machine learning model that provides a mapping relationship between a bone conduction data set derived from a particular bone conduction audio signal and one or more equivalent air conduction data sets derived from a particular equivalent air conduction audio signal. The system may further use the trained machine learning model based on the bone conduction audio signal and the air conduction audio signal to determine a target equivalent air conduction data set corresponding to the bone conduction audio signal. The target equivalent air conduction data set may represent semantic content of the user's voice. The system may further cause an audio signal output device to output a target audio signal representing the user's voice based on the target equivalent air conduction data set.

[0004] In some embodiments, to determine a target equivalent air conduction data set corresponding to the bone conducted audio signal based on the bone conducted audio signal and the air conducted audio signal, the system may use a trained machine learning model based on the bone conducted audio signal to determine one or more equivalent air conduction data sets corresponding to the bone conducted audio signal, and the system may further identify the target equivalent air conduction data set from the one or more equivalent air conduction data sets based on the air conducted audio signal.

[0005] In some embodiments, to determine one or more equivalent air conduction data sets corresponding to a bone conducted audio signal using a trained machine learning model based on the bone conducted audio signal, the system can input the bone conducted audio signal to the trained machine learning model to obtain one or more equivalent air conduction data sets corresponding to the bone conducted audio signal.

[0006] In some embodiments, the system can extract bone conduction acoustic features from the bone conduction audio signal to determine one or more equivalent air conduction data sets corresponding to the bone conduction audio signal using a trained machine learning model based on the bone conduction audio signal. The system can further determine the bone conduction data set based on the bone conduction acoustic features. The system can further input the bone conduction data set to the trained machine learning model to obtain one or more equivalent air conduction data sets corresponding to the bone conduction audio signal.

[0007] In some embodiments, to identify a target equivalent air conduction data set from one or more equivalent air conduction data sets based on the air conduction audio signal, the system can extract air conduction acoustic features from the air conduction audio signal. The system can further determine an air conduction data set based on the air conduction acoustic features. For each of the one or more equivalent air conduction data sets, the system can further determine a degree of association between the equivalent air conduction data set and the air conduction data set. The system can further determine an equivalent air conduction data set of the one or more equivalent air conduction data sets that has a maximum degree of association with the air conduction data set. The system can further determine a target equivalent air conduction data set based on one of the one or more equivalent air conduction data sets.

[0008] In some embodiments, the system can convert the target equivalent air conduction data set into a target equivalent air conduction audio signal to cause the audio signal output device to output a target audio signal representing a user voice based on the target equivalent air conduction data set. The system can further determine a target audio signal based on the target equivalent air conduction audio signal. The system can further cause the audio signal output device to output the target audio signal.

[0009] In some embodiments, the noise level of the target audio signal may be less than the noise level of the air conducted audio signal.

[0010] In some embodiments, to determine a target audio signal based on a target equivalent air-conducted audio signal, the system may generate the target audio signal by adding noise to the target equivalent air-conducted audio signal.

[0011] In some embodiments, to generate a trained machine learning model, the system may perform one or more of the following operations: The system may obtain at least two sets of training data. Each of the at least two training data sets may include a first audio signal and a second audio signal representing the same audio sample. The first audio signal may be obtained by a bone conduction audio collection device, and the second audio signal may be obtained by an air conduction audio collection device in a noise-free state. The system may train a preliminary machine learning model using the multiple sets of training data to obtain a trained machine learning model.

[0012] In some embodiments, a preliminary machine learning model can be constructed based on a hidden Markov model.

[0013] In some embodiments, to train the preliminary machine learning model using a plurality of sets of training data to obtain a trained machine learning model, the system can extract a first acoustic feature from a first audio signal. The system can further extract a second acoustic feature from a second audio signal. The system can further determine one or more first bone conduction data sets based on the first acoustic feature. The system can further determine one or more second air conduction data sets based on the second acoustic feature. The system can further determine a plurality of pairs of air conduction data sets and bone conduction data sets, each pair including a first bone conduction data set and a second air-bone conduction data set representing the same meaning. The system can further establish a mapping relationship between each of the one or more first bone conduction data sets and the one or more second air conduction data sets based on the plurality of pairs of air conduction data and bone conduction data.

[0014] In some embodiments, each of the one or more first bone conduction data sets and the one or more second air conduction data sets may include syllables of one or more phonemes.

[0015] In some examples, the trained machine learning model may provide a mapping relationship between each syllable of the bone conducted audio signal and one or more syllables of the air conducted audio signal.

[0016] In some embodiments, the mapping relationship between each syllable in the bone conduction audio data and the one or more syllables in the one or more equivalent air conduction data sets may include a probability of matching each syllable in the bone conduction audio data with each of the one or more syllables in the one or more equivalent air conduction data sets.

[0017] According to another aspect of the present application, a system for audio signals may be provided. The system may include at least one storage medium and at least one processor in communication with the at least one storage medium. The at least one storage medium may include an instruction set. When executing the instruction set, the at least one processor may cause the system to perform one or more of the following operations: The system may acquire a bone conduction audio signal acquired by a bone conduction sensor and an air conduction audio signal acquired by an air conduction sensor. The bone conduction audio signal and the air conduction audio signal may represent a user voice. The system may further acquire a trained machine learning model. The system may generate a target audio signal representing the user voice using the trained machine learning model based on the bone conduction audio signal and the air conduction audio signal. The system may further cause an audio signal output device to output the target audio signal. A noise level of the target audio signal may be less than a noise level of the air conduction audio signal.

[0018] According to yet another aspect of the present application, a method for an audio signal may be provided. The method may include acquiring a bone conduction audio signal acquired by a bone conduction sensor and an air conduction audio signal acquired by an air conduction sensor. The bone conduction audio signal and the air conduction audio signal may represent a user's voice. The method may further include acquiring a trained machine learning model that provides a mapping relationship between a bone conduction data set derived from a particular bone conduction audio signal and one or more equivalent air conduction data sets derived from a particular equivalent air conduction audio signal. The method may further include determining a target equivalent air conduction data set corresponding to the bone conduction audio signal using the trained machine learning model based on the bone conduction audio signal and the air conduction audio signal. The target equivalent air conduction data set may represent semantic content of the user's voice. The method may further include causing an audio signal output device to output a target audio signal representing the user's voice based on the target equivalent air conduction data set.

[0019] According to yet another aspect of the present application, a method for an audio signal may be provided. The method may include acquiring a bone conduction audio signal acquired by a bone conduction sensor and an air conduction audio signal acquired by an air conduction sensor. The bone conduction audio signal and the air conduction audio signal may represent a user's voice. The method may further include acquiring a trained machine learning model. The method may further include generating a target audio signal representing the user's voice using the trained machine learning model based on the bone conduction audio signal and the air conduction audio signal. The method may further include causing an audio signal output device to output the target audio signal having a noise level less than a noise level of the air conduction audio signal.

[0020] According to yet another aspect of the present application, a system for an audio signal may be provided. The system may include an acquisition module, a determination module, and an output module. The acquisition module may be configured to acquire a bone conduction audio signal acquired by a bone conduction sensor, an air conduction audio signal acquired by an air conduction sensor, and a trained machine learning model that provides a mapping relationship between a bone conduction data set derived from a particular bone conduction audio signal and one or more equivalent air conduction data sets derived from a particular equivalent air conduction audio signal. The bone conduction audio signal and the air conduction audio signal may represent a user's voice. The determination module may be configured to determine a target equivalent air conduction data set corresponding to the bone conduction audio signal using the trained machine learning model, where the target equivalent air conduction data set represents a semantic content of the user's voice based on the bone conduction audio signal and the air conduction audio signal. The output module may be configured to cause an audio signal output device to output a target audio signal representing the user's voice based on the target equivalent air conduction data set.

[0021] According to yet another aspect of the present application, a system for an audio signal may be provided. The system may include an acquisition module, a decision module, and an output module. The acquisition module may be configured to acquire a bone conduction audio signal acquired by a bone conduction sensor, an air conduction audio signal acquired by an air conduction sensor, and a trained machine learning model. The bone conduction audio signal and the air conduction audio signal may represent a user's voice. The decision module may be configured to generate a target audio signal representing the user's voice using the trained machine learning model based on the bone conduction audio signal and the air conduction audio signal. The output module may be configured to cause an audio signal output device to output the target audio signal. A noise level of the target audio signal may be less than a noise level of the air conduction audio signal.

[0022] According to yet another aspect of the present application, a non-transitory computer-readable medium may be provided. The non-transitory computer-readable medium may include an instruction set for an audio signal. When executed by at least one processor, the instruction set may direct the at least one processor to perform a method. The method may include acquiring a bone conduction audio signal acquired by a bone conduction sensor and an air conduction audio signal acquired by an air conduction sensor. The bone conduction audio signal and the air conduction audio signal may represent a user voice. The method may further include acquiring a trained machine learning model that provides a mapping relationship between a bone conduction data set derived from a particular bone conduction audio signal and one or more equivalent air conduction data sets derived from a particular equivalent air conduction audio signal. The method may further include determining a target equivalent air conduction data set corresponding to the bone conduction audio signal using the trained machine learning model based on the bone conduction audio signal and the air conduction audio signal. The target equivalent air conduction data set may represent semantic content of the user voice. The method may further include causing an audio signal output device to output a target audio signal representing the user voice based on the target equivalent air conduction data set.

[0023] According to yet another aspect of the present application, a non-transitory computer-readable medium may be provided. The non-transitory computer-readable medium may include a set of instructions for an audio signal. When executed by at least one processor, the set of instructions may direct the at least one processor to perform a method. The method may include acquiring a bone conduction audio signal acquired by a bone conduction sensor and an air conduction audio signal acquired by an air conduction sensor. The bone conduction audio signal and the air conduction audio signal may represent a user voice. The method may further include acquiring a trained machine learning model. The method may further include generating a target audio signal representing the user voice using the trained machine learning model based on the bone conduction audio signal and the air conduction audio signal. The method may further include causing an audio signal output device to output the target audio signal having a noise level less than a noise level of the air conduction audio signal.

[0024] Additional features of the present application may be set forth in part in the description that follows. Additional features of the present application will become apparent to those skilled in the art upon study of the following description and the corresponding drawings, or upon understanding the manufacture or operation of the embodiments. Features of the present application may be realized or learned by practice or use of the various aspects of the methods, means and combinations of the specific embodiments described below.

[0025] The present application will be further illustrated by exemplary embodiments, which are illustrated in more detail in the figures, which are illustrative rather than limiting, and in which like numbers represent similar structures. [Brief description of the drawings]

[0026] [Figure 1] FIG. 1 is a schematic diagram of an example audio signal generation system in accordance with some embodiments of the present application. [Diagram 2]FIG. 2 is a schematic diagram of exemplary hardware and software components of a computing device in accordance with some embodiments of the present application. [Diagram 3] 1 is a schematic diagram of an exemplary audio collection device according to some embodiments of the present application. [Figure 4] FIG. 2 is a block diagram of an example processing device in accordance with some embodiments of the present application. [Diagram 5] FIG. 2 is a block diagram of another example processing device in accordance with some embodiments of the present application. [Figure 6] 1 is a schematic flowchart of an exemplary process for generating an audio signal, in accordance with some embodiments of the present application. [Figure 7] 1 is a schematic flow chart of an exemplary process for determining a target equivalent air conduction data set corresponding to a bone conducted audio signal, according to some embodiments of the present application; [Figure 8] 1 is a schematic flowchart of an exemplary process for generating a trained machine learning model, in accordance with some embodiments of the present application. [Figure 9] FIG. 2 is a schematic diagram of an example preliminary machine learning model according to some embodiments of the present application. [Figure 10] 1 is a schematic flowchart of another exemplary process for generating an audio signal, in accordance with some embodiments of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] In order to more clearly describe the technical means of the embodiments of the present application, the drawings necessary for the description of the embodiments will be briefly described below. However, those skilled in the art can understand that the present application can be practiced without these details. In other circumstances, well-known methods, procedures, systems, assemblies, and / or circuits are described at a relatively high level to avoid unnecessarily obscuring each aspect of the present application. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined in the present application may be applied to other embodiments and application scenarios without departing from the principles and scope of the present application. Thus, the present application is not intended to be limited to the described embodiments, but is to be accorded the widest scope consistent with the scope of the claims.

[0028] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not limiting. As used herein, the singular forms "a", "one" and "the" may include the plural unless the context clearly dictates otherwise. It will be further understood that the terms "comprise" and "containing", when used herein, refer only to the presence of the stated features, integers, steps, operations, assemblies and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, assemblies, components and / or combinations thereof.

[0029] It will be understood that the terms "system," "engine," "unit," "module," and / or "block" used herein are ways to distinguish between various assemblies, elements, parts, portions, or assemblies at different levels in ascending order, however, other terms may be used in place of the above terms if the same purpose is achieved.

[0030] Generally, the words "module", "unit" or "block" as used in the specification refer to logic embodied in hardware or firmware, or a set of software instructions. The modules, units or blocks described herein may be implemented as software and / or hardware, and may be stored in any type of non-transitory computer-readable medium or other storage device. In some embodiments, the software modules / units / blocks may be compiled and linked into an executable program. It will be understood that the software modules may be called from other modules / units / blocks or themselves, and / or may be called in response to a detected event or interrupt. The software modules / units / blocks configured to run on a computing device may be provided on a computer-readable medium, such as an optical disk, a digital video disk, a flash memory drive, a magnetic disk, or any other tangible medium, or may be provided as a digital download (and may initially be stored in a compressed or installable format, and need to be installed, decompressed or decrypted before execution). Here, the software code may be partially or completely stored in the storage device of the computing device that performs the operation, and applied to the operation of the computing device. The software instructions may be embedded in firmware, such as an erasable programmable read-only memory (EPROM). It will be further understood that the hardware modules / units / blocks may be included in connected logic components, such as gates and flip-flops, and / or may include programmable units, such as programmable gate arrays or processors. The modules / units / blocks or functions of the computing devices described herein may be implemented as software modules / units / blocks, but may also be represented in hardware or firmware.Generally, the modules / units / blocks described herein refer to logic modules / units / blocks that may be combined with other modules / units / blocks or divided into sub-modules / sub-units / sub-blocks, regardless of the physical configuration or storage device. The description may apply to a system, engine, or part thereof.

[0031] When a unit, engine, module or block is referred to as being "located on," "connected to," or "coupled to" another unit, engine, module or block, it will be understood that it may be directly located on, connected to, coupled to, or in communication with the other unit, engine, module or block, or that intermediate units, engines, modules or blocks may be present, unless the context clearly dictates otherwise. As used herein, the term "and / or" may include any one or more of the associated listed items or combinations thereof.

[0032] These and other features, characteristics and the function and method of operation of the associated structural elements, combinations of parts and economies of manufacture of the present application may become more apparent from the following description of the drawings, all of which form a part of the specification of this application. It is understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of the present application. It is understood that the drawings are not drawn to scale.

[0033] The flowcharts used herein illustrate operations performed by systems according to some embodiments of the present application. It will be understood that the operations in the flowcharts do not have to be performed in order. Instead, the steps may be processed in reverse order or simultaneously. Also, one or more other operations may be added to these flowcharts. One or more operations may be removed from these flowcharts.

[0034] An aspect of the present application relates to a system and method for generating an audio signal. The system can obtain a bone conduction audio signal obtained by a bone conduction sensor and an air conduction audio signal obtained by an air conduction sensor. The bone conduction audio signal and the air conduction audio signal can represent a user's voice. The system can further obtain a trained machine learning model that provides a mapping relationship between bone conduction data derived from the bone conduction audio signal and one or more equivalent air conduction data sets. The system can further use the trained machine learning model based on the bone conduction audio signal and the air conduction audio signal to determine a target equivalent air conduction data set corresponding to the bone conduction audio signal. The target equivalent air conduction audio data set can represent the semantic content of the user's voice. The system can further cause the audio signal output device to output a target audio signal representing the user's voice based on the target equivalent air conduction data set. Compared to the bone conduction audio signal, the system and method of the present application can generate a target audio signal with higher fidelity, which includes more frequency components than the bone conduction audio signal. Compared to the air conduction audio signal, the system and method of the present application can generate a target audio signal with a noise level lower than the noise level of the air conduction audio signal.

[0035] 1 is a schematic diagram of an exemplary audio signal generation system 100 according to some embodiments of the present application. The audio signal generation system 100 may include an audio collection device 110, a server 120, a terminal 130, a storage device 140, and a network 150.

[0036] The audio collecting device 110 can collect the user's voice, voice or speech to obtain audio data (e.g., audio signals) when the user speaks. For example, when the user speaks, the user's voice can cause air vibrations around the user's mouth and / or vibrations of the user's body tissues (e.g., skull). The audio collecting device 110 can receive and convert the vibrations into electrical signals (e.g., analog signals or digital signals), also referred to as audio data. The audio data may be transmitted in the form of electrical signals to the server 120, the terminal 130 and / or the storage device 140 via the network 150. In some embodiments, the audio collecting device 110 may include a recorder, earphones (e.g., Bluetooth earphones), wired earphones, hearing aid devices, etc.

[0037] In some embodiments, the audio collecting device 110 may be connected to a speaker via a wireless connection (e.g., network 150) and / or a wired connection. The audio data may be transmitted to the speaker for playing and / or reproducing the user voice. In some embodiments, the speaker and the audio collecting device 110 may be integrated into one device, e.g., earphones. In some embodiments, the audio collecting device and the speaker may be separate from each other. For example, the audio collecting device 110 may be attached to a first terminal (e.g., earphones) and the speaker may be attached to another terminal (e.g., terminal 130).

[0038] In some embodiments, the audio collecting device may include a bone conduction microphone 112 and an air conduction microphone 114. The bone conduction microphone 112 may include one or more bone conduction sensors (also referred to as a bone conduction assembly) that collect audio signals (also referred to as bone conduction audio signals) conducted through a user's skeleton (e.g., skull) when the user speaks. The bone conduction audio signals may be generated by the bone conduction microphone 112, which collects vibration signals of the user's skeleton (e.g., skull) when the user speaks. In some embodiments, the one or more bone conduction sensors may form a bone conduction sensor array. In some embodiments, the bone conduction microphone 112 may be placed on and / or in contact with an area of ​​the user's body to collect bone conduction data. The area of ​​the user's body may include the forehead, neck (e.g., throat), face (e.g., area around the mouth, chin), top of the head, mastoid process, area around or inside the ear, temples, etc., or any combination thereof. For example, the bone conduction microphone 112 may be located at and / or in contact with the tragus, pinna, inner ear canal, outer ear canal, etc. In some embodiments, one or more characteristics of the bone conduction data may vary depending on the region of the user's body that the bone conduction microphone 112 is located at and / or in contact with. For example, bone conduction data collected by a bone conduction microphone 112 located in the area around the ear has higher energy than a bone conduction data set collected by a bone conduction microphone 112 located on the forehead.

[0039] The air conduction microphone 114 may include one or more air conduction sensors (also referred to as an air conduction assembly) that collect audio signals conducted through the air (also referred to as air conduction audio signals) when a user speaks. In some embodiments, the one or more air conduction sensors may form an air conduction sensor array. In some embodiments, the air conduction microphone 114 may be located within a certain distance (e.g., 0 centimeters, 1 centimeter, 2 centimeters, 5 centimeters, 10 centimeters, 20 centimeters, etc.) from the user's mouth. One or more characteristics of the air conduction data (e.g., the average amplitude of the air conduction data) may vary depending on the distance between the air conduction microphone 114 and the user's mouth. For example, the greater the distance between the air conduction microphone 114 and the user's mouth, the smaller the average amplitude of the air conduction microphone dataset may be.

[0040] In some embodiments, the bone conduction microphone 112 and the air conduction microphone 114 may be integrated into one device (e.g., an audio collection device). In some embodiments, the bone conduction microphone 112 and the air conduction microphone 114 may be integrated into different devices.

[0041] In some embodiments, server 120 may be a single server or a collection of servers. The collection of servers may be centralized (e.g., a data center) or distributed (e.g., server 120 may be a distributed system). In some embodiments, server 120 may be local or remote. For example, server 120 may access information and / or data stored in terminal 130 and / or storage device 140 via network 150. Also, for example, server 120 may be directly connected to terminal 130 and / or storage device 140 to access stored information and / or data. In some embodiments, server 120 may be implemented in a cloud platform. By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof. In some embodiments, server 120 may be implemented in a computing device 200, which may have one or more assemblies as shown in FIG. 2 of the present application.

[0042] In some embodiments, the server 120 may include a processing unit 122. The processing unit 122 may process information and / or data related to the generation of the audio signal to perform one or more functions described herein. For example, the processing unit 122 may obtain a bone conduction audio signal collected by the bone conduction microphone 112 and an air conduction audio signal collected by the air conduction microphone 114. The bone conduction audio signal and the air conduction audio signal may represent a user's voice. The processing unit 122 may further obtain a trained machine learning model that provides a mapping relationship between a bone conduction data set derived from a particular bone conduction audio signal (e.g., the bone conduction audio signal collected by the bone conduction microphone 112) and one or more equivalent air conduction data sets. The processing unit 122 may use the trained machine learning model to generate a target equivalent air conduction data set corresponding to the bone conduction audio signal based on the bone conduction audio signal and the air conduction audio signal. The target equivalent air conduction audio data set may represent the semantic content of the user's voice, and its noise level is less than the noise level of the air conduction audio signal. The processing unit 122 may further cause an audio signal output device to output a target audio signal representative of the user's voice based on the target equivalent air conduction data set.

[0043] A trained machine learning model as used herein may be updated from time to time (e.g., periodically or aperiodically) based on a sample set that is at least partially different from the original sample set that determined the original trained machine learning model. For example, the trained machine learning model may be updated based on a sample set that includes new samples not included in the original sample set. In some embodiments, the determination and / or update of the trained machine learning model may be performed on a processing device, and the application of the trained machine learning model may be performed on a different processing device. In some embodiments, the determination and / or update of the trained machine learning model may be performed on a processing device of a system different from the system 100, or on a server different from the processing device 122 that includes the application on which the trained machine learning model is executed. For example, the determination and / or update of the machine learning model may be performed on a first system of a vendor that provides and / or maintains such machine learning models and / or has access to training samples for determining and / or updating the trained machine learning model, and the generation of the audio signal based on the machine learning model may be performed on a second system of a client of the vendor. In some embodiments, the determination and / or update of the trained machine learning model may be performed online in response to a request for audio signal generation. In some embodiments, the determination and / or update of the trained machine learning model may be performed offline.

[0044] In some embodiments, the processing device 122 may include one or more processing engines (e.g., a single-chip processing engine or a multi-chip processing engine). By way of example only, the processing device 122 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), an application specific instruction set processor (ASIP), a graphics processing unit (GPU), a physical processing unit (PPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a microcontroller unit, a reduced instruction set computer (RISC), a microprocessor, or the like, or any combination thereof. In some embodiments, the processing device 122 may be integrated into the audio collecting device 110.

[0045] In some embodiments, the terminal 130 may include a mobile device 130-1, a tablet computer 130-2, a laptop computer 130-3, a vehicle's built-in device 130-4, a wearable device 130-5, etc., or any combination thereof. In some embodiments, the mobile device 130-1 may include a smart home device, a smart mobile device, a virtual reality device, an augmented reality device, etc., or any combination thereof. In some embodiments, the smart home device may include a smart lighting device, a smart appliance control device, a smart surveillance device, a smart television, a smart video camera, an intercom, etc., or any combination thereof. In some embodiments, the smart mobile device may include a smartphone, a personal digital assistant (PDA), a gaming device, a navigation device, a point of sale device (POS), etc., or any combination thereof. In some embodiments, the virtual reality device and / or the augmented reality device may include a virtual reality helmet, a virtual reality glasses, a virtual reality eye mask, an augmented reality helmet, an augmented reality glasses, an augmented reality eye mask, etc., or any combination thereof. For example, the virtual reality device and / or the augmented reality device may include Google Glass®, Oculus Rift, Hololens®, Gear VR®, etc. In some embodiments, the in-vehicle device 130-4 may include an in-vehicle computer, an in-vehicle television, etc. In some embodiments, the terminal 130 may be a device having positioning technology to position the passenger and / or the terminal 130. In some embodiments, the wearable device 130-5 may include a smart bracelet, smart footwear, smart glasses, smart helmet, smart watch, smart clothing, smart backpack, smart accessories, etc., or any combination thereof. In some embodiments, the audio collecting device 110 and the terminal 130 may be integrated into one device.

[0046] The storage device 140 can store data and / or instructions. For example, the storage device 140 can store a plurality of sets of voice sample data, one or more machine learning models, trained machine learning models and / or constructed filters, audio data collected by the bone conduction microphone and the air conduction microphone 114, and the like. In some embodiments, the storage device 140 can store data acquired from the terminal 130 and / or the audio collecting device 110. In some embodiments, the storage device 140 can store data and / or instructions for performing the exemplary methods described herein, which may be executed by the server 120. In some embodiments, the storage device 140 may include mass memory, removable memory, volatile read-write memory, read-only memory (ROM), and the like, or any combination thereof. Exemplary mass memory may include magnetic disks, optical disks, solid-state drives, and the like. Exemplary removable memory may include flash drives, floppy disks, optical disks, memory cards, compact disks, magnetic tapes, and the like. Exemplary volatile read-write memory may include random access memory (RAM). Exemplary RAM may include dynamic random access memory (DRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), static random access memory (SRAM), thyristor random access memory (T-RAM), zero capacitor random access memory (Z-RAM), etc. Exemplary ROM may include mask read only memory (MROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), compact disc read only memory (CD-ROM), digital versatile disc read only memory, etc. In some embodiments, the storage device 140 may be implemented in a cloud platform.By way of example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-cloud, etc., or any combination thereof.

[0047] In some embodiments, the storage device 140 may be connected to a network 150 and communicate with one or more components of the audio signal generation system 100 (e.g., the audio collecting device 110, the server 120, and the terminal 130). The one or more components of the audio signal generation system 100 may access data or instructions stored in the storage device 140 via the network 150. In some embodiments, the storage device 140 may be directly connected to or in communication with one or more components of the audio signal generation system 100 (e.g., the audio collecting device 110, the server 120, and the terminal 130). In some embodiments, the storage device 140 may be part of the server 120.

[0048] The network 150 can facilitate the exchange of information and / or data. In some embodiments, one or more components of the audio signal generating system 100 (e.g., the audio collecting device 110, the server 120, the terminal 130, and the storage device 140) can transmit information and / or data to other components of the audio signal generating system 100 via the network 150. For example, the server 120 can obtain bone conduction data and air conduction data from the terminal 130 via the network 150. In some embodiments, the network 150 can be any form of wired or wireless network, or a combination thereof. By way of example only, the network 150 can include a cable network, a wired network, an optical fiber network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network, a ZigBee network, a near field communication (NFC) network, or the like, or any combination thereof. In some embodiments, network 150 may include one or more network access points, such as, for example, wired or wireless network access points, such as base stations and / or Internet switching points, by which one or more components of audio signal production system 100 can be connected to network 150 to exchange data and / or information.

[0049] Those skilled in the art will understand that the elements (or components) of the audio signal generating system 100 may operate by electrical and / or electromagnetic signals. For example, when the bone conduction microphone 112 transmits bone conduction data to the server 120, the processor of the bone conduction microphone 112 may generate an electrical signal encoding the bone conduction data. The processor of the bone conduction microphone 112 may then transmit the electrical signal to an output port. When the bone conduction microphone 112 communicates with the server 120 via a wired network, the output port may be physically connected to a cable, which may further transmit the electrical signal to an input port of the server 120. When the bone conduction microphone 112 communicates with the server 120 via a wireless network, the output port of the bone conduction microphone 112 may be one or more antennas that convert the electrical signal into an electromagnetic signal. Similarly, the air conduction microphone 114 may transmit air conduction data to the server 120 via an electrical or electromagnetic signal. In an electronic device such as the terminal 130 and / or the server 120, when a processor thereof processes, transmits, and / or executes instructions, the instructions and / or operations are performed via electrical signals. For example, when a processor retrieves or retrieves data from a storage medium, the processor may transmit electrical signals to a read / write device of the storage medium, which may read structured data in the storage medium or write structured data to the storage medium. The structured data may be transmitted to the processor in the form of electrical signals via a bus of the electronic device. Here, the electrical signal may be one electrical signal, a series of electrical signals, and / or at least two discontinuous electrical signals.

[0050] FIG. 2 shows a schematic diagram of an exemplary computing device according to some embodiments of the present application. The computing device may be a computer configured to implement any particular system according to some embodiments of the present application, such as the processing unit 122 of FIG. 1, and / or a computer having a particular function. The computing device 200 may implement any component that performs one or more functions disclosed in the present application. For example, the server 120 may be implemented in a hardware device, a software program, firmware, or any combination thereof, of a computer such as the computing device 200. For simplicity, FIG. 2 shows only one computing device. In some embodiments, the functionality of the computing device may be implemented by a group of similar platforms in a distributed mode to distribute the processing load of the system.

[0051] The computing device 200 may include a communication port 250 that can be connected to a network capable of data communication. The computing device 200 may further include a processor 220 configured to execute instructions and including one or more processors. The schematic computer platform may include an internal communication bus 210, different types of program storage units and data storage units (e.g., a hard disk 270, a read only memory (ROM) 230, a random access memory (RAM) 240), various data files applicable for computer processing and / or communication, and some program instructions executable by the processor 220. The computing device 200 may further include an I / O device 260 that can support input and output of data streams between the computing device 200 and other components. The computing device 200 may also receive programs and data via a communication network.

[0052] 3 is a schematic diagram of an exemplary audio collecting device according to some embodiments of the present application. As shown in FIG. 3, the audio collecting device 300 may include a collecting module 310 and a signal processing module 320.

[0053] The collection module 310 may be configured to collect a vibration signal generated when a user speaks and convert the vibration signal into an electrical signal (also referred to as an audio signal). The vibration signal may represent the user's voice. Specifically, by transmitting the vibration signal generated when a user speaks in the form of mechanical vibrations to one or more assemblies of the collection module 310 (e.g., a diaphragm of a microphone assembly), the one or more assemblies of the collection module 310 may generate corresponding mechanical vibrations.

[0054] For illustrative purposes, the collection module 310 may include a bone conduction assembly 311 and / or an air conduction assembly 312. The bone conduction assembly 311 may include a vibration assembly, a magnet assembly, and a coil. The vibration assembly may collect vibration signals (also referred to as bone conduction vibration signals or sound waves) that are generated when a user speaks and are conducted through the user's skeleton (e.g., skull) as described elsewhere in this application (e.g., FIG. 1 and its description). The bone conduction assembly 311 may convert the vibration signals into electrical signals (i.e., audio signals). For example, vibration of a diaphragm may cause the coil to vibrate. The coil may vibrate in a magnetic field generated by the magnet assembly to generate an electrical signal.

[0055] The air-conducting assembly 312 can collect vibration signals conducted through the air when a user speaks (also referred to as air-conducted vibration signals), as described elsewhere herein (e.g., FIG. 1 and its description). The air-conducting assembly 312 may include a vibration assembly, a magnet assembly, and a coil. As described elsewhere herein (e.g., FIG. 1 and its description), the vibration component can collect vibration signals conducted through the air (also referred to as bone-conducted vibration signals or sound waves), as described elsewhere herein (e.g., FIG. 1 and its description). The air-conducting assembly 312 can convert the vibration signals into electrical signals (i.e., audio signals). For example, the vibration of the diaphragm may cause the coil to vibrate. The coil can vibrate in a magnetic field generated by the magnet assembly to generate an electrical signal.

[0056] In some embodiments, the bone conduction assembly 311 and the air conduction assembly 312 may be two independent functional devices or two independent assemblies of a single device. As used herein, the independence of the first device and the second device indicates that the operation of the first / second device is not due to the operation of the second / first device, in other words, the operation of the first / second device is not the result of the operation of the second / first device. Taking the bone conduction assembly and the air conduction assembly as examples, the air conduction assembly and the bone conduction assembly are independent because when a user speaks, the vibration of the air drives the air conduction assembly to generate an air conduction vibration signal, and when a user speaks, the vibration of the bone structure drives the bone conduction assembly to generate a bone conduction vibration signal.

[0057] In some embodiments, the bone-conducted audio signal or the air-conducted audio signal may be represented by a superposition of multiple waves (e.g., sine waves, harmonics, etc.) having different frequencies and / or intensities (i.e., amplitudes). As used herein, a wave having a particular frequency may be referred to as a frequency component having a particular frequency. In some embodiments, the frequency components included in the bone-conducted audio signal acquired by the bone-conducting assembly 321 may be in a frequency range of 0 Hz to 20 kHz, 20 Hz to 10 kHz, 20 Hz to 4000 Hz, 20 Hz to 3000 Hz, 1000 Hz to 3500 Hz, 1000 Hz to 3000 Hz, 1500 Hz to 3000 Hz, etc. In some embodiments, the frequency components included in the air-conducted audio signal acquired by the air-conducting assembly 322 may be in a frequency range of 0 Hz to 20 kHz, or 20 Hz to 20 kHz, or 1000 Hz to 10 kHz, etc.

[0058] The signal processing module 320 may be electrically coupled to the bone conduction assembly 311 and / or the air conduction assembly 312. The signal processing module 320 may receive the vibration signal (i.e., the audio signal) from the collection module 310 and process the vibration signal (i.e., the audio signal). For example, the signal processing module 320 may process the bone conduction audio signal and the air conduction audio signal according to process 600, process 700, and / or process 1000.

[0059] As another example, the signal processing module 320 may process the audio signal by performing various signal processing operations, such as sampling, digitization, compression, frequency allocation, frequency modulation, encoding, etc., or a combination thereof.

[0060] In some embodiments, the signal processing module 320 may include the same or a different processor as the processing unit 122 .

[0061] It should be noted that the above description of the audio collecting device 300 is merely for explanation and is not intended to limit the scope of the present application. Those skilled in the art may make various changes and modifications based on the description of the present application. However, these changes and modifications do not depart from the scope of the present application. For example, the audio collecting device 300 may include an output module configured to transmit a processed audio signal (e.g., a target audio signal described elsewhere in the present application) to an output device (e.g., a terminal equipped with a speaker (e.g., a bone conduction speaker and / or an air conduction speaker)). As another example, the output module may include a speaker (e.g., a bone conduction speaker and / or an air conduction speaker) configured to output the processed audio signal.

[0062] 4 is a block diagram of an example processing device 122 according to some embodiments of the present application. The processing device 122 may be the example processing device 122 described with reference to FIG 1. In some embodiments, the processing device 122 may be configured to apply one or more machine learning models in generating the audio signal.

[0063] As shown in FIG. 4, the processing unit 122 may include an acquisition module 402 , a determination module 404 , and an output module 406 .

[0064] The acquisition module 402 may be configured to acquire information about the audio signal generating system 100. For example, the acquisition module 402 may acquire a bone conduction audio signal acquired by a bone conduction sensor and an air conduction audio signal acquired by an air conduction sensor. More description regarding acquiring a bone conduction audio signal and an air conduction audio signal may be found elsewhere in this application. See, for example, operations 602 and 604 in FIG. 6 and their associated description. As another example, the acquisition module 402 may acquire a trained machine learning model. The trained machine learning model provides a mapping relationship between a bone conduction data set derived from a particular bone conduction audio signal and one or more equivalent air conduction data sets derived from a particular equivalent air conduction signal corresponding to the particular bone conduction audio signal. More description regarding the trained machine learning model may be found elsewhere in this application. See, for example, operation 606 in FIG. 6 and its associated description.

[0065] Based on the bone conduction audio signal and the air conduction audio signal, the determination module 404 can determine a target equivalent air conduction data set corresponding to the bone conduction audio signal based on a trained machine learning model. More description regarding determining the target equivalent air conduction data set can be found elsewhere in this application. See operation 608 in FIG. 6 and its related description.

[0066] The decision module 404 can further generate a target audio signal representative of the user's voice based on the bone conduction audio signal and the air conduction audio signal using a trained machine learning model. More information regarding the generation of the target audio signal can be found elsewhere in this application. See operation 1008 of FIG. 10 and its associated description.

[0067] The output module 406 may be configured to cause an audio signal output device to output the target audio signal. More information regarding outputting the target audio signal can be found elsewhere in this application. See, for example, operation 610 in FIG. 6 and operation 1010 in FIG. 10 and their associated descriptions.

[0068] 5 is a block diagram of an example processing device 122 according to some embodiments of the present application. The processing device 122 may be the example processing device 122 described with reference to FIG. 1. In some embodiments, the processing device 122 may be configured to generate one or more machine learning models. In some embodiments, the processing device 122 described in FIG. 4 and the processing device 122 described in FIG. 5 may each be implemented in a processing unit. Alternatively, the processing device 122 described in FIG. 4 and / or the processing device 122 described in FIG. 5 may be implemented in the same computing device (e.g., computing device 200).

[0069] As shown in FIG. 5, the processing unit 122 may include an acquisition module 502 and a model generation module 504 .

[0070] The acquisition module 502 may be configured to acquire at least two training data. In some embodiments, each of the multiple training data sets may include a first audio signal and a second audio signal representing the same audio sample. In some embodiments, each of the multiple training data sets may include a bone conduction audio signal, an air conduction audio signal, and a reference air conduction audio signal. In training the preliminary machine learning model, the bone conduction audio signal and the air conduction audio signal in each training data set may be used as an input of the preliminary machine learning model, and the reference air conduction audio signal may be used as a desired output of the preliminary machine learning model. More description regarding acquiring training data can be found elsewhere in this application. See, for example, operation 802 of FIG. 8, operation 1006 of FIG. 10, and their associated descriptions.

[0071] The model generation module 504 may be configured to generate one or more trained machine learning models by training the preliminary machine learning model using multiple sets of training data. In some embodiments, one or more machine learning models may be generated according to a machine learning algorithm. The machine learning algorithm may include, but is not limited to, an artificial neural network algorithm, a deep learning algorithm, a decision tree algorithm, an association rule algorithm, an inductive logic programming algorithm, a support vector machine algorithm, a clustering algorithm, a Bayesian network algorithm, a reinforcement learning algorithm, a representation learning algorithm, a similarity metric learning algorithm, a sparse dictionary learning algorithm, a genetic algorithm, a rule-based machine learning algorithm, and the like, or any combination thereof. The machine learning algorithm for generating one or more machine learning models may be a supervised learning algorithm, a semi-supervised learning algorithm, an unsupervised learning algorithm, and the like. More description regarding generating one or more machine learning models may be found elsewhere in this application. See, for example, operation 804 of FIG. 8, operation 1006 of FIG. 10, and their associated descriptions.

[0072] It should be noted that the above description is for illustrative purposes only and is not intended to limit the scope of the present application. Those skilled in the art may make various changes and modifications based on the description of the present application. However, these changes and modifications do not depart from the scope of the present application. In some embodiments, the processing device 122 described in FIG. 4 and / or the processing device 122 described in FIG. 5 may share two or more modules, and any one of the modules may be divided into two or more units. For example, the processing device 122 described in FIG. 4 and the processing device 122 described in FIG. 5 may share the same acquisition module, that is, the acquisition module 402 and the acquisition module 502 are the same module. In some embodiments, the processing device 122 shown in FIG. 1 and / or the processing device 122 described in FIG. 4 may include one or more additional modules, such as a storage module (not shown) for storing data. In some embodiments, the processing device 122 described in FIG. 4 and the processing device 122 described in FIG. 5 may be integrated into one processing device 122.

[0073] FIG. 6 is a schematic flow chart of an exemplary process for generating an audio signal, according to some embodiments of the present application. In some embodiments, the process 600 may be implemented as instructions (e.g., an application program) stored in the storage device 140, the ROM 230, the RAM 240, or the memory 390. FIG. 6 is a schematic flow chart of an exemplary process for generating an audio signal, according to some embodiments of the present application. The process operations shown below are for illustrative purposes only. In some embodiments, the process 600 may be completed by one or more additional operations not described and / or one or more operations not discussed. Also, the order of operations of the process 600 shown in FIG. 6 and described below is not limited.

[0074] At 602, the processing unit 122 (eg, the acquisition module 402) can acquire a bone conduction audio signal acquired by a bone conduction sensor.

[0075] As used herein, a bone conduction sensor may be any sensor (e.g., bone conduction microphone 112) that can acquire vibration signals conducted through a user's skeleton (e.g., skull) when the user speaks, as described elsewhere in this application (e.g., FIG. 1 and its description). The bone conduction sensor or any other device (e.g., amplifier, analog-to-digital converter (ADC), etc.) can convert the vibration signals acquired by the bone conduction sensor into audio signals (also referred to as audio data). The audio signals acquired by the bone conduction sensor may be referred to as bone conduction audio signals. An audio signal is data that includes the semantic content of speech.

[0076] In some embodiments, the processing device 122 may acquire the bone conduction audio signal in real time or periodically from the bone conduction sensor (e.g., the bone conduction microphone 112), the terminal 130, the storage device 140, or any other storage device via the network 150. For example, the bone conduction audio signal may be acquired and / or generated by the bone conduction sensor when the user speaks within a time period (e.g., 20 ms, 30 ms, 40 ms, etc.) that is less than a threshold value. The bone conduction sensor may directly transmit the bone conduction audio signal to the processing device 122 in real time. As used herein, the bone conduction sensor transmitting the bone conduction audio signal to the processing device 122 in real time means that the acquisition time at which the bone conduction sensor acquires the bone conduction audio signal and the reception time at which the processing device 122 acquires the bone conduction audio signal are less than a threshold value (e.g., 20 ms, 20 ms, 30 ms, 40 ms, etc.). As another example, the bone conduction sensor may transmit the bone conduction audio signal periodically, for example, every second, every minute, every hour, etc.

[0077] A bone conduction audio signal may be represented by a superposition of multiple waves (e.g., sine waves, harmonics, etc.) having different frequencies and / or intensities (i.e., amplitudes). As used herein, a wave having a particular frequency may be referred to as a frequency component having a particular frequency. In some embodiments, the frequency components included in the bone conduction audio signal collected by the bone conduction sensor may be within a frequency range of 0 Hz to 20 kHz, 20 Hz to 10 kHz, 20 Hz to 4000 Hz, 20 Hz to 3000 Hz, 1000 Hz to 3500 Hz, 1000 Hz to 3000 Hz, 1500 Hz to 3000 Hz, etc.

[0078] In some embodiments, the bone conduction audio signal may be collected and / or generated by fixing a bone conduction sensor to an area of ​​the user's body and / or contacting the bone conduction sensor to the user's skin. The area of ​​the user's body that contacts the bone conduction sensor to collect the bone conduction audio signal may include, but is not limited to, the forehead, neck (e.g., throat), mastoid process, area around or inside the ear, temples, face (e.g., area around the mouth, chin), top of the head, etc. For example, the bone conduction microphone 112 may be located at and / or contact the tragus, pinna, inner ear canal, outer ear canal, etc. In some embodiments, the bone conduction audio signal may be different depending on different areas of the user's body that contact the bone conduction sensor. For example, different areas of the user's body that contact the bone conduction sensor may cause changes in the frequency content of the bone conduction audio signal, the acoustic characteristics (e.g., amplitude of the frequency content), noise included in the bone conduction audio signal, etc. For example, the signal strength of a bone conduction audio signal collected by a bone conduction sensor located at the neck is greater than the signal strength of a bone conduction audio signal collected by a bone conduction sensor located at the tragus, which in turn is greater than the signal strength of a bone conduction audio signal collected by a bone conduction sensor located at the ear canal. As another example, a bone conduction audio signal collected by a first bone conduction sensor located at an area around the user's ear may contain more frequency components than a bone conduction audio signal having the same configuration but collected simultaneously by a second bone conduction sensor located at the top of the user's head. In some embodiments, the bone conduction audio signal may be collected by a bone conduction sensor located at an area of ​​the user's body, and the specific pressure applied by the bone conduction sensor may range from 0 Newtons to 1 Newtons, or from 0 Newtons to 0.8 Newtons, etc. For example, the bone conduction audio signal may be collected by a bone conduction sensor located at the tragus of the user's body, and the specific pressure applied by the bone conduction sensor may range from 0 Newtons to 1 Newtons, or from 0 Newtons to 0.8 Newtons, etc.When the bone conduction sensor applies different pressures to the same region of the user's body, the frequency content of the bone conduction audio signal, the acoustic features (e.g., the amplitude of the frequency content), the noise in the bone conduction audio signal, etc. may change. For example, when the pressure increases from 0 N to 0.8 N, the signal strength of the bone conduction data may gradually increase, and then the increase in the signal strength may slow down until it is saturated.

[0079] The bone-conducted audio signal may represent the speech of a user, i.e., the user's voice. In other words, the bone-conducted audio signal may include acoustic features and / or semantic information that may reflect or indicate the semantic content of the user's voice. As used herein, the semantic information represented by the audio signal (e.g., the bone-conducted audio signal) may refer to the semantic content recognized from the audio data using a voice recognition technique. The semantic information recognized from the audio signal (e.g., the bone-conducted audio signal) using a voice recognition technique may be referred to as the estimated semantic content. The semantic content of the user's voice may be referred to as the actual semantic content. The estimated semantic content recognized from the audio signal (e.g., the bone-conducted audio signal) may be the same as or similar to the actual semantic content of the user's voice. As used herein, the semantic content of the audio signal (i.e., the estimated semantic content) reflecting or indicating the actual semantic content of the user's voice refers to a similarity or relatedness between the estimated semantic content recognized from the audio signal (e.g., the bone-conducted audio signal) and the actual semantic content of the user's voice being greater than a threshold value (e.g., 90%, 95%, 99%, etc.).

[0080] The acoustic features of the bone-conducted audio signal may include one or more features related to duration, one or more features related to energy, one or more features related to fundamental frequency, one or more features related to frequency spectrum, one or more features related to phase spectrum, etc. The features related to duration may be referred to as duration features. Exemplary duration features may include speaking rate, short-term average zero-crossing rate, etc. The features related to energy may be referred to as energy or amplitude features. Exemplary energy or amplitude features may include short-term average energy, short-term average amplitude, short-term energy slope, average amplitude rate of change, short-term maximum amplitude, etc. The features related to fundamental frequency may be referred to as fundamental frequency features. Exemplary fundamental frequency features may include fundamental frequency, fundamental frequency of fundamental frequency, mean fundamental frequency, maximum fundamental frequency, fundamental frequency range, etc. Exemplary features related to frequency spectrum may include formant features, Linear Prediction Cepstrum Coefficients (LPCC), Mel-Frequency Cepstrum Coefficients (MFCC), etc. Exemplary features associated with the phase spectrum may include the instantaneous phase, the initial phase, and the like.

[0081] In some embodiments, the bone conducted audio signal may be a time domain audio signal, a frequency domain audio signal, etc. The bone conducted audio signal may include an analog signal or a digital signal.

[0082] In some embodiments, the bone conduction audio signal may be processed to obtain one or more bone conduction data sets. In some embodiments, the bone conduction audio signal may be divided into one or more portions, each portion corresponding to a time period. For example, a framing operation may be performed on the bone conduction audio signal based on a time window. One bone conduction data set may include an audio frame. In some embodiments, a feature extraction operation may be performed on the bone conduction audio signal to obtain one or more bone conduction data sets, and / or an audio recognition operation may be performed based on the extracted acoustic features to obtain one or more bone conduction data sets. The bone conduction data sets may include bone conduction acoustic features that may form a phoneme, a phoneme consisting of the bone conduction acoustic features, a syllable consisting of one or more phonemes, a word consisting of one or more syllables, a phrase consisting of one or more words, a sentence consisting of one or more phrases, etc., or any combination thereof.

[0083] At 604, the processing unit 122 (eg, the acquisition module 402) may acquire an air-conduction audio signal acquired by an air-conduction sensor.

[0084] An air conduction sensor as used herein may be any sensor (e.g., air conduction microphone 114) capable of acquiring a vibration signal conducted through the air when a user speaks, as described elsewhere in this application (e.g., FIG. 1 and its description). The vibration signal acquired by the air conduction sensor may be converted into an audio signal (also referred to as audio data) by the air conduction sensor or any other device (e.g., an amplifier, an analog-to-digital converter (ADC), etc.). The audio signal acquired by the air conduction sensor may be referred to as an air conduction audio signal.

[0085] In some embodiments, the processing unit 122 may acquire air-conducted audio signals in real time or periodically from an air-conducted sensor (e.g., air-conducted microphone 114), the terminal 130, the storage device 140, or any other storage device via the network 150. When a user speaks, the air-conducted audio signals may be acquired and / or generated by the air-conducted sensor. In some embodiments, the air-conducted sensor may be fixed within a certain distance threshold (e.g., 0 centimeters, 1 centimeter, 2 centimeters, 5 centimeters, 10 centimeters, 20 centimeters, etc.) from the user's mouth to collect the air-conducted audio signals. In some embodiments, the air-conducted audio signals (e.g., the average amplitude of the air-conducted audio signals) may vary depending on the distance between the air-conducted sensor and the user's mouth.

[0086] In some embodiments, the air conducted audio signal may be an audio signal in the time domain, an audio signal in the frequency domain, etc. The air conducted audio signal may include an analog signal or a digital signal.

[0087] The air-conducted audio signal may be represented by a superposition of multiple waves (e.g., sine waves, harmonics, etc.) having different frequencies and / or intensities (i.e., amplitudes). In some embodiments, the frequency components included in the air-conducted audio signal collected by the air-conducted sensor may be in a frequency range such as 0 Hz to 20 kHz, 20 Hz to 20 kHz, or 1000 Hz to 10 kHz. When a user speaks, the air-conducted audio signal may be collected and / or generated by the air-conducted data. The air-conducted audio signal may represent the user's speech, i.e., the user's meaning. For example, the air-conducted audio signal may include acoustic features and / or semantic information that may reflect the semantic content of the user's voice. As described in operation 602, the acoustic features of the air-conducted audio signal may include one or more features related to duration, one or more features related to energy, one or more features related to fundamental frequency, one or more features related to frequency spectrum, one or more features related to phase spectrum, etc.

[0088] In some embodiments, the bone conduction audio signal and the air conduction audio signal may represent the same user voice by different frequency components. The bone conduction data and the air conduction data may include the same or different estimated semantic information, which may reflect the actual semantic content of the same user voice. The estimated semantic information recognized from the air conduction audio signal may be the same as or different from the estimated semantic information recognized from the bone conduction audio signal. The bone conduction audio signal and the air conduction audio signal representing the same user voice may mean that the bone conduction audio signal and the air conduction audio signal are simultaneously collected by the bone conduction sensor and the air conduction sensor, respectively, when the user speaks. In some embodiments, the bone conduction audio signal collected by the bone conduction sensor may include a first frequency component. The air conduction audio signal may include a second frequency component. In some embodiments, the second frequency component of the air conduction audio signal may include at least a portion of the first frequency component. The acoustic features of the air conduction audio signal may be the same as or different from the acoustic features of the bone conduction audio signal. For example, the amplitude of a particular frequency component of the bone conduction audio signal may be different from the amplitude of a particular frequency component of the air conduction audio signal. As another example, the number of frequency components in the bone-conducted audio signal that are less than a frequency point (e.g., 2000 Hz) or in a frequency range (e.g., 20 Hz to 2000 Hz) may be greater than the number of frequency components in the air-conducted audio signal that are less than the frequency point (e.g., 2000 Hz) or in the frequency range (e.g., 20 Hz to 2000 Hz). The number of frequency components in the bone-conducted audio signal that are greater than a frequency point (e.g., 3000 Hz) or in the frequency range (e.g., 3000 Hz to 20 kHz) may be less than the number of frequency components in the air-conducted audio signal that are greater than the frequency point (e.g., 3000 Hz) or in the frequency range (e.g., 3000 Hz to 20 kHz).As used herein, a greater number of frequency components in a bone conducted audio signal below a frequency point (e.g., 2000 Hz) or within a frequency range (e.g., 20 Hz to 2000 Hz) than a frequency component in an air conducted audio signal below that frequency point (e.g., 2000 Hz) or within that frequency range (e.g., 20 Hz to 2000 Hz) may mean that the count or number of frequency components in the bone conducted audio signal below a frequency point (e.g., 2000 Hz) or within a frequency range (e.g., 20 Hz to 2000 Hz) is greater than the count or number of frequency components in the air conducted audio signal below that frequency point (e.g., 2000 Hz) or within that frequency range (e.g., 20 Hz to 2000 Hz).

[0089] In some embodiments, the air conduction audio signal can be processed to obtain one or more air conduction data sets similar or the same as the one or more bone conduction data sets. The air conduction data sets can represent the user's speech, i.e. the user's meaning. The air conduction data sets can include air conduction acoustic features that can form phonemes, phonemes consisting of air conduction acoustic features, syllables consisting of one or more phonemes, words consisting of one or more syllables, phrases consisting of one or more words, sentences consisting of one or more phrases, etc., or any combination thereof.

[0090] In some embodiments, the processing unit 122 may pre-process at least one of the bone conducted audio signal or the air conducted audio signal. Exemplary pre-processing operations may include domain transformation operations, signal calibration operations, speech enhancement operations, etc.

[0091] At 606, the processing unit 122 (e.g., the acquisition module 402) may acquire a trained machine learning model that provides a mapping relationship signal between a bone conduction data set derived from a particular bone conduction audio signal and one or more equivalent air conduction data sets derived from a particular equivalent air conduction signal corresponding to the particular bone conduction audio signal.

[0092] As used herein, a particular bone conduction audio signal refers to an audio signal (e.g., the bone conduction audio signal acquired in operation 602) acquired by a bone conduction sensor (e.g., the bone conduction sensor described in operation 602) when a particular user speaks. A particular equivalent air conduction signal corresponding to a particular bone conduction audio signal refers to an audio signal generated by an air conduction sensor (e.g., the air conduction sensor described in operation 604) when a bone conduction sensor is used to acquire a particular user's voice and at the same time an air conduction sensor is used to acquire a particular user's voice under noise-free conditions. The particular bone conduction audio signal and the particular equivalent air conduction audio signal may represent the same voice of a particular user. The noise-free condition may refer to one or more noise evaluation parameters (e.g., a noise standard curve, a statistical noise level, etc.) meeting a condition, e.g., below a threshold value. Semantic information represented by a bone conduction data set derived from a particular bone conduction audio signal may be similar or the same as semantic information represented by each of one or more equivalent air conduction data sets derived from the particular equivalent air conduction audio signal. The similarity or degree of association between the semantic information represented by the bone conduction dataset and the semantic information represented by each of the one or more equivalent air conduction datasets may exceed a threshold, e.g., 90%, 95%, 99%, etc.

[0093] In some embodiments, the mapping relationship between the bone conduction dataset and the one or more equivalent air conduction datasets may indicate a matching relationship between the bone conduction dataset and each of the one or more equivalent air conduction datasets. As used herein, the matching relationship between the bone conduction dataset and each of the one or more equivalent air conduction datasets may refer to the bone conduction dataset matching each of the one or more equivalent air conduction datasets. If the similarity or relevance between the semantic information represented by the bone conduction dataset and the semantic information represented by the equivalent air conduction dataset exceeds a threshold value, such as 90%, 95%, 99%, etc., the bone conduction dataset and the equivalent air conduction dataset may be matched.

[0094] In some embodiments, a bone conduction data set derived from a particular bone conduction signal may include a set of acoustic features extracted from the particular bone conduction signal, or a phoneme of one or more acoustic features, or a syllable of one or more phonemes, or a word of one or more syllables, etc. In some embodiments, an equivalent air conduction data set derived from a particular equivalent air conduction signal may include a set of acoustic features extracted from the particular equivalent air conduction signal, or a phoneme of one or more acoustic features, or a syllable of one or more phonemes, or a word of one or more syllables, etc.

[0095] In some embodiments, the mapping relationship between the bone conduction data set and the one or more equivalent air conduction data sets may include a mapping relationship between acoustic features, a mapping relationship between phonemes, a mapping relationship between syllables, a mapping relationship between words, etc., or any combination thereof. The mapping relationship between acoustic features refers to a matching relationship between a set of acoustic features in a particular bone conduction audio signal and each of the set or more acoustic features in a particular equivalent air conduction audio signal. The mapping relationship between phonemes refers to a matching relationship between each phoneme in a particular bone conduction audio signal and each of the one or more phonemes in a particular equivalent air conduction audio signal. The mapping relationship between syllables refers to a matching relationship between each syllable in a particular bone conduction audio signal and each of the one or more syllables in a particular equivalent air conduction audio signal. The mapping relationship between words refers to a matching relationship between each word in a particular bone conduction audio signal and each of the one or more words in a particular equivalent air conduction audio signal.

[0096] In some embodiments, the mapping relationship between the bone conduction dataset and the one or more equivalent air conduction datasets may include one or more matching probabilities that the bone conduction dataset matches with the one or more equivalent air conduction datasets, respectively. The matching probability that the bone conduction dataset matches with one equivalent air conduction dataset may indicate a similarity or relatedness between the semantic information represented by the bone conduction dataset and the semantic information represented by the equivalent air conduction dataset. The greater the similarity or relatedness between the semantic information represented by the bone conduction dataset and the semantic information represented by the equivalent air conduction dataset, the greater the matching probability may be between the bone conduction dataset and the equivalent air conduction dataset. In some embodiments, the similarity or relatedness between the semantic information represented by the bone conduction dataset and the semantic information represented by the equivalent air conduction dataset may be equal to or specified as the matching probability that the bone conduction dataset matches with the equivalent air conduction dataset.

[0097] By way of example only, the mapping relationship between syllables may include one or more matching probabilities that each syllable in a particular bone conducted audio signal matches with one or more syllables in a particular equivalent air conducted audio signal, respectively.

[0098] In some embodiments, the trained machine learning model may be configured to determine information (e.g., one or more matching probabilities) associated with the one or more equivalent air conduction data sets. In some embodiments, the trained machine learning model may be configured to perform speech recognition operations on the bone conduction audio signal and / or the air conduction audio signal to recognize the one or more bone conduction data sets (e.g., a set of acoustic features, one or more phonemes, one or more syllables, one or more words, sentences, etc.), as described elsewhere herein. In some embodiments, the trained machine learning model may be configured to determine information (e.g., one or more matching probabilities) associated with the one or more equivalent air conduction data sets based on the recognized bone conduction data sets. More description of the trained machine learning model may be found elsewhere herein (e.g., FIG. 7 and its description).

[0099] In some embodiments, the trained machine learning model may be reconstructed based on a neural network model. Exemplary neural network models may include a Convolutional Neural Network (CNN) model, a Recurrent Neural Network (RNN) model, a Long Short-Term Memory Network (LSTM) model, and the like. In some embodiments, the trained machine learning model may include a speech recognition model. Exemplary speech recognition models may include a Hidden Markov Model (HMM), a speech recognition model based on Dynamic Time Warping (DTW), an artificial neural network model, an end-to-end automatic speech recognition model, and the like, or any combination thereof.

[0100] In some embodiments, the processing unit 122 may obtain the trained machine learning model from one or more components of the audio signal generating system 100 (e.g., the storage device 140, the terminal 130) or an external source via a network (e.g., the network 150). For example, the trained machine learning model may be trained in advance by a computing device (e.g., the processing unit 122) and stored in a storage device (e.g., the storage device 140) of the audio signal generating system 100. The processing unit 122 may access the storage device to retrieve the trained machine learning model. In some embodiments, the trained machine learning model may be generated according to a machine learning algorithm described elsewhere in this application (e.g., FIG. 4 and its related description). More description regarding generating a trained machine learning model may be found elsewhere in this application (e.g., FIG. 8 and its description).

[0101] In some embodiments, when the bone conduction sensor acquires the bone conduction audio signal, as described in operation 602, the processing unit 122 can obtain a trained machine learning model based on the region of the user's body where the bone conduction sensor is located and / or the particular pressure applied by the bone conduction sensor. The region of the user's body in contact with the bone conduction sensor that acquires the bone conduction audio signal may include, but is not limited to, the forehead, neck (e.g., throat), mastoid, area around or inside the ear, temples, face (e.g., area around the mouth, chin), top of the head, etc. For example, multiple trained machine learning models corresponding to different body regions and / or different pressures applied by the bone conduction sensor to the user may be stored in a database, and when the bone conduction sensor acquires the bone conduction audio signal, the processing unit 122 can obtain a trained machine learning model corresponding to the same region of the user's body where the bone conduction sensor is located, as described in operation 602.

[0102] At 608, the processing unit 122 (e.g., the determination module 404) may determine, based on the bone conduction audio signal and the air conduction audio signal, a target equivalent air conduction data set corresponding to the bone conduction audio signal using the trained machine learning model.

[0103] The target equivalent air conduction data set corresponding to the bone conducted audio signal may represent a semantic content. The semantic content represented by the target equivalent air conduction data set may represent an actual semantic content of the user's speech.

[0104] In some embodiments, the target equivalent air conduction data set may include one or more phonemes comprised of air conducted acoustic features extracted from the equivalent air conducted audio signal, or air conducted acoustic features extracted from the equivalent air conducted audio signal, or one or more syllables comprised of one or more phonemes, or one or more words comprised of one or more syllables, or one or more phrases comprised of one or more words, or one or more sentences comprised of one or more phrases, etc., or any combination thereof. The target equivalent air conduction data set, the bone conducted audio signal obtained in operation 602, and the air conducted audio signal obtained in operation 604 may represent the same voice of the user.

[0105] In some embodiments, the bone conduction audio signal may include one or more bone conduction data sets as described in operation 602. In some embodiments, for each of the one or more bone conduction data sets, the processing unit 122 may determine a target equivalent air conduction data set corresponding to the bone conduction data set. The target equivalent air conduction data sets corresponding to the one or more bone conduction data sets may form a target equivalent air conduction data set corresponding to the bone conduction audio signal.

[0106] In some embodiments, for each of the one or more bone conduction data sets, the processing unit 122 can determine one or more equivalent air conduction data sets corresponding to the bone conduction data set using a trained machine learning model based on the bone conduction audio signal. The processing unit 122 can further identify a target equivalent air conduction data set corresponding to the bone conduction data set from the one or more equivalent air conduction data sets based on the air conduction audio signal obtained in operation 604.

[0107] In some embodiments, the processing unit 122 can input the bone conducted audio signal to a trained machine learning model, and the trained machine learning model can output a matching probability of each of the one or more equivalent air conduction data sets corresponding to the bone conducted audio signal and / or the one or more equivalent air conduction data sets matching the bone conducted audio signal. The processing unit 122 can determine a target equivalent air conduction data set corresponding to the bone conducted audio signal from the one or more equivalent air conduction data sets based on the air conducted audio signal obtained in operation 604.

[0108] In some embodiments, the processing unit 122 can input the bone conducted audio signal and the air conducted audio signal to a trained machine learning model, and the trained machine learning model can output a target equivalent air conducted data set that corresponds to the bone conducted audio signal.

[0109] In some embodiments, the processing unit 122 can input the one or more bone conduction data sets to a trained machine learning model, and the trained machine learning model can output one or more equivalent air conduction data sets corresponding to each of the one or more bone conduction data sets and / or a matching probability of matching each of the one or more equivalent air conduction data sets with the bone conduction data set. The processing unit 122 can determine a target equivalent air conduction data set corresponding to the bone conduction data set from the one or more equivalent air conduction data sets based on the one or more air conduction data sets included in the air conduction audio signal acquired in operation 604.

[0110] In some embodiments, the processing unit 122 can input one or more bone conduction data sets and one or more air conduction data sets into a trained machine learning model, and the trained machine learning model can output a target equivalent air conduction data set that corresponds to the bone conduction data set.

[0111] Further discussion regarding determining the target equivalent air conductance data set can be found elsewhere in this application (eg, FIG. 7 and its discussion).

[0112] At 610, the processing unit 122 (eg, the output module 406) can cause an audio signal output device to output a target audio signal representative of a user's voice based on the target equivalent air conduction data set.

[0113] In some embodiments, the processing unit 122 may use a speech synthesis algorithm (also called a text-to-speech algorithm) to convert the target equivalent air-conducted data set into an equivalent air-conducted audio signal, e.g., in the time domain, frequency domain, etc. Exemplary speech synthesis algorithms may include a Linear Predictive Coding (LPC) algorithm, a Pitch Synchronous Over Lapand Add (PSOLA) algorithm, etc.

[0114] The processing unit 122 can determine a target audio signal based on the equivalent air conducted audio signal. In some embodiments, the processing unit 122 can designate the equivalent air conducted audio signal as the target audio signal.

[0115] In some embodiments, the processing unit 122 may perform one or more post-processing operations on the equivalent air-conducted audio signal. For example, the processing unit 122 may add additional information to the equivalent air-conducted audio signal based on the air-conducted audio signal. Exemplary additional information may include one or more background noises similar to those included in the air-conducted audio signal (e.g., rain noise, wind noise, etc.), emotional characteristics of a user (e.g., anger, fear, sadness, joy, etc.), acoustic characteristics of a user, etc., or any combination thereof. The processing unit 122 may designate the post-processed equivalent air-conducted audio signal as a target audio signal, which may improve the fidelity and intelligibility of the determined target audio signal. In some embodiments, the additional information may be referred to as noise.

[0116] In some embodiments, the noise level of the target audio signal may be less than the noise level of the air conducted audio signal. In some embodiments, the target audio signal may include more frequency components than the bone conducted audio signal obtained in operation 602. In some embodiments, the semantic content represented by the target audio signal is more similar to the actual semantic content of the user's speech than the semantic information or content represented by the bone conducted audio signal obtained in operation 602 is more similar to the actual semantic content of the user's speech.

[0117] The processing unit 122 can cause an audio signal output device to output the target audio signal. In some embodiments, the processing unit 122 can send a signal via the network 150 to a client terminal (e.g., terminal 130), the storage device 140, and / or any other storage device (not shown in the audio signal generation system 100). The signal may include the target audio signal. The signal may be configured to instruct the client terminal to play the target audio signal.

[0118] It should be noted that the above description of the process 600 is for illustrative purposes only and is not intended to limit the scope of the present application. Those skilled in the art may make various changes and modifications based on the description of the present application. However, these changes and modifications do not depart from the scope of the present application. For example, the operations 602 and 604 may be integrated into a single operation.

[0119] FIG. 7 is a schematic flow chart of an exemplary process for determining a target equivalent air conduction data set corresponding to a bone conduction audio signal, according to some embodiments of the present application. In some embodiments, the process 700 may be implemented as instructions (e.g., application programs) stored in the storage device 140, the ROM 230, the RAM 240, or the memory 390. The processing device 122, the processor 220, and / or the CPU 340 may execute a set of instructions, and when the processing device 122, the processor 220, and / or the CPU 340 executes the instructions, the processing device 122, the processor 220, and / or the CPU 340 may be configured to perform the process 700. The operations of the process shown below are for illustration purposes only. In some embodiments, the process 700 may be completed by one or more additional operations not described and / or one or more operations not discussed. Also, the order of the operations of the process 700 shown in FIG. 7 and described below is not limited. In some embodiments, one or more operations of the process 700 may be performed to realize at least a part of the operations of the operation 608 described with reference to FIG. 6.

[0120] At 702, the processing unit 122 (e.g., the determination module 404) may determine, based on the bone conduction audio signal, one or more equivalent air conduction data sets corresponding to the bone conduction audio signal using a trained machine learning model. The bone conduction audio signal may be obtained by operation 602 described with reference to FIG.

[0121] In some embodiments, the trained machine learning model can provide a mapping relationship between a bone conduction data set derived from a particular bone conduction audio signal and one or more equivalent air conduction data sets derived from a particular equivalent air conduction signal corresponding to the particular conducted audio signal. More description of the trained machine learning model can be found elsewhere in this application (e.g., FIG. 6 and its description).

[0122] In some embodiments, the trained machine learning model may include a first part performing speech recognition and a second part providing a mapping relationship between a bone conduction data set derived from a particular bone conduction audio signal and one or more equivalent air conduction data sets derived from a particular equivalent air conduction signal corresponding to the particular conducted audio signal, the second part may further determine one or more equivalent air conduction data sets corresponding to a bone conduction data set based on the mapping relationship.

[0123] In some embodiments, the first portion may include a speech recognition model configured to generate a bone conduction data set from the bone conduction audio signal as described in operation 602. Exemplary speech recognition models may include a Hidden Markov Model (HMM), an end-to-end model, a Connectionist Temporal Classification (CTC) model, or the like, or any combination thereof.

[0124] In some embodiments, the first portion of the trained machine learning model may be configured to extract acoustic features from a particular audio signal (e.g., a bone conduction audio signal) to perform speech recognition. The recognition result of the particular audio signal obtained using the first portion of the trained machine learning model may include one or more data sets representing semantic information of the particular audio signal in the form of text. The acoustic features may be extracted from the particular audio signal using an acoustic feature extraction algorithm. Exemplary acoustic feature extraction algorithms may include an Auto Correlation Function (ACF) algorithm, an Average Amplitude Difference Function (AMDF) algorithm, a nonlinear feature extraction algorithm based on a Teager Energy Operator (TEO), a Linear Predictive Analysis (LPC) algorithm, a deep learning algorithm (e.g., Laplacian Eigenmap, Principal Component Analysis (PCA), Local Preserved Projection (LPP), etc.).

[0125] The second part of the trained machine learning model may be configured to generate one or more equivalent air conduction data sets based on the mapping relationship and the bone conduction data set. In some embodiments, the second part may express the mapping relationship in the form of a list. For example, the list may include a plurality of reference bone conduction data sets and a plurality of reference equivalent air conduction data sets. Each of the plurality of reference bone conduction data sets may correspond to one or more reference equivalent air conduction data sets. The list may include a matching probability that the reference bone conduction data set matches each of the one or more reference equivalent air conduction data sets. The second part of the trained machine learning model may generate one or more equivalent air conduction data sets based on the list and the bone conduction data set. For example, the second part of the trained machine learning model may identify a reference bone conduction data set in the list that is the same as or similar to the bone conduction data set, obtain one or more reference equivalent air conduction data sets from the list that correspond to the reference bone conduction data set, and designate the one or more reference equivalent air conduction data sets as one or more equivalent air conduction data sets that correspond to the bone conduction data set.

[0126] In some embodiments, the second portion may express the mapping relationship as a function, and in some embodiments, the second portion of the trained machine learning model may be configured to use the function to generate one or more equivalent air conduction data sets corresponding to the bone conduction data sets.

[0127] In some examples, the bone conduction audio signal may be input to a trained machine learning model. The trained machine learning model may extract bone conduction acoustic features from the bone conduction audio signal. As described in operation 602, the bone conduction acoustic features may include one or more features related to duration, one or more features related to energy, one or more features related to fundamental frequency, one or more features related to frequency spectrum, one or more features related to phase spectrum, etc. The trained machine learning model may determine a bone conduction data set based on the bone conduction acoustic features. For example, the trained machine learning model may determine one or more bone conduction acoustic features, which may form a phoneme as the bone conduction data set. As another example, the trained machine learning model may designate one or more phonemes of the bone conduction acoustic features as the bone conduction data set. Also, for example, the trained machine learning model may designate one or more syllables of the one or more phonemes as the bone conduction data set. As a further example, the trained machine learning model may designate one or more words of one or more syllables as the bone conduction data set. As a further example, the trained machine learning model may designate one or more phrases of one or more words as the bone conduction data set. As another example, the trained machine learning model can specify one or more sentences of one or more phrases as a bone conduction dataset.

[0128] The trained machine learning model can determine one or more equivalent air conduction data sets and / or information related to the one or more equivalent air conduction data sets based on the bone conduction data set and the mapping relationship provided by the trained machine learning model. For example, a first portion of the trained machine learning model can determine a bone conduction data set from a bone conduction audio signal. For example, the first portion of the trained machine learning model can determine a bone conduction data set based on the bone conduction audio signal. A second portion of the trained machine learning model can determine one or more equivalent air conduction data sets corresponding to the bone conduction data set based on the comparison.

[0129] For example, the second part of the trained machine learning model can determine a similarity or relatedness between the bone conduction data set and each of a plurality of reference bone conduction data sets in the list. The second part of the trained machine learning model can determine a reference bone conduction data set among a plurality of reference bone conduction data sets in the list that has a maximum similarity or relatedness with the bone conduction data set. The second part of the trained machine learning model can output one or more reference equivalent air conduction data sets corresponding to a reference bone conduction data set in the list, the reference bone conduction data set having a maximum similarity or correlation with the bone conduction data set derived from the bone conduction audio signal. In some embodiments, the second part of the trained machine learning model can output a matching probability between one or more reference equivalent air conduction data sets in the list and the reference bone conduction data set, the reference bone conduction data set having a maximum similarity or correlation with the bone conduction data set derived from the bone conduction audio signal.

[0130] In some embodiments, the processing unit 122 may designate one or more reference equivalent air conduction data sets in the list corresponding to the reference bone conduction data set as one or more equivalent air conduction data sets corresponding to the bone conduction audio data set.

[0131] In some embodiments, the processing device 122 may determine a portion of the one or more reference equivalent air conduction data sets as the one or more equivalent air conduction data sets corresponding to the bone conduction audio data set based on information (e.g., matching probability) about the one or more reference equivalent air conduction data sets. The processing device 122 may determine the one or more equivalent air conduction data sets based on the one or more reference equivalent air conduction data sets and the one or more matching probabilities. For example, the processing device 122 may determine the one or more reference equivalent air conduction data sets with matching probabilities greater than a certain threshold (e.g., 0.9, 0.8, 0.7, etc.) as the one or more equivalent air conduction data sets. As another example, the one or more matching probabilities corresponding to the one or more reference equivalent air conduction data sets may be sorted in descending order, and the processing device 122 may determine the one or more reference equivalent air conduction data sets with the top rankings as the one or more equivalent air conduction data sets according to the descending order.

[0132] At 704, the processing unit 122 (eg, the determination module 404) may identify a target equivalent air conduction data set from the one or more equivalent air conduction data sets based on the air conduction audio signal.

[0133] The air conducted audio signal and the bone conducted audio signal may represent the same user voice with different frequency components. The air conducted audio signal may be obtained as described in operation 604 with reference to FIG.

[0134] In some embodiments, the processing unit 122 can determine a set of air conduction audio data by processing the air conduction audio signal. The air conduction data set and the bone conduction data set may be derived from at least a portion of the air conduction audio signal and at least a portion of the bone conduction audio signal simultaneously acquired by the air conduction sensor and the bone conduction sensor, respectively. In other words, the air conduction data set and the bone conduction data set can represent the actual semantic content of the same user voice. More description of the air conduction data set can be found elsewhere in this application (e.g., FIG. 6 and its description).

[0135] In some embodiments, the processing unit 122 may determine the air-conducted data set by extracting acoustic features (also referred to as air-conducted acoustic features) from the air-conducted audio signal. In some embodiments, the air-conducted acoustic features may be extracted from the air-conducted audio signal using a speech recognition model or an acoustic feature extraction algorithm, as described elsewhere herein (e.g., in operation 702 and related description). In some embodiments, the air-conducted acoustic features may be extracted from the air-conducted audio signal using a first portion of a trained machine learning model. As described in operation 602, the air-conducted acoustic features may include one or more features related to duration, one or more features related to energy, one or more features related to fundamental frequency, one or more features related to frequency spectrum, one or more features related to phase spectrum, etc.

[0136] The processing unit 122 can determine the air conduction data set based on the air conduction acoustic features. For example, the determination of the air conduction data set based on the air conduction acoustic features may be performed similarly to the determination of the bone conduction data set based on the bone conduction acoustic features described in connection with operation 702, and the description thereof will not be repeated here.

[0137] The processing unit 122 can identify the target equivalent air conduction data set from one or more equivalent air conduction data sets by determining a degree of association between each equivalent air conduction data set and the air conduction data set. In some embodiments, the processing unit 122 can determine a degree of association between the equivalent air conduction data set and the air conduction data set by using a similarity algorithm. Exemplary similarity algorithms may include a cosine similarity algorithm, a Jaccard coefficient algorithm, a Dice coefficient algorithm, and the like. In some embodiments, the processing unit 122 can determine a degree of association between the equivalent equivalent air conduction data set and the air conduction data set using a machine learning model for determining semantic similarity. Exemplary machine learning models for determining semantic similarity may include a Deep Structured Semantic Model (DSSM), a Convolutional Latent Semantic Model (CLSM), a Long-Short-Term Memory-Deep Structured Semantic Model (LSTM-DSSM), and the like, or any combination thereof.

[0138] The processing unit 122 may determine an equivalent air conductance data set that is the most related to the one or more equivalent air conductance data sets among the one or more equivalent air conductance data sets as an initial target equivalent air conductance data set. The processing unit 122 may determine a target equivalent air conductance data set based on the initial target equivalent air conductance data set.

[0139] In some embodiments, the processing unit 122 can designate the initial target equivalent air conductance data set as the target equivalent air conductance data set.

[0140] In some embodiments, the processing unit 122 may perform one or more post-processing operations on the initial target equivalent air conduction data set. For example, the processing unit 122 may perform a semantic calibration operation on the initial target equivalent air conduction data set based on the air conduction data set. The semantic calibration operation refers to adjusting or calibrating the semantic information of the initial target air conduction data set based on the semantic information of the air conduction data set so that the semantic information of the initial target air conduction data set and the semantic information of the air conduction data set are the same or similar. The processing unit 122 may designate the post-processed initial target equivalent air conduction data set as the target equivalent air conduction data set.

[0141] It should be noted that the above description of the process 700 is for illustrative purposes only and is not intended to limit the scope of the present application. Those skilled in the art may make various changes and modifications based on the description of the present application. However, these changes and modifications do not depart from the scope of the present application. For example, the process 700 may further include an operation of determining a bone conduction data set from the bone conduction audio signal and / or an operation of determining an air conduction data set from the air conduction audio signal.

[0142] FIG. 8 is a schematic flow chart of an exemplary process for generating a trained machine learning model according to some embodiments of the present application. In some embodiments, the process 800 may be implemented as instructions (e.g., application programs) stored in the storage device 140, the ROM 230, the RAM 240, or the memory 390. The processing device 122, the processor 220, and / or the CPU 340 may execute a set of instructions, and the processing device 122, the processor 220, and / or the CPU 340 may be configured to perform the process 800 when executing the instructions. In some embodiments, the trained machine learning model according to the operation 606 described with reference to FIG. 6 may be obtained based on the process 800. In some embodiments, the process 800 may be performed by another device or system other than the audio signal generating system 100, for example, a device or system of a vendor or manufacturer of the trained machine learning model. For the purpose of explanation, the implementation of the process 800 by the processing device 122 is described as an example.

[0143] At 802, the processing unit 122 (eg, the acquisition module 502) can acquire multiple sets of training data.

[0144] In some embodiments, each of the multiple sets of training data may include a first audio signal and a second audio signal representing the same speech sample.

[0145] In some embodiments, when generating a sound sample under a noise-free condition, the first audio signal and the second audio signal are collected simultaneously by a bone conduction audio collecting device (e.g., the bone conduction microphone 112) and an air conduction audio collecting device (e.g., the air conduction microphone 114) and stored in one or more components of the audio signal generating system 100 (e.g., the storage device 140, the terminal 130) or an external source. The processing device 122 can obtain the first audio signal and the second audio signal from one or more components of the audio signal generating system 100. As used herein, the noise-free condition may refer to one or more noise assessment parameters (e.g., a noise standard curve, a statistical noise level, etc.) meeting a condition, e.g., below a threshold value. The first audio signal may be referred to as a bone conduction audio signal, and the second audio signal may be referred to as an air conduction audio signal.

[0146] In some embodiments, in the process of training the preliminary machine learning model, the bone conduction audio signal in the set of training data may be used as an input of the preliminary machine learning model, and the air conduction audio signal corresponding to the bone conduction audio signal in the audio sample may be used as a desired output of the preliminary machine learning model. In some embodiments, the bone conduction audio signal and the air conduction audio signal in the set of training data may be used as inputs of the preliminary machine learning model.

[0147] In some embodiments, the bone conduction audio signals in each of the plurality of training data may be collected by a bone conduction sensor located at the same region of the user's (e.g., tester's) body (e.g., the region around the ear). In some embodiments, the region of the body where the bone conduction sensor is located when collecting the bone conduction data for training the machine learning model may match and / or be the same as the region of the body where the bone conduction sensor is located when collecting the bone conduction data (e.g., the first audio data) used in applying the trained machine learning model. For example, the region of the user's (e.g., tester's) body where the bone conduction sensor is located when collecting the bone conduction audio signals in each of the plurality of sets of training data may be the same as the region of the user's body where the bone conduction sensor is located when collecting the bone conduction audio signals described in operation 602. As another example, if the region of the user's body where the bone conduction sensor is located when collecting the bone conduction audio signals described in operation 602 is the neck, the region of the body where the bone conduction sensor is located when collecting the bone conduction data used in the training process of the trained machine learning model is the neck.

[0148] When collecting multiple sets of training data (e.g., a tester), the location of the region of the user's body where the bone conduction sensor is located may affect the correspondence between the bone conduction data derived from the bone conduction audio signal and the one or more air conduction data derived from the air conduction audio signal in each set of training data. In some embodiments, multiple bone conduction sensors of the same configuration may be located in different regions of the body, such as the mastoid process, temples, top of the head, and ear canal. When a user speaks, the multiple bone conduction sensors can simultaneously collect bone conduction data. Multiple training sets can be formed based on the bone conduction audio signals collected by the multiple bone conduction sensors. Each of the multiple training sets may include multiple sets of training data collected by one of the multiple bone conduction sensors and the air conduction sensor. Each of the multiple sets of training data may include a bone conduction audio signal and an air conduction audio signal representing the same voice. Each of the multiple training sets can train a machine learning model to obtain a trained machine learning model. Multiple trained machine learning models can be obtained based on the multiple training sets. The multiple trained machine learning models can provide different mapping relationships between specific bone conduction data and air conduction data. For example, the same bone conduction data can be input to multiple trained machine learning models to generate different air conduction data. In some embodiments, bone conduction data collected by different bone conduction sensors with different configurations can be different. Thus, the bone conduction sensors that collect bone conduction data for training a machine learning model and the bone conduction sensors that collect bone conduction audio signals used to apply the trained machine learning models may match and / or have the same configuration.

[0149] At 804, the processing unit 122 (e.g., the model generation module 508) may train the preliminary machine learning model using the multiple sets of training data to obtain a trained machine learning model.

[0150] The preliminary machine learning model may be referred to as a model to be trained. The preliminary machine learning model may be any type of model (e.g., HMM model) described elsewhere in this application (e.g., FIG. 6 and related description). In some embodiments, the preliminary machine learning model may be a machine learning model that has not been trained using a training set. In some embodiments, the preliminary machine learning model may be a trained machine learning model that has been trained using a training set, the training set including training data different from the sets of training data obtained in operation 802. In some embodiments, the processing unit 122 may obtain the preliminary machine learning model from one or more components of the audio signal generation system 100 (e.g., storage unit 140, terminal 130) or an external source (e.g., a third-party database) via a network (e.g., network 150).

[0151] In some embodiments, the preliminary machine learning model may be constructed based on a neural network model (e.g., a multi-layer perceptron), a statistical model (e.g., a hidden Markov model (HMM)), etc., or a combination thereof. In some embodiments, the preliminary machine learning model may include a multi-layer structure. For example, the preliminary machine learning model may include an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. In some embodiments, the hidden layers may include one or more convolutional layers, one or more rectified linear unit layers (ReLU layers), one or more pooling layers, one or more fully connected layers, etc., or any combination thereof. As used herein, a model layer may refer to an algorithm or a function for processing input data of a layer. Different layers may perform different types of processing on their respective inputs. A successive layer may use output data from a layer before the successive layer as input data. In some embodiments, the convolutional layer may include multiple kernels for extracting features. In some embodiments, each of the multiple kernels may filter a portion (i.e., a region). The pooling layer may take the output of the convolutional layer as input. The pooling layer may include a plurality of pooling nodes that can sample the output of the convolution layer to reduce the computational load of data processing and accelerate data processing. In some embodiments, the size of the matrix representing the input data may be reduced in the pooling layer. The fully connected layer may include a plurality of neurons. The neurons may be connected to the pooling nodes in a pool. In the fully connected layer, a plurality of vectors corresponding to the plurality of pooling nodes may be determined based on the training samples, and a plurality of weighting factors may be assigned to the plurality of vectors. The output layer may determine an output based on the vector and the weighting factor obtained from the fully connected layer.

[0152] In some embodiments, each layer may include one or more nodes. In some embodiments, each node may be connected to one or more nodes of the previous layer. The number of nodes in each layer may be the same or different. In some embodiments, each node may correspond to an activation function. As used herein, an activation function of a node may define an output of the node given an input or a set of inputs. In some embodiments, each connection between two nodes of the plurality of nodes in the preliminary machine learning model may transmit a signal from one node to the other node. In some embodiments, each connection may correspond to a weight. As used herein, a weight corresponding to a connection may increase or decrease the strength or influence of the signal in the connection.

[0153] The machine learning model may include multiple parameters such as architecture parameters, learning parameters, etc. Exemplary architecture parameters of the machine learning model may include the size of the kernel of a layer, the total number (or number) of layers, the node count (or number) of each layer, the learning rate, the batch size, epochs, etc. Exemplary learning parameters may include the connection weight between two connecting nodes, the deviation vector associated with the node, etc.). Before training, the machine learning model may have one or more initial parameter values. In training the machine learning model, the learning parameters of the machine learning model may be updated. Before the update process, the values ​​of the learning parameters of the machine learning model may be initialized. For example, the connection weights and / or bias vectors of the preliminary machine learning model may be initialized by assigning random values ​​in the range of -1 to 1. As another example, all the connection weights of the preliminary machine learning model may be assigned the same value in the range of -1 to 1, e.g., 0. As another example, the deviation vectors of the nodes in the preliminary machine learning model may be initialized by assigning random values ​​in the range of 0 to 1. In some embodiments, the parameters of the preliminary machine learning model may be initialized according to a Gaussian random algorithm, a Xavier algorithm, etc.

[0154] In some embodiments, the processing unit 122 may train the preliminary machine learning model using a training algorithm to obtain a trained machine learning model. Exemplary training algorithms may include a backpropagation algorithm, a gradient descent algorithm, a Newton algorithm, a quasi-Newton algorithm, a Levenberg-Marquardt algorithm, a conjugate gradient algorithm, etc., or a combination thereof.

[0155] In some examples, training the preliminary machine learning model may include multiple iterations. In each iteration, the first audio signal or the bone conduction dataset may be used as an input of the preliminary machine learning model, and the air conduction dataset may be used as a desired output of the preliminary machine learning model. The preliminary machine learning model may extract a first acoustic feature from the first audio signal and a second acoustic feature from the second audio signal. For example, the preliminary machine learning model may extract the first acoustic feature and the second acoustic feature using a sub-model (e.g., a speech recognition model as described in operation 702). As another example, the preliminary machine learning model may extract the first acoustic feature and the second acoustic feature using an acoustic feature extraction algorithm described elsewhere in this application (e.g., in operation 702 and its associated description). As described in operation 602, the first acoustic feature and the second acoustic feature may include one or more features related to duration, one or more features related to energy, one or more features related to fundamental frequency, one or more features related to frequency spectrum, one or more features related to phase spectrum, etc. The preliminary machine learning model (e.g., a first portion of the preliminary machine learning model) can determine one or more first bone conduction data sets based on the first acoustic features and one or more second air conduction data sets based on the second acoustic features. Each of the one or more first bone conduction data sets and the one or more second air conduction data sets may include one or more phonemes of the acoustic features, one or more syllables of the one or more phonemes, one or more words of the one or more syllables, one or more phrases of the one or more words, one or more sentences of the one or more phrases, etc., or any combination thereof.

[0156] The preliminary machine learning model (e.g., a second portion of the preliminary machine learning model) can determine a plurality of pairs of air conduction data and bone conduction data. Each pair may include a first bone conduction data set and a second air conduction data set that represent the same meaning. In some embodiments, the preliminary machine learning model can determine a matching probability of matching each of the one or more first bone conduction data sets with each of the one or more second air conduction data sets. The matching probability of matching the first bone conduction data set with the second air conduction data set may indicate a similarity or correlation between the semantic information represented by the first bone conduction data set and the semantic information represented by the second air conduction data set. The similarity or relatedness between the semantic information represented by the first bone conduction data set and the semantic information represented by the second air conduction data set may be a matching probability of matching the first bone conduction data set with the second air conduction data set. By way of example only, the preliminary machine learning model may determine a degree of association between each of the one or more first bone conduction data sets and each of the one or more second air conduction data sets using similar algorithms described elsewhere herein (e.g., operation 704 and related discussions thereof). The preliminary machine learning model may determine the degree of association as a matching probability.

[0157] In training the machine learning model, the processing unit 122 may iteratively update parameter values ​​of the preliminary machine learning model based on multiple training samples. Updating the learning parameters of the machine learning model may be referred to as updating the machine learning model. For example, the processing unit 122 may perform one or more iterations to update the model parameters of the machine learning model until a termination condition is met. In some embodiments, the preliminary machine learning model may use a cost function to determine a difference or error associated with the matching probability and a threshold value. If the value of the cost function does not satisfy the condition, the processing unit 122 may adjust the parameter values ​​of the preliminary machine learning model in the current iteration based on the value of the cost function using a backpropagation algorithm.

[0158] The processing unit 122 (e.g., the model generation module 504) can determine whether a termination condition is met. The termination condition can indicate whether the machine learning model is sufficiently trained. The termination condition can be related to a cost function or an iteration count of the training process. For example, the processing unit 122 can determine a cost function of the machine learning model and determine a value of the cost function based on a difference between an estimated output and an actual or desired output (i.e., a reference output). The processing unit 122 can also determine that the termination condition is met when the value of the cost function is less than a threshold. The threshold may be a default setting of the system 100 or may be adjusted in different situations. As another example, the termination condition can be met when the value of the cost function converges. Convergence is considered to have occurred if the change in the cost function value is less than a threshold (e.g., a constant) in two or more consecutive iterations. As yet another example, the processing unit 122 can determine that the termination condition is met when a predetermined number (or count) of iterations have been performed in the training process.

[0159] In response to determining that the termination condition is met, the processing unit 122 may designate the machine learning model having the updated parameter values ​​in the previous iteration as a trained machine learning model (e.g., a trained machine learning model). On the other hand, in response to determining that the termination condition is not met, the processing unit 122 may update at least a portion of the parameter values ​​of the machine learning model based on the evaluation result. For example, the processing unit 122 may update the values ​​of the learning parameters (S) of the machine learning model based on the values ​​according to a cost function such as an inverse decay algorithm. The processing unit 122 may continue to perform the next iteration until the termination condition is met. In the next iteration, the processing unit 122 may obtain another set of training data. Once the termination condition is met in a particular iteration, the machine learning model in the particular iteration having the updated values ​​of the learning parameters may be designated as a trained machine learning model (e.g., a trained machine learning model).

[0160] The trained machine learning model (e.g., the second part of the preliminary machine learning model) can provide a mapping relationship between each of the one or more first bone conduction data sets and at least one of the one or more second air conduction data sets based on the multiple pairs of air conduction data and bone conduction data. In some examples, each of the multiple pairs of bone conduction data and air conduction data may refer to a pair of bone conduction data and air conduction data having a correspondence relationship. As used herein, the correspondence relationship may refer to the bone conduction data set corresponding to the air conduction data set. If the similarity or relatedness between the semantic information represented by the bone conduction data set and the semantic information represented by the air conduction data set exceeds a threshold, e.g., 90%, 95%, 99%, etc., the bone conduction data set may correspond to the air conduction data set. The preliminary machine learning model can establish a mapping relationship between each of the one or more first bone conduction data sets and at least one of the one or more second air conduction data sets based on the correspondence relationship. The mapping relationship may include a mapping relationship between acoustic features, a mapping relationship between phonemes, a mapping relationship between syllables, a mapping relationship between words, etc., or any combination thereof. Further discussion of the mapping relationships can be found elsewhere in this application (eg, FIG. 6 and its discussion).

[0161] For example, the one or more first bone conduction data sets include syllable A and syllable B, and the one or more second air conduction data sets include syllable C, syllable D, syllable E, syllable F, and syllable G. The multiple pairs of bone conduction audio data and air conduction data include AC, AD, BE, BF, and BG. A and C, A and D, B and E, B and F, and B and G may be pairs of bone conduction audio data and air conduction data having a corresponding or matching relationship. The mapping relationship can be expressed by the following formula:

[0162]

number

[0163] In some embodiments, the preliminary machine learning model may include a first portion (also referred to as a first sub-model) and a second portion (also referred to as a second sub-model). The first portion may be trained to obtain a speech recognition model. The first portion may decompose the audio signal into one or more units (also referred to as an acoustic vector) by extracting acoustic features of the audio signal. Each unit may be referred to as a set of audio data derived from the audio signal. The unit of the audio signal (i.e., a set of audio data) may include a phoneme of acoustic features, a syllable of one or more phonemes, a word of one or more syllables, a phrase of one or more words, a sentence of one or more phrases, etc., or any combination thereof. For example, the first audio signal may be decomposed in the first portion to obtain a plurality of bone conduction data sets (i.e., a plurality of bone conduction units), and the second audio signal may be decomposed in the first portion to obtain a plurality of air conduction data sets (i.e., a plurality of air conduction units). In some embodiments, the first portion of the preliminary machine learning model may include an encoder including a machine learning model (e.g., a conventional neural network model, a deep neural network model, etc.).

[0164] The second part may be trained to establish a mapping relationship between the plurality of bone conduction data sets and the plurality of air conduction data sets. The mapping relationship between the plurality of bone conduction data sets and the plurality of air conduction data sets may be referred to as a matching relationship between each of the plurality of bone conduction data sets and one or more of the plurality of air conduction data sets. In some embodiments, when a degree of association between a bone conduction data set and an air conduction data set exceeds a threshold, the bone conduction data set can be matched with the air conduction data set, and a matching relationship can be established between the bone conduction data set and the air conduction data set.

[0165] In some embodiments, the degree of association between one bone conduction data set and one air conduction data set can be expressed as a degree of similarity between the bone conduction data set and the air conduction data set. In some embodiments, the second part may include a similarity determination function. The second part may use a similarity determination function to determine a degree of association between each of the plurality of bone conduction data and each of the plurality of air conduction data. Exemplary similarity determination functions may include a cosine similarity function, a Jaccard coefficient function, a Dice coefficient function, and the like. In some embodiments, the second part may include a machine learning model for determining semantic similarity. Exemplary machine learning models for determining semantic similarity may include a deep structure semantic model (DSSM), a convolutional latent semantic model (CLSM), a long-short-term memory deep structure semantic model (LSTM-DSSM), and the like, or any combination thereof.

[0166] By way of example only, FIG. 9 is a schematic diagram of an exemplary preliminary machine learning model according to some embodiments of the present application. As shown in FIG. 9, the preliminary machine learning model may include an input layer 902, an encoder 904, a decoder 906, and an output layer 908. The encoder 904 may be referred to as a first part (i.e., a first sub-model) of the preliminary machine learning model. The input layer 902 may be configured to input an audio signal. The encoder 904 may be configured to convert the input audio signal (e.g., a first audio signal and a second audio signal) into a sequence of acoustic vectors. In some embodiments, the encoder 904 may include a neural network model (e.g., CNN, DNN, etc.) of multiple layers V1, V2, ..., VL. The decoder 906 may be configured to determine one or more acoustic vectors corresponding to an audio signal that is most likely to correspond to each acoustic vector among a sequence of acoustic vectors corresponding to another audio signal. The decoder 906 may include a neural network model (e.g., CNN, DNN, etc.), a Bayesian network model (e.g., HMM), etc. As shown in FIG. 9, the decoder 906 may include an HMM. The parameters of the HMM may be expressed as (p, A, B), where p refers to a reference initial probability matrix, A refers to a hidden state transition matrix, and B refers to a probability matrix for transitioning from a hidden state to an observed state. The hidden state transfer matrix A may include transition probabilities of hidden states, for example, a transition probability a01 from hidden state S0 to hidden state S1, a transition probability a12 from hidden state S1 to hidden state S2, ..., a transition probability ak(k+1) from hidden state Sk to hidden state Sk+1. The probability matrix B transmitted from a hidden state to an observed state may include transition probabilities from a hidden state to an observed state, a transition probability b1(y0) from hidden state S1 to observed state Y0, a transition probability b1(y1) from hidden state S1 to observed state y1, ..., a transition probability bk(yk) from hidden state Sk to observed state yk. For each iteration, the input layer 902 may input a set of training data, an audio signal (eg, a first audio signal and a second audio signal, eg, an audio waveform).The encoder 904 may extract acoustic features of each audio signal (e.g., the first audio signal and the second audio signal) to form a first acoustic vector sequence corresponding to the first audio signal and a second acoustic vector sequence corresponding to the second audio signal. In some embodiments, the acoustic vectors in the acoustic vector sequence corresponding to the audio signals may include acoustic features of each unit of the audio signal (e.g., a phoneme consisting of acoustic features, a syllable consisting of one or more phonemes, a word consisting of one or more syllables, a phrase consisting of one or more words, a sentence consisting of one or more phrases). The decoder 906 may determine a hidden state sequence (i.e., S0, S1, S2, ..., Sk, Sk+1) based on the parameters of the decoder 906 and the observed state sequence (i.e., y0, y1, y2, ..., yk, yk+1). In other words, the decoder 906 may determine one or more acoustic vectors in the second acoustic vector sequence of the second audio signal that are most likely to correspond to each acoustic vector in the first acoustic vector sequence corresponding to the first audio signal based on the parameters of the decoder 906. The decoder 906 may then establish a mapping relationship between one or more acoustic vectors in the second audio signal corresponding to the second audio signal and each acoustic vector in the first audio signal sequence corresponding to the first audio signal. The decoder 906 may, for example, adjust parameters of the decoder 906 and the encoder 904 according to the Baum-Welch algorithm to obtain an optimal solution (i.e., one or more acoustic vectors in the second audio signal sequence corresponding to the second audio signal that are most likely to be each acoustic vector in the first audio signal sequence corresponding to the first audio signal). In some embodiments, the decoder 906 may use a maximum likelihood estimation algorithm to determine the optimal solution (i.e., one or more acoustic vectors in the second audio signal sequence of the second audio signal that are most likely to be each acoustic vector in the first audio signal sequence corresponding to the first audio signal).

[0167] It should be noted that the above description of the process 800 is merely for explanation and is not intended to limit the scope of the present application. Those skilled in the art can make various changes and modifications based on the description of the present application. However, these changes and modifications do not depart from the scope of the present application. For example, the process 800 may further include an operation of initializing parameter values ​​of the preliminary machine learning model and / or an operation of storing the trained machine learning model.

[0168] FIG. 10 is a schematic flow chart of an exemplary process for generating an audio signal according to some embodiments of the present application. In some embodiments, the process 1000 may be implemented as instructions (e.g., application programs) stored in the storage device 140, the ROM 230, the RAM 240, or the memory 390. The processing device 122, the processor 220, and / or the CPU 340 may execute a set of instructions, and when the processing device 122, the processor 220, and / or the CPU 340 executes the instructions, the processing device 122, the processor 220, and / or the CPU 340 may be configured to perform the process 1000. The operations of the process shown below are for illustration purposes only. In some embodiments, the process 1000 may be completed by one or more additional operations not described and / or one or more operations not discussed. Also, the order of operations of the process 1000 shown in FIG. 10 and described below is not limited.

[0169] At 1002, the processing unit 122 (eg, the acquisition module 402) can acquire a bone conduction audio signal acquired by a bone conduction sensor.

[0170] In some embodiments, the processing unit 122 may acquire bone conduction audio signals in real time or periodically from a bone conduction sensor (e.g., bone conduction microphone 112), the terminal 130, the storage device 140, or any other storage device via the network 150. As the user speaks, the bone conduction audio signals may be acquired and / or generated by the bone conduction sensor. In some embodiments, operation 1002 may be the same as or similar to operation 602 of process 600 shown in FIG. 6.

[0171] At 1004, the processing unit 122 (eg, the acquisition module 402) may acquire an air-conduction audio signal acquired by an air-conduction sensor.

[0172] In some embodiments, the processing unit 122 may acquire air conduction audio signals in real time or periodically from an air conduction sensor (e.g., air conduction microphone 114), the terminal 130, the storage device 140, or any other storage device via the network 150. When a user speaks, the air conduction audio signals may be acquired and / or generated by the air conduction sensor. In some embodiments, operation 1004 may be the same as or similar to operation 604 of process 600 shown in FIG. 6.

[0173] At 1006, the processing unit 122 (eg, the acquisition module 402) may acquire the trained machine learning model.

[0174] In some embodiments, the trained machine learning model may be reconstructed based on a neural network model (e.g., a deep learning model). Exemplary deep learning models may include a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a long short-term memory network (LSTM) model, or the like, or a combination thereof.

[0175] In some embodiments, the processing unit 122 may obtain the trained machine learning model from one or more components of the audio signal generation system 100 (e.g., storage device 140, terminal 130) or an external source via a network (e.g., network 150). For example, the trained machine learning model may be previously trained by a computing device (e.g., processing unit 122) and stored in a storage device (e.g., storage device 140) of the audio signal generation system 100. The processing unit 122 may access the storage device to retrieve the trained machine learning model.

[0176] In some embodiments, the trained machine learning model may be generated by the processing unit 122 or another computing device (e.g., a computing device of a vendor of the trained machine learning model) according to a training algorithm. Exemplary training algorithms may include a gradient descent algorithm, a Newton algorithm, a quasi-Newton algorithm, a Levenberg-Marquardt algorithm, a conjugate gradient algorithm, or the like, or a combination thereof.

[0177] By way of example only, the trained machine learning model may be obtained by a process. The process may include obtaining a plurality of sets of training data and a preliminary machine learning model, and training the preliminary machine learning model using the plurality of sets of training data. Each of the plurality of sets of training data may include a bone conduction audio signal, an air conduction audio signal, and a reference air conduction audio signal. In training the preliminary machine learning model, the bone conduction audio signal and the air conduction audio signal in each set of training data may be used as an input of the preliminary machine learning model, and the reference air conduction audio signal may be used as a desired output of the preliminary machine learning model. The bone conduction audio signal, the air conduction audio signal, and the reference air conduction audio signal in each set of training data may represent the same voice of a user. In each set of training data, the reference air conduction audio signal may include a noise level that is less than the noise level of the air conduction audio signal.

[0178] In some embodiments, the bone conduction audio signal and the reference air conduction audio signal in a set of training data may be obtained by a bone conduction sensor and an air conduction sensor, respectively, when a user speaks under noise-free conditions. The air conduction audio signal in the training data set may be obtained by adding noise to the reference air conduction audio signal in the training data set.

[0179] In some embodiments, the bone conduction audio signals and the air conduction audio signals in a set of training data may be acquired by a bone conduction sensor and an air conduction sensor, respectively, when a user speaks under noise conditions, and a reference air conduction audio signal in the training data set may be obtained by removing noise from the air conduction audio signals in the training data set.

[0180] In some embodiments, the bone conduction audio signal and the air conduction audio signal in a set of training data may be acquired by a bone conduction sensor and an air conduction sensor, respectively, when a user speaks under noise conditions, and the air conduction sensors may each acquire a reference air conduction audio signal in the training data set under noise-free conditions.

[0181] In some embodiments, the machine learning model may include a multi-layer structure. For example, the machine learning model may include an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. In some embodiments, the hidden layer may include one or more convolution layers, one or more rectified linear unit layers (ReLU layers), one or more pooling layers, one or more fully connected layers, etc., or any combination thereof. As used herein, a layer of a model may refer to an algorithm or function for processing input data of the layer. Different layers may perform different types of processing on their respective inputs. A successive layer may use output data from a layer before the successive layer as input data. In some embodiments, the convolution layer may include multiple kernels for extracting features. In some embodiments, each of the multiple kernels may filter a portion (i.e., a region). The pooling layer may take the output of the convolution layer as input. The pooling layer may include multiple pooling nodes that can sample the output of the convolution layer to reduce the computational load of data processing and accelerate data processing. In some embodiments, the size of the matrix representing the input data in the pooling layer may be reduced. The fully connected layer may include at least two neurons. The neurons may be connected to the pooling nodes in a pool. In the fully connected layer, a plurality of vectors corresponding to the plurality of pooling nodes may be determined based on the training samples, and a plurality of weighting factors may be assigned to the plurality of vectors. The output layer may determine an output based on the vectors and the weighting factors obtained from the fully connected layer.

[0182] In some embodiments, each layer may include one or more nodes. In some embodiments, each node may be connected to one or more nodes of the previous layer. The number of nodes in each layer may be the same or different. In some embodiments, each node may correspond to an activation function. As used herein, an activation function of a node may define an output of the node given an input or a set of inputs. In some embodiments, each connection between two nodes of the plurality of nodes in the preliminary machine learning model may transmit a signal from one node to the other node. In some embodiments, each connection may correspond to a weight. As used herein, a weight corresponding to a connection may increase or decrease the strength or influence of the signal in the connection.

[0183] The machine learning model may include at least two parameters, such as architecture parameters and learning parameters. Exemplary architecture parameters of the machine learning model may include the size of the kernel of a layer, the total number (or number) of layers, the node count (or number) of each layer, the learning rate, the batch size, epochs, etc. Exemplary learning parameters may include the connection weight between two connecting nodes, the bias carrier for the node, etc.). Before training, the machine learning model may have one or more initial parameter values. In training the machine learning model, the learning parameters of the machine learning model may be updated. Before the update process, the values ​​of the learning parameters of the machine learning model may be initialized. For example, the connection weights and / or deviation vectors of the nodes of the preliminary machine learning model may be initialized by assigning random values ​​in the range of -1 to 1. As another example, all the connection weights of the preliminary machine learning model may be assigned the same value in the range of -1 to 1, such as 0. As an example, the deviation vectors of the nodes in the preliminary machine learning model may be initialized by assigning random values ​​in the range of 0 to 1. In some embodiments, the parameters of the preliminary machine learning model may be initialized according to a Gaussian random algorithm, a Xavier algorithm, etc.

[0184] Training the preliminary machine learning model may include one or more iterations, iteratively updating parameter values ​​of the machine learning preliminary model based on the training data until a termination condition is met in an iteration. Exemplary termination conditions may be that a value of a loss function obtained in a particular iteration is less than a threshold, that a particular number of iterations have been performed, that the loss function converges such that the difference between values ​​of the loss function obtained in a previous iteration and a current iteration is within a threshold, etc. The loss function may measure a difference between an estimated air-conducted audio signal predicted by the preliminary machine learning model in an iteration based on the bone-conducted audio signal and the air-conducted audio signal in each set of training data and a reference air-conducted audio signal in the training data set. For example, the bone-conducted audio signal and the air-conducted audio signal in each set of training data may be input to the preliminary machine learning model, and the preliminary machine learning model may output a predicted or estimated air-conducted audio signal. The loss function may measure a difference between a predicted air-conducted audio signal in each set of training data and a reference second air-conducted audio signal. The parameter values ​​of the preliminary machine learning model may be adjusted, for example, using a back-propagation algorithm based on the value of the loss function in the current iteration. Exemplary loss functions may include focal loss function, logarithmic loss function, cross entropy loss, Dice ratio, etc. If the termination condition is not met in the current iteration, the processing unit 122 may further update the preliminary machine learning model used in the next iteration (also referred to as updating parameter values ​​of the preliminary machine learning model), for example according to a backpropagation algorithm. If the termination condition is met in the current iteration, the processing unit 122 may designate the preliminary machine learning model in the current iteration as the trained machine learning model.

[0185] The trained machine learning model can provide a mapping relationship between the specific bone conduction audio signal and the specific air conduction audio signal, and an equivalent air conduction audio signal corresponding to the specific bone conduction audio signal. The processing unit 122 can determine an equivalent air conduction audio signal corresponding to the specific bone conduction audio signal based on the mapping relationship, the specific bone conduction audio signal, and the specific air conduction audio signal. When a user speaks, the specific bone conduction audio signal and the specific air conduction audio signal can be acquired by the bone conduction sensor and the air conduction sensor, respectively. The specific bone conduction audio signal and the specific air conduction audio signal can represent the same user voice. The equivalent air conduction audio signal corresponding to the specific bone conduction audio signal may have a noise level smaller than the noise level of the specific air conduction audio signal.

[0186] At 1008, the processing unit 122 (e.g., the determination module 404) may generate a target audio signal representative of a user's voice using the trained machine learning model based on the bone conducted audio signal and the air conducted audio signal. In some embodiments, the noise level of the target audio signal may be less than the noise level of the air conducted audio signal.

[0187] In some embodiments, the bone conducted audio signal and / or the air conducted audio signal may be input to a trained machine learning model, which may directly output an equivalent air conducted audio signal. In some embodiments, the processing unit 122 may designate the equivalent air conducted audio signal as a target audio signal. In some embodiments, the processing unit 122 may process the equivalent air conducted audio signal to obtain the target audio signal, as described elsewhere herein.

[0188] In some embodiments, the bone conduction audio signal and / or the air conduction audio signal may be input to a trained machine learning model, and the trained machine learning model may output information related to the target audio signal. For example, the trained machine learning model may output a set of audio data. The processing unit 122 may convert the set of audio data into the target audio signal using an algorithm (e.g., the speech synthesis algorithm described in operation 610).

[0189] At 1010, the processing unit 122 (eg, the output module 406) can cause an audio signal output device to output a target audio signal.

[0190] In some embodiments, the processing unit 122 may transmit a signal over the network 150 to a client terminal (e.g., terminal 130), to the storage device 140, and / or to any other storage device (not shown in the audio signal generation system 100). The signal may include a target audio signal. The signal may be configured to instruct the client terminal to play the target audio signal.

[0191] It should be noted that the above description is merely for the purpose of explanation and is not intended to limit the scope of the present application. Those skilled in the art can make various changes and modifications based on the description of the present application. However, these changes and modifications do not depart from the scope of the present application.

[0192] Although the basic concept has been described above, it is clear to those skilled in the art who read this application that the above disclosure of the invention is presented only as an example and is not intended to limit the present application. Although not expressly described herein, those skilled in the art can make various changes, improvements and modifications to the present application. These changes, improvements and modifications are intended to be suggested by the present application and therefore still fall within the spirit and scope of the exemplary embodiments of the present application.

[0193] Furthermore, certain terms are used herein to describe embodiments of the present application. For example, "one embodiment," "an embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of the present application. Thus, it is emphasized and understood that references to "one embodiment," "one embodiment," or "one alternative embodiment" more than once in various parts of this specification do not necessarily all refer to the same embodiment. Also, certain features, structures, or characteristics of one or more embodiments of the present application may be combined as appropriate.

[0194] Moreover, as will be appreciated by those skilled in the art, aspects of the present application may be illustrated and described in several patentable classes or contexts, including any new and useful process, machine, manufacture or combination of matter, or any new and useful improvement thereto. Accordingly, aspects of the present application may be embodied entirely in hardware, generally referred to herein as a "unit," "module," or "system," software (including firmware, resident software, microcode, etc.), or a combination of software and hardware. Moreover, aspects of the present application may take the form of a computer program product embodied in one or more computer-readable medium(s) containing computer-readable program code.

[0195] The non-transitory computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave for carrying computer-readable program code. Such propagated signals may include various forms, such as electromagnetic signals, optical signals, or suitable combinations thereof. The computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, apparatus, or device to realize the communication, propagation, or transmission of a program used. The program code on the computer-readable signal medium may be propagated in any suitable medium, including wireless, cable, fiber optic cable, RF, etc., or any combination of the above media.

[0196] Computer program code for carrying out the operations of aspects of the present application may be coded in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., the "C" programming language, traditional procedural programming languages ​​such as Visual Basic, Fortran, Perl, COBOL, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages. The program code may be executed entirely on the user's computer, on the user's computer as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), connected to an external computer (e.g., using a network service provider's network), in a cloud computing environment, or provided as a service, e.g., Software as a Service (SaaS).

[0197] Additionally, unless expressly stated in the claims, the recited order, use of alphanumeric characters, or other designations of the processing elements or sequences described herein are not intended to limit the order of procedures and methods of the present application. While the above disclosure describes through various examples what are presently believed to be various useful embodiments of the invention, it will be understood that such details are merely illustrative and that the appended claims are not limited to the disclosed embodiments, but on the contrary are intended to cover all modifications and equivalent combinations within the spirit and scope of the embodiments of the present application. For example, the system assembly described above may be implemented by a hardware device, but may also be implemented as a software-only solution, for example, by installing the described system on an existing server or mobile device.

[0198] Similarly, in the foregoing description of embodiments of the present application, it will be understood that various features may be combined in a single embodiment, drawing, or description thereof in order to simplify the application and facilitate understanding of one or more of the embodiments of each invention. However, this method of the present application should not be interpreted as reflecting an intention that the claimed subject matter requires more features than are expressly recited in each claim. Rather, the claimed subject matter should have fewer features than a single embodiment described above.

[0199] In some embodiments, the numbers used to describe and protect the quantities, characteristics, and so forth of some embodiments of the present application may be understood to be modified by the terms "about", "approximately" or "substantially". For example, "about", "approximately" or "substantially" indicate that the stated value is allowed to vary by ±20%, unless otherwise specified. Thus, in some embodiments, all numerical parameters used in the specification and claims are approximations that may vary depending on the characteristics required for a particular embodiment. In some embodiments, the numerical parameters should be applied with the stated number of significant digits and ordinary rounding techniques. In some embodiments, the numerical ranges and parameters for determining the ranges are approximations; however, in specific embodiments, such numerical values ​​are set as precisely as possible.

[0200] All patents, patent applications, published patent applications, and other materials (articles, books, specifications, publications, records, things, and / or the like) mentioned herein are incorporated herein by reference in their entirety for all purposes, except for any charging documents related to said documents, any such documents that are inconsistent with or contradictory to this document, or any such documents that sooner or later limit the broad scope of the claims related to this document. For example, in the event of any inconsistency or discrepancy between the description, definition, and / or use of a term related to any of the incorporated materials and the term related herein, the description, definition, and / or term used herein shall control for the term in the present specification.

[0201] Finally, it should be understood that the embodiments described herein are merely illustrative of the principles of the embodiments of the present application. Other variations may be within the scope of the present application. Thus, by way of example, and not of limitation, alternative configurations of the embodiments of the present application may be considered consistent with the teachings of the present application. Thus, the embodiments of the present application are not limited to the embodiments expressly introduced and described herein. [Explanation of symbols]

[0202] 100 Audio signal generation system 110 Audio Collector 112 Bone Conduction Microphone 114 Air Conduction Microphone 120 Servers 122 Processing equipment 130 terminals 140 Storage device 150 Network 200 computing devices 210 Internal Communication Bus 220 Processor 230 Read Only Memory 240 Random Access Memory 250 communication port 260 I / O Devices 270 Hard Disk 300 Audio Collector 310 Collection Module 320 signal processing module 402 acquisition module 404 Decision Module 406 Output Module 502 Acquisition Module 504 Model Generation Module

Claims

1. at least one storage medium containing an instruction set; and at least one processor in communication with the at least one storage medium, the instruction set providing to the at least one processor: obtaining a bone conduction audio signal representative of a user's voice captured by a bone conduction sensor; obtaining an air conducted audio signal representative of the user's voice obtained by an air conducted sensor; Obtaining a trained machine learning model that provides a mapping relationship between a bone conduction data set derived from a particular bone conduction audio signal and an equivalent air conduction data set derived from a particular equivalent air conduction audio signal; determining a target equivalent air conduction data set representing semantic content of the user's speech corresponding to the bone conduction audio signal using the trained machine learning model based on the bone conduction audio signal and the air conduction audio signal, wherein determining the target equivalent air conduction data set corresponding to the bone conduction audio signal includes: determining, based on the bone conducted audio signal, one or more equivalent air conduction data sets corresponding to the bone conducted audio signal using the trained machine learning model; determining, based on the air conducted audio signal, identifying the target equivalent air conducted data set from the one or more equivalent air conducted data sets; and causing an audio signal output device to output a target audio signal representative of said user voice based on said target equivalent air conduction data set.

2. determining, based on the bone conducted audio signal, one or more equivalent air conduction data sets corresponding to the bone conducted audio signal using the trained machine learning model, 2. The system of claim 1, further comprising inputting the bone conducted audio signal into the trained machine learning model to obtain the one or more equivalent air conduction data sets corresponding to the bone conducted audio signal.

3. determining, based on the bone conducted audio signal, one or more equivalent air conduction data sets corresponding to the bone conducted audio signal using the trained machine learning model, extracting bone-conducted acoustic features from the bone-conducted audio signal; determining the bone conduction data set based on the bone conduction acoustic signature; and and inputting the bone conduction data set into the trained machine learning model to obtain the one or more equivalent air conduction data sets corresponding to the bone conduction audio signal.

4. Identifying the target equivalent air conduction data set from the one or more equivalent air conduction data based on the air conduction audio signal includes: extracting air-conducted acoustic features from the air-conducted audio signal; determining an air conducted data set based on the air conducted acoustic signature; and determining, for each of the one or more equivalent air conduction data sets, a degree of association between the equivalent air conduction data set and the air conduction data set; determining an equivalent air conduction data set from among said equivalent air conduction data sets that has a maximum degree of correlation with said air conduction data set; and determining the target equivalent air conductance data set based on the equivalent air conductance data set.

5. causing an audio signal output device to output a target audio signal representative of the user's voice based on the target equivalent air conduction data set, converting the target equivalent air conduction data set into a target equivalent air conduction audio signal; determining the target audio signal based on the target equivalent air conducted audio signal; and causing the audio signal output device to output the target audio signal.

6. The system of claim 5 , wherein the noise level of the target audio signal is less than the noise level of the air conducted audio signal.

7. determining the target audio signal based on a target equivalent air conducted audio signal, 6. The system of claim 5, further comprising generating the target audio signal by adding noise to the target equivalent air conducted audio signal.

8. The trained machine learning model is obtaining a plurality of sets of training data, each set including a first audio signal and a second audio signal representing the same audio sample, the first audio signal being collected by a bone conduction audio collection device and the second audio signal being collected by an air conduction audio collection device under noise-free conditions; training a preliminary machine learning model using the plurality of sets of training data to obtain the trained machine learning model.

9. The system of claim 8 , wherein the preliminary machine learning model is constructed based on a hidden Markov model.

10. Training a preliminary machine learning model using the plurality of sets of training data to obtain the trained machine learning model, extracting a first acoustic feature from the first audio signal; extracting second acoustic features from the second audio signal; and determining one or more first bone conduction data sets based on the first acoustic features; determining one or more second air conduction data sets based on the second acoustic signature; and determining a plurality of pairs of air conduction data and bone conduction data, each pair including a first pair of air-bone conduction data and a second pair of air-bone conduction data representing the same meaning as the first pair of bone conduction data; and establishing a mapping relationship between each of the one or more first bone conduction data sets and at least one of the one or more second air conduction data sets based on the plurality of pairs of air conduction data sets and bone conduction data sets.

11. 11. The system of claim 10, wherein each of the one or more first bone conduction data sets and the one or more second air conduction data sets includes syllables of one or more phonemes.

12. 2. The system of claim 1, wherein the trained machine learning model provides a mapping relationship between each syllable of the bone conducted audio signal and one or more syllables of the air conducted audio signal.

13. 13. The system of claim 12, wherein the mapping relationship between each syllable in the bone conduction dataset and one or more syllables in the one or more equivalent air conduction datasets includes a probability of matching each syllable in the bone conduction dataset with each of the one or more syllables in the one or more equivalent air conduction datasets.

14. 1. A method implemented by a computing device including at least one processor and at least one storage device, comprising: obtaining a bone conduction audio signal representative of a user's voice captured by a bone conduction sensor; obtaining an air conducted audio signal representative of the user's voice obtained by an air conducted sensor; obtaining a trained machine learning model that provides a mapping relationship between a bone conduction data set derived from a particular bone conduction audio signal and one or more equivalent air conduction data sets derived from a particular equivalent air conduction audio signal; determining a target equivalent air conduction data set representing semantic content of the user's speech corresponding to the bone conduction audio signal using the trained machine learning model based on the bone conduction audio signal and the air conduction audio signal, wherein determining the target equivalent air conduction data set corresponding to the bone conduction audio signal includes: determining, based on the bone conducted audio signal, one or more equivalent air conduction data sets corresponding to the bone conducted audio signal using the trained machine learning model; determining, based on the air conducted audio signal, identifying the target equivalent air conducted data set from the one or more equivalent air conducted data sets; and causing an audio signal output device to output a target audio signal representative of the user's voice based on the target equivalent air conduction data set.

Images