Voice enhancement method and device based on audio-visual bone fusion

Abstract

The invention discloses a voice enhancement method and device based on audio-visual bone fusion. The method comprises the following steps: acquiring a voice enhancement training data set; performing training processing on a preset voice enhancement model by using the voice enhancement training data set to obtain a trained voice enhancement model; acquiring a multi-modal data set of the user in a speaking state; the multi-modal data set comprises video data, air conduction data and bone conduction data; the multi-modal data set is acquired based on the same time reference; and processing the multi-modal data set by using the trained voice enhancement model to obtain enhanced voice information. According to the method, the three-mode information can be effectively utilized to cope with voice enhancement challenges in a high-noise environment, and remarkable effects are achieved in the aspects of specific speaker enhancement and speaker separation. In addition, interference caused by illumination change and human head posture movement in audio-visual voice enhancement and challenge of high-frequency information loss in air bone voice enhancement are overcome, and the stability and accuracy of voice enhancement are further optimized.

Description

Technical Field

[0001] The present invention relates to the technical field of speech enhancement, and in particular to a speech enhancement method and device based on audio-visual bone fusion. Background Art

[0002] At present, clear voice communication is crucial in daily life and professional scenarios, especially in complex environments such as high noise backgrounds. However, environmental noise, such as traffic and crowd noise, often reduces the clarity of voice signals and affects the efficiency of communication. Therefore, developing effective speech enhancement technology to recover high-quality speech from noisy environments is a key challenge in this field. Traditional speech enhancement technology mainly focuses on improving speech quality through single-modal speech denoising processing. This method relies on analyzing noise characteristics and separating noise from speech signals to obtain clearer speech output. Although this method has certain advantages in processing speed and cost, it does not perform well under conditions of limited information, especially in strong noise environments. In the face of strong noise environments, single-modal speech enhancement often has poor results because effective speech information is easily submerged by background noise.

[0003] Audiovisual fusion technology integrates visual and auditory information to simulate the natural process of humans using visual cues (such as observing lip movements and facial expressions) to assist auditory understanding in noisy environments, especially in noisy environments, making up for the lack of effective information in audio. This technology uses a deep learning model to learn the correlation between audiovisual signals, fuses multimodal data at different feature levels, and effectively improves speech clarity and intelligibility in background noise. Although audiovisual fusion technology has shown broad application prospects in many application fields such as improving the performance of hearing assistive devices, multimedia content production, and voice control systems, it still faces challenges in its implementation, such as lighting conditions, the impact of object occlusion on the quality of visual information, and the problem of synchronization of audiovisual data streams. Bone conduction (BC) technology uses sensors to pick up the vibration signal of the human larynx or skull when a person speaks. Since the vibration signal is not interfered by noise in the air, it effectively avoids environmental noise. Although bone conduction microphones have a natural resistance to background noise, the recovery of high-frequency information is still a challenge due to their bandwidth limitations.

[0004] How to effectively integrate multiple types of information for the purpose of speech enhancement in complex environments while overcoming the bandwidth limitations of bone conduction signals and the occlusion and lighting conditions of visual signals is an issue that needs to be addressed urgently. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a speech enhancement method and device based on audio-visual bone fusion, which is aimed at the speech enhancement goal in complex environments, effectively fuses multiple types of information, and overcomes the bandwidth limitation of bone conduction signals and the occlusion problem of visual signals and lighting conditions, thereby improving the real-time and accuracy of speech enhancement.

[0006] In order to solve the above technical problems, in a first aspect of an embodiment of the present invention, a speech enhancement method based on visual and auditory bone fusion is disclosed, comprising:

[0007] S1, obtain the speech enhancement training data set;

[0008] S2, using the speech enhancement training data set to train a preset speech enhancement model to obtain a trained speech enhancement model;

[0009] S3, acquiring a multimodal data set of the user in a speaking state; the multimodal data set includes video data, air conduction data, and bone conduction data; the multimodal data set is acquired based on the same time reference;

[0010] S4, using the trained speech enhancement model, processing the multimodal data set to obtain enhanced speech information; the enhanced speech information represents the user's speech information after speech enhancement processing.

[0011] The speech enhancement training data set includes a speech enhancement data set, an audio-visual data set, and an air-bone data set; the speech enhancement data set includes an air conduction data sequence and corresponding label information; the audio-visual data set includes a video data sequence and corresponding label information, an air conduction data sequence and corresponding label information; the air-bone data set includes an air conduction data sequence and corresponding label information, a bone conduction data sequence and corresponding label information;

[0012] The category information of the speech enhancement training data set includes speech enhancement data, audio-visual data, and air-bone data.

[0013] The speech enhancement model includes: an encoding module, an air-bone fusion module and a comprehensive fusion module;

[0014] The encoding module is used to encode the air conduction data, the bone conduction data and the video data to obtain encoding information;

[0015] The air-bone fusion module is connected to the encoding module and the comprehensive fusion module respectively, and is used to fuse the encoding information of the air conduction data and the encoding information of the bone conduction data to obtain air-bone fusion information;

[0016] The comprehensive fusion module is used to fuse the air-bone fusion information and the encoding information of the video data to obtain enhanced voice information;

[0017] The encoding module includes an air conduction encoding submodule, a bone conduction encoding submodule, and a video encoding submodule.

[0018] The air-bone fusion module includes a first input module, a first self-attention module, a first multi-head attention module, a first addition module, a first fusion output module, a second input module, a second self-attention module, a second multi-head attention module, a second addition module, and a second fusion output module;

[0019] The first fusion output module is used to output a fusion air conduction signal;

[0020] The second fusion output module is used to output a fusion bone conduction signal;

[0021] The first input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the air conduction coding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the first input module is connected to the input port of the first self-attention module; the second output port of the first input module is connected to the second input port of the first addition module;

[0022] The first self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable obtained by self-attention encoding the coded information output by the air conduction coding submodule; the second output port is used to output the K variable obtained by self-attention encoding the coded information output by the air conduction coding submodule; the third output port is used to output the V variable obtained by self-attention encoding the coded information output by the air conduction coding submodule; the first output port of the first self-attention module is connected to the first input port of the first multi-head attention module; the second output port of the first self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the first self-attention module is connected to the third input port of the second multi-head attention module;

[0023] The first multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the first multi-head attention module is connected to the first input port of the first addition module;

[0024] The first addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the first fusion output module;

[0025] The second input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the bone conduction coding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the second input module is connected to the input port of the second self-attention module; the second output port of the second input module is connected to the second input port of the first addition module;

[0026] The second self-attention module comprises an input port, a first output port, a second output port, and a third output port; the first output port is used to output a Q variable obtained by self-attention encoding the coded information output by the bone conduction coding submodule; the second output port is used to output a K variable obtained by self-attention encoding the coded information output by the bone conduction coding submodule; the third output port is used to output a V variable obtained by self-attention encoding the coded information output by the bone conduction coding submodule; the first output port of the second self-attention module is connected to the first input port of the second multi-head attention module; the second output port of the second self-attention module is connected to the second input port of the first multi-head attention module; the third output port of the second self-attention module is connected to the third input port of the first multi-head attention module;

[0027] The second multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the second multi-head attention module is connected to the first input port of the second addition module;

[0028] The second addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the second fusion output module.

[0029] The comprehensive fusion module includes a first fusion submodule, a second fusion submodule, a third fusion submodule, a fifth splicing fusion module, and a one-dimensional deconvolution network module;

[0030] The first fusion submodule includes: a first PReLU activation function module, a first normalization module, a first fully connected layer, a second PReLU activation function module, a second normalization module, a first splicing and fusion module, a second fully connected layer, a third PReLU activation function module, a third fully connected layer, a fourth PReLU activation function module, a third normalization module, a first LSTM network module, a fourth normalization module, a second splicing and fusion module, a fourth fully connected layer, a first Sigmoid activation function module, and a first Hadamard inner product module;

[0031] The output end of the first PReLU activation function module is connected to the input end of the first normalization module; the output end of the first normalization module is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the second PReLU activation function module; the output end of the second PReLU activation function module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the first splicing and fusion module; the output end of the first splicing and fusion module is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the third PReLU activation function module; the output end of the third PReLU activation function module is connected to the input end of the third fully connected layer; The output end of the fully connected layer is connected to the input end of the fourth PReLU activation function module; the output end of the fourth PReLU activation function module is connected to the input end of the third normalization module; the output end of the third normalization module is connected to the input end of the first LSTM network module; the output end of the first LSTM network module is connected to the input end of the fourth normalization module; the output end of the fourth normalization module is connected to the input end of the second splicing and fusion module; the output end of the second splicing and fusion module is connected to the input end of the fourth fully connected layer; the output end of the fourth fully connected layer is connected to the input end of the first Sigmoid activation function module; the output end of the first Sigmoid activation function module is connected to the input end of the first Hada code inner product module;

[0032] The second fusion submodule includes: a fifth PReLU activation function module, a fifth normalization module, a fifth fully connected layer, a sixth PReLU activation function module, a sixth normalization module, a third splicing and fusion module, a sixth fully connected layer, a seventh PReLU activation function module, a seventh fully connected layer, an eighth PReLU activation function module, a seventh normalization module, a second LSTM network module, an eighth normalization module, a fourth splicing and fusion module, an eighth fully connected layer, a second Sigmoid activation function module, and a second Hadamard inner product module;

[0033] The output end of the fifth PReLU activation function module is connected to the input end of the fifth normalization module; the output end of the fifth normalization module is connected to the input end of the fifth fully connected layer; the output end of the fifth fully connected layer is connected to the input end of the sixth PReLU activation function module; the output end of the sixth PReLU activation function module is connected to the input end of the sixth normalization module; the output end of the sixth normalization module is connected to the input end of the third splicing and fusion module; the output end of the third splicing and fusion module is connected to the input end of the sixth fully connected layer; the output end of the sixth fully connected layer is connected to the input end of the seventh PReLU activation function module; the output end of the seventh PReLU activation function module is connected to the input end of the seventh fully connected layer; The output end of the fully connected layer is connected to the input end of the eighth PReLU activation function module; the output end of the eighth PReLU activation function module is connected to the input end of the seventh normalization module; the output end of the seventh normalization module is connected to the input end of the second LSTM network module; the output end of the second LSTM network module is connected to the input end of the eighth normalization module; the output end of the eighth normalization module is connected to the input end of the fourth splicing and fusion module; the output end of the fourth splicing and fusion module is connected to the input end of the eighth fully connected layer; the output end of the eighth fully connected layer is connected to the input end of the second Sigmoid activation function module; the output end of the second Sigmoid activation function module is connected to the input end of the second Hada code inner product module;

[0034] The third fusion submodule includes: a ninth fully connected layer, a ninth PReLU activation function module, a ninth normalization module, a tenth fully connected layer, a tenth PReLU activation function module, an eleventh fully connected layer, an eleventh PReLU activation function module, a tenth normalization module, a third LSTM network module, and an eleventh normalization module;

[0035] The output end of the ninth fully connected layer is connected to the input end of the ninth PReLU activation function module; the output end of the ninth PReLU activation function module is connected to the input end of the ninth normalization module; the output end of the ninth normalization module is connected to the input end of the tenth fully connected layer; the output end of the tenth fully connected layer is connected to the input end of the tenth PReLU activation function module; the output end of the tenth PReLU activation function module is connected to the input end of the eleventh fully connected layer; the output end of the eleventh fully connected layer is connected to the input end of the eleventh PReLU activation function module; the output end of the eleventh PReLU activation function module is connected to the input end of the tenth normalization module; the output end of the tenth normalization module is connected to the input end of the third LSTM network module; the output end of the third LSTM network module is connected to the input end of the eleventh normalization module;

[0036] The output end of the first fusion output module is connected to the input end of the first PReLU activation function module and the input end of the first Hadamard code inner product module respectively;

[0037] The output end of the second fusion output module is connected to the input end of the fifth PReLU activation function module and the input end of the second Hada code inner product module respectively;

[0038] The output end of the ninth normalization module is connected to the input end of the first splicing and fusion module and the input end of the third splicing and fusion module respectively;

[0039] The output end of the eleventh normalization module is connected to the input end of the second splicing and fusion module and the input end of the fourth splicing and fusion module respectively;

[0040] The output end of the first Hada code inner product module and the output end of the second Hada code inner product module are respectively connected to the input end of the fifth splicing and fusion module;

[0041] The output end of the fifth splicing and fusion module is connected to the input end of the first-dimensional deconvolution network module;

[0042] The first PReLU activation function module is used to input the fused air conduction signal; the input end of the first PReLU activation function module is connected to the output end of the first fusion output module;

[0043] The fifth PReLU activation function module is used to input the fused bone conduction signal; the input end of the fifth PReLU activation function module is connected to the output end of the second fusion output module;

[0044] The ninth fully connected layer is used to input encoding information of video data; the input end of the ninth fully connected layer is connected to the output end of the video encoding submodule;

[0045] The one-dimensional deconvolution network module is used to output enhanced speech information.

[0046] The method of using the speech enhancement training data set to train a preset speech enhancement model to obtain a trained speech enhancement model includes:

[0047] Obtain category information of the speech enhancement training data set; when the category information is speech enhancement data, determine the learning rate value of the second self-attention module to be α, and determine the learning rate value of the second multi-head attention module to be Determine the learning rate value of the third LSTM network module to be α, where α≤0.1,

[0048] When the category information is audio-visual data, the learning rate value of the first self-attention module is determined to be α, and the learning rate value of the first multi-head attention module is determined to be

[0049] When the category information is air-bone data, determining that the learning rate value of the third LSTM network module is α;

[0050] Initialize the number of training iterations;

[0051] Inputting the data sequence of the speech enhancement training data set as input data into the speech enhancement model;

[0052] Using the speech enhancement model, the input data is processed to obtain enhanced speech information;

[0053] Performing difference calculation processing on the obtained enhanced speech information and the label information corresponding to the input data to obtain a difference value;

[0054] Determine whether the difference value satisfies a convergence condition, and obtain a first determination result;

[0055] When the first judgment result is no, judging whether the training iteration number value is equal to the training number threshold, and obtaining a second judgment result;

[0056] When the second judgment result is no, determining that the model training state does not meet the training termination condition;

[0057] When the second judgment result is yes, determining that the model training state satisfies the training termination condition;

[0058] When the first judgment result is yes, determining that the model training state satisfies a training termination condition;

[0059] When the model training state does not meet the termination training condition, the parameters of the air-bone fusion module and the comprehensive fusion module are updated using the parameter update model, the training iteration number value is increased by 1, and the execution is triggered to input the data sequence of the speech enhancement training data set as input data into the speech enhancement model;

[0060] When the model training status satisfies the training termination condition, the training process of the speech enhancement model is completed to obtain a trained speech enhancement model.

[0061] The parameter updating model is:

[0062]

[0063] θ←θ+v;

[0064] In the formula, x (l) is the label information of the i-th input data, is the loss function, v is the parameter update value, θ is the parameter of the air-bone fusion module and the comprehensive fusion module, η is the initial parameter learning rate, α is the momentum angle parameter, 0≤α≤1, It means to find the partial derivative of variable θ, f(x (l) ; θ) represents the enhanced speech information calculated by the speech enhancement model for the i-th input data, f(·) is the calculation function corresponding to the speech enhancement model; exp represents the power operation of the constant e; η and α are preset values.

[0065] In a second aspect of an embodiment of the present invention, a speech enhancement device based on visual and auditory bone fusion is disclosed, the device comprising:

[0066] A memory storing executable program code;

[0067] a processor coupled to the memory;

[0068] The processor calls the executable program code stored in the memory to execute the speech enhancement method based on audiovisual bone fusion.

[0069] According to a third aspect of an embodiment of the present invention, a computer storable medium is disclosed, wherein the computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the speech enhancement method based on visual and audio-bone fusion.

[0070] According to a fourth aspect of the embodiments of the present invention, an information data processing terminal is disclosed, wherein the information data processing terminal is used to implement the speech enhancement method based on visual and audio bone fusion.

[0071] The beneficial effects of the present invention are:

[0072] The present invention achieves the enhancement of the original speaker's voice quality in a high-noise environment, and is applicable to the environments used by all original speaker audio-visual speech enhancement tasks or air-bone speech enhancement environment tasks;

[0073] The present invention realizes speech enhancement in special environments such as when whispering and keeping quiet is required, so that good quality gaseous speech can be obtained in special environments, and instructions / commands can be quickly and unimpededly transmitted to other people / devices.

[0074] The present invention also realizes speech enhancement in complex environments and can enhance the speech of a specific speaker in complex environments.

[0075] The present invention effectively reduces the complexity of fusion through a three-stage fusion process and realizes the full integration of information from different modalities. The present invention can effectively utilize tri-modal information to cope with the challenge of speech enhancement in high-noise environments, and achieves remarkable results in specific speaker enhancement and speaker separation. In addition, the present invention also overcomes the interference caused by changes in illumination and human head posture movement in audio-visual speech enhancement, as well as the challenge of missing high-frequency information in air-bone speech enhancement, further optimizing the stability and accuracy of speech enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a schematic diagram of the speech enhancement process of the three-stage fusion of audiovisual and bone according to the present invention;

[0077] Figure 2 It is a structural diagram of the air-bone fusion module of the present invention;

[0078] Figure 3 It is a structural diagram of the comprehensive fusion module of the present invention;

[0079] Figure 4 Schematic diagram of the model training process of the present invention. DETAILED DESCRIPTION

[0080] In order to better understand the content of the present invention, an embodiment is given here.

[0081] Figure 1 It is a schematic diagram of the speech enhancement process of the three-stage fusion of audiovisual and bone according to the present invention; Figure 2 It is a structural diagram of the air-bone fusion module of the present invention; Figure 3 It is a structural diagram of the comprehensive fusion module of the present invention; Figure 4 Schematic diagram of the model training process of the present invention.

[0082] A first aspect of an embodiment of the present invention discloses a speech enhancement method based on visual and auditory bone fusion, comprising:

[0083] S1, obtain the speech enhancement training data set;

[0084] S2, using the speech enhancement training data set to train a preset speech enhancement model to obtain a trained speech enhancement model;

[0085] S3, acquiring a multimodal data set of the user in a speaking state; the multimodal data set includes video data, air conduction data, and bone conduction data; the multimodal data set is acquired based on the same time reference;

[0086] S4, using the trained speech enhancement model, processing the multimodal data set to obtain enhanced speech information; the enhanced speech information represents the user's speech information after speech enhancement processing.

[0087] The speech enhancement training data set includes a speech enhancement data set, an audio-visual data set, and an air-bone data set; the speech enhancement data set includes an air conduction data sequence and corresponding label information; the audio-visual data set includes a video data sequence and corresponding label information, an air conduction data sequence and corresponding label information; the air-bone data set includes an air conduction data sequence and corresponding label information, a bone conduction data sequence and corresponding label information;

[0088] Category information of the speech enhancement training data set, including speech enhancement data, audio-visual data, and air-bone data;

[0089] The speech enhancement dataset may be the AISHELL-3 dataset; the audio-visual dataset may be the CN-CVS dataset; and the air-bone dataset may be the ESMB (Elevoc Simultaneously-recorded Microphone / Bone-sensor) speech corpus dataset.

[0090] The speech enhancement model includes: an encoding module, an air-bone fusion module and a comprehensive fusion module;

[0091] The encoding module is used to encode the air conduction data, the bone conduction data and the video data to obtain encoding information;

[0092] The air-bone fusion module is connected to the encoding module and the comprehensive fusion module respectively, and is used to fuse the encoding information of the air conduction data and the encoding information of the bone conduction data to obtain air-bone fusion information;

[0093] The comprehensive fusion module is used to fuse the air-bone fusion information and the encoding information of the video data to obtain enhanced voice information;

[0094] The encoding module includes an air conduction encoding submodule, a bone conduction encoding submodule, and a video encoding submodule;

[0095] The video encoding submodule is implemented using the ShuffleNetV2 network;

[0096] The air conduction coding submodule and the bone conduction coding submodule are both implemented by using a one-dimensional convolutional neural network;

[0097] The expression of the encoding process is:

[0098] Encoder(v)=ShuffleNetV2(visual,w v ),

[0099] Encoder(a)=Conv1d(audio,w a ),

[0100] Encoder(b)=Conv1d(bone,w b ),

[0101] Among them, w v Represents the weight vector of the ShuffleNetV2 network, w a represents the weight vector of the one-dimensional convolutional neural network for air conduction coding, w b represents the weight vector of the one-dimensional convolutional neural network used for bone conduction coding, v is the abbreviation of visual, a is the abbreviation of audio, b is the abbreviation of bone, Encoder(v), Encoder(a) and Encoder(b) represent the coding results of v, a and b respectively;

[0102] The calculation process of the air-bone fusion module is expressed as follows:

[0103] Self(a)=SelfAttention(query Encoder(a) ,key Encoder(a) ,value Encoder(a) )+Encoder(a),

[0104] Self(b)=SelfAttention(query Encoder(b) ,key Encoder(b) ,value Encoder(b) )+Encoder(b)

[0105] EarlyFusion(a)=MultiHeadAttention(query self(a) ,key self(b) ,value self(b) )+Self(a)

[0106] EarlyFusion(b)=MultiHeadAttention(query self(b) ,key self(a) ,value self(a) )+Self(b),

[0107] Among them, query Encoder(a) 、key Encoder(a) 、value Encoder(a) Respectively represent the Q variable, K variable and V variable obtained after the encoding information output by the air conduction encoding submodule is self-attention encoded;

[0108] SelfAttention(query Encoder(a) ,key Encoder(a) ,value Encoder(a) ) is the output of the self-attention module and the output of the autonomous module, and we can express it item by item to get:

[0109] query self(a) 、key Encoder(a) value Encoder(a) ;

[0110] The air-bone fusion module includes a first input module, a first self-attention module, a first multi-head attention module, a first addition module, a first fusion output module, a second input module, a second self-attention module, a second multi-head attention module, a second addition module, and a second fusion output module;

[0111] The first fusion output module is used to output a fusion air conduction signal;

[0112] The second fusion output module is used to output a fusion bone conduction signal;

[0113] The first input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the air conduction coding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the first input module is connected to the input port of the first self-attention module; the second output port of the first input module is connected to the second input port of the first addition module;

[0114] The first self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the coding information output by the air conduction coding submodule is self-attention-coded; the second output port is used to output the K variable after the coding information output by the air conduction coding submodule is self-attention-coded; the third output port is used to output the V variable after the coding information output by the air conduction coding submodule is self-attention-coded; the first output port of the first self-attention module is connected to the first input port of the first multi-head attention module; the second output port of the first self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the first self-attention module is connected to the third input port of the second multi-head attention module;

[0115] The first multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the first multi-head attention module is connected to the first input port of the first addition module;

[0116] The first addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the first fusion output module;

[0117] The second input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the bone conduction coding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the second input module is connected to the input port of the second self-attention module; the second output port of the second input module is connected to the second input port of the first addition module;

[0118] The second self-attention module comprises an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable after the coding information output by the bone conduction coding submodule is self-attention-coded; the second output port is used to output the K variable after the coding information output by the bone conduction coding submodule is self-attention-coded; the third output port is used to output the V variable after the coding information output by the bone conduction coding submodule is self-attention-coded; the first output port of the second self-attention module is connected to the first input port of the first multi-head attention module; the second output port of the second self-attention module is connected to the second input port of the first multi-head attention module; the third output port of the second self-attention module is connected to the third input port of the first multi-head attention module;

[0119] The second multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the second multi-head attention module is connected to the first input port of the second addition module;

[0120] The second addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the second fusion output module;

[0121] The comprehensive fusion module includes a first fusion submodule, a second fusion submodule, a third fusion submodule, a fifth splicing fusion module, and a one-dimensional deconvolution network module;

[0122] The first fusion submodule includes: a first PReLU activation function module, a first normalization module, a first fully connected layer, a second PReLU activation function module, a second normalization module, a first splicing and fusion module, a second fully connected layer, a third PReLU activation function module, a third fully connected layer, a fourth PReLU activation function module, a third normalization module, a first LSTM network module, a fourth normalization module, a second splicing and fusion module, a fourth fully connected layer, a first Sigmoid activation function module, and a first Hadamard inner product module;

[0123] The output end of the first PReLU activation function module is connected to the input end of the first normalization module; the output end of the first normalization module is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the second PReLU activation function module; the output end of the second PReLU activation function module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the first splicing and fusion module; the output end of the first splicing and fusion module is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the third PReLU activation function module; the output end of the third PReLU activation function module is connected to the input end of the third fully connected layer; The output end of the fully connected layer is connected to the input end of the fourth PReLU activation function module; the output end of the fourth PReLU activation function module is connected to the input end of the third normalization module; the output end of the third normalization module is connected to the input end of the first LSTM network module; the output end of the first LSTM network module is connected to the input end of the fourth normalization module; the output end of the fourth normalization module is connected to the input end of the second splicing and fusion module; the output end of the second splicing and fusion module is connected to the input end of the fourth fully connected layer; the output end of the fourth fully connected layer is connected to the input end of the first Sigmoid activation function module; the output end of the first Sigmoid activation function module is connected to the input end of the first Hada code inner product module;

[0124] The second fusion submodule includes: a fifth PReLU activation function module, a fifth normalization module, a fifth fully connected layer, a sixth PReLU activation function module, a sixth normalization module, a third splicing and fusion module, a sixth fully connected layer, a seventh PReLU activation function module, a seventh fully connected layer, an eighth PReLU activation function module, a seventh normalization module, a second LSTM network module, an eighth normalization module, a fourth splicing and fusion module, an eighth fully connected layer, a second Sigmoid activation function module, and a second Hadamard inner product module;

[0125] The output end of the fifth PReLU activation function module is connected to the input end of the fifth normalization module; the output end of the fifth normalization module is connected to the input end of the fifth fully connected layer; the output end of the fifth fully connected layer is connected to the input end of the sixth PReLU activation function module; the output end of the sixth PReLU activation function module is connected to the input end of the sixth normalization module; the output end of the sixth normalization module is connected to the input end of the third splicing and fusion module; the output end of the third splicing and fusion module is connected to the input end of the sixth fully connected layer; the output end of the sixth fully connected layer is connected to the input end of the seventh PReLU activation function module; the output end of the seventh PReLU activation function module is connected to the input end of the seventh fully connected layer; The output end of the fully connected layer is connected to the input end of the eighth PReLU activation function module; the output end of the eighth PReLU activation function module is connected to the input end of the seventh normalization module; the output end of the seventh normalization module is connected to the input end of the second LSTM network module; the output end of the second LSTM network module is connected to the input end of the eighth normalization module; the output end of the eighth normalization module is connected to the input end of the fourth splicing and fusion module; the output end of the fourth splicing and fusion module is connected to the input end of the eighth fully connected layer; the output end of the eighth fully connected layer is connected to the input end of the second Sigmoid activation function module; the output end of the second Sigmoid activation function module is connected to the input end of the second Hada code inner product module;

[0126] The third fusion submodule includes: a ninth fully connected layer, a ninth PReLU activation function module, a ninth normalization module, a tenth fully connected layer, a tenth PReLU activation function module, an eleventh fully connected layer, an eleventh PReLU activation function module, a tenth normalization module, a third LSTM network module, and an eleventh normalization module;

[0127] The output end of the ninth fully connected layer is connected to the input end of the ninth PReLU activation function module; the output end of the ninth PReLU activation function module is connected to the input end of the ninth normalization module; the output end of the ninth normalization module is connected to the input end of the tenth fully connected layer; the output end of the tenth fully connected layer is connected to the input end of the tenth PReLU activation function module; the output end of the tenth PReLU activation function module is connected to the input end of the eleventh fully connected layer; the output end of the eleventh fully connected layer is connected to the input end of the eleventh PReLU activation function module; the output end of the eleventh PReLU activation function module is connected to the input end of the tenth normalization module; the output end of the tenth normalization module is connected to the input end of the third LSTM network module; the output end of the third LSTM network module is connected to the input end of the eleventh normalization module;

[0128] The output end of the first fusion output module is connected to the input end of the first PReLU activation function module and the input end of the first Hadamard code inner product module respectively;

[0129] The output end of the second fusion output module is connected to the input end of the fifth PReLU activation function module and the input end of the second Hada code inner product module respectively;

[0130] The output end of the ninth normalization module is connected to the input end of the first splicing and fusion module and the input end of the third splicing and fusion module respectively;

[0131] The output end of the eleventh normalization module is connected to the input end of the second splicing and fusion module and the input end of the fourth splicing and fusion module respectively;

[0132] The output end of the first Hada code inner product module and the output end of the second Hada code inner product module are respectively connected to the input end of the fifth splicing and fusion module;

[0133] The output end of the fifth splicing and fusion module is connected to the input end of the first-dimensional deconvolution network module;

[0134] The first PReLU activation function module is used to input a fused air conduction signal;

[0135] The fifth PReLU activation function module is used to input the fusion bone conduction signal;

[0136] The ninth fully connected layer is used to input encoding information of video data;

[0137] The one-dimensional deconvolution network module is used to output enhanced speech information.

[0138] The speech enhancement training data set includes a speech enhancement data subset, an audio-visual data set, and an air-bone data set; the speech enhancement data set includes an air conduction data sequence and corresponding label information; the audio-visual data set includes a video data sequence and corresponding label information, an air conduction data sequence and corresponding label information; the air-bone data set includes an air conduction data sequence and corresponding label information, a bone conduction data sequence and corresponding label information;

[0139] Category information of the speech enhancement training data set, including speech enhancement data, audio-visual data, and air-bone data;

[0140] The method of using the speech enhancement training data set to train a preset speech enhancement model to obtain a trained speech enhancement model includes:

[0141] Obtain category information of the speech enhancement training data set; when the category information is speech enhancement data, determine the learning rate value of the second self-attention module to be α, and determine the learning rate value of the second multi-head attention module to be Determine the learning rate value of the third LSTM network module to be α, where α≤0.1,

[0142] When the category information is audio-visual data, the learning rate value of the first self-attention module is determined to be α, and the learning rate value of the first multi-head attention module is determined to be

[0143] When the category information is air-bone data, determining that the learning rate value of the third LSTM network module is α;

[0144] Initialize the number of training iterations;

[0145] Inputting the data sequence of the speech enhancement training data set as input data into the speech enhancement model;

[0146] Using the speech enhancement model, the input data is processed to obtain enhanced speech information;

[0147] Performing difference calculation processing on the obtained enhanced speech information and the label information corresponding to the input data to obtain a difference value;

[0148] Determine whether the difference value satisfies a convergence condition, and obtain a first determination result;

[0149] When the first judgment result is no, judging whether the training iteration number value is equal to the training number threshold, and obtaining a second judgment result;

[0150] When the second judgment result is no, determining that the model training state does not meet the training termination condition;

[0151] When the second judgment result is yes, determining that the model training state satisfies the training termination condition;

[0152] When the first judgment result is yes, determining that the model training state satisfies a training termination condition;

[0153] When the model training state does not meet the termination training condition, the parameters of the air-bone fusion module and the comprehensive fusion module are updated using the parameter update model, the training iteration number value is increased by 1, and the execution is triggered to input the data sequence of the speech enhancement training data set as input data into the speech enhancement model;

[0154] When the model training status satisfies the training termination condition, the training process of the speech enhancement model is completed to obtain a trained speech enhancement model.

[0155] The difference value satisfies the convergence condition, which means that the difference value is less than a preset convergence threshold; the difference value does not satisfy the convergence condition, which means that the difference value is not less than the preset convergence threshold.

[0156] The parameter updating model is:

[0157]

[0158] θ←θ+v;

[0159] In the formula, x (l) is the label information of the i-th input data, is the loss function, v is the parameter update value, θ is the parameter of the air-bone fusion module and the comprehensive fusion module, η is the initial parameter learning rate, α is the momentum angle parameter, 0≤α≤1, It means to find the partial derivative of variable θ, f(x (i) ; θ) represents the enhanced speech information calculated by the speech enhancement model for the i-th input data, f(·) is the calculation function corresponding to the speech enhancement model; exp represents the power operation of the constant e; η and α are preset values.

[0160] The method of processing the multimodal data set using the trained speech enhancement model to obtain enhanced speech information includes:

[0161] The multimodal data set includes a plurality of frames of multimodal data;

[0162] Acquire current frame data of the multimodal data set;

[0163] Using the trained speech enhancement model, the multimodal data of the current frame is processed to obtain prediction result data of N subsequent frames of the current frame;

[0164] The trained speech enhancement model processes the multimodal data of the current frame and the previous N frames to obtain N+1 frames of enhanced speech information;

[0165] A circulant matrix is ​​constructed using enhanced speech information of a current frame and its previous N frames and subsequent N frames; the i-th row of the circulant matrix is ​​N+1 frame enhanced speech information obtained by multimodal data of the subsequent i-1 frame of the current frame and the N frames preceding the subsequent i-1 frame;

[0166] Extract the sub-diagonal vector of the circulant matrix;

[0167] The sampling sequence of the normalized Gaussian distribution function is used to perform inner product operation with the sub-diagonal vector to obtain enhanced speech information of the current frame data.

[0168] Taking the current frame as the 10th frame as an example, the expression of the constructed circulant matrix is:

[0169]

[0170] The numbers in the matrix represent the prediction result data of the sequence frame.

[0171] Multimodal fusion is a method with far-reaching impact. By comprehensively utilizing information from different sources, multimodal fusion can provide a more comprehensive data view than a single modality, thereby increasing the accuracy and reliability of the system's decision-making. Different types of data complement each other's shortcomings, reducing the deviation or error that may be introduced by relying on a single data source. In addition, when one modality is disturbed or fails, the multimodal system can rely on other modalities to continue to operate normally, ensuring the continuity and robustness of the system. With the improvement of embedded chip capabilities, it has become possible to synchronously collect audio-visual bone data on wearable devices and perform fusion processing. The information of the audio-visual modality can fill the missing high-frequency information in the bone conduction modality, while the air-bone modality effectively solves the similarity problem of different phonemes in the video. By utilizing the advantages and disadvantages of the three modalities, high-quality speech enhancement can be achieved in complex environments.

[0172] In a second aspect of an embodiment of the present invention, a speech enhancement device based on visual and auditory bone fusion is disclosed, the device comprising:

[0173] A memory storing executable program code;

[0174] a processor coupled to the memory;

[0175] The processor calls the executable program code stored in the memory to execute the speech enhancement method based on audiovisual bone fusion.

[0176] According to a third aspect of an embodiment of the present invention, a computer storable medium is disclosed, wherein the computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the speech enhancement method based on visual and audio-bone fusion.

[0177] According to a fourth aspect of the embodiments of the present invention, an information data processing terminal is disclosed, wherein the information data processing terminal is used to implement the speech enhancement method based on visual and audio bone fusion.

[0178] To this end, the present invention innovatively proposes a speech enhancement method for the fusion of visual, audio and bone trimodal. In addition, considering the difficulty of fusion caused by the heterogeneity between modalities, the present invention designs a three-stage fusion method. In the first stage, air-bone fusion is performed. Because the air-bone modalities are closer in nature than the visual information, a fusion method based on the cross-attention mechanism is designed. Through the allocation of weights and the weighting of features, air-conduction features and bone-conduction features with each other's characteristics are obtained respectively; in the second stage, the visual information is introduced. In order to fully realize the heterogeneous fusion of visual signals and speech signals, after obtaining the visual modality embedding vector, two branches are designed to realize multi-layer fusion with bone-conduction features and air-conduction features respectively, and air-conduction or bone-conduction features are densely introduced in each fusion layer, so as to achieve full fusion; in the third stage, the visual-air features, visual-bone features fused with the fused visual information, and the air-bone features obtained in the first stage are further fused through the weight allocation of the gated network to obtain enhanced speech. Through this three-stage fusion process, the present technology effectively reduces the complexity of fusion and realizes the full integration of different modal information. Through the above technical improvements, the present invention can effectively utilize the information of the three modes to cope with the challenge of speech enhancement in high-noise environments, and achieve remarkable results in specific speaker enhancement and speaker separation. In addition, the present invention also overcomes the interference caused by illumination changes and human head posture movement in audio-visual speech enhancement, as well as the challenge of lack of high-frequency information in air-bone speech enhancement, further optimizing the stability and accuracy of speech enhancement.

[0179] In summary, the present invention proposes an innovative three-stage three-modal fusion speech enhancement method by comprehensively utilizing the three modal information and overcoming the fusion difficulty caused by modal heterogeneity, which provides an effective solution for achieving high-quality speech enhancement.

[0180] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A speech enhancement method based on audio-visual bone fusion, characterized in that: include: S1, obtain the speech enhancement training data set; S2, using the speech enhancement training data set to train a preset speech enhancement model to obtain a trained speech enhancement model; S3, collecting a multimodal data set of the user in a speaking state; The multimodal data set includes video data, air conduction data and bone conduction data; The multimodal data set is collected based on the same time reference; S4, using the trained speech enhancement model to process the multimodal data set to obtain enhanced speech information; The enhanced voice information represents the user's speech voice information after voice enhancement processing.

2. The method for speech enhancement based on visual and auditory bone fusion as claimed in claim 1, characterized in that: The speech enhancement training data set includes a speech enhancement data set, an audio-visual data set, and a gas-bone data set; The speech enhancement data set includes an air conduction data sequence and corresponding label information; the audio-visual data set includes a video data sequence and corresponding label information, an air conduction data sequence and corresponding label information; The air-bone data set includes an air conduction data sequence and corresponding label information, a bone conduction data sequence and corresponding label information; The category information of the speech enhancement training data set includes speech enhancement data, audio-visual data, and air-bone data.

3. The speech enhancement method based on visual and auditory bone fusion as claimed in claim 1, characterized in that: The speech enhancement model includes: an encoding module, an air-bone fusion module and a comprehensive fusion module; The encoding module is used to encode the air conduction data, the bone conduction data and the video data to obtain encoding information; The air-bone fusion module is connected to the encoding module and the comprehensive fusion module respectively, and is used to fuse the encoding information of the air conduction data and the encoding information of the bone conduction data to obtain air-bone fusion information; The comprehensive fusion module is used to fuse the air-bone fusion information and the encoding information of the video data to obtain enhanced voice information; The encoding module includes an air conduction encoding submodule, a bone conduction encoding submodule, and a video encoding submodule.

4. The method for speech enhancement based on visual and auditory bone fusion as claimed in claim 3, characterized in that: The air-bone fusion module includes a first input module, a first self-attention module, a first multi-head attention module, a first addition module, a first fusion output module, a second input module, a second self-attention module, a second multi-head attention module, a second addition module, and a second fusion output module; The first fusion output module is used to output a fusion air conduction signal; The second fusion output module is used to output a fusion bone conduction signal; The first input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the air conduction coding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the first input module is connected to the input port of the first self-attention module; the second output port of the first input module is connected to the second input port of the first addition module; The first self-attention module includes an input port, a first output port, a second output port, and a third output port; the first output port is used to output the Q variable obtained by self-attention encoding the coded information output by the air conduction coding submodule; the second output port is used to output the K variable obtained by self-attention encoding the coded information output by the air conduction coding submodule; the third output port is used to output the V variable obtained by self-attention encoding the coded information output by the air conduction coding submodule; the first output port of the first self-attention module is connected to the first input port of the first multi-head attention module; the second output port of the first self-attention module is connected to the second input port of the second multi-head attention module; the third output port of the first self-attention module is connected to the third input port of the second multi-head attention module; The first multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the first multi-head attention module is connected to the first input port of the first addition module; The first addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the first fusion output module; The second input module includes an input port, a first output port, and a second output port, and is used to receive the coded information output by the bone conduction coding submodule, and output the coded information through the first output port and the second output port respectively; the first output port of the second input module is connected to the input port of the second self-attention module; the second output port of the second input module is connected to the second input port of the first addition module; The second self-attention module comprises an input port, a first output port, a second output port, and a third output port; the first output port is used to output a Q variable obtained by self-attention encoding the coded information output by the bone conduction coding submodule; the second output port is used to output a K variable obtained by self-attention encoding the coded information output by the bone conduction coding submodule; the third output port is used to output a V variable obtained by self-attention encoding the coded information output by the bone conduction coding submodule; the first output port of the second self-attention module is connected to the first input port of the second multi-head attention module; the second output port of the second self-attention module is connected to the second input port of the first multi-head attention module; the third output port of the second self-attention module is connected to the third input port of the first multi-head attention module; The second multi-head attention module comprises a first input port, a second input port, a third input port and an output port; the output port of the second multi-head attention module is connected to the first input port of the second addition module; The second addition module is used to sum the input variable of the first input port and the input variable of the second input port to obtain a sum variable, and send the sum variable to the second fusion output module.

5. The method for speech enhancement based on visual and auditory bone fusion as claimed in claim 3, characterized in that: The comprehensive fusion module includes a first fusion submodule, a second fusion submodule, a third fusion submodule, a fifth splicing fusion module, and a one-dimensional deconvolution network module; The first fusion submodule includes: a first PReLU activation function module, a first normalization module, a first fully connected layer, a second PReLU activation function module, a second normalization module, a first splicing and fusion module, a second fully connected layer, a third PReLU activation function module, a third fully connected layer, a fourth PReLU activation function module, a third normalization module, a first LSTM network module, a fourth normalization module, a second splicing and fusion module, a fourth fully connected layer, a first Sigmoid activation function module, and a first Hadamard inner product module; The output end of the first PReLU activation function module is connected to the input end of the first normalization module; the output end of the first normalization module is connected to the input end of the first fully connected layer; the output end of the first fully connected layer is connected to the input end of the second PReLU activation function module; the output end of the second PReLU activation function module is connected to the input end of the second normalization module; the output end of the second normalization module is connected to the input end of the first splicing and fusion module; the output end of the first splicing and fusion module is connected to the input end of the second fully connected layer; the output end of the second fully connected layer is connected to the input end of the third PReLU activation function module; the output end of the third PReLU activation function module is connected to the input end of the third fully connected layer; The output end of the fully connected layer is connected to the input end of the fourth PReLU activation function module; the output end of the fourth PReLU activation function module is connected to the input end of the third normalization module; the output end of the third normalization module is connected to the input end of the first LSTM network module; the output end of the first LSTM network module is connected to the input end of the fourth normalization module; the output end of the fourth normalization module is connected to the input end of the second splicing and fusion module; the output end of the second splicing and fusion module is connected to the input end of the fourth fully connected layer; the output end of the fourth fully connected layer is connected to the input end of the first Sigmoid activation function module; the output end of the first Sigmoid activation function module is connected to the input end of the first Hada code inner product module; The second fusion submodule includes: a fifth PReLU activation function module, a fifth normalization module, a fifth fully connected layer, a sixth PReLU activation function module, a sixth normalization module, a third splicing and fusion module, a sixth fully connected layer, a seventh PReLU activation function module, a seventh fully connected layer, an eighth PReLU activation function module, a seventh normalization module, a second LSTM network module, an eighth normalization module, a fourth splicing and fusion module, an eighth fully connected layer, a second Sigmoid activation function module, and a second Hadamard inner product module; The output end of the fifth PReLU activation function module is connected to the input end of the fifth normalization module; the output end of the fifth normalization module is connected to the input end of the fifth fully connected layer; the output end of the fifth fully connected layer is connected to the input end of the sixth PReLU activation function module; the output end of the sixth PReLU activation function module is connected to the input end of the sixth normalization module; the output end of the sixth normalization module is connected to the input end of the third splicing and fusion module; the output end of the third splicing and fusion module is connected to the input end of the sixth fully connected layer; the output end of the sixth fully connected layer is connected to the input end of the seventh PReLU activation function module; the output end of the seventh PReLU activation function module is connected to the input end of the seventh fully connected layer; The output end of the fully connected layer is connected to the input end of the eighth PReLU activation function module; the output end of the eighth PReLU activation function module is connected to the input end of the seventh normalization module; the output end of the seventh normalization module is connected to the input end of the second LSTM network module; the output end of the second LSTM network module is connected to the input end of the eighth normalization module; the output end of the eighth normalization module is connected to the input end of the fourth splicing and fusion module; the output end of the fourth splicing and fusion module is connected to the input end of the eighth fully connected layer; the output end of the eighth fully connected layer is connected to the input end of the second Sigmoid activation function module; the output end of the second Sigmoid activation function module is connected to the input end of the second Hada code inner product module; The third fusion submodule includes: a ninth fully connected layer, a ninth PReLU activation function module, a ninth normalization module, a tenth fully connected layer, a tenth PReLU activation function module, an eleventh fully connected layer, an eleventh PReLU activation function module, a tenth normalization module, a third LSTM network module, and an eleventh normalization module; The output end of the ninth fully connected layer is connected to the input end of the ninth PReLU activation function module; the output end of the ninth PReLU activation function module is connected to the input end of the ninth normalization module; the output end of the ninth normalization module is connected to the input end of the tenth fully connected layer; the output end of the tenth fully connected layer is connected to the input end of the tenth PReLU activation function module; the output end of the tenth PReLU activation function module is connected to the input end of the eleventh fully connected layer; the output end of the eleventh fully connected layer is connected to the input end of the eleventh PReLU activation function module; the output end of the eleventh PReLU activation function module is connected to the input end of the tenth normalization module; the output end of the tenth normalization module is connected to the input end of the third LSTM network module; the output end of the third LSTM network module is connected to the input end of the eleventh normalization module; The output end of the first fusion output module is connected to the input end of the first PReLU activation function module and the input end of the first Hadamard code inner product module respectively; The output end of the second fusion output module is connected to the input end of the fifth PReLU activation function module and the input end of the second Hada code inner product module respectively; The output end of the ninth normalization module is connected to the input end of the first splicing and fusion module and the input end of the third splicing and fusion module respectively; The output end of the eleventh normalization module is connected to the input end of the second splicing and fusion module and the input end of the fourth splicing and fusion module respectively; The output end of the first Hada code inner product module and the output end of the second Hada code inner product module are respectively connected to the input end of the fifth splicing and fusion module; The output end of the fifth splicing and fusion module is connected to the input end of the first-dimensional deconvolution network module; The first PReLU activation function module is used to input the fused air conduction signal; the input end of the first PReLU activation function module is connected to the output end of the first fusion output module; The fifth PReLU activation function module is used to input the fused bone conduction signal; the input end of the fifth PReLU activation function module is connected to the output end of the second fusion output module; The ninth fully connected layer is used to input encoding information of video data; the input end of the ninth fully connected layer is connected to the output end of the video encoding submodule; The one-dimensional deconvolution network module is used to output enhanced speech information.

6. The method for speech enhancement based on visual and auditory bone fusion as claimed in claim 3, characterized in that: The method of using the speech enhancement training data set to train a preset speech enhancement model to obtain a trained speech enhancement model includes: Obtain category information of the speech enhancement training data set; when the category information is speech enhancement data, determine the learning rate value of the second self-attention module to be α, and determine the learning rate value of the second multi-head attention module to be Determine the learning rate value of the third LSTM network module to be α, where When the category information is audio-visual data, the learning rate value of the first self-attention module is determined to be α, and the learning rate value of the first multi-head attention module is determined to be When the category information is air-bone data, determining that the learning rate value of the third LSTM network module is α; Initialize the number of training iterations; Inputting the data sequence of the speech enhancement training data set as input data into the speech enhancement model; Using the speech enhancement model, the input data is processed to obtain enhanced speech information; Performing difference calculation processing on the obtained enhanced speech information and the label information corresponding to the input data to obtain a difference value; Determine whether the difference value satisfies a convergence condition, and obtain a first determination result; When the first judgment result is no, judging whether the training iteration number value is equal to the training number threshold, and obtaining a second judgment result; When the second judgment result is no, determining that the model training state does not meet the training termination condition; When the second judgment result is yes, determining that the model training state satisfies the training termination condition; When the first judgment result is yes, determining that the model training state satisfies a training termination condition; When the model training state does not meet the termination training condition, the parameters of the air-bone fusion module and the comprehensive fusion module are updated using the parameter update model, the training iteration number value is increased by 1, and the execution is triggered to input the data sequence of the speech enhancement training data set as input data into the speech enhancement model; When the model training state satisfies the training termination condition, the training process of the speech enhancement model is completed to obtain a trained speech enhancement model.

7. The method for speech enhancement based on visual and auditory bone fusion as claimed in claim 6, characterized in that: The parameter updating model is: θ←θ+v; In the formula, x (l) is the label information of the i-th input data, is the loss function, v is the parameter update value, θ is the parameter of the air-bone fusion module and the comprehensive fusion module, η is the initial parameter learning rate, α is the momentum angle parameter, 0≤α≤1, It means to find the partial derivative of variable θ, f(x (l) ; θ) represents the enhanced speech information calculated by the speech enhancement model for the i-th input data, f(.) is the calculation function corresponding to the speech enhancement model; exp represents the power operation of the constant e; η and α are preset values.

8. A speech enhancement device based on visual and auditory bone fusion, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the speech enhancement method based on audiovisual bone fusion as described in any one of claims 1-7.

9. A computer storable medium, characterized in that: The computer storable medium stores computer instructions, which, when called, are used to execute the speech enhancement method based on audiovisual bone fusion as described in any one of claims 1-7.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the speech enhancement method based on audiovisual bone fusion as described in any one of claims 1-7.

Images