A Cloud-Edge-Terminal Collaborative Hearing Inference Method with Controllable Multimodal Perception Flow

Abstract

The present invention relates to a cloud-edge-end collaborative hearing aid inference method with controllable multi-modal perception streams. First, GRU is used for audiovisual event recognition, visual and auditory features are extracted respectively, and under the condition of controlling the visual features, a recurrent neural network is used for audiovisual event recognition. At the decision-making layer, the results of audiovisual information are fused by means of dynamic weighting. According to the influence of visual perception amount on the recognition accuracy and the influence of the change of audiovisual information weight on the final result, two parameters meeting the requirements are determined according to the actual situation. Secondly, cross-modal data generation is carried out based on DCGAN, visual and auditory features are extracted, and a part of visual features are fused with auditory features. Under the condition of controlling the visual information perception amount for fusion, a generative adversarial network is used for cross-modal data generation. According to the evaluation method of the performance of the generative adversarial network and the influence of visual information perception amount on the quality of the generated samples, parameters meeting the actual requirements are determined.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence hearing aids in the context of cloud-edge-end collaboration for deep learning models and their multi-modal audio-visual data, and specifically to a method for collaborative hearing aid using multi-modal data with controllable perception metrics through deep learning technology in the cloud-edge-end collaboration framework. It includes audio-visual event recognition technology and cross-modal data generation technology based on deep learning. Background Art

[0002] As an important branch of machine learning, deep learning has achieved rich and remarkable results in multiple interdisciplinary fields in recent years. Among them, in the field of assisted medicine, artificial intelligence-assisted hearing aid technology for hearing-impaired people has always received much attention. This technology aims to use technologies and methods in the field of artificial intelligence to enable hearing-impaired people to regain the perception of sound, or achieve an effect equivalent to perceiving sound, that is, to enable hearing-impaired people to obtain the results of recognition and classification of sound events. Among the two target effects, the former tends to have a direct effect on the human ear and belongs to a relatively traditional hearing aid method. Technologies such as intelligent hearing aids and cochlear implants achieve this effect. However, although hearing-impaired people do obtain more or less the perception of sound, that is, they hear the sound, such methods are often costly and lack comfort for users; the latter transfers the key issue to the recognition and classification of sound events. Its basic idea is to first identify the type of event represented by the sound, and then transmit the recognition result to the hearing-impaired person in a non-sound way. In this process, it does not care about the perception of sound itself by deaf-mutes, but can achieve an effect similar to the degree of perceiving sound.

[0003] In existing hearing aid methods, it is common to use single-modal sound data for event recognition and classification. Although many studies have achieved good results in this field, in a more complex reasoning environment, the reasoning ability using single-modal data is still limited. Considering the diversity of data types and the complementarity of data properties in the real world, using multi-modal data for event recognition can improve the accuracy of recognition to a certain extent. At the same time, the complementarity of data is also of great significance for the expression of sound events. However, introducing multi-modal data also brings some problems, among which the more important ones include the perception power consumption problem of high-dimensional data, the fusion problem of heterogeneous data, and the balance problem between power consumption and efficiency. Summary of the Invention

[0004] Technical Problems to be Solved

[0005] In order to avoid the deficiencies of the prior art, the present invention provides a cloud-edge-end collaborative hearing aid reasoning method with controllable multi-modal perception flow.

[0006] Technical Solution

[0007] A cloud-edge-terminal collaborative hearing aid inference method with controllable multi-modal perception stream, characterized by including a GRU-based audio-visual event recognition method and a DCGAN-based cross-modal data generation method:

[0008] The GRU-based audio-visual event recognition method: Use GRU for audio-visual event recognition, extract visual and auditory features respectively, use a recurrent neural network for audio-visual event recognition under the condition of controlling visual features, and fuse the results of audio-visual information in a dynamic weighting manner at the decision layer. According to the influence of visual perception quantity on recognition accuracy and the influence of the change of audio-visual information weight on the final result, determine two parameters that meet the requirements according to the actual situation;

[0009] The DCGAN-based cross-modal data generation method: Based on DCGAN for cross-modal data generation, especially suitable for scenarios with few data samples, extract audio-visual features, and fuse a part of visual features with auditory features. Use a generative adversarial network for cross-modal data generation under the condition of controlling the visual information perception quantity for fusion. Determine the parameters that meet the actual requirements according to the evaluation method of the performance of the generative adversarial network and the influence of visual information perception quantity on the quality of generated samples;

[0010] Set the parameters obtained by the two methods in the GRU classification network and the DCGAN cross-modal generator respectively to achieve a balance between perception power consumption and system performance, and finally present the audio-visual event to the user, that is, the hearing-impaired person, in a visual way.

[0011] A further technical solution of the present invention: The GRU-based audio-visual event recognition method includes the following modules:

[0012] Module 1: Data collection and preprocessing

[0013] Deploy this module at the data acquisition terminal to realize the immediate acquisition and processing of data; the collected data should include two types of visual data and auditory data, but both are uniformly represented as audio-visual in the sensor, so it is necessary to separate the audio-visual signals. In addition, in order to more simply express visual information and control the visual information perception quantity, it is also necessary to extract the key frames of the video, use the number of frames as the representative of the visual information perception quantity, and the information contained in the key frames will be used as the basis for the feature extraction module;

[0014] Module 2: Feature extraction of audio-visual information

[0015] For the audio signal, extract its MFCC features: perform pre-emphasis, framing, windowing, discrete Fourier transform, Mel filtering, logarithmic operation, discrete cosine transform, and dynamic differential feature extraction on the audio data to obtain the MFCC and differential MFCC features of the audio; for the video data, extract its image features from the key frames obtained in the preprocessing stage as the visual features of the data, and this process is carried out through the VGG neural network;

[0016] Module Three: Gated Recurrent Unit Classification Network and Classification Result Fusion

[0017] The pre-trained GRU network is used as the classifier of the features, and the features of the audio data and the video data are respectively used for inference. The obtained results are fused at the decision layer through the following formula to obtain the predicted probability that a certain audio-visual event belongs to a certain category after fusion

[0018] p = t·p a +(1 - t)·p v

[0019] where the probabilities of predicting a certain event as a certain category using the audio features and the video features are p a and p v , respectively, and t represents the variable confidence weight;

[0020] Module Four: Determine the Final Parameters According to the Fusion Result

[0021] There are two variables in the analysis of the results, namely the confidence weight t and the number of key frames representing the amount of visual information perception. By fixing the number of key frames and changing the confidence weight t, the change results of multiple groups of prediction accuracies are obtained, which reflect the contribution degrees of the auditory information and the visual information to the recognition accuracy; while fixing the confidence weight t and changing the number of key frames, it reflects the influence of the amount of visual information perception on the recognition accuracy; in the process of determining the optimal values of these two parameters, the user has a high degree of autonomy and can select the parameter values that can achieve a relatively ideal accuracy and are within the power consumption limit requirements according to the actual situation.

[0022] A further technical solution of the present invention: The cross-modal data generation method based on DCGAN includes the following modules:

[0023] Module One: Data Acquisition and Preprocessing

[0024] Deploy this module at the data acquisition terminal to achieve immediate acquisition and processing of data; the collected data is uniformly presented as audio-visual in the data sensor, so it is necessary to separate the audio-visual signals. In addition, it is also necessary to extract the key frames of the video. The number of frames is used as a representative of the visual information perception amount. At the same time, the information contained in the key frames will be used as the basis for the feature extraction module. In addition, it is necessary to fuse the features of a large number of key frames, regarded as the features of real video data, as one of the inputs of the discriminant network in the generative adversarial network. Use a small number of key frames as an incomplete representation of the video data, control the number of these key frames, fuse their features with the audio features, and use them as the input of the generative network to assist the audio features in cross-modal data reasoning, and then study the impact of the number of these key frames on the final network performance;

[0025] Module 2: Feature Extraction and Fusion of Visual and Auditory Information

[0026] Convolutional neural networks are used to extract the features of audio-visual signals. For the fusion of these two types of features, first, a simple concatenate connection operation is performed to obtain a feature vector with a higher dimension, and then the PCA algorithm is used for feature dimensionality reduction to obtain a lower-dimensional fusion feature;

[0027] Module 3: Deep Convolutional Generative Adversarial Network as the Generator

[0028] A small number of key frames are subjected to image feature extraction to obtain feature F’ visual , together with the feature F of the audio data audio , are sent to the feature processing and fusion layer, and after processing, the fused audio-visual feature F a-v is obtained. The features F visual extracted from a large number of key frames are used to represent the features of real visual data; then, the fused audio-visual feature F a-v is used as the input of the generative network. The generative network generates forged pictures by learning the fused features, and its visual feature is F * visual , F * visual and the real visual features are sent into the discriminant network together for it to distinguish the authenticity. Through multiple rounds of training of the generative network and the adversarial network, the generation ability of the generative network is improved, and finally, it is expected to obtain a visual information generation network with high quality;

[0029] Module 4: Quality Assessment of Generated Data and Final Parameter Determination

[0030] Evaluate the quality of the generated visual information through two methods: subjective qualitative and objective quantitative, determine the number of training rounds that can meet the requirements, and represent the perceived amount of visual information by controlling the number of input key frames to determine the number of key frames that meet the requirements of data quality and power consumption limitations.

[0031] A computer system, characterized in that it includes: one or more processors, and a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the above method.

[0032] A computer-readable storage medium, characterized in that it stores computer-executable instructions, and the instructions are used to implement the above method when executed.

[0033] Beneficial effects

[0034] A hearing aid method provided by the present invention is based on two deep learning models to respectively perform hearing aid through audiovisual event recognition and cross-modal data generation under the condition that the perceived amount is controllable. Among them, multi-modal audiovisual data is used, a recurrent neural network model is used as a classifier for audiovisual event recognition, and a generative adversarial network model is used as a generator for cross-modal data generation. In addition, a cloud-edge-end collaborative computing framework is established in the method to handle data heterogeneity problems and adjust the computing capabilities of each part of the network model.

[0035] The present invention utilizes the characteristics of multi-modal data and many network models of deep learning to represent the information contained in sound data with other modal data to achieve the effect and purpose of hearing aid. Brief description of the drawings

[0036] The drawings are only for the purpose of showing specific embodiments and are not considered to be a limitation of the present invention. Throughout the drawings, the same reference numerals represent the same components.

[0037] [[ID=2३]] Figure 1 It is the basic research content framework of the present invention;

[0038] Figure 2 It is the basic process framework of the present invention for audiovisual event recognition based on a recurrent neural network;

[0039] Figure 3 It is the basic process framework of the present invention for cross-modal data generation based on a generative adversarial network. Detailed implementation manners

[0040] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0041] The present invention utilizes the following principles: The present invention is mainly targeted at hearing-impaired persons. Based on deep learning technology, it studies artificial intelligence hearing aid methods using multi-modal audio-visual data in the framework of cloud-edge-end collaboration. Under two distinct deep learning network frameworks, according to the characteristics of the networks, research is carried out respectively with two different ideas and mechanisms. Under the recurrent neural network framework, the gated recurrent unit network is used as a feature classifier to classify the extracted auditory and visual features respectively; under the generative adversarial network framework, images of real video frames are simulated and generated based on cross-modal data of DCGAN. Conditions for controlling the perception amount of visual data are proposed in both methods, and the key frame features of the video are used as visual features, with the number of key frames as the representation of the visual information perception amount.

[0042] As Figure 1 shown, the present invention includes an audio-visual event recognition method based on GRU and a cross-modal data generation method based on DCGAN:

[0043] The described audio-visual event recognition method based on GRU: First, GRU is used for audio-visual event recognition, and visual and auditory features are extracted respectively. Under the condition of controlling visual features, a recurrent neural network is used for audio-visual event recognition. At the decision-making layer, the results of audio-visual information are fused by means of dynamic weighting. According to the influence of visual perception amount on the recognition accuracy and the influence of the change of audio-visual information weight on the final result, two parameters that meet the requirements are determined according to the actual situation.

[0044] The described cross-modal data generation method based on DCGAN: Cross-modal data generation is carried out based on DCGAN, which is especially suitable for scenarios with fewer data samples. Visual and auditory features are extracted, and a part of the visual features are fused with the auditory features. Under the condition of controlling the perception amount of visual information used for fusion, a generative adversarial network is used for cross-modal data generation. According to the evaluation method of the performance of the generative adversarial network and the influence of the visual information perception amount on the quality of the generated samples, parameters that meet the actual requirements are determined. [[ID=!5]]

[0045] Apply the parameter values determined by the above two methods to the parameter settings of the GRU classification network and the DCGAN cross-modal generator respectively. For the GRU classification network, the determination of the parameters enables it to achieve a balance between classification accuracy and perceived power consumption, accurately identify the categories of audiovisual events without generating excessive power consumption, and convey the results to the hearing-impaired in the form of text or images. For the DCGAN cross-modal generator, the determination of the parameters enables it to achieve a balance between the quality of generated samples and the amount of information perceived, and realize the generation from auditory data to visual data at the cost of lower perceived power consumption, and directly present the generated results to the hearing-impaired. Both methods enable the hearing-impaired to perceive events without hearing sounds.

[0046] As Figure 2 shown, for the method of using a recurrent neural network model to identify audiovisual events, its process includes the following modules:

[0047] Module 1: Data acquisition and preprocessing. Deploy this module at the data acquisition terminal to achieve the immediate acquisition and processing of data. The collected data should include both visual data and auditory data, but both are uniformly represented as audio-visual in the sensor, so it is necessary to separate the audio-visual signals. In addition, in order to more simply express the visual information and control the amount of visual information perceived, it is also necessary to extract the key frames of the video, use the number of frames as the representative of the amount of visual information perceived, and the information contained in the key frames will be used as the basis for the feature extraction module.

[0048] Module 2: Feature extraction of visual and auditory information. For the audio signal, extract its MFCC features: perform pre-emphasis, framing, windowing, discrete Fourier transform, Mel filtering, logarithmic operation, discrete cosine transform, and dynamic differential feature extraction on the audio data to obtain the MFCC and differential MFCC features of the audio, so as to better express the static and dynamic characteristics of the sound. For the video data, use the key frames obtained in the preprocessing stage to extract its image features as the visual features of the data, and this process is carried out through the VGG neural network.

[0049] Module 3: Gated recurrent unit classification network and classification result fusion. The pre-trained GRU network is used as the classifier of the features, and the audio data features and video data features are used for inference respectively. The obtained results are fused at the decision layer through the following formula to obtain the predicted probability that a certain audiovisual event belongs to a certain category after fusion

[0050] p = t · p a +(1 - t) · p v

[0051] where the probabilities of predicting a certain event as a certain category using audio features and video features are p a and p v, where t represents a variable confidence weight.

[0052] Module Four: Determine according to the final parameters of the fusion result. There are two variables in the result analysis, namely the confidence weight t and the number of key frames representing the visual information perception amount. By fixing the number of key frames and changing the confidence weight t, multiple sets of change results of the prediction accuracy are obtained, which reflects the contribution degree of auditory information and visual information to the recognition accuracy; while fixing the confidence weight t and changing the number of key frames reflects the influence of the visual information perception amount on the recognition accuracy. In the process of determining the optimal values of these two parameters, the user has a high degree of autonomy and can select parameter values that can achieve a relatively ideal accuracy and are within the power consumption limit requirements according to the actual situation.

[0053] Such as Figure 3 shown, for the method of cross-modal data generation using a generative adversarial network, its process includes the following modules:

[0054] Module One: Data acquisition and preprocessing. Deploy this module at the data acquisition terminal to achieve the immediate acquisition and processing of data. The data collected by the sensors uniformly appears as audio-visual, so it is necessary to separate the audio-visual signals. In addition, it is also necessary to extract the key frames of the video. The number of frames is used as a representative of the visual information perception amount, and the information contained in the key frames will be used as the basis for the feature extraction module. In addition, the features of a large number of key frames need to be fused and regarded as the features of real video data, which is one of the inputs of the discriminator network in the generative adversarial network. Use a small number of key frames as an incomplete representation of the video data, control the number of these key frames, fuse their features with the audio features, and use them as the input of the generative network to assist the audio features in cross-modal data reasoning, and then study the influence of the number of these key frames on the final network performance.

[0055] Module Two: Feature extraction and fusion of visual and auditory information. Convolutional neural networks are used to extract the features of audio-visual signals. For the fusion of these two types of features, first, a simple concatenate connection operation is performed to obtain a feature vector with a higher dimension, and then the PCA algorithm is used for feature dimensionality reduction to obtain a lower-dimensional fused feature.

[0056] Module Three: The deep convolutional generative adversarial network as a generator. A small number of key frames are subjected to image feature extraction to obtain feature F’ visual , which, together with the feature F audio of the audio data, is sent to the feature processing and fusion layer. After processing, the fused audio-visual feature F a-v is obtained, and the features F visual extracted from a large number of key frames are used to represent the features of real visual data; then, the fused audio-visual feature F a-vAs the input of the generation network, the generation network generates forged images by learning and fusing features, and its visual feature is F * visual , F * visual Together with the real visual features, they are fed into the discriminant network to distinguish between true and false. Through multiple rounds of training of the generation network and the adversarial network, the generation ability of the generation network is improved. Finally, it is expected to obtain a generation network of visual information with high quality.

[0057] Module 4: Quality assessment of the generated data and final parameter determination. The quality of the generated visual information is evaluated by subjective qualitative and objective quantitative methods to determine the number of training rounds that can meet the requirements. The perceptual amount of visual information is represented by controlling the number of input key frames, and the number of key frames that meets the requirements of data quality and power consumption limitations is determined.

[0058] As described above, it is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or replacements, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. A cloud-edge-end collaborative hearing inference method with controllable multi-modal perception stream, characterized in that Including a GRU-based audio-visual event recognition method and a DCGAN-based cross-modal data generation method: The GRU-based audio-visual event recognition method: Utilize GRU for audio-visual event recognition, extract visual and auditory features respectively, use a recurrent neural network for audio-visual event recognition under the condition of controlling visual features, and fuse the results of audio-visual information in the decision layer by means of dynamic weighting. According to the influence of visual perception quantity on recognition accuracy and the influence of the change of audio-visual information weight on the final result, determine two parameters that meet the requirements according to the actual situation; The DCGAN-based cross-modal data generation method: Based on DCGAN for cross-modal data generation, especially suitable for scenarios with fewer data samples, extract audio-visual features, and make a part of visual features fuse with auditory features. Under the condition of controlling the visual information perception quantity for fusion, use a generative adversarial network for cross-modal data generation. Determine the parameters that meet the actual requirements according to the evaluation method of the performance of the generative adversarial network and the influence of visual information perception quantity on the quality of generated samples; Set the parameters obtained by the two methods in the GRU classification network and the DCGAN cross-modal generator respectively to achieve a balance between perception power consumption and system performance, and finally present the audio-visual event to the user, i.e., the hearing-impaired person, in a visual way.

2. A cloud-edge-end collaborative hearing aid inference method with controllable multi-modal perception stream according to claim 1, characterized in that The GRU-based audio-visual event recognition method described above includes the following modules: Module 1: Data acquisition and preprocessing Deploy this module at the data acquisition terminal to achieve immediate data acquisition and processing; The collected data should include two types: visual data and auditory data, but both are uniformly presented as audio-visual in the sensor, so it is necessary to separate the audio-visual signal. In addition, in order to more simply express visual information and control the visual information perception quantity, it is also necessary to extract key frames of the video, use the number of frames as a representative of the visual information perception quantity, and at the same time, the information contained in the key frames will be used as the basis for the feature extraction module; Module 2: Feature extraction of audio-visual information For the audio signal, extract its MFCC features: Through pre-emphasis, framing, windowing, discrete Fourier transform, Mel filtering, logarithmic operation, discrete cosine transform, and dynamic differential feature extraction of audio data, obtain the MFCC and differential MFCC features of the audio; For video data, use the key frames obtained in the preprocessing stage to extract its image features as the visual features of the data, and this process is carried out through the VGG neural network; Module 3: Gated recurrent unit classification network and classification result fusion The pre-trained GRU network is used as the classifier of features, and the audio data features and video data features are used for inference respectively. The obtained results are fused in the decision layer through the following formula to obtain the predicted probability that a certain audio-visual event belongs to a certain category after fusion p = t·p a +(1 - t)·p v The probabilities predicted for a certain type of event using audio features and video features are p a and p v , where t represents a variable confidence weight; Module 4: Determine the final parameters according to the fusion result There are two variables in the analysis of the results, namely the confidence weight t and the number of key frames representing the amount of visual information perception. By fixing the number of key frames and changing the confidence weight t, the change results of multiple groups of prediction accuracies are obtained, which reflect the contribution degrees of auditory information and visual information to the recognition accuracy; while fixing the confidence weight t and changing the number of key frames reflects the influence of the amount of visual information perception on the recognition accuracy; in the process of determining the optimal values of these two parameters, the user has a high degree of autonomy and can select parameter values that can achieve a relatively ideal accuracy and are within the power consumption limit requirements according to the actual situation.

3. A cloud-edge-end collaborative hearing aid inference method with controllable multi-modal perception stream according to claim 1, characterized in that The described cross-modal data generation method based on DCGAN includes the following modules: Module 1: Data collection and preprocessing Deploy this module at the data acquisition terminal to achieve the immediate acquisition and processing of data; the collected data sensors are uniformly represented as audio-visual videos, so it is necessary to separate the audio-visual signals. In addition, it is also necessary to extract the key frames of the video. The number of frames is used as a representative of the amount of visual information perception. At the same time, the information contained in the key frames will be used as the basis for the feature extraction module. In addition, the features of a large number of key frames need to be fused and regarded as the features of real video data, which is one of the inputs of the discriminator network in the generative adversarial network. Use a small number of key frames as an incomplete representation of the video data, control the number of these key frames, fuse their features with the audio features, and use them as the input of the generative network to assist the audio features in cross-modal data reasoning, and then study the impact of the number of these key frames on the final network performance; Module 2: Feature extraction and fusion of visual and auditory information Convolutional neural networks are used to extract the features of audio-visual signals. For the fusion of these two types of features, first, a simple concatenate connection operation is performed to obtain a feature vector with a higher dimension, and then the PCA algorithm is used for feature dimensionality reduction to obtain a lower-dimensional fusion feature; Module 3: Deep convolutional generative adversarial network as a generator A small number of key frames are subjected to image feature extraction to obtain feature F'. visual , together with the feature F of the audio data audio , are fed into the feature processing and fusion layer, and after processing, the fused audio-visual feature F is obtained a-v , while the features F extracted from a large number of key frames visual are used to represent the features of real visual data; then, the fused audio-visual feature F a-v is used as the input of the generation network. The generation network generates forged pictures by learning the fused features, and its visual feature is F * visual , F * visual and the real visual features are fed into the discriminant network together for it to distinguish between true and false. Through multiple rounds of training of the generation network and the adversarial network, the generation ability of the generation network is improved, and ultimately, it is expected to obtain a visual information generation network with high quality; Module 4: Quality evaluation of generated data and final parameter determination Evaluate the quality of the generated visual information through two methods: subjective qualitative and objective quantitative, determine the number of training rounds that can meet the requirements, represent the amount of visual information perception by controlling the number of input key frames, and determine the number of key frames that meet the requirements of data quality and power consumption limit.

4. A computer system, characterized in that Including: One or more processors, a computer-readable storage medium for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in claim 1.

5. A computer-readable storage medium, characterized in that Store computer-executable instructions that are used to implement the method described in claim 1 when executed.

Images