Multi-mode synchronous fusion speech recognition system

Abstract

The invention discloses a multi-modal synchronous fusion speech recognition system, and belongs to the technical field of multi-modal information fusion. A multi-modal information processing platform; the fusion weight adjusting module is used for monitoring the quality and characteristics of the input signal in real time, dynamically adjusting the fusion weight of each module and ensuring that the fusion weight can adapt to the change of the input signal; an adaptive fusion algorithm, the fusion weight adjustment module feeds back to the adaptive fusion algorithm in real time, and the adaptive fusion algorithm can dynamically adjust a fusion strategy according to characteristics of different modal information to ensure an optimal fusion effect; and the environment noise processing module is used for identifying and analyzing the characteristics of the environment noise so as to facilitate effective suppression in the fusion process. Through three-time fusion adjustment of the multi-modal information processing platform, the fusion weight adjustment module and the adaptive fusion algorithm, optimal fusion of voice, images and physiological signals is ensured, and the accuracy of information identification is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal information fusion, and in particular relates to a multimodal synchronous fusion speech recognition system. Background Art

[0002] The rapid development of deep learning technology, especially the introduction of attention mechanism, has brought revolutionary changes to speech recognition technology, bringing the research and application of this field to a new level. However, despite the significant progress, the single-modality speech recognition system still seems to be unable to cope with the complex environment of the real world.

[0003] First of all, the noisy environment is a major problem faced by speech recognition technology. Whether it is background noise indoors or outdoors, or noisy scenes of multiple people talking, it will seriously affect the clarity of the audio signal, thereby reducing the accuracy of speech recognition. In order to overcome this challenge, researchers began to explore the use of video information as an auxiliary means to enhance the robustness of speech recognition by capturing visual features such as the speaker's lip movements and facial expressions. The addition of video information can not only provide an additional source of information when the audio signal is disturbed, but also alleviate the problem of homophones to a certain extent, because different words are often accompanied by different lip shape changes. However, it is not easy to effectively integrate audio and video information. On the one hand, the characterization process of visual information is relatively complex and diverse. Unlike the relatively stable acoustic features in audio signals (such as MFCC), the visual features in video signals (such as lip contours, textures, motion trajectories, etc.) are more diverse and difficult to quantify. Therefore, how to extract the most significant visual features that improve speech recognition performance and effectively match them with audio features (such as MFCC) has become an important topic in the study of AVSR systems.

[0004] On the other hand, audio and video signals often have mismatches in rate and information length. Audio signals are continuous and time-sensitive, while video signals may present discontinuous information due to factors such as frame rate and encoding method. This mismatch not only increases the difficulty of information fusion, but may also cause confusion and increased load in the audio and video information processing process.

[0005] In addition, the AVSR system also needs to consider the diversity and complexity of actual application scenarios. For example, in a conference system, in addition to basic speech recognition functions, it is also necessary to realize functions such as automatic switching of multiple speakers, automatic generation of meeting minutes, and real-time tracking of meeting progress. The realization of these functions requires the AVSR system to have a high level of intelligence and automation, and to be able to accurately understand the content of the meeting, identify the identities of the participants, and analyze the mood of the meeting.

[0006] However, the existing AVSR system has mismatched audio and video rates and different information lengths during use, which makes information fusion difficult and easily causes confusion in audio and video information processing, leading to increased load, low recognition matching, and low accuracy. Summary of the invention

[0007] The purpose of the present invention is to propose a multimodal synchronous fusion speech recognition system in order to solve the problems of the existing AVSR system in which the audio and video rates do not match and the information lengths are different, making information fusion difficult, easily causing confusion in audio and video information processing, resulting in increased load, low recognition matching degree, and low accuracy.

[0008] In order to achieve the above object, the present invention adopts the following technical solution: a multi-modal synchronous fusion speech recognition system, comprising:

[0009] A data acquisition module, used for collecting multimodal information;

[0010] Multimodal information processing platforms for processing, integrating and analyzing information from different sources and types;

[0011] The fusion weight adjustment module is used to monitor the quality and characteristics of the input signal in real time, dynamically adjust the fusion weight of each module, and ensure that the fusion weight can adapt to the changes of the input signal; the fusion weight adjustment module includes an adaptive fusion unit, which is used to learn the characteristics of the collected signal and automatically adjust the fusion weight of each modal information accordingly to improve the accuracy of information recognition:

[0012] Adaptive fusion algorithm: the fusion weight adjustment module provides real-time feedback to the adaptive fusion algorithm, and the adaptive fusion algorithm can dynamically adjust the fusion strategy according to the characteristics of different modal information to ensure the best fusion effect;

[0013] and environmental noise processing modules, which are used to identify and analyze the characteristics of environmental noise to facilitate its effective suppression during the fusion process.

[0014] As a further description of the above technical solution:

[0015] The data acquisition module comprises a voice module, an environmental noise module, an image module and a physiological signal module, wherein the voice module, the environmental noise module, the image module and the physiological signal module are used to collect voice, environmental noise, image and physiological signals respectively.

[0016] As a further description of the above technical solution:

[0017] The multimodal information processing platform includes a preprocessing module, a feature extraction module, a fusion processing module and a decision output module, wherein the preprocessing module performs preliminary processing such as denoising, filtering, calibration and enhancement on the original data for each module; the feature extraction module is used to extract the feature information of each modal data; the fusion processing module uses a multimodal feature fusion algorithm to integrate the feature information; and the decision output module outputs a corresponding decision or inference based on the fusion result.

[0018] As a further description of the above technical solution:

[0019] The voice module, the environmental noise module, the image module and the physiological signal module are respectively configured with corresponding sensors and acquisition devices.

[0020] As a further description of the above technical solution:

[0021] The adaptive fusion unit includes a training unit, which is used to train a large number of speech, image, biological signal and environmental noise samples to learn the characteristics of input signals under different conditions.

[0022] As a further description of the above technical solution:

[0023] The fusion weight is calculated using the least square method, information entropy or fuzzy logic.

[0024] As a further description of the above technical solution:

[0025] The fusion weight adjustment module adopts a real-time feedback mechanism to respond to changes in the input signal in real time and adjust the fusion strategy accordingly.

[0026] As a further description of the above technical solution:

[0027] The voice module includes at least one microphone, the image module includes at least one camera, and the physiological signal module includes at least one physiological signal sensor.

[0028] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0029] 1. In the present invention, multimodal information such as speech, environmental noise, images and physiological signals is collected through a data collection module, and the collected multimodal information is preliminarily processed by denoising, filtering, calibration and enhancement through a multimodal information processing platform, and then the feature information of each modal data is extracted, the extracted multimodal information is fused, and the fusion result is decided and transmitted to the fusion weight adjustment module. In the fusion weight adjustment module, a training unit is set in an adaptive fusion unit and a large number of samples are stored. The training unit is used to train a large number of speech, image, biological signal and environmental noise samples, and the characteristics of the input signal under different conditions are learned, so that the adaptive fusion unit dynamically adjusts the characteristics of the learned collected signal according to the characteristics of the learned collected signal. Adjust the fusion weight of each modal information, monitor the characteristics of the input signal and quality in real time, and provide real-time feedback to the adaptive fusion algorithm. The adaptive fusion algorithm can dynamically adjust the fusion strategy according to the characteristics of different modal information to ensure the best fusion effect. In the process of multimodal information fusion, the environmental noise processing module effectively suppresses the noise in the multimodal information to ensure that the multimodal information can be presented in the clearest and most complete form. This system collects, processes, and fuses multiple information such as voice, environmental noise, images, physiological signals, etc. within the same time frame, and produces the final recognition or decision-making results. This synchronization ensures that the system can integrate information from different sources in real time and improve the accuracy and robustness of information processing.

[0030] 2. In the present invention, three fusion adjustments are performed through a multimodal information processing platform, a fusion weight adjustment module, and an adaptive fusion algorithm, and a decision or inference is outputted based on the fusion result for the first time; the second fusion is to use deep learning to train the fusion network and optimize its automatic learning and fusion weight allocation capabilities; the third fusion is to monitor the signal quality in real time during the fusion process, and use an adaptive fusion algorithm to dynamically adjust the fusion weights, thereby ensuring the best fusion of voice, image and physiological signals, improving the matching degree and accuracy of information recognition, and solving the problems of existing audio and video information processing chaos, increased load, low recognition matching degree, and low accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0032] Figure 1 This is a flow chart of multimodal information fusion in a multimodal synchronous fusion speech recognition system. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0035] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0037] In the description of the embodiments of the present invention, it should be noted that the terms "upper", "inner", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, or are directions or positional relationships in which the inventive product is usually placed when in use. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as a limitation on the present invention.

[0038] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0039] See also Figure 1 The present invention provides a technical solution: a multi-modal synchronous fusion speech recognition system, comprising:

[0040] A data acquisition module, used for collecting multimodal information;

[0041] Multimodal information processing platforms for processing, integrating and analyzing information from different sources and types;

[0042] A fusion weight adjustment module is used to monitor the quality and characteristics of the input signal in real time, dynamically adjust the fusion weight of each module, and ensure that the fusion weight can adapt to the changes of the input signal; the fusion weight adjustment module includes an adaptive fusion unit, and the adaptive fusion unit is used to learn the characteristics of the collected signal and automatically adjust the fusion weight of each modal information accordingly to improve the accuracy of information recognition;

[0043] Adaptive fusion algorithm: the fusion weight adjustment module provides real-time feedback to the adaptive fusion algorithm, and the adaptive fusion algorithm can dynamically adjust the fusion strategy according to the characteristics of different modal information to ensure the best fusion effect;

[0044] and environmental noise processing modules, which are used to identify and analyze the characteristics of environmental noise to facilitate its effective suppression during the fusion process.

[0045] The data acquisition module comprises a voice module, an environmental noise module, an image module and a physiological signal module, wherein the voice module, the environmental noise module, the image module and the physiological signal module are used to collect voice, environmental noise, image and physiological signals respectively.

[0046] The multimodal information processing platform includes a preprocessing module, a feature extraction module, a fusion processing module and a decision output module, wherein the preprocessing module performs preliminary processing such as denoising, filtering, calibration and enhancement on the original data for each module; the feature extraction module is used to extract the feature information of each modal data; the fusion processing module uses a multimodal feature fusion algorithm to integrate the feature information; and the decision output module outputs a corresponding decision or inference based on the fusion result.

[0047] The voice module, the environmental noise module, the image module and the physiological signal module are respectively configured with corresponding sensors and acquisition devices.

[0048] The adaptive fusion unit includes a training unit, which is used to train a large number of speech, image, biological signal and environmental noise samples to learn the characteristics of input signals under different conditions.

[0049] The fusion weight is calculated using the least square method, information entropy or fuzzy logic.

[0050] The fusion weight adjustment module adopts a real-time feedback mechanism to respond to changes in the input signal in real time and adjust the fusion strategy accordingly.

[0051] The voice module includes at least one microphone, the image module includes at least one camera, and the physiological signal module includes at least one physiological signal sensor.

[0052] Working principle: The data acquisition module collects multimodal information such as speech, environmental noise, images and physiological signals, and performs preliminary processing of denoising, filtering, calibration and enhancement on the collected multimodal information through the multimodal information processing platform. The feature information of each modal data is extracted, the extracted multimodal information is fused, and the fusion result is transmitted to the fusion weight adjustment module for decision making. In the fusion weight adjustment module, a training unit is set in the adaptive fusion unit and a large number of samples are stored. The training unit is used to train a large number of speech, image, biological signal and environmental noise samples, learn the characteristics of input signals under different conditions, so that the adaptive fusion unit can dynamically adjust the characteristics of the learned collected signals according to the characteristics of the learned collected signals. The fusion weight of each modal information is adjusted, and the characteristics of the input signal and quality are monitored in real time, and the feedback is given to the adaptive fusion algorithm in real time. The adaptive fusion algorithm can dynamically adjust the fusion strategy according to the characteristics of different modal information to ensure the best fusion effect. In the process of multimodal information fusion, the environmental noise processing module effectively suppresses the noise in the multimodal information to ensure that the multimodal information can be presented in the clearest and most complete form. This system collects, processes, and fuses multiple information such as voice, environmental noise, images, physiological signals, etc. within the same time frame, and produces the final recognition or decision-making results. This synchronization ensures that the system can integrate information from different sources in real time and improve the accuracy and robustness of information processing.

[0053] And through three fusion adjustments of multimodal information processing platform, fusion weight adjustment module and adaptive fusion algorithm, the decision or inference is output according to the fusion result for the first time; the second fusion is to use deep learning to train the fusion network and optimize its automatic learning and fusion weight allocation capabilities; the third fusion is to monitor the signal quality in real time during the fusion process, and use the adaptive fusion algorithm to dynamically adjust the fusion weight, ensuring the optimal fusion of speech, image and physiological signals and improving the accuracy of information recognition.

[0054] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multimodal synchronous fusion speech recognition system, characterized in that: include: A data acquisition module, used for collecting multimodal information; Multimodal information processing platforms for processing, integrating and analyzing information from different sources and types; A fusion weight adjustment module is used to monitor the quality and characteristics of the input signal in real time, dynamically adjust the fusion weight of each module, and ensure that the fusion weight can adapt to the changes of the input signal; the fusion weight adjustment module includes an adaptive fusion unit, and the adaptive fusion unit is used to learn the characteristics of the collected signal and automatically adjust the fusion weight of each modal information accordingly to improve the accuracy of information recognition; Adaptive fusion algorithm: the fusion weight adjustment module provides real-time feedback to the adaptive fusion algorithm, and the adaptive fusion algorithm can dynamically adjust the fusion strategy according to the characteristics of different modal information to ensure the best fusion effect; and environmental noise processing modules, which are used to identify and analyze the characteristics of environmental noise to facilitate its effective suppression during the fusion process.

2. A multimodal synchronous fusion speech recognition system according to claim 1, characterized in that: The data acquisition module comprises a voice module, an environmental noise module, an image module and a physiological signal module, wherein the voice module, the environmental noise module, the image module and the physiological signal module are used to collect voice, environmental noise, image and physiological signals respectively.

3. A multimodal synchronous fusion speech recognition system according to claim 2, characterized in that: The multimodal information processing platform includes a preprocessing module, a feature extraction module, a fusion processing module and a decision output module, wherein the preprocessing module performs preliminary processing of denoising, filtering, calibration and enhancement on the original data for each module; the feature extraction module is used to extract feature information of each modal data; the fusion processing module uses a multimodal feature fusion algorithm to integrate feature information; The decision output module outputs corresponding decisions or inferences according to the fusion results.

4. A multimodal synchronous fusion speech recognition system according to claim 3, characterized in that: The voice module, the environmental noise module, the image module and the physiological signal module are respectively configured with corresponding sensors and acquisition devices.

5. A multi-modal synchronous fusion speech recognition system according to claim 1, characterized in that: The adaptive fusion unit includes a training unit, which is used to train a large number of speech, image, biological signal and environmental noise samples to learn the characteristics of input signals under different conditions.

6. A multi-modal synchronous fusion speech recognition system according to claim 1, characterized in that: The fusion weight is calculated using the least square method, information entropy or fuzzy logic.

7. A multi-modal synchronous fusion speech recognition system according to claim 1, characterized in that: The fusion weight adjustment module adopts a real-time feedback mechanism to respond to changes in the input signal in real time and adjust the fusion strategy accordingly.

8. A multi-modal synchronous fusion speech recognition system according to claim 2, characterized in that: The voice module includes at least one microphone, the image module includes at least one camera, and the physiological signal module includes at least one physiological signal sensor.

Images