Intelligent scene recognition method for real-time captioning and translation of electronic whiteboard

Abstract

This invention belongs to the field of intelligent recognition technology, specifically relating to an intelligent scene recognition method for real-time captioning and translation on electronic whiteboards. It trains a scene recognition model and a text recognition model using a specific speech training set, achieving mutual synergy between the two models' outputs. This simplifies the training set and improves the training accuracy by verifying the outputs of both models. By recognizing the speaker's voice, it filters out background noise and reduces interference, further influencing speech translation. Furthermore, the cooperation between the two models achieves accurate scene recognition of the target speech, resulting in precise translated text based on different scenes. By obtaining scene recognition and the corresponding text recognition from the two models, the error rate is reduced.

Description

Technical Field

[0001] This invention belongs to the field of intelligent recognition technology, specifically relating to an intelligent scene recognition method for real-time subtitle and translation on electronic whiteboards. Background Technology

[0002] With the acceleration of globalization and the deepening of cultural integration, multilingual scenarios have become the norm. Among these, online voice translation based on the user's device has been widely used to overcome language barriers.

[0003] Online voice translation on interactive whiteboards generally involves two steps. The first is speech recognition, which converts the user's input speech signal in the first language into the first text. The second is online translation of the first text using machine translation to obtain the second text in the second language. Finally, the second text information in the second language is displayed on the interactive whiteboard, completing the process of converting speech in the first language into text in the second language.

[0004] In existing solutions, due to the high complexity of the scenarios, the speech scene must be identified first, and then the speech in the specific scenario must be translated into the first text. This has a prerequisite. If the scenario prediction is wrong, the subsequent speech recognition model will process the audio based on the wrong scenario parameters, which will lead to a significant increase in the translation error rate. Moreover, the error is difficult to correct in subsequent steps, which undoubtedly poses a great challenge to the accuracy of scenario recognition.

[0005] Furthermore, due to the complexity of speech signals, the diversity of language features, and the differences in environmental interference in different scenarios, the accuracy of speech recognition decreases, which in turn seriously affects the semantic fidelity, professional adaptability, and user experience of subsequent translation. This is especially true in real-time speech translation scenarios on electronic whiteboards, where one speaker is speaking while others who are not speakers are also talking. Although the noise from the discussion in the audience does not affect the speaker's speech, the equipment that collects the speech usually also captures the noise from the discussion in the audience. The noise from the discussion in the audience can not only interfere with the misjudgment of the scene, but also affect the accuracy of speech translation. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention provides an intelligent scene recognition method for real-time subtitles and translation on electronic whiteboards. This method solves the problem that the complexity of speech signals, the diversity of language features, and the differences in environmental interference in different scenarios lead to a decrease in speech recognition accuracy, which in turn seriously affects the semantic fidelity, professional adaptability, and user experience of subsequent translations.

[0007] The objective of this invention can be achieved through the following technical solution: an intelligent scene recognition method for real-time subtitle and translation on an electronic whiteboard, comprising the following steps:

[0008] S1: Obtain the speaker's voice as the target voice;

[0009] S2: Extract features from the target speech to obtain the speech features of the target speech;

[0010] S3: Input the speech features into a preset scene recognition model to obtain a scene recognition result. Input the speech features and scene recognition result into a preset text recognition model to obtain a first text recognition result based on scene recognition.

[0011] Preferably, confirming the speaker's voice includes:

[0012] Collect the speaker's vocal characteristics and microphone position in the scene, and collect the microphone sound source;

[0013] Based on the speaker's timbre characteristics, the speaker's speech is separated from the acquired microphone sound source and used as the target speech.

[0014] Preferably, the scene recognition model is constructed based on a decision tree matching model to represent the mapping relationship between different scene types and the speech of the first language;

[0015] The text recognition model uses a recurrent convolutional neural network to construct a mapping relationship between the first language's speech and the first text under different scene types.

[0016] Preferably, determining the scene model and the text recognition model includes:

[0017] Extract the first language speech training set from the first language speech data;

[0018] The first language speech training set is used to train the scene recognition model and the text recognition model respectively. The first language speech training set is input into the text recognition model to obtain the first text recognition result. The first language speech training set is input into the scene recognition model to obtain the scene recognition result. The scene recognition result is input into the text recognition model. The text recognition model re-recognizes the first speech training set according to the scene recognition result to obtain the second text recognition result.

[0019] The text recognition model and scene recognition model are optimized with the goal of minimizing the difference between the first text recognition result and the second text recognition result.

[0020] Preferably, in step S1, the text recognition model includes a cascaded recurrent neural network and a convolutional neural network. The convolutional neural network model constructs the relationship between speech features and text in different scenarios, while the recurrent neural network model obtains the correlation between speech features in different scenarios.

[0021] Preferably, the first language speech training set includes the original speech of the first language, various scenarios corresponding to the original speech, and a set of text annotations for the original speech in each scenario.

[0022] Preferably, in step S2, obtaining the first language speech training set further includes:

[0023] The acquired first language speech signal is segmented into speech data frames of a set length, and Hamming windows are added to the speech signal after segmentation. The MFCC features of the speech signal are extracted in each Hamming window.

[0024] Preferably, in step S1, the scene recognition model constructed using the decision tree matching model includes N layers, wherein each node in the first layer is a speech signal frame, and each node in the second to Nth layers is a scene word corresponding to each speech signal frame. The scene word node in the Nth layer is derived by mapping N speech signal frames. After training, the scene recognition model obtains the mapping probability between speech signal frame nodes and scene word nodes.

[0025] Preferably, in the scene recognition model, each speech data frame corresponds to a node, and the feature dimension is defined as D. In the second to the Nth layer, each node represents the words corresponding to the speech data frames of multiple scenes. The node data of the next layer is generated by mapping the node data of the previous layer. Each prime word node in the Nth layer is generated by feature fusion of the first N-1 speech signal frames.

[0026] Preferably, in step S2, inputting the scene recognition result number into the text recognition model includes:

[0027] Feature extraction involves normalizing the scene type features and speech features, and unifying the dimensions of the scene features and speech features.

[0028] Generate weights by fusing the scene type feature vector and the speech feature vector to obtain a fused feature vector, using the following formula:

[0029] F f =σ(W·S)⊙F a +[1-σ(W·S)]⊙F b ;

[0030] Among them, F f To fuse feature vectors, F a and F b These are the scene type feature vector and the speech feature vector, respectively. σ is the Sigmoid function, S is the scene type vector predicted by the scene recognition model, and W is the learnable weight matrix of the text recognition model.

[0031] The beneficial effects of this invention are as follows:

[0032] By recognizing the speaker's voice, filtering out background noise and reducing noise interference, the speech translation is further influenced.

[0033] Furthermore, a specific speech training set was used to train the scene recognition model and the text recognition model separately, and the output results of the two models were mutually coordinated. The training set was simplified, and the text recognition model and the scene recognition model were trained with the goal of minimizing the difference between the first text recognition result and the second text recognition result. The output results of the two models were used to verify each other, thereby improving the training accuracy of the two models.

[0034] By combining the correspondence between speech and text, and between speech and scene, in different scenarios, specific translation questions can be obtained based on speech in different scenarios, thereby improving the adaptability of the same speech in different scenarios and increasing the translation accuracy. Attached Figure Description

[0035] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0036] Figure 1 This is a block diagram of the method structure of the present invention. Detailed Implementation

[0037] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description of the specific implementation methods, structures, features and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0038] In real-time speech translation scenarios on electronic whiteboards, the semantics of the same speech content may become ambiguous in different application scenarios due to differences in professional fields, context, and user intent. For example, in an educational setting, when a teacher mentions "cell," it should be translated as "cell" in a biology class, but might refer to "battery" in a physics class, requiring the use of a subject-specific terminology model to differentiate. Similarly, in various technical conferences, "monitor" is translated as "display" in hardware discussions but as "monitoring system" in software monitoring scenarios, requiring the use of different IT-specific translation quotations based on the participants' backgrounds. Simply translating according to the meaning of the words in the first language can easily lead to ambiguity; therefore, dynamically adapting to a domain-specific translation model is necessary to ensure the accuracy of the subtitles.

[0039] In real-time speech translation scenarios on electronic whiteboards, there exists a situation where one speaker is speaking while others who are not speakers are also speaking in hushed tones. However, the translation function of the electronic whiteboard only needs to translate the speaker's speech in real time. Therefore, it is necessary to quickly locate the target speech based on the speaker's position in a noisy environment.

[0040] Please see Figure 1 This embodiment provides an intelligent scene recognition method for real-time captioning and translation on electronic whiteboards, including the following steps:

[0041] S1: Obtain the speaker's voice as the target voice;

[0042] This includes first determining the speaker's location, which can be the microphone's pickup position. In specific usage scenarios, the speaker's voice can be limited to the microphone. To better ensure the uniqueness of the speaker's identity, location, facial recognition information, and voice feature information are used to identify the speaker. Here, voice feature refers to the features extracted from the speaker's voice that can distinguish the voices of different people in the acoustic field, such as MFCC features or voiceprint embedding methods to distinguish the speaker's voice features.

[0043] The speaker is identified by matching their location, facial recognition information, and voice characteristics.

[0044] Based on the speaker's timbre characteristics, the speaker's voice is separated from the acquired sound source as the target voice. Before acquiring the target voice that needs to be translated, the speaker's identity is identified based on the speaker's location, facial recognition information, and timbre characteristics. This enables better separation of the voice that needs to be translated from the target voice that contains noise in the subsequent acquisition, reducing the difficulty of extraction.

[0045] S2: Obtain the target speech to be recognized, and extract its features to obtain the speech features of the target speech;

[0046] S3: Input the speech features of the target speech into the scene recognition model to obtain the current scene type corresponding to the target speech. Input the current scene type into the text recognition model to output the first text recognition result. The first text recognition result here is the text recognition result of the target speech in its specific scene, which serves as the first text for translation on the electronic whiteboard. Subsequently, the first text is translated online by machine translation to obtain the second text in the second language as the translation result. Finally, the second text information in the second language is displayed on the electronic whiteboard, completing the process of converting speech in the first language into text in the second language. The scene model and text recognition model here are generated by building and training the machine learning model using historical data.

[0047] Furthermore, to improve the accuracy and synergy of the predictions from the two learning models, the construction and training of the scene model and the text recognition model include:

[0048] Step 1: Model Construction. A scene recognition model is constructed using a decision tree matching model to establish the mapping relationship between different scene types and the first language speech. A text recognition model is constructed using a recurrent convolutional neural network to establish the mapping relationship between the first language speech and the first text under different scene types. The scene recognition model and the text recognition model are then cascaded. Specifically, the text recognition model involves a cascaded recurrent neural network and a convolutional neural network. The convolutional neural network model establishes the relationship between speech features and text under different scenes, while the recurrent neural network model obtains the correlation between speech features under different scenes.

[0049] Specifically, the scene recognition model constructed using the decision tree matching model includes N layers. In the first layer, each node is a speech signal frame, and in the second to Nth layers, each node is a scene word corresponding to each speech signal frame. The scene word node in the Nth layer is derived from the mapping of N speech signal frames. After training, the scene recognition model obtains the mapping probability between speech signal frame nodes and scene word nodes.

[0050] Each speech data frame corresponds to a node, and the feature dimension is defined as D. In layers 2 to N, each node represents the words corresponding to multiple speech signal frames in different scenarios. The node data of the next layer is generated by mapping the node data of the previous layer. Each prime word node in layer N is generated by feature fusion of the first N-1 speech signal frames.

[0051] Step 2: Model Training

[0052] S21: Obtain the first language speech training set:

[0053] The first language speech training set includes the original speech of the first language, various scenarios corresponding to the original speech, and a set of text annotations for the original speech in each scenario.

[0054] That is, the acquired first language speech training set needs to collect the original speech of the first language in different scenarios, and include the text annotation set of the original speech of the first language in different scenarios.

[0055] Among them, the scenarios for collecting first language speech are diverse speech samples covering the target scenarios, such as multi-person dialogues, different noise environments, and speech rate variations, to achieve scenario diversity. When acquiring first language speech, it is necessary to ensure that the original speech is tested in various scenarios. The original speech here uses a standard audio format to sample the initial speech of the first language, thereby ensuring that the acquired training set contains the mapping relationship between speech and scenarios, as well as the text standard mapped by the speech in different scenarios.

[0056] S22: Train the scene recognition model and the text recognition model separately using the first language speech training set:

[0057] S221: Obtain the first text recognition result:

[0058] The first language speech training set is input into the text recognition model to obtain the first text recognition result. Here, the text recognition model is a mapping relationship between the first language speech and the first text in different scenarios. The first language speech in the scenario is not explicitly stated in the text recognition model, and the output is the first text recognition result with the first language speech as the variable.

[0059] S222: Input the first language speech training set into the scene recognition model to obtain the scene recognition result;

[0060] S223: Obtain the second text recognition result:

[0061] The input to the text recognition model is transformed into a first-language speech signal with scene recognition results, which is then input into the text recognition model for iterative training to obtain the second text recognition result.

[0062] S224: Train the text recognition model and the scene recognition model with the goal of minimizing the difference between the first text recognition result and the second text recognition result, and iteratively optimize them to obtain a dynamic balance between the scene recognition model and the text recognition model.

[0063] By using a specific speech training set, a scene recognition model and a text recognition model are trained separately. The output results of the two models are then synergistically evaluated, simplifying the training set. The goal is to minimize the difference between the first and second text recognition results. The outputs of the two models are then compared to improve their training accuracy. Through the cooperation between the two models, accurate scene recognition of the target speech is achieved, leading to precise translations based on different scenes. Furthermore, by obtaining scene recognition and the corresponding text recognition results from both models, the error rate is reduced.

[0064] Specifically, in step S2, obtaining the first language speech training set also includes:

[0065] a: Data preprocessing

[0066] The acquired speech signal in the first language is divided into fixed-length data frames according to time. Because the signal needs to be truncated and distinguished according to the length of time, when the signal is truncated, high-frequency components (Gibbs phenomenon) will be introduced due to abrupt changes at the signal edges, resulting in false frequency components in the signal spectrum, which will affect the accuracy of subsequent feature recognition based on the signal spectrum. Therefore, it is necessary to add a Hamming window to the segmented speech signal and extract MFCC (Mel-frequency cepstral coefficients) features to facilitate subsequent analysis of the target speech.

[0067] When adding a Hamming window, the window length should be determined based on the pitch period of the first language speech signal to avoid insufficient spectral resolution caused by an excessively short window period. Specifically, determining the Hamming window length based on the pitch period of the target speech requires comprehensive consideration of the characteristics of the pitch period and the window function in frequency domain analysis, balancing time resolution and frequency domain resolution to effectively reduce spectral leakage and improve the accuracy of speech signal analysis.

[0068] Furthermore, in step S2, the scene recognition results are input into the text recognition model, including:

[0069] The scene type feature vector and the speech feature vector are fused to obtain a fused feature vector. Through a gating mechanism, the speech features and scene features are dynamically combined, including:

[0070] Feature extraction involves adjusting the representation of scene information in the feature space to make it compatible with speech features. This standardization process involves ensuring consistency in dimensionality and distribution. The scene feature vector can be transformed into W·S through matrix multiplication in linear transformation, where S is the scene type vector predicted by the scene recognition model and W is the learnable weight matrix of the text recognition model. This process unifies the dimensionality between the scene feature vector and the speech feature vector.

[0071] Mapping the scene features to the speech features dimension achieves consistency between the two features in terms of dimension, thus achieving dimensional compatibility between scene features and speech features.

[0072] The generation of gating weights involves aligning the scene vector with the speech features in terms of dimensions to generate a dynamic set of weight vectors, set as follows:

[0073] g = σ(W·S);

[0074] Where each element g in the weight vector seti Representing speech features F b The activation strength of the i-th dimension, F b It can be a frame-level feature extracted from the speech feature vector, 1-g i Represents the scene feature vector F a Activation intensity, F a By extracting features from the scene recognition model, σ is the Sigmoid function, and the output value is mapped to [0,1].

[0075] The speech feature vector and the scene feature vector are weighted using g and 1-g respectively, and then summed to obtain the fused feature vector F. f Its mathematical expression is:

[0076] F f =g⊙F b +(1-g)⊙F b , where g⊙F b For the dominant phonological part, (1-g)⊙F b As the scene type dominates the part, in g i When →1, the feature fusion vector mainly depends on speech features, in g i When the value approaches 0, the feature fusion vector mainly depends on the scene type feature.

[0077] By using a gating function to fuse scene type features and speech features, the expressive power of the subsequent text recognition model is improved. A linear transformation relationship is used to fuse the dimensions of the scene feature vector and the speech feature vector, thereby improving the accuracy of the model operation. By setting the weight values ​​between speech features and scene features in different applicable environments through the fused feature vector, dynamic adjustment of the weight values ​​is achieved to realize dynamic synergy between speech recognition and scene recognition, thereby improving the robustness and adaptability of the model in complex applicable situations.

[0078] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An intelligent scene recognition method for real-time subtitle and translation on an electronic whiteboard, characterized in that: Includes the following steps: S1: Obtain the speaker's voice as the target voice; S2: Extract features from the target speech to obtain the speech features of the target speech; S3: Input the speech features into a preset scene recognition model to obtain a scene recognition result; input the speech features and scene recognition result into a preset text recognition model to obtain a first text recognition result based on scene recognition. The scene recognition model is built based on a decision tree matching model and is used to represent the mapping relationship between different scene types and the speech of the first language. The text recognition model uses a recurrent convolutional neural network to construct a mapping relationship between the first language's speech and the first text under different scene types; Determining the scene recognition model and text recognition model includes: Extract the first language speech training set from the speech data of the first language; The scene recognition model and the text recognition model are trained using the first language speech training set, respectively. The first language speech training set is input into the text recognition model to obtain a first text recognition result. The first language speech training set is input into the scene recognition model to obtain a scene recognition result. The scene recognition result is input into the text recognition model. The text recognition model re-recognizes the first language speech training set according to the scene recognition result to obtain a second text recognition result. The text recognition model and scene recognition model are optimized with the goal of minimizing the difference between the first text recognition result and the second text recognition result; The scene recognition model constructed using the decision tree matching model includes N layers, where each node in the first layer is a speech signal frame, and each node in the second to Nth layers is a scene word corresponding to each speech signal frame. The scene word node in the Nth layer is derived by mapping N speech signal frames. After training, the scene recognition model obtains the mapping probability between speech signal frame nodes and scene word nodes.

2. The intelligent scene recognition method for real-time subtitle and translation on an electronic whiteboard according to claim 1, characterized in that: Obtaining the speaker's voice includes: Collect the speaker's vocal characteristics and microphone position in the scene, and collect the microphone sound source; Based on the speaker's timbre characteristics, the speaker's speech is separated from the acquired microphone sound source and used as the target speech.

3. The intelligent scene recognition method for real-time subtitle and translation on an electronic whiteboard according to claim 1, characterized in that: The text recognition model comprises a cascaded recurrent neural network and a convolutional neural network. The convolutional neural network model constructs the relationship between speech features and text in different scenarios, while the recurrent neural network model obtains the correlation between speech features in different scenarios.

4. The intelligent scene recognition method for real-time subtitle and translation on an electronic whiteboard according to claim 1, characterized in that: The first language speech training set includes the original speech of the first language, various scenarios corresponding to the original speech, and a set of text annotations for the original speech in each scenario.

5. The intelligent scene recognition method for real-time subtitle and translation on an electronic whiteboard according to claim 1, characterized in that: Obtaining the first language speech training set also includes: The acquired speech signal in the first language is segmented into speech data frames of a set length, and Hamming windows are added to the speech signal after segmentation. The MFCC features of the speech signal are extracted in each Hamming window.

6. The intelligent scene recognition method for real-time subtitle and translation on an electronic whiteboard according to claim 3, characterized in that: In the scene recognition model, each speech signal frame corresponds to a node, and the feature dimension is defined as D. In the second to the Nth layer, each node represents the words corresponding to the speech signal frames of multiple scenes. The node data of the next layer is generated by mapping the node data of the previous layer. Each word node of the Nth layer is generated by feature fusion of the first N-1 speech signal frames.

7. The intelligent scene recognition method for real-time subtitle and translation on an electronic whiteboard according to claim 1, characterized in that: The scene recognition results are input into the text recognition model, including: Feature extraction involves normalizing the scene type features and speech features, and unifying the dimensions of the scene features and speech features. Generate weights by fusing the scene type feature vector and the speech feature vector to obtain a fused feature vector, using the following formula: ; in, To fuse feature vectors, and These are scene type feature vectors and speech feature vectors, respectively. Here, S is the sigmoid function, S is the scene type vector predicted by the scene recognition model, and W is the learnable weight matrix of the text recognition model.

Images