Intelligent scene recognition method for real-time subtitle and translation of electronic whiteboard

Abstract

The invention belongs to the technical field of intelligent recognition, and particularly relates to an intelligent scene recognition method for real-time subtitles and translations of an electronic whiteboard, which comprises the following steps of: training a scene recognition model and a text recognition model by using a specific voice training set, and realizing mutual collaboration of output results in the two models. A training set is simplified, verification is carried out between output results of the two models, and the training precision of the two models is improved. The speech translation is further influenced by recognizing the speech of the speaker, filtering the noise in the scene and reducing the noise interference scene recognition, and the precise recognition of the scene of the target speech is realized through the mutual cooperation of the two models, so that the precise translation text is obtained according to different scenes, and the speech translation efficiency is improved. And the scene recognition and the text recognition corresponding to the scene recognition result are obtained between the two models, so that the error rate is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent recognition technology, and in particular relates to an intelligent scene recognition method for real-time subtitles and translation of an electronic whiteboard. Background Art

[0002] With the acceleration of globalization and the deepening of cultural integration, the use of multiple languages has become the norm. To overcome language barriers, online voice translation based on the user end has been widely used.

[0003] Online voice translation on an electronic whiteboard generally involves two steps. The first is voice recognition, which converts the voice signal in the first language input by the user into a first text. The second is online translation of the first text through machine translation to obtain a second text in the second language as the translation result. Finally, the second text information in the second language is displayed on the electronic whiteboard, completing the process of converting the voice in the first language into text in the second language.

[0004] In the existing solution, due to the high complexity of the scene, the scene of the speech is first identified, and then the speech in the specific scene is translated into the first text. This is a prerequisite. If the scene prediction is wrong, the subsequent speech recognition model will process the audio based on the wrong scene parameters, resulting in a significant increase in the translation error rate, and the error is difficult to correct in subsequent steps. This undoubtedly brings great challenges to the accuracy of scene recognition.

[0005] Moreover, due to the complexity of voice signals, diversity of language features and differences in environmental interference in different scenarios, the accuracy of voice recognition decreases, which in turn has a serious impact on the semantic fidelity, professional adaptability and user experience of subsequent translation. This is especially true in the real-time voice translation scenario of an electronic whiteboard, where there is a speaker speaking and other non-speakers are also talking. Although the discussion sounds in the audience do not affect the speaker's speech, the voice collection equipment will usually also collect the sound recognition of the discussion in the audience. The noise collected from the discussion in the audience will not only interfere with the misjudgment of the scene, but also affect the accuracy of the voice translation. Summary of the Invention

[0006] In order to solve the above-mentioned problems existing in the prior art, the present invention provides an intelligent scene recognition method for real-time subtitles and translation on an electronic whiteboard, which solves the problem that the complexity of voice signals, the diversity of language features and the differences in environmental interference in different scenarios lead to a decrease in voice recognition accuracy, which in turn seriously affects the semantic fidelity, professional adaptability and user experience of subsequent translation.

[0007] The purpose of the present invention can be achieved by the following technical solution: an intelligent scene recognition method for real-time subtitles and translation of an electronic whiteboard, comprising the following steps:

[0008] S1: Acquire the speaker's voice as the target voice;

[0009] S2: Extract the target speech features to obtain the speech features of the target speech;

[0010] S3: Inputting the speech feature into a preset scene recognition model to obtain a scene recognition result, inputting the speech feature and the scene recognition result into a preset text recognition model to obtain a first text recognition result based on scene recognition.

[0011] Preferably, confirming the speaker's voice includes:

[0012] Collect the speaker's timbre characteristics and microphone position in the scene, and collect the microphone sound source;

[0013] Based on the timbre characteristics of the speaker, the speaker's speech is separated from the acquired microphone sound source as the target speech.

[0014] Preferably, the scene recognition model is constructed based on a decision tree matching model to represent the mapping relationship between different scene types and speech in the first language;

[0015] The text recognition model uses a recurrent convolutional neural network to construct a mapping relationship between the speech of the first language and the first text in different scenario types.

[0016] Preferably, determining the scene model and the text recognition model includes:

[0017] extracting a first language speech training set from the first language speech data;

[0018] Using the first language speech training set to train a scene recognition model and a text recognition model respectively, inputting the first language speech training set into the text recognition model to obtain a first text recognition result, inputting the first language speech training set into the scene recognition model to obtain a scene recognition result, inputting the scene recognition result into the text recognition model, and having the text recognition model re-recognize the first speech training set according to the scene recognition result to obtain a second text recognition result;

[0019] The text recognition model and the scene recognition model are optimized with the goal of minimizing the gap between the first text recognition result and the second text recognition result.

[0020] Preferably, in step S1, the text recognition model includes a cascade of a recurrent neural network and a convolutional neural network, the convolutional neural network model constructs the relationship between speech features and text in different scenarios, and the recurrent neural network model obtains the correlation between speech features in different scenarios.

[0021] Preferably, the first language speech training set includes original speech in the first language, various scenes corresponding to the original speech, and a set of text annotations of the original speech in various scenes.

[0022] Preferably, in step S2, obtaining the first language speech training set further includes:

[0023] The acquired first language speech signal is divided into speech data frames of a set length, a Hamming window is added to the speech signal after framing, and MFCC features of the speech signal are extracted in each Hamming window.

[0024] Preferably, in step S1, the decision tree matching model is used to construct a scene recognition model, which includes N layers, wherein each node of the first layer is a speech signal frame, and each node of the second to Nth layers is a scene vocabulary corresponding to each speech signal frame. The scene vocabulary nodes in the Nth layer expression corresponding layer are obtained by mapping N speech signal frames, and the scene recognition model obtains the mapping probability between the speech signal frame nodes and the scene vocabulary nodes after training.

[0025] Preferably, in the scene recognition model, each of the speech data frames corresponds to a node, and the feature dimension is defined as D. In the 2nd to Nth layers, each of the nodes represents the vocabulary corresponding to the speech data frames corresponding to multiple scenes. The node data of the latter layer is generated by mapping the node data of the previous layer, and each prime vocabulary node of the Nth layer is generated by feature fusion of the first N-1 speech signal frames.

[0026] Preferably, in step S2, inputting the scene recognition result number into the text recognition model includes:

[0027] Feature extraction, normalizing the scene type features and speech features and unifying the scene feature dimensions and speech feature dimensions;

[0028] Generate weights and fuse the scene type feature vector and the speech feature vector to obtain a fused feature vector. The formula is:

[0029] F f =σ(W·S)⊙F a +[1-σ(W·S)]⊙F b ;

[0030] Among them, F f is the fusion feature vector, F a and F b are the scene type feature vector and the speech feature vector respectively, σ is the Sigmoid function, S is the scene type vector predicted by the scene recognition model, and W is the learnable weight matrix of the text recognition model.

[0031] The beneficial effects of the present invention are:

[0032] By recognizing the speaker's voice, the noise in the scene is filtered out and the noise interference with the scene recognition is reduced, which further affects the translation of the voice.

[0033] A specific speech training set is also used to train the scene recognition model and the text recognition model respectively, and to achieve mutual coordination in the output results of the two models, simplify the training set, and train the text recognition model and the scene recognition model with the goal of minimizing the gap between the first text recognition result and the second text recognition result. The output results of the two models are used to verify each other to improve the training accuracy of the two models.

[0034] By combining the correspondence between speech and text, and between speech and scenes in different scenarios, we can get specific translation questions based on the speech in different scenarios, improve the adaptability of the same speech in different scenarios, and improve the translation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] To facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0036] Figure 1 This is a structural diagram of the method of the present invention. DETAILED DESCRIPTION

[0037] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0038] In real-time speech translation on electronic whiteboards, the semantics of the same speech content in different application scenarios can be ambiguous due to differences in professional fields, context, and user intent. For example, in an educational setting, when a teacher mentions "cell," it should be translated as "cell" in a biology class, but may refer to "battery" in a physics class, requiring the use of subject-specific terminology models to distinguish them. Similarly, in various technical conferences, "monitor" may be translated as "display" in hardware discussions and "monitoring system" in software monitoring scenarios, requiring the use of IT-specific translation models based on the attendees' context. Direct translation based on the corresponding meaning of the word in the first language can easily lead to translation ambiguity, so dynamic adaptation of domain-specific translation models is necessary to ensure subtitle accuracy.

[0039] In the real-time voice translation scenario of an electronic whiteboard, there is a scene where one speaker is speaking and other non-speakers are also whispering. However, the translation of the electronic whiteboard only needs to translate the speaker's voice in real time. Therefore, it is necessary to quickly locate the target voice in the noisy environment.

[0040] See also Figure 1 This embodiment provides an intelligent scene recognition method for real-time subtitles and translation on an electronic whiteboard, comprising the following steps:

[0041] S1: Acquire the speaker's voice as the target voice;

[0042] This includes first determining the speaker's position, which can be the set microphone's pickup position. In specific usage scenarios, the speaker's voice using the microphone can be limited to the target voice. In order to better limit the uniqueness of the speaker's identity, the speaker is identified using position, face recognition information, and timbre feature information. The timbre feature here is a feature extracted from the speaker's voice that can distinguish the timbre of different people in the acoustic field, such as MFCC features or using voiceprint embedding to distinguish the speaker's timbre features.

[0043] Identify the speaker based on the speaker's position, facial recognition information, and voice characteristics;

[0044] Based on the speaker's timbre characteristics, the speaker's voice is separated from the acquired sound source as the target voice. Before obtaining the target voice to be translated, the identity of the speaker to be translated is identified and determined based on the speaker's position, face recognition information and timbre characteristics. This allows for better separation of the voice to be translated from the subsequently acquired target voice with noise, reducing the difficulty of extraction.

[0045] S2: Obtain the target speech to be recognized, and extract the features of the target speech to obtain the speech features of the target speech;

[0046] S3: Input the speech features of the target speech into the scene recognition model to obtain the current scene type corresponding to the target speech, input the current scene type into the text recognition model, and output the first text recognition result. The first text recognition result here is the text recognition result of the target speech in its specific scene, which is used as the first text for electronic whiteboard translation. Subsequently, the first text is translated online through machine translation to obtain a second text in the second language as the translation result. Finally, the second text information in the second language is displayed on the electronic whiteboard, completing the process of converting the speech in the first language to the text in the second language. The scene model and text recognition model here use historical data to build and train the machine learning model.

[0047] Furthermore, to improve the prediction accuracy and synergy of the two learning models, building and training the scene model and text recognition model includes:

[0048] Step 1: Model construction, using a decision tree matching model to build a scene recognition model that maps the relationship between different scene types and the first language speech, and using a recurrent convolutional neural network to build a text recognition model that maps the relationship between the first language speech and the first text under different scene types, and cascade the scene recognition model and the text recognition model; wherein, the text recognition model includes a cascade between a recurrent neural network and a convolutional neural network, the convolutional neural network model builds the relationship between speech features and text under different scenes, and the recurrent neural network model obtains the correlation between speech features under different scenes

[0049] Specifically, a decision tree matching model is used to construct a scene recognition model, which includes N layers, wherein each node of the first layer is a speech signal frame, and each node from the second layer to the Nth layer is a scene vocabulary corresponding to each speech signal frame. The scene vocabulary nodes in the Nth layer are obtained by mapping N speech signal frames. After training, the scene recognition model obtains the mapping probability between the speech signal frame nodes and the scene vocabulary nodes.

[0050] Each speech data frame corresponds to a node, and the feature dimension is defined as D. In the 2nd to Nth layers, each node represents the vocabulary corresponding to the speech signal frames corresponding to multiple scenes. The node data of the latter layer is generated by mapping the node data of several nodes in the previous layer. Each prime vocabulary node in the Nth layer is generated by feature fusion of the previous N-1 speech signal frames.

[0051] Step 2: Model training:

[0052] S21: Get the first language speech training set:

[0053] The first language speech training set includes the original speech of the first language, various scenes corresponding to the original speech, and a set of text annotations of the original speech in various scenes.

[0054] That is, the acquired first language speech training set needs to be collected for the original speech of the first language in different scenarios, and includes text annotation sets of the original speech of the first language in different scenarios.

[0055] Among them, the scenarios for collecting the first language speech cover diverse speech samples of the target scenarios, such as multi-person conversations, different noise environments, changes in speech speed, etc., to achieve scenario diversification. When obtaining the first language speech, it is necessary to ensure that the original speech is tested in various scenarios, and the original speech here uses a standard audio format to sample the initial speech of the first language, so as to ensure that the obtained training set contains the mapping relationship between speech and scenarios, and also contains the text standards mapped by the speech in different scenarios.

[0056] S22: Use the first language speech training set to train the scene recognition model and text recognition model respectively:

[0057] S221: Obtaining the first text recognition result:

[0058] Input the first language speech training set into the text recognition model to obtain a first text recognition result. Here, the text recognition model is a mapping relationship between the first language speech and the first text in different scenarios. Here, the first language speech in the first language speech scenario is not determined. The first text recognition result with the first language speech as a variable is output.

[0059] S222: Inputting the first language speech training set into the scene recognition model to obtain a scene recognition result;

[0060] S223: Obtaining the second text recognition result:

[0061] The input of the text recognition model is converted into the first language speech signal with scene recognition results, which is input into the text recognition model and iteratively trained to obtain the second text recognition result.

[0062] S224: Training the text recognition model and the scene recognition model with the goal of minimizing the gap between the first text recognition result and the second text recognition result, iteratively optimizing to achieve a dynamic balance between the scene recognition model and the text recognition model;

[0063] By using a specific speech training set to train the scene recognition model and the text recognition model separately, and achieving mutual coordination in the output results of the two models, the training set is simplified. The text recognition model and the scene recognition model are trained with the goal of minimizing the gap between the first text recognition result and the second text recognition result. The output results of the two models are used to verify each other to improve the training accuracy of the two models. Through the mutual coordination between the two models, the scene of the target speech is accurately recognized, and then accurate translation text is obtained according to different scenarios. By obtaining the scene recognition and the text recognition corresponding to the scene recognition results between the two models, the error rate is reduced.

[0064] Specifically, in step S2, obtaining the first language speech training set further includes:

[0065] a: Data preprocessing

[0066] The acquired speech signal of the first language is divided into data frames of fixed length according to time. Because the signal needs to be truncated and distinguished according to the length of time, when the signal is truncated, high-frequency components will be introduced due to the sudden change of the signal edge (Gibbs phenomenon), resulting in false frequency components in the signal spectrum, which in turn affects the accuracy of subsequent feature recognition based on the signal spectrum. Therefore, it is necessary to add a Hamming window to the framed speech signal and extract MFCC (Mel-frequency cepstral coefficient) features to facilitate subsequent analysis of the target speech.

[0067] When adding a Hamming window, the window length should be determined based on the pitch period of the first language's speech signal to avoid insufficient spectral resolution due to a too short window period. Specifically, the window length of the Hamming window should be determined based on the pitch period of the target speech. This requires a comprehensive consideration of the pitch period and the characteristics of the window function in frequency domain analysis to balance time resolution and frequency domain resolution, effectively reducing spectral leakage and improving the accuracy of speech signal analysis.

[0068] Furthermore, in step S2, the scene recognition results are input into the text recognition model, including:

[0069] The scene type feature vector and the speech feature vector are fused to obtain a fused feature vector. Through the gating mechanism, the speech features and scene features are dynamically combined, including:

[0070] Feature extraction: To adjust the representation of scene information in the feature space to make it compatible with speech features, the scene type features and speech features are normalized. This normalization includes dimensionality and distribution consistency. The scene feature vector can be obtained by matrix multiplication in a linear transformation to obtain W·S, where S is the scene type vector predicted by the scene recognition model and W is the learnable weight matrix of the text recognition model. This unifies the dimensions of the scene feature vector and the speech feature vector.

[0071] Mapping the scene feature's own dimension to the speech feature dimension to achieve consistency in the feature dimensions of the two, and achieving dimensional compatibility between the scene feature and speech feature;

[0072] The generation of gated weights generates a dynamic set of weight vectors from the scene vector after dimensional alignment with the speech features, which are set as:

[0073] g=σ(W·S);

[0074] Among them, each element g in the weight vector seti Represents the speech feature F b The activation strength of the i-th dimension in , F b It can be the frame-level features extracted from the speech feature vector, 1-g i Represents the scene feature vector F a The activation strength, F a Through the features extracted in the scene recognition model, σ is the Sigmoid function, which maps the output value to [0,1];

[0075] Use the gate weights g and 1-g to weight the speech feature vector and scene feature vector respectively, and add them together to obtain the fusion feature vector F f , its mathematical expression is:

[0076] F f =g⊙F b +(1-g)⊙F b , where g⊙F b is the dominant part of speech, (1-g)⊙F b It is the dominant part of the scene type. i →1, the feature fusion vector mainly depends on the speech features. i →0, the feature fusion vector mainly depends on the scene type features;

[0077] By using a gating function to fuse scene type features and speech features, the expressive power of the subsequent text recognition model is improved, and the linear transformation relationship is used to fuse the dimensions of the scene feature vector and the dimensions of the speech feature vector to improve the accuracy of the model operation. By setting the fusion feature vector to the weight values between the speech features and scene features in different applicable environments, and by dynamically adjusting the weight values, dynamic coordination between speech recognition and scene recognition is achieved, thereby improving the robustness and adaptability of the model in complex applicable situations.

[0078] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as above in terms of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can, without departing from the scope of the technical solution of the present invention, make some changes or modifications to equivalent embodiments using the technical contents disclosed above. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. An intelligent scene recognition method for real-time captioning and translation on an electronic whiteboard, characterized by: The following steps are involved: S1: Acquire the speaker's voice as the target voice; S2: Extract the target speech features to obtain the speech features of the target speech; S3: Inputting the speech feature into a preset scene recognition model to obtain a scene recognition result, inputting the speech feature and the scene recognition result into a preset text recognition model to obtain a first text recognition result based on scene recognition.

2. The intelligent scene recognition method for real-time captioning and translation of an electronic whiteboard according to claim 1, characterized in that: Confirmation of speaker voice includes: Collect the speaker's timbre characteristics and microphone position in the scene, and collect the microphone sound source; Based on the timbre characteristics of the speaker, the speaker's speech is separated from the acquired microphone sound source as the target speech.

3. The intelligent scene recognition method for real-time captioning and translation of an electronic whiteboard according to claim 1, characterized in that: The scene recognition model is constructed based on a decision tree matching model to represent the mapping relationship between different scene types and speech in the first language; The text recognition model uses a recurrent convolutional neural network to construct a mapping relationship between the speech of the first language and the first text in different scenario types.

4. The intelligent scene recognition method for real-time captioning and translation of an electronic whiteboard according to claim 1, characterized in that: Determining the scene model and the text recognition model includes: extracting a first language speech training set from the first language speech data; Using the first language speech training set to train a scene recognition model and a text recognition model respectively, inputting the first language speech training set into the text recognition model to obtain a first text recognition result, inputting the first language speech training set into the scene recognition model to obtain a scene recognition result, inputting the scene recognition result into the text recognition model, and having the text recognition model re-recognize the first speech training set according to the scene recognition result to obtain a second text recognition result; The text recognition model and the scene recognition model are optimized with the goal of minimizing the gap between the first text recognition result and the second text recognition result.

5. The intelligent scene recognition method for real-time captioning and translation of an electronic whiteboard according to claim 1, characterized in that: In step S1, the text recognition model includes a cascade of a recurrent neural network and a convolutional neural network, wherein the convolutional neural network model constructs the relationship between speech features and text in different scenarios, and the recurrent neural network model obtains the correlation between speech features in different scenarios.

6. The intelligent scene recognition method for real-time captioning and translation of an electronic whiteboard according to claim 1, characterized in that: The first language speech training set includes original speech in the first language, various scenes corresponding to the original speech, and a text annotation set of the original speech in each scene.

7. The intelligent scene recognition method for real-time captioning and translation of an electronic whiteboard according to claim 3, characterized in that: In step S2, obtaining the first language speech training set further includes: The acquired first language speech signal is divided into speech data frames of a set length, a Hamming window is added to the speech signal after framing, and MFCC features of the speech signal are extracted in each Hamming window.

8. The intelligent scene recognition method for real-time captioning and translation of an electronic whiteboard according to claim 4, characterized in that: In step S1, the decision tree matching model is used to construct a scene recognition model, which includes N layers, wherein each node of the first layer is a speech signal frame, and each node of the second to Nth layers is a scene vocabulary corresponding to each speech signal frame. The scene vocabulary nodes in the Nth layer expression corresponding layer are obtained by mapping N speech signal frames. After training, the scene recognition model obtains the mapping probability between the speech signal frame nodes and the scene vocabulary nodes.

9. The intelligent scene recognition method for real-time captioning and translation of an electronic whiteboard according to claim 5, characterized in that: In the scene recognition model, each speech data frame corresponds to a node, and the feature dimension is defined as D. In the 2nd to Nth layers, each node represents the vocabulary corresponding to the speech data frames corresponding to multiple scenes. The node data of the latter layer is generated by mapping the node data of the previous layer. Each prime vocabulary node of the Nth layer is generated by feature fusion of the first N-1 speech signal frames.

10. The intelligent scene recognition method for real-time captioning and translation of an electronic whiteboard according to claim 1, characterized in that: In step S2, the scene recognition result is input into the text recognition model, including: Feature extraction, normalizing the scene type features and speech features and unifying the scene feature dimensions and speech feature dimensions; Generate weights and fuse the scene type feature vector and the speech feature vector to obtain a fused feature vector. The formula is: F f =σ(W·S)⊙F a +[1-σ(W·S)]⊙F b ; Among them, F f is the fusion feature vector, F a and F b are the scene type feature vector and the speech feature vector respectively, σ is the Sigmoid function, S is the scene type vector predicted by the scene recognition model, and W is the learnable weight matrix of the text recognition model.

Images