A method and system for multilingual speech content recognition

Abstract

This application relates to a multilingual speech content recognition method and system, belonging to the field of speech signal processing technology. The recognition method includes: acquiring raw audio stream data and performing noise reduction filtering to segment it into multiple audio segments; extracting acoustic feature vectors from the audio segments; inputting the audio segments into a speech recognition model group to obtain text segments and confidence scores; fusing all text segments to generate preliminary recognized text and segmenting it into text blocks, combining real-time received semantic feedback tags, and translating it block by block into target language text through a streaming translation model; performing bidirectional translation verification on text blocks with confidence scores below a set threshold; inputting the translated text stream and acoustic feature vectors into a semantic analysis model, fusing the output results to obtain semantic feedback tags; constructing a cross-modal association graph through a graph neural network, outputting key information summaries, and triggering alarm signals. This application can improve speech recognition accuracy while ensuring recognition efficiency and perform risk response.

Description

Technical Field

[0001] This application relates to the field of speech signal processing technology, and in particular to a method and system for multilingual speech content recognition. Background Technology

[0002] With the acceleration of globalization and the diversification of multimedia information dissemination methods, multilingual voice data from conferencing systems, social media live streaming, remote collaboration platforms, and security monitoring equipment has experienced explosive growth. How to efficiently and accurately extract structured semantic information from continuous audio streams in complex real-world scenarios and achieve cross-language understanding and key event recognition has become one of the core challenges in the field of intelligent voice processing.

[0003] Currently, traditional speech recognition and translation systems mostly employ a serial architecture, where speech is first converted to text, and then translation and semantic analysis are performed independently. The lack of effective collaboration between modules leads to significant limitations in addressing real-world problems such as noise interference, accent differences, language mixing, and semantic ambiguity. In practical applications, the raw audio stream is often accompanied by strong background noise, such as traffic noise, air conditioning sounds, and overlapping speech from multiple people. These non-speech components severely interfere with the energy distribution and time-frequency characteristics of the speech signal. Furthermore, due to significant acoustic differences between different languages, dialects, and even individual pronunciation habits, a single speech recognition model struggles to maintain stable recognition performance in multilingual scenarios. Especially when dealing with low-resource languages ​​or regional accents, recognition accuracy drops sharply, impacting subsequent processing efficiency. This is particularly problematic in responding to sudden public opinion events or potential violations, hindering rapid response and the ability to proactively trigger appropriate risk warnings. Summary of the Invention

[0004] To address the aforementioned technical issues, this application provides a multilingual speech content recognition method and system.

[0005] Firstly, this application provides a multilingual speech content recognition method, which adopts the following technical solution:

[0006] A multilingual speech content recognition method, the recognition method comprising:

[0007] The system acquires raw audio stream data and performs noise reduction filtering. It then segments the audio stream into multiple audio segments using a silence detection algorithm and generates a timestamp for each audio segment.

[0008] Extract the acoustic feature vector of the audio segment;

[0009] The audio segment is input into a group of speech recognition models deployed in parallel, and the text segment and confidence score output by each speech recognition model are obtained.

[0010] The weighted weights are calculated based on the confidence scores, and all text fragments are merged to generate preliminary identified text with timestamps.

[0011] The initially identified text is segmented into text blocks according to the timestamp, and combined with the semantic feedback tags received in real time, it is translated into target language text block by block through a streaming translation model;

[0012] Based on the target language text translated block by block, bidirectional translation verification is performed on text blocks with confidence scores below a set threshold to generate a translated text stream.

[0013] The translated text stream and the acoustic feature vector are input into the semantic analysis model to calculate the sentiment polarity score, sensitivity level and keyword set, and then the semantic feedback label is obtained and fed back to the streaming translation model.

[0014] By fusing the acoustic feature vectors and semantic feedback labels, a cross-modal association graph is constructed using a graph neural network, and a timestamped summary of key information is output.

[0015] An alarm signal is triggered based on the key information summary and pushed to an external system.

[0016] By adopting the above technical solutions, high-precision recognition of multilingual speech content, cross-modal semantic understanding, and risk warning functions have been achieved. The system realizes modal complementarity and information reuse in multiple stages. For example, acoustic features are used for both recognition and sentiment correction, semantic feedback labels drive translation optimization, and graph neural networks integrate text and acoustic information to form a high-order semantic structure, fully embodying a hybrid intelligent paradigm of "data-driven + knowledge-guided." This technical solution improves recognition accuracy and translation quality while constructing an intelligent speech analysis platform with self-regulating capabilities suitable for complex real-world scenarios, particularly applicable to high-value application areas such as broadcast media public opinion monitoring, public safety early warning, and multinational corporate compliance audits.

[0017] Secondly, this application provides a multilingual speech content recognition system, which adopts the following technical solution:

[0018] A multilingual speech content recognition system, the recognition system comprising:

[0019] The acquisition and processing module is used to acquire raw audio stream data and perform noise reduction filtering, segment it into multiple audio segments using a silence detection algorithm, and generate a timestamp for each audio segment.

[0020] The feature extraction module is used to extract the acoustic feature vector of the audio segment;

[0021] The speech recognition module is used to input the audio segment into a group of speech recognition models deployed in parallel, and obtain the text segment and confidence score output by each speech recognition model;

[0022] The weighted fusion module is used to calculate the weighted weights based on the confidence scores and fuse all text fragments to generate preliminary identified text with timestamps;

[0023] The streaming translation module is used to segment the initially identified text into text blocks according to the timestamp, and translate them into target language text block by block through the streaming translation model, combined with the semantic feedback tags received in real time.

[0024] The translation verification module is used to perform bidirectional translation verification on text blocks whose confidence scores are lower than a set threshold based on the target language text translated block by block, and generate a translated text stream;

[0025] The semantic feedback module is used to input the translated text stream and the acoustic feature vector into the semantic analysis model, calculate the sentiment polarity score, sensitivity level and keyword set, fuse them to obtain semantic feedback labels and feed them back to the streaming translation model;

[0026] The summary output module is used to fuse the acoustic feature vectors and semantic feedback labels, construct a cross-modal association graph through a graph neural network, and output a summary of key information with timestamps.

[0027] The alarm module is used to trigger alarm signals based on the key information digest and push them to external systems.

[0028] Thirdly, this application provides a computer-readable storage medium, which adopts the following technical solution:

[0029] A computer-readable storage medium storing a computer program that can be loaded by a processor and executed as in any of the methods in the first aspect. Attached Figure Description

[0030] Figure 1 This is a first flowchart illustrating a multilingual speech content recognition method according to one embodiment of this application.

[0031] Figure 2 This is a second flowchart illustrating a multilingual speech content recognition method according to one embodiment of this application.

[0032] Figure 3 This is a schematic diagram of the third process of a multilingual speech content recognition method according to one embodiment of this application.

[0033] Figure 4 This is a schematic diagram of the fourth process of a multilingual speech content recognition method according to one embodiment of this application.

[0034] Figure 5 This is a schematic diagram of the fifth process of a multilingual speech content recognition method according to one embodiment of this application.

[0035] Figure 6 This is a schematic diagram of the sixth process of a multilingual speech content recognition method according to one embodiment of this application.

[0036] Figure 7 This is a schematic diagram of the seventh process of a multilingual speech content recognition method according to one embodiment of this application. Detailed Implementation

[0037] To make the purpose, technical solution, and advantages of this application clearer, the following description is provided in conjunction with the appendix. Figure 1-7 The present application will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the application.

[0038] This application discloses a method for multilingual speech content recognition.

[0039] Reference Figure 1 A multilingual speech content recognition method, the recognition method includes:

[0040] Step S101: Collect the original audio stream data and perform noise reduction filtering. Segment the audio into multiple audio segments using a silence detection algorithm and generate a timestamp for each audio segment.

[0041] The core objective of this step is to transform the continuous, noisy raw audio signal into a clearly structured, discrete speech unit that can be processed by subsequent modules. In practical applications, the raw audio stream typically comes from remote conferencing systems, public address systems, social media live streams, or security monitoring equipment, and its sampling rate is generally 16kHz or higher to ensure the integrity of the voice information.

[0042] Since background noise (such as traffic noise, air conditioning noise, and conversations between multiple people) is common in real-world scenarios, direct speech recognition will significantly reduce accuracy. Therefore, it is necessary to first use noise reduction and filtering methods in digital signal processing technology, such as spectral subtraction, Wiener filtering, or time-frequency masking models based on deep learning, to suppress non-speech components in the audio spectrum.

[0043] Furthermore, after noise reduction, the system can use the Voice Activity Detection (VAD) algorithm to determine the start and end points of speech activity, thereby segmenting long audio segments into several fragments containing valid speech. The VAD algorithm is typically implemented based on energy thresholds, zero-crossing rates, or more advanced neural network models, with the goal of avoiding over-segmentation while preserving complete semantic units.

[0044] At the same time, the system adds a precise timestamp to each audio segment, recording its start and end times in the original audio stream (e.g., "00:01:23.450–00:01:27.890"). This time information is not only used for subsequent text and speech alignment, but also provides a spatiotemporal location basis for the key information summary in the final output, ensuring that the output of the entire system is traceable and has temporal consistency.

[0045] Step S102: Extract the acoustic feature vector of the audio segment;

[0046] The acoustic eigenvectors include Mel frequency cepstral coefficients, fundamental frequency, and energy value;

[0047] Specifically, the construction of acoustic feature vectors relies on the short-time analysis theory of speech signals, which involves windowing each audio segment with a frame length of 20–30 milliseconds (commonly Hamming window) and setting overlaps between frames (e.g., 10 milliseconds) to enhance feature continuity.

[0048] Based on this, the system extracts three core features: Mel-frequency cepstral coefficients (MFCC), fundamental frequency (F0), and energy. MFCC simulates the nonlinear characteristics of human hearing's perception of different frequencies. By mapping a linear spectrum to the Mel scale and performing a discrete cosine transform, a set of low-dimensional and highly discriminative feature parameters is obtained, which is widely used in speech recognition tasks as the basic input for phoneme classification. Fundamental frequency reflects the frequency of vocal cord vibration and is an important clue for judging intonation, emotional state (such as anger or calmness), and the speaker's gender, especially crucial for semantic discrimination in tonal languages ​​such as Chinese. Energy represents the amplitude intensity of the speech signal and can be used to distinguish between voiced and unvoiced sounds, and between emphasized and weakly pronounced words.

[0049] These three types of features together constitute a multi-dimensional vector sequence, which not only preserves the segmental information of speech but also contains prosodic features, providing rich underlying representation support for subsequent speech recognition, sentiment analysis, and cross-modal modeling.

[0050] Step S103: Input the audio segment into the parallel deployed speech recognition model group and obtain the text segment and confidence score output by each speech recognition model;

[0051] This design breaks through the limitations of traditional single ASR (Automatic Speech Recognition) systems by introducing the concept of model ensemble, aiming to improve recognition robustness and adaptability to complex and ever-changing language environments. Parallel deployment refers to multiple heterogeneous or homogeneous speech recognition models running simultaneously on the same input audio. Examples include end-to-end models based on CTC (Connectionist Temporal Classification) (such as DeepSpeech), sequence-to-sequence models based on Transformer (such as Conformer), and sub-models specifically optimized for dialects (such as models dedicated to Cantonese and Sichuanese).

[0052] It's important to note that the model selection is not random, but rather a dynamic decision based on audio spectral characteristics: by rapidly classifying the input audio's MFCC mean, frequency band energy distribution, or pitch contour, the system can determine whether it belongs to a specific dialect region or accent type, thereby activating the corresponding dedicated model for recognition. Each model, while outputting the transcribed text, also returns a confidence score, typically calculated from the probability sequence output by the model's last softmax layer using a weighted average or maximum path probability, to measure the reliability of the text result. This multi-model collaborative mechanism effectively mitigates the recognition bias of single models in terms of accent, speech rate, or domain vocabulary, and is particularly suitable for cross-regional, multilingual speech content recognition scenarios.

[0053] Step S104: Calculate the weighted weight based on the confidence score, and merge all text fragments to generate preliminary identified text with timestamps;

[0054] Specifically, the fusion weights are calculated based on the confidence scores of each model, and all text segments are aligned by timestamp. The fused text with a global timestamp is then generated based on the fusion weights.

[0055] This fusion process is not a simple voting or splicing process, but rather employs a weighted strategy based on confidence level squared normalization. The specific formula for calculating the weights is as follows: , where Conf i This represents the confidence score of the i-th model. The mathematical significance of this formula lies in amplifying the influence of high-confidence models while suppressing the interference of low-confidence models. The squaring operation makes the weight distribution more concentrated on the best-performing model, avoiding misjudgments caused by multiple medium-confidence models jointly dominating the market.

[0056] In this embodiment of the application, the timestamp alignment step includes: detecting speech overlap intervals between adjacent text segments, aligning phoneme boundaries in the overlap intervals using a dynamic time warping algorithm, and directly inheriting the original timestamp markers for non-overlapping intervals.

[0057] Furthermore, within the same timestamp interval, candidate texts are weighted and voted on based on fusion weights; when the confidence score of the text with the highest weight exceeds 0.8, it is directly adopted; otherwise, a cross-model verification process based on phoneme similarity is triggered. The final generated preliminary recognized text not only contains complete sentence content but also inherits the timestamp information of the original audio segment, forming a mapping relationship of "time interval - text content," laying the foundation for subsequent text segmentation and streaming translation by time window.

[0058] Step S105: The initially identified text is segmented into text blocks according to the timestamp, and combined with the semantic feedback tags received in real time, it is translated into target language text block by block through a streaming translation model;

[0059] In this model, text blocks are typically continuous text units divided according to semantic boundaries (such as sentence termination marks) or fixed durations (such as segments every 5 seconds), ensuring that the translation process maintains both contextual coherence and meets the requirements of low-latency real-time performance. Streaming translation models differ from traditional whole-sentence translation architectures. They employ an incremental decoding strategy, enabling partial translation output even before the input has fully finished. Typical structures include attention-based Transformer-Transducer or Recurent Neural Network Transducer (RNN-T), suitable for applications such as simultaneous conference interpretation and real-time captioning.

[0060] More importantly, the translation process is not isolated but dynamically incorporates "semantic feedback labels" as context-adjusting factors. These labels, derived from the system's internal closed-loop feedback mechanism, contain information such as sentiment polarity correction coefficients and sensitivity level encodings, which can be used to adjust translation strategies. For example, when a high sensitivity level is detected, a more conservative word substitution mechanism is automatically activated, or the tone intensity is increased to preserve the original meaning when the sentiment is negative. This translation approach, which integrates external semantic signals, allows the system to transcend the limitations of formal equivalence in traditional machine translation, moving towards functional equivalence and improving semantic fidelity in cross-language communication.

[0061] Step S106: Based on the block-by-block translated target language text, perform bidirectional translation verification on text blocks with confidence scores below a set threshold to generate a translated text stream;

[0062] Among them, bidirectional translation verification refers to the process of translating the source language → target language translation result back into the target language → source language, and then comparing the semantic consistency between the back-translated text and the original text.

[0063] For example, if the original sentence is "This project has significant compliance risks," after being translated into English as "This project has significant compliance risks," and then translated back into Chinese as "This project has serious compliance issues," the system assesses the similarity between the two Chinese texts by calculating the cosine similarity of their semantic vectors. If this value is lower than a preset threshold (e.g., 0.85), it is determined that there is potential semantic drift, triggering a manual review process. This mechanism is particularly suitable for high-risk fields such as law, medicine, and finance, and can effectively identify translation errors caused by ambiguous words, cultural differences, or model illusions.

[0064] It should be noted that the "translated text stream" generated in this step is not a simple concatenation result, but a structured data stream that has undergone verification, filtering, anomaly marking, and version management. It has auditable and traceable engineering attributes, providing high-quality input for subsequent semantic analysis.

[0065] Step S107: Input the translated text stream and acoustic feature vector into the semantic analysis model, calculate the sentiment polarity score, sensitivity level and keyword set, fuse them to obtain semantic feedback labels and feed them back to the streaming translation model;

[0066] This step marks a significant advancement in the system's understanding, moving from "speech-to-text" to a deeper level of "text-to-semantic analysis." Semantic analysis models typically employ a multimodal fusion architecture, such as concatenating or interacting with the text semantic vectors extracted by BERT-like pre-trained language models at a high level, thereby achieving joint modeling of "speech content" and "speech style."

[0067] Specifically, the sentiment polarity score is based on a text sentiment dictionary or the output of a pre-trained classifier, and can be dynamically adjusted using an acoustic correction mechanism. For example, when a speaker's speaking speed increases, pitch rises, and energy increases, the system automatically enhances its assessment of sentiment intensity, avoiding underestimation of intense emotions based solely on the surface meaning of the text. Sensitivity levels are determined through a combination of a rule engine and deep learning, covering dimensions such as politically sensitive words, violent tendencies, and privacy breaches, and outputting a graded code (e.g., 0–5). The keyword set is generated using TF-IDF, TextRank, or BERT keyword extraction algorithms to identify core topics.

[0068] Furthermore, these outputs are encapsulated as "semantic feedback labels" in the form of quadruples: {sentiment polarity correction coefficient, sensitivity level encoding, keyword hash value, timestamp interval}, which are then re-injected into the streaming translation model as closed-loop feedback signals to form a dynamic optimization loop.

[0069] Step S108: Integrate acoustic feature vectors and semantic feedback labels, construct a cross-modal association graph through a graph neural network, and output a summary of key information with timestamps;

[0070] Specifically, the system uses timestamps as nodes and each text block, along with its corresponding acoustic feature vector and semantic feedback label, as node attributes to construct a temporal semantic graph. Edge connections in the graph are established based on semantic similarity (e.g., keyword overlap, consistent sentiment trends) or temporal proximity. Subsequently, a multi-head graph attention mechanism is used to calculate the association weights between nodes, enabling high-impact events (e.g., sudden, highly sensitive statements) to propagate and influence surrounding nodes through attention.

[0071] In this embodiment, the system can form a "high-risk event cluster" by aggregating all nodes with a sensitivity level greater than 1, and reflect the overall credibility of the cluster by the average confidence score of the nodes within the cluster. The final output key information summary adopts a standardized JSON format, including time interval, aggregated topic text, severity score and acoustic evidence (such as fundamental frequency peak), realizing end-to-end transformation from raw speech to decision support information.

[0072] Step S109: Trigger an alarm signal based on the key information summary and push it to an external system.

[0073] This alarm mechanism not only relies on static threshold judgments but can also be dynamically adjusted based on historical data. For example, it can automatically escalate the warning level when multiple medium-level events occur consecutively within a specific time period. Alarm signals can be pushed to public opinion monitoring platforms, security management systems, or human operator terminals via API interfaces, triggering emergency plans, recording and archiving, or immediate intervention measures. The entire process embodies a highly automated, intelligent, and scalable design philosophy.

[0074] The above implementation achieves high-precision recognition of multilingual speech content, cross-modal semantic understanding, and risk warning functions. The system realizes modal complementarity and information reuse in multiple stages. For example, acoustic features are used for both recognition and sentiment correction, semantic feedback labels drive translation optimization, and graph neural networks integrate text and acoustic information to form a high-order semantic structure, fully embodying a hybrid intelligent paradigm of "data-driven + knowledge-guided." This technical solution improves recognition accuracy and translation quality while constructing an intelligent speech analysis platform with self-regulating capabilities suitable for complex real-world scenarios, particularly applicable to high-value application areas such as broadcast media public opinion monitoring, public safety early warning, and multinational corporate compliance audits.

[0075] Reference Figure 2As one implementation of step S103, the step of inputting audio segments into a group of parallel-deployed speech recognition models and obtaining text segments and confidence scores output by each speech recognition model includes:

[0076] Step S201: Receive audio segments with timestamps and acoustic feature vectors;

[0077] Step S202: Based on the audio spectrum characteristics of the acoustic feature vectors, dynamically load at least two speech recognition models from the pre-trained model library to form a parallel speech recognition model group.

[0078] Dynamic loading refers to the system not using a fixed type of recognition model in advance, but deciding in real time which models to use in the recognition process based on the acoustic performance of the current audio segment.

[0079] In this embodiment, the system first extracts the Mel spectrogram of the audio segment and focuses on analyzing the energy distribution ratio of the 0.3–1kHz frequency band. This band covers the main energy concentration areas of vowels and voiced sounds in human speech, especially in southern dialects (such as Cantonese and Minnan) and some Middle Eastern languages ​​(such as Arabic), where the proportion of low-frequency energy is significantly higher than in common languages ​​such as Mandarin or English due to the abundance of nasal and guttural sounds. When the system detects that the energy of this frequency band accounts for more than 40% of the total spectrum energy, it determines that the speech may belong to a specific dialect or accent region, and then loads the corresponding dialect-specific speech recognition model (such as the Cantonese Conformer model or the Uyghur Transformer-Transducer model) from the local or cloud model library; otherwise, it loads the general speech recognition model by default (such as the multilingual Whisper-large-v3 trained based on LibriSpeech).

[0080] More importantly, the system does not simply load a single model, but rather constructs a group of recognition models that run in parallel with "at least two" models. For example, it simultaneously enables a general model and a dialect model to cope with code-switching phenomena that may occur in mixed contexts. This dynamic loading mechanism is essentially a meta-learning strategy based on acoustic cues, which enables the system to pre-configure the optimal model resources in unknown language environments, significantly improving its adaptability to non-standard pronunciations, regional accents, and multilingual mixed scenarios.

[0081] Step S203: Simultaneously input the audio segment into each speech recognition model in the speech recognition model group;

[0082] Among them, this operation utilizes the parallel computing capabilities of modern computing hardware (such as GPUs or multi-core CPUs) to complete the synchronous inference of multiple models within milliseconds. Each model independently runs its internal neural network structure (which may be an end-to-end model based on CTC, an RNN-T sequence transducer, or a Transformer architecture) and outputs the corresponding text transcription results.

[0083] It should be noted that due to differences in training data, language coverage, and acoustic modeling methods among different models, even for the same audio input, the output texts may vary in terms of vocabulary selection, grammatical structure, and even semantic deviations. For example, a general model may mis-recognize the Cantonese spoken sentence "我哋行先啦" as "我们走先吧", while a Cantonese-specific model can accurately restore it as "我们先走了". This diversity increases the uncertainty of the results on the one hand and provides a rich source of candidate information for subsequent fusion on the other hand. Therefore, parallel input is not only a performance optimization measure but also a key design for enhancing the system's fault tolerance and semantic coverage breadth.

[0084] Step S204: Obtain the text segments output by each speech recognition model and their corresponding confidence scores.

[0085] Among them, the confidence score represents the probability estimate of the correctness of the text segment by the speech recognition model. This score is usually generated by the decoder inside the model. For example, in a model based on the attention mechanism, it can be quantified by the geometric mean of the maximum likelihood probabilities of the target tokens at all time steps; in a CTC model, it can be calculated based on the best path probability or the highest-scoring path in prefix beam search for normalization. The confidence score is not a simple binary judgment but a continuous value between 0 and 1, reflecting the "self-confidence" of the model in its judgment under the current input conditions.

[0086] For example, when the audio is clear, the speech rate is moderate, and the vocabulary appears frequently in the training set, the model may give a high confidence score above 0.9; while in the presence of high background noise, a strong accent of the speaker, or the use of rare terms, the confidence score may be below 0.6. This score serves as the core basis for subsequent fusion decisions and functions as a quality assessment, enabling the system to distinguish which model outputs are more valuable for reference. Especially in the multi-model parallel scenario, the confidence score becomes a unified scale for measuring the performance differences of models and provides a computable basis for weighted fusion.

[0087] The above implementation achieves high accuracy and strong adaptability of the multilingual speech recognition system in complex real-world environments. By using acoustic features in advance for dynamic model loading, it breaks through the limitations of static deployment of traditional ASR models. This not only improves the robustness of dialect, accent, and mixed language recognition but also ensures the traceability and temporal consistency of output results through the continuous use of timestamps. It is particularly suitable for challenging application scenarios such as cross-border meeting recording, border voice monitoring, and minority language services.

[0088] Reference Figure 3 As one implementation of step S105, the step of segmenting the initially identified text into text blocks according to timestamps and translating them block by block into target language text using a streaming translation model, combined with real-time received semantic feedback tags, includes:

[0089] Step S301: Receive the initial recognition text stream with timestamps;

[0090] The initial identification of the text stream includes text fragments and associated timestamps and confidence scores.

[0091] Specifically, the text stream is typically input continuously in an event-driven manner, with each text segment representing a semantic unit that has been transcribed into speech, such as “We are discussing the project budget” or “This decision has compliance risks”, and accompanied by precise timestamps (such as “00:01:23.450–00:01:26.800”) to identify its start and end positions in the original audio.

[0092] At the same time, each segment also carries a confidence score, which is output by the speech recognition model during the decoding process. This score reflects its probability estimate of the correctness of the current transcription result, and the value generally ranges between 0 and 1. It can be used to evaluate the reliability of the text segment. For example, in scenarios with low background noise and clear pronunciation, the model may give a high confidence score of 0.9 or higher; while in cases with complex accents or excessively fast speech, the score may be lower than 0.6.

[0093] Step S302: Aggregate text fragments into text blocks according to timestamp continuity, with each text block covering a time window of a preset duration;

[0094] Since speech recognition models typically output text fragments in short sentences or with incomplete semantic boundaries, direct translation may lead to problems such as improper sentence segmentation, missing context, or inconsistent terminology. Therefore, the system analyzes the time interval between adjacent text fragments to determine whether they belong to the same semantic paragraph: when the timestamp interval between two fragments is less than 200 milliseconds, it is considered that there is no obvious silence between them, and they are very likely to belong to the same sentence or continuous expression, and therefore should be merged into a larger text block.

[0095] For example, if "We need to" and "Complete this task today" appear at "00:02:15.300" and "00:02:15.480" respectively, with an interval of only 180 milliseconds, the system determines that they can be merged and generates a new time window marker, with the start time being the start time of the former and the end time being the end time of the latter, forming a unified interval of "00:02:15.300–00:02:16.200".

[0096] Understandably, this time-continuous aggregation strategy not only preserves the rhythmic features of the original speech but also avoids semantic fragmentation caused by recognition delays or overly fine segmentation. This allows the translation model to decode within a more complete context, improving the coherence and accuracy of the translation. Furthermore, the preset time window (e.g., dividing the text into blocks every 3–5 seconds) also sets a fixed processing granularity for streaming processing, facilitating system resource scheduling and latency control.

[0097] Step S303: Receive semantic feedback tags transmitted in real time. The semantic feedback tags include sentiment polarity scores and sensitivity level markers.

[0098] Among them, the emotional polarity score is usually expressed in numerical form, such as "-1" representing negative emotions, "0" representing neutral emotions, and "+1" representing positive emotions. It can also be expanded into multi-level classifications (such as anger, anxiety, joy, etc.). The sensitivity level label is used to indicate whether the text content involves high-risk topics such as politics, security, privacy, or compliance. It is often represented by a graded code (such as 0-5 levels), with higher levels indicating greater potential risks.

[0099] It's important to note that these labels are not statically configured but dynamically updated as the audio content evolves, with clearly defined effective time intervals. For example, in a meeting speech, when the speaker mentions "this policy may trigger mass incidents," the semantic analysis model immediately generates a label with a sensitivity level of 4 and a negative sentiment polarity, marked with a time range of "00:03:10.000–00:03:25.000." The introduction of such labels allows the translation process to move beyond literal translation, enabling the perception of the emotional tone and potential risks behind the discourse, providing a basis for dynamic adjustments to subsequent translation strategies.

[0100] Step S304: Dynamically bind the semantic feedback tags that overlap the current text block with the time window;

[0101] Since the generation of semantic feedback tags has a certain delay (affected by factors such as model inference and data transmission), its time interval may not be completely aligned with the current text block. Therefore, the system needs to calculate the overlap ratio of the time windows of the two: that is, take the intersection length of the time interval of the text block and the semantic tag and divide it by the union length.

[0102] In this embodiment of the application, when the ratio exceeds 60%, the two are considered to have a significant temporal correlation and should be bound together. For example, if the time of a text block is "00:03:12.000–00:03:18.000" and the time of a sensitive tag is "00:03:10.000–00:03:25.000", the intersection is 6 seconds and the union is 15 seconds, with an overlap ratio of 40%, which does not reach the threshold and is not bound at this time; however, if the time of another tag is "00:03:11.000–00:03:19.000", the intersection is 7 seconds and the union is 9 seconds, with a ratio of approximately 78%, which meets the condition and is bound together.

[0103] More importantly, the system can also introduce a decay coefficient for the bound semantic feedback tags. Where Δt is the time difference between the current processing time and the tag generation time, T max The preset maximum effective duration (e.g., 5 seconds) is used. This mechanism simulates the human cognitive pattern of information timeliness: newer tags have stronger influence, and their effect gradually weakens over time. For example, a tag generated 3 seconds ago has a γ=0.4, and only 40% of its influence weight is retained. This prevents outdated or delayed semantic information from interfering with the current translation decision, thereby improving the system's real-time response capability and logical consistency.

[0104] Step S305: Input the bound current text block and semantic feedback tags into the streaming translation model to generate target language text;

[0105] Among them, streaming translation models are usually based on streaming variants of the Transformer architecture (such as Transformer-Transducer or Streaming Conformer), which have incremental decoding capabilities and can start outputting part of the translation before the input is completely finished, meeting the requirements of low-latency real-time translation.

[0106] It's important to note that, unlike traditional translation models that rely solely on the source language text, the streaming translation model in this approach deeply integrates semantic feedback information. Specifically, the system transforms the sensitivity level markers in the semantic labels into attention mask vectors, which are injected into the decoder as additional control signals. For example, when a high sensitivity level is detected, the model enhances its attention to certain keywords (such as "protest" and "leak") in the attention mechanism, while suppressing alternative translations that may cause ambiguity. In the Transformer decoding layer, this mask vector is concatenated with the word embedding vectors through a gating mechanism to dynamically adjust the representation weights of each word, achieving "risk-aware translation."

[0107] Furthermore, for text blocks with a confidence score below 0.7, the system automatically activates a bidirectional attention verification mechanism: that is, an inverse attention path is introduced between the encoder and decoder, forcing the model to re-examine the alignment between the source text and the target translation, and identify whether there are omissions, mistranslations, or semantic drift. This confidence-based adaptive verification strategy enables the translation system to proactively enhance its robustness under low-quality input conditions, rather than passively accepting erroneous premises.

[0108] Step S306: Output the target language text stream with timestamps and retain the confidence score labels.

[0109] The output text stream maintains the same structured format as the input, with each target text block carrying its corresponding time window marker (e.g., "00:01:23.450–00:01:26.800") and the confidence score generated during the original speech recognition stage.

[0110] Understandably, this design ensures the integrity and traceability of the information chain: downstream applications (such as caption display, public opinion monitoring, or manual review) can not only know "what was said," but also "how certain" it is and "when it was said." For example, in international conference simultaneous interpretation systems, high-confidence translations can be directly displayed, while low-confidence parts are presented in a semi-transparent or annotated manner to prompt users to refer to them with caution. In addition, retaining the confidence score provides a quantitative basis for subsequent quality assessment, model training feedback, and manual intervention, forming a closed-loop optimization mechanism.

[0111] The above implementation achieves intelligent and contextualized streaming translation in multilingual speech recognition. By deeply integrating temporal continuity analysis and semantic feedback mechanisms, it overcomes the limitations of traditional machine translation's "text-to-text" approach, enabling the system to make dynamic decisions under the collaborative guidance of multiple dimensions such as sentiment, sensitivity risk, and recognition quality. In particular, the label binding strategy based on temporal overlap ratio and the attenuation coefficient design effectively solve the mismatch problem caused by semantic feedback delay in streaming scenarios; while mechanisms such as converting sensitivity levels into attention masks and confidence-driven bidirectional verification significantly improve the accuracy and security of translation.

[0112] Reference Figure 4 As one implementation of step S106, the step of performing bidirectional translation verification on text blocks with confidence scores lower than a set threshold based on the block-by-block translated target language text, and generating the translated text stream, includes:

[0113] Step S401: Receive the target language text stream and associated confidence score output by the streaming translation model;

[0114] Among them, the target language text block is generated by a pre-order streaming translation model. For example, a piece of text after Chinese speech recognition, "This project has significant compliance risks", is translated into English in real time as "This project has significant compliance risks", and is accompanied by a confidence score, which is used to quantify the reliability of the translation.

[0115] Step S402, when the confidence score of the text block is lower than the preset score threshold, input the target language text corresponding to the text block into the reverse translation model to generate a back-translated source language text;

[0116] Among them, traditional translation systems often adopt the "once output and publish" mode, which is prone to semantic distortion due to model hallucinations, mis-translation of ambiguous words or disordered word order. However, this application introduces a conditional trigger mechanism to only perform additional verification on text blocks at risk, which not only ensures the overall processing efficiency but also improves the accuracy of key information.

[0117] It should be noted that the preset score threshold is not a fixed constant, but a judgment boundary dynamically adjusted according to the context. Specifically, the system can calculate the standard deviation σ of the confidence scores of multiple text blocks within the same time window to measure their degree of fluctuation, and construct a dynamic threshold in combination with the mean μ , where k is a configurable risk coefficient (such as 1.0, 1.5 or 2.0), which is used to adjust the verification sensitivity.

[0118] For example, in a meeting speech, if the translation confidence of most sentences is stable above 0.85 and the standard deviation is small, the threshold is high, and only very low-scoring texts are triggered for verification; while in the scenario of breaking news broadcasting, if the confidence fluctuates violently (such as dropping from 0.9 to 0.5 suddenly), it indicates that the environment is complex or the content has changed suddenly. At this time, σ increases, and T h automatically moves down, allowing more marginal cases to enter the verification process. This dynamic threshold mechanism based on statistical distribution enables the system to have an adaptive ability and can intelligently balance accuracy and resource consumption in different contexts.

[0119] In the embodiments of this application, the reverse translation model refers to a neural machine translation (NMT) model with a training direction opposite to that of the original translation model. For example, if the original model is Chinese→English, then the reverse model is English→Chinese. This model receives the generated English translation "This project has significant compliance risks" and attempts to re-translate it back to Chinese. Ideally, it should be restored to "该项目存在重大合规风险" or an expression with highly similar semantics. However, due to differences in vocabulary mapping, grammatical structure, and domain knowledge among different models, the back-translation results may deviate, such as "这个计划有严重的合规问题" or "本项目面临合规性威胁". These differences are not defects but key clues revealing the uncertainty of translation.

[0120] It can be understood that by comparing the semantic consistency between the back-translated text and the original source language text, the system can indirectly evaluate the quality of forward translation. If the two are highly similar, it indicates that the translation process is reversible and stable; if there are significant differences, it suggests that there may be semantic drift or loss of key information.

[0121] Step S403, calculate the similarity between the original text block and the back-translated source language text;

[0122] First, the system performs stemming or lemmatization processing on the two texts to eliminate the interference caused by surface changes such as tenses, singular / plural forms, and derivative affixes. For example, "compliance risks" and "compliance risk" are unified into the same root form. Subsequently, the basic similarity S base is calculated based on the edit distance algorithm, that is, the minimum number of insertions, deletions, or replacements required to convert one string to another, and after normalization, it is used as a matching metric at the literal level.

[0123] Meanwhile, the system uses a pre-trained word vector model (such as Word2Vec, FastText, or BERT embedding) to convert the text into high-dimensional semantic vectors and calculates the cosine similarity S sem between their average sentence vectors to capture deep semantic relationships. The final similarity adopts a weighted combination method, giving higher weight to word form matching to ensure term accuracy, while retaining the semantic dimension to tolerate reasonable expression differences. For example, even if "隐患" is translated as "问题", as long as the overall semantic trend is consistent, a high S value can still be obtained. This hybrid similarity calculation strategy takes into account both precision and robustness, avoiding misjudgments caused by a single metric.

[0124] Step S404, correct the target language text corresponding to the text block according to the similarity calculation result;

[0125] In the embodiments of this application, when the similarity When the translation result is considered to be highly consistent with the original text, and the translation process is reversible and stable, the original translated text is retained and marked as "verified"; when When the error occurs, it indicates that there is a certain deviation but it has not yet reached an unacceptable level. The system then triggers a multi-model voting mechanism to seek a better solution: it calls at least two independently trained translation models to re-translate the same source text and generates a set of candidate translations.

[0126] To select the optimal result from the candidate translations, the system constructs a weighted voting matrix. W i E is the weight for adapting the historical accuracy or domain of the i-th model. i The algorithm outputs word embedding vectors for the translated text. Finally, the candidate translation with the highest cosine similarity to the aggregated vector V is selected as the correction result, achieving data-driven objective decision-making and avoiding interference from subjective preferences.

[0127] When S < 0.6, it indicates that the translation has seriously deviated from the original meaning. The system marks it as "text to be reviewed," suspends automatic output, and suspends it to the manual correction interface, awaiting intervention from external experts. This three-level correction strategy not only achieves a refined hierarchical response to the degree of error but also enhances the system's self-optimization capability in scenarios with moderate uncertainty through a voting matrix.

[0128] Step S405: Output the target language text stream with correction markers as the translated text stream.

[0129] The text stream not only includes the final, verified or corrected translation, but also includes rich metadata tags such as "original translation," "verified," "corrected by vote," or "awaiting manual review," for downstream modules to process. Furthermore, all intermediate data during the verification process (such as back-translated text, similarity scores, and voting records) can be archived for iterative model training, forming a feedback loop for continuous optimization.

[0130] The above implementation achieves high fidelity and robustness of the multilingual speech translation system in uncertain environments, upgrades the traditional one-way translation paradigm to a closed-loop architecture with self-verification capabilities, and achieves adaptive perception of contextual risks through a dynamic threshold mechanism. It also improves judgment accuracy by using word form and semantic dual-dimensional similarity calculation and designs a gradient response strategy from automatic correction to manual intervention, effectively preventing potential harm caused by mistranslation of key information.

[0131] Reference Figure 5, as an implementation of step S107, the steps of inputting the translated text stream and the acoustic feature vectors into the semantic analysis model, calculating the sentiment polarity score, sensitivity level, and keyword set, and integrating to obtain the semantic feedback label and feeding it back to the streaming translation model include:

[0132] Step S501, receiving the translated text stream and the associated timestamp and confidence score;

[0133] Among them, the translated text stream is generated in real time by the previous streaming translation model. For example, translating a Chinese speech recognition result "This policy has triggered strong public dissatisfaction" into English "This policy has triggered strong public dissatisfaction", and continuously outputting it in a streaming manner according to time segments. Each text block carries accurate timestamp information (such as "00:03:12.400–00:03:16.800") to identify its spatio-temporal position in the original audio, and at the same time is accompanied by a confidence score, reflecting the system's comprehensive judgment on the accuracy of the text content.

[0134] Step S502, synchronously obtaining the acoustic feature vectors aligned with the timestamp;

[0135] Among them, these acoustic feature vectors include low-level acoustic parameters such as Mel-frequency cepstral coefficients (MFCC), fundamental frequency (F0), energy value (Energy), and spectral centroid, which can reflect non-verbal behavior characteristics such as the intonation fluctuations, volume changes, and speech rate rhythms of the speaker. Due to the possible slight offset between the speech signal and the text content on the time axis (such as due to recognition delay or decoding buffer), the system ensures that the obtained acoustic feature vectors are accurately aligned with the current text block to be analyzed through the timestamp alignment mechanism.

[0136] For example, when processing the English translation in the interval of "00:03:12.400–00:03:16.800", the system extracts the acoustic frame sequence in the same time period from the cache and performs average pooling or temporal maximum pooling operations to generate a feature vector with a fixed dimension.

[0137] Step S503, inputting the translated text stream into the semantic analysis model, and outputting the initial sentiment polarity score and keyword set;

[0138] This semantic analysis model is typically built upon a pre-trained language model (such as mBERT, XLM-RoBERTa, or multilingual BART), possessing cross-linguistic semantic understanding capabilities and able to handle mixed inputs from multiple languages, including Arabic, Russian, Chinese, and Spanish. The model extracts deep semantic representations of the text through an encoder and integrates a sentiment classification head and keyword extraction module at the top layer. Sentiment polarity scores are typically output as continuous numerical values ​​(e.g., -1.0 to +1.0), with negative values ​​representing negative sentiment, positive values ​​representing positive sentiment, and values ​​near zero representing neutrality. The keyword set is extracted using attention weight saliency analysis, TF-IDF weighting, or sequence labeling methods, covering core entities (e.g., "policy," "protest"), action verbs (e.g., "oppose," "appeal"), and descriptive phrases (e.g., "serious consequences").

[0139] For example, for the translation "This policy has triggered strong public dissatisfaction," the model might output an initial sentiment score of -0.7, with the keyword set {policy, public, dissatisfaction, triggered}. While this initial analysis result is valuable, it relies solely on surface-level textual information and easily overlooks the emotional intensity cues inherent in the speech. Therefore, further refinement by integrating acoustic modalities is necessary.

[0140] Step S504: Input the acoustic feature vector into the acoustic correction module to generate acoustic emotion auxiliary coefficients;

[0141] The core function of the acoustic correction module is to extract dynamic features strongly correlated with emotional state from the speech signal and convert them into quantitative factors that can be used to correct the sentiment judgment of the text. Specifically, the system first extracts the fundamental frequency (F0) and energy value from the acoustic feature vector. These two are widely regarded as the core acoustic cues of emotional expression: the rate of change of the fundamental frequency (i.e., the speed at which the pitch rises or falls) is significantly correlated with high-intensity emotions such as excitement and anger; while the peak value and duration of the energy value reflect the speaker's volume intensity and emotional stability.

[0142] For example, when a speaker's tone of voice rises sharply and their volume increases continuously, it often indicates anger or anxiety. The system inputs the fundamental frequency change rate into a pre-trained LSTM network, which is trained on a large amount of emotion-annotated speech data. This network can map nonlinear time-series changes into an emotion intensity factor α∈[0,1], where a higher value indicates a stronger emotion.

[0143] Simultaneously, the system calculates an emotion stability coefficient β based on the duration for which the peak energy value exceeds a preset threshold (e.g., mean + 2 standard deviations). Frequent fluctuations and long durations result in a high β value, indicating emotional instability. The final generated acoustic emotion auxiliary coefficient... It combines the effects of both strength and stability, where T max This is a normalization constant to ensure that γ is within a reasonable range. The design of this coefficient reflects the modeling of emotional complexity: even if the volume is high (high α), if the fluctuation is drastic (high β), the system will reduce its emotional influence to prevent misjudgment.

[0144] Step S505: Correct the initial emotional polarity score based on the acoustic emotional auxiliary coefficient to generate the final emotional polarity score;

[0145] This correction process is not a simple weighted average, but rather employs a nonlinear formula for confidence level adjustment: S text The initial sentiment score output by the text model is ΔS, which is a preset sentiment offset (e.g., 0.3), representing the maximum correction that acoustic information can provide. The coefficient λ is dynamically adjusted based on the confidence score of the current text block: when the confidence score is below 0.6, λ is 0.8, indicating that the text itself is unreliable and should rely more on acoustic cues for compensation; when the confidence score is above 0.6, λ is 0.5, indicating that the text is credible and acoustic information is only used for auxiliary fine-tuning.

[0146] For example, if a text misidentifies "dissatisfaction" as "satisfaction" due to a recognition error, but its acoustic features show a rapid tone and a sudden increase in volume, the system can significantly lower the original sentiment score of +0.5 to -0.6 by using high γ and high λ, thus correcting the text misjudgment. This adaptive correction mechanism effectively improves the system's robustness in low-quality input or semantically ambiguous scenarios, achieving collaborative judgment of "words" and "tone".

[0147] Step S506: Match the keyword set with the dynamically updated pre-set sensitive word library and output the sensitivity level;

[0148] Specifically, the process employs a three-layer sensitive word matching architecture, balancing accuracy, scalability, and contextual understanding. The first layer is the foundational layer, which directly matches a predefined list of sensitive words (such as "strike," "leak," and "subversion") to ensure rapid identification of high-risk words. The second layer is the derivative layer, which uses a thesaurus, lemmatization, and contextual embedding models (such as Sentence-BERT) to detect variant forms of sensitive words. For example, "strike," "work stoppage," and "industrial action" are uniformly categorized as expressions related to "strike." The third layer is the combination layer, which identifies the structure of sensitive phrases through co-occurrence patterns, such as fixed collocations like "procedural injustice," "election fraud," and "capital flight." Even if individual words are not sensitive, their combination can have high-risk semantics.

[0149] Furthermore, the system determines the matching frequency N at each level. i and preset weight w i (For example, the weight of the base layer is 0.5, the weight of the derived layer is 0.3, and the weight of the combined layer is 0.2) Calculate the overall sensitivity level. It outputs a continuous or graded score (e.g., 0–5 levels) to quantify the risk level.

[0150] It should be noted that this sensitive word database is not statically configured, but continuously evolves through a dynamic update mechanism: the system receives high-risk text fragment tags from the manual review system, automatically extracts new sensitive words and their contextual features (such as co-occurring words and grammatical structures), and adds them to the database when the same features appear more than 5 times within 24 hours. This mechanism enables the system to learn on its own, quickly respond to emerging public opinion topics or avoid strategic expressions (such as homophones and abbreviations), significantly enhancing its adaptability and control capabilities in long-term operation.

[0151] Step S507: Aggregate the final sentiment polarity score, sensitivity level, and keyword set to generate a semantic feedback tag with a timestamp.

[0152] The semantic feedback label can be used not only to feed back to the streaming translation model to adjust the translation strategy (such as enabling conservative wording in highly sensitive scenarios), but also as input to the graph neural network to build a cross-modal association graph, supporting event clustering and alarm generation.

[0153] The above implementation achieves high-order semantic understanding capabilities of the multilingual speech recognition system in complex real-world scenarios. It integrates acoustic emotion auxiliary coefficients into the emotion polarity correction process in a non-linear manner and prevents over-correction through a confidence-driven adjustment mechanism. At the same time, it designs a three-level sensitive word matching architecture of basic, derived, and combined levels, combined with a frequency-triggered dynamic lexicon update mechanism, which significantly improves the system's response speed and recognition accuracy to emerging risks.

[0154] Reference Figure 6 As one implementation of step S108, the steps of fusing acoustic feature vectors and semantic feedback labels, constructing a cross-modal association graph through a graph neural network, and outputting a timestamped summary of key information include:

[0155] Step S601: Receive a semantic feedback tag with a timestamp and an acoustic feature vector aligned with the timestamp;

[0156] Step S602: Define the timestamp as a graph node, and map the sentiment polarity score, sensitivity level, and keyword set to the sentiment attribute, risk attribute, and tag set attribute of the graph node, respectively.

[0157] In this graph, the nodes are not simply data containers, but semantic-acoustic complexes carrying multidimensional attributes. The sentiment attribute reflects the emotional tone of the time unit and can be used for subsequent sentiment trend analysis; the risk attribute serves as a quantitative indicator of potential threats, directly influencing the selection of subsequent clustering strategies; and the tag set attribute constitutes the semantic fingerprint of the node, which is a key basis for judging semantic coherence and thematic consistency.

[0158] The mapping process described above is essentially a reconstruction of the feature space, integrating scattered semantic outputs into a unified graph node representation, making each graph node an information unit with self-descriptive capabilities. For example, in a diplomatic debate, a timestamp node might have a sentiment score of -0.8 and a sensitivity level of 4. This indicates that the graph node not only involves a highly sensitive topic but also expresses extremely negative emotions, making it a potential conflict trigger. Such graph nodes will be given higher attention weight in subsequent graph analysis, reflecting a natural transition from semantic parsing to initial risk screening.

[0159] Step S603: Decompose the acoustic feature vector into fundamental frequency component and energy component, which serve as the acoustic properties of the graph nodes.

[0160] The fundamental frequency, or fundamental tone of speech, is directly related to the vibration rate of the speaker's vocal cords and is an important indicator for measuring intonation and emotional intensity. For example, anger or excitement is often accompanied by an overall upward shift or increased fluctuation of the fundamental frequency. Energy, on the other hand, reflects the loudness or intensity of speech and is often used to judge emphasis, shouting, or emotional outbursts. Decoupling the acoustic feature vector into these two core components helps improve the interpretability and modeling accuracy of the features, avoiding interference between different physical dimensions. These two acoustic attributes are attached to the corresponding timestamp nodes, giving them not only semantic identity but also an acoustic personality.

[0161] Step S604: Calculate the semantic correlation degree based on the label set attributes of adjacent graph nodes, calculate the acoustic correlation degree based on the acoustic attributes of adjacent graph nodes, fuse the weights of the edges between the generated graph nodes, and construct a cross-modal correlation graph.

[0162] In the cross-modal association graph, the graph nodes represent timestamp units, and the edges represent the semantic association strength across timestamps.

[0163] Specifically, the introduction of a graph structure is one of the core innovations of this scheme. It breaks away from the dependence of traditional sequence models (such as RNNs and Transformers) on local adjacency relationships, instead adopting a global topological perspective to model the dynamic semantic evolution in speech streams. In this graph, the connections between nodes are not fixed but dynamically generated based on both semantic and acoustic similarity. Specifically, the edge weights are determined by the semantic association weight W. s Acoustic association weight W a The two systems are integrated: semantic association is obtained by calculating the Jaccard similarity of the keyword sets of adjacent nodes, which measures the continuity of topics and the degree of overlap of concepts; acoustic association is calculated by the correlation coefficient of the fundamental frequency components, which reflects the consistency or abruptness of speech expression style.

[0164] More importantly, the system introduces a dynamic fusion coefficient α to adjust the relative importance of semantic and acoustic weights. This coefficient is adaptively adjusted by the sensitivity level of the node. When the sensitivity level is high (e.g., ≥3), the system automatically reduces the value of α to enhance the influence of acoustic features. This is because in high-risk scenarios, abnormal tone of voice (e.g., trembling, rapid speech) often reveals the true intention more than the literal meaning. Conversely, in low-sensitivity scenarios, semantic coherence is more important.

[0165] Step S605: Based on the risk attributes and sentiment attributes of all graph nodes, detect the graph node clusters that meet the risk conditions;

[0166] Among them, meeting the risk conditions means meeting the conditions of "sensitivity level exceeding the preset threshold" or "emotional polarity score below the negative threshold". These nodes have certain early warning value when they exist alone, but what is truly significant is the situation where multiple high-risk nodes appear consecutively in time and form a semantic closed loop, i.e., clustering.

[0167] To identify such patterns, the system employs a multi-head graph attention mechanism to calculate the influence score of each node. This mechanism quantifies a node's propagation ability and control within its local network by learning the attention weights of neighboring nodes towards the central node. Nodes with high influence scores are often located at key positions in the semantic chain, such as topic turning points, emotional outbursts, or logical climaxes. The system further sets dual screening conditions: only nodes with an influence score greater than 0.7 and a sensitivity level of no less than 2 participate in cluster formation. This design avoids misjudging isolated noise points, ensuring that the clustering results represent composite events of "high influence + high risk."

[0168] Specifically, the clustering process itself can employ density-based graph segmentation algorithms (such as GraphSAGE clustering or spectral clustering) to aggregate highly correlated high-risk nodes into subgraph structures, with each subgraph corresponding to a potential key event fragment.

[0169] Step S606: Aggregate the label set attributes and acoustic attributes of each graph node within the cluster to generate a key information summary with start and end timestamps.

[0170] The final output of the key information summary includes start and end timestamps, core keywords, risk level, and acoustic evidence fields, forming a structured report with clear spatiotemporal positioning, focused content, and a complete chain of evidence.

[0171] In the above implementation, a "semantic-acoustic" dual-channel perception system is constructed. Using timestamps as spatiotemporal anchors, the speech stream is transformed into graph nodes with rich attributes. A semantic evolution graph is built using a dynamically weighted cross-modal association mechanism. Then, a graph attention-driven risk clustering algorithm is combined to identify high-risk event segments with practical significance, ultimately generating a structured summary that combines semantic focus and acoustic verifiability. The entire scheme achieves end-to-end automated extraction from raw speech to key information, not only improving the information utilization rate of traditional speech recognition systems but also enhancing the insight and response capabilities to implicit risks in complex multilingual environments.

[0172] Reference Figure 7 As one implementation of step S109, the step of triggering an alarm signal based on the key information digest and pushing it to an external system includes:

[0173] Step S701: Generate a risk code based on the combination of the sensitivity level and sentiment polarity score of the key information summary;

[0174] The sensitivity level and sentiment polarity score are combined and encoded to form a multidimensional risk identifier, namely "risk code". In essence, it is a semantically enhanced classification label that goes beyond the coarse-grained mode of single threshold judgment.

[0175] For example, a speech segment with a sensitivity level of 3 but an emotion score of -0.9 might be coded as "R3E-" (indicating medium-to-high risk and extremely negative emotion), while another segment of the same level 3 but with neutral emotion would be coded as "R3N". Although both belong to the same sensitivity level, their response strategies should differ. This composite coding method allows the system to capture the most potentially destructive combination of "high sensitivity + strong emotion," avoiding false positives or false negatives caused by relying solely on static thresholds.

[0176] In addition, the risk coding design also takes scalability into account, allowing for the introduction of more dimensions (such as acoustic anomaly degree and speaker identity weight) for dynamic upgrades in the future, ensuring that the alarm system has long-term adaptability.

[0177] Step S702: Based on the risk code matching of the pre-set emergency response protocol library, load the corresponding handling template;

[0178] The emergency response protocol library stores standardized handling procedures for different types of risk events. Each protocol is bound to a specific risk code range, covering response level, responsible party, operation steps, communication instructions, and other content.

[0179] For example, when the risk code is "R4E-", the system automatically matches the "Major Diplomatic Conflict Early Warning and Response Plan". This template may stipulate actions such as immediately notifying high-level diplomatic officials, initiating multilingual translation support, and retrieving relevant national background information. For low-risk neutral events such as "R2N", the system may only trigger internal archiving processes without manual intervention. By mapping abstract risk states to specific operational guidelines, the system constructs a closed-loop control link from perception to execution, greatly improving the efficiency and consistency of response in emergencies.

[0180] Step S703: When the risk code exceeds the preset level, automatically associate with the geographic information system to obtain population density data of the affected area and generate a graded push instruction.

[0181] This step breaks through the limitations of traditional speech analysis systems, which are confined to the dimensions of time and content, by evaluating language events within a real physical space. For example, in a live broadcast of an international conference, if a speaker makes strong remarks involving "local military operations" and is judged as an R4 high-risk event, the system will not only identify the semantic content but also, in conjunction with the conference agenda, the geographical location of the speaking country and the topic, call a GIS interface to query information such as population distribution, transportation networks, and infrastructure density in the relevant region (such as disputed border areas).

[0182] Specifically, population density, as a key indicator, directly affects the potential public reach and the degree of social impact of an event. High-density areas imply potential risks of public opinion escalation or public safety pressures, requiring priority response; low-density areas may only require monitoring without immediate intervention. By integrating voice content with geospatial data, the system implements a three-dimensional risk assessment model of "semantics-emotion-space," enabling subsequent information delivery to move beyond indiscriminate broadcasting and instead achieve intelligent distribution based on the scope of impact.

[0183] The above implementation achieves deep integration from speech content recognition to intelligent emergency response. The system can not only accurately capture high-risk semantic segments in multilingual speech, but also automatically generate context-adaptive alarm strategies and information distribution schemes by combining emotional intensity, spatial distribution, and organizational structure. Especially in complex cross-cultural, multilingual, and highly dynamic environments, this mechanism effectively compensates for the lag and subjectivity of manual monitoring, providing solid technical support for building an intelligent, explainable, and traceable voice security protection system.

[0184] As one implementation method of hierarchical push instructions, it specifically includes: pushing key information summaries and raw audio stream data to the core area, pushing keyword sets and sensitivity levels to the peripheral area, and pushing cross-modal association diagrams to the regulatory center.

[0185] The core idea of ​​this tiered push mechanism is "minimum information adaptation" and "maximum responsibility matching," that is, to provide just the right amount of information granularity based on the responsibilities and information processing capabilities of different audiences, so as to avoid information overload or missing key information.

[0186] In this embodiment, the system first identifies the spatial scope of the event's impact and divides it into "core areas" (such as directly related regions), "peripheral areas" (adjacent or indirectly related regions), and "monitoring centers" (such as central command platforms or public opinion monitoring departments). For the receiving end in the core area (such as local emergency management offices or security units), the system pushes complete key information summaries and original audio stream data, enabling them to grasp the overall picture of the event and conduct secondary analysis by reviewing the original audio, ensuring frontline personnel have sufficient evidence for on-site handling. For relevant institutions in the peripheral area (such as surrounding city management departments), only keyword sets and sensitivity levels are pushed, such as "Keywords: sanctions, immediate action; risk level: level 4," conveying necessary early warning signals while preventing irrelevant details from causing unnecessary panic or misinterpretation. For high-level decision-making nodes such as monitoring centers, the system pushes the topological structure of a cross-modal association graph, displaying the semantic and acoustic association paths between various timestamp nodes within the event, helping experts analyze the event's evolution logic and identify potential manipulation patterns or long-term trends. This hierarchical distribution strategy not only optimizes information dissemination efficiency but also conforms to the organizational principles of "flat command and differentiated authorization" in modern emergency management systems.

[0187] This application also discloses a multilingual speech content recognition system.

[0188] A multilingual speech content recognition system, the recognition system comprising:

[0189] The acquisition and processing module is used to acquire raw audio stream data and perform noise reduction filtering, segment it into multiple audio segments using a silence detection algorithm, and generate a timestamp for each audio segment.

[0190] The feature extraction module is used to extract the acoustic feature vectors of audio segments;

[0191] The speech recognition module is used to input audio segments into a group of speech recognition models deployed in parallel, and obtain the text segments and confidence scores output by each speech recognition model.

[0192] The weighted fusion module is used to calculate weights based on confidence scores and fuse all text fragments to generate preliminary recognized text with timestamps.

[0193] The streaming translation module is used to segment the initially identified text into text blocks according to timestamps, and combine the semantic feedback tags received in real time to translate the blocks into target language text one by one through the streaming translation model;

[0194] The translation verification module is used to perform bidirectional translation verification on text blocks whose confidence scores are lower than a set threshold based on the block-by-block translated target language text, and generate a translated text stream.

[0195] The semantic feedback module is used to input the translated text stream and acoustic feature vector into the semantic analysis model, calculate the sentiment polarity score, sensitivity level and keyword set, fuse them to obtain semantic feedback labels and feed them back to the streaming translation model.

[0196] The summary output module is used to fuse acoustic feature vectors and semantic feedback labels, construct a cross-modal association graph through a graph neural network, and output a summary of key information with timestamps.

[0197] The alarm module is used to trigger alarm signals based on key information summaries and push them to external systems.

[0198] The multilingual speech content recognition system of this application embodiment can implement any of the above-mentioned multilingual speech content recognition methods, and the specific working process of each module in the multilingual speech content recognition system can refer to the corresponding process in the above-mentioned method embodiments.

[0199] In the several embodiments provided in this application, it should be understood that the provided methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for example, the division of a certain module is merely a logical functional division, and in actual implementation there may be other division methods, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0200] This application also discloses a computer-readable storage medium.

[0201] A computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above in any of the multilingual speech content recognition methods.

[0202] The computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device; the program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0203] In this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0204] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude a plurality. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce a good effect.

[0205] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only one example of a series of equivalent or similar features.

Claims

1. A method for multilingual speech content recognition, characterized in that, The identification method includes: The system acquires raw audio stream data and performs noise reduction filtering. It then segments the audio stream into multiple audio segments using a silence detection algorithm and generates a timestamp for each audio segment. Extract the acoustic feature vector of the audio segment; The audio segment is input into a group of speech recognition models deployed in parallel, and the text segment and confidence score output by each speech recognition model are obtained. The weighted weights are calculated based on the confidence scores, and all text fragments are merged to generate preliminary identified text with timestamps. The text fragments in the initially identified text are aggregated into text blocks according to the timestamp, and combined with the semantic feedback tags received in real time, they are translated into target language text block by block through a streaming translation model; Based on the target language text translated block by block, bidirectional translation verification is performed on text blocks with confidence scores below a set threshold to generate a translated text stream. The translated text stream and the acoustic feature vector are input into the semantic analysis model to calculate the sentiment polarity score, sensitivity level and keyword set, and then the semantic feedback label is obtained and fed back to the streaming translation model. By fusing the acoustic feature vectors and semantic feedback labels, a cross-modal association graph is constructed using a graph neural network, and a timestamped summary of key information is output. An alarm signal is triggered based on the key information summary and pushed to an external system.

2. The multilingual speech content recognition method according to claim 1, characterized in that, The steps of inputting the audio segment into a group of parallel-deployed speech recognition models and obtaining the text segment and confidence score output by each speech recognition model include: Receive timestamped audio segments and acoustic feature vectors; Based on the audio spectrum characteristics of the acoustic feature vectors, at least two speech recognition models are dynamically loaded from the pre-trained model library to form a parallel speech recognition model group. The audio segment is simultaneously input into each speech recognition model in the speech recognition model group; The text segment output by each speech recognition model and its corresponding confidence score are obtained.

3. The multilingual speech content recognition method according to claim 1, characterized in that, The steps of aggregating text fragments from the initially identified text into text blocks according to the timestamps, and translating them block by block into target language text using a streaming translation model, in conjunction with the semantic feedback tags received in real time, include: Receive a preliminary identified text stream with timestamps; wherein the preliminary identified text stream includes text fragments and associated timestamps and confidence scores; The text segments are aggregated into text blocks according to the continuity of timestamps, and each text block covers a time window of a preset duration; Receive semantic feedback tags transmitted in real time; wherein the semantic feedback tags include sentiment polarity scores and sensitivity level markers; Dynamically bind semantic feedback labels that overlap with the current text block and the time window; The bound current text block and semantic feedback labels are input into the streaming translation model to generate target language text. Output a time-stamped target language text stream while retaining confidence score labels.

4. The multilingual speech content recognition method according to claim 3, characterized in that, Based on the block-by-block translated target language text, the steps of performing bidirectional translation verification on text blocks with confidence scores below a set threshold to generate a translated text stream include: Receive the target language text stream and associated confidence scores output by the streaming translation model; When the confidence score of a text block is lower than a preset score threshold, the target language text corresponding to the text block is input into the reverse translation model to generate the source language text to be translated back. The similarity between the original text block and the back-translated source language text is calculated. The target language text corresponding to the text block is corrected based on the similarity calculation results; Output the target language text stream with correction markers as the translated text stream.

5. The multilingual speech content recognition method according to claim 1, characterized in that, The steps of inputting the translated text stream and the acoustic feature vector into a semantic analysis model, calculating the sentiment polarity score, sensitivity level, and keyword set, fusing them to obtain semantic feedback labels, and feeding them back to the streaming translation model include: Receive the translated text stream along with its associated timestamps and confidence scores; Simultaneously acquire acoustic feature vectors aligned with the timestamp; The translated text stream is input into a semantic analysis model, which outputs an initial sentiment polarity score and a set of keywords. The acoustic feature vector is input into the acoustic correction module to generate acoustic emotion auxiliary coefficients; The initial emotion polarity score is corrected based on the acoustic emotion auxiliary coefficient to generate the final emotion polarity score; The keyword set is matched with a dynamically updated pre-set sensitive word library, and the sensitivity level is output. By aggregating the final sentiment polarity score, sensitivity level, and keyword set, a semantic feedback tag with a timestamp is generated.

6. The multilingual speech content recognition method according to claim 5, characterized in that, The steps of fusing the acoustic feature vectors and semantic feedback labels, constructing a cross-modal association graph through a graph neural network, and outputting a timestamped summary of key information include: Receive a semantic feedback tag with a timestamp and an acoustic feature vector aligned with the timestamp; The timestamp is defined as a graph node, and the sentiment polarity score, sensitivity level, and keyword set are respectively mapped to the sentiment attribute, risk attribute, and tag set attribute of the graph node; The acoustic feature vector is decomposed into fundamental frequency components and energy components, which are used as the acoustic properties of the graph nodes. Semantic association degree is calculated based on the label set attributes of adjacent graph nodes, acoustic association degree is calculated based on the acoustic attributes of adjacent graph nodes, and the weights of the edges between the generated graph nodes are fused to construct a cross-modal association graph. Based on the risk and sentiment attributes of all graph nodes, detect graph node clusters that meet the risk conditions; The label set attributes and acoustic attributes of each graph node within the cluster are aggregated to generate a key information summary with start and end timestamps.

7. The multilingual speech content recognition method according to claim 6, characterized in that, The steps of triggering an alarm signal and pushing it to an external system based on the key information digest include: A risk code is generated based on the combination of the sensitivity level and sentiment polarity score of the key information summary; Based on the risk code matching pre-set emergency response protocol library, load the corresponding handling template; When the risk code exceeds the preset level, the system automatically retrieves population density data of the affected area from the geographic information system and generates a tiered push instruction.

8. The multilingual speech content recognition method according to claim 7, characterized in that, The generation of tiered push instructions includes: pushing key information summaries and raw audio stream data to the core area, pushing keyword sets and sensitivity levels to the peripheral area, and pushing cross-modal association diagrams to the regulatory center.

9. A multilingual speech content recognition system, characterized in that, The identification system includes: The acquisition and processing module is used to acquire raw audio stream data and perform noise reduction filtering, segment it into multiple audio segments using a silence detection algorithm, and generate a timestamp for each audio segment. The feature extraction module is used to extract the acoustic feature vector of the audio segment; The speech recognition module is used to input the audio segment into a group of speech recognition models deployed in parallel, and obtain the text segment and confidence score output by each speech recognition model; The weighted fusion module is used to calculate the weighted weights based on the confidence scores and fuse all text fragments to generate preliminary identified text with timestamps; The streaming translation module is used to aggregate text fragments in the initially identified text into text blocks according to the timestamp, and translate them into target language text block by block through the streaming translation model, combined with the semantic feedback tags received in real time. The translation verification module is used to perform bidirectional translation verification on text blocks whose confidence scores are lower than a set threshold based on the target language text translated block by block, and generate a translated text stream; The semantic feedback module is used to input the translated text stream and the acoustic feature vector into the semantic analysis model, calculate the sentiment polarity score, sensitivity level and keyword set, fuse them to obtain semantic feedback labels and feed them back to the streaming translation model; The summary output module is used to fuse the acoustic feature vectors and semantic feedback labels, construct a cross-modal association graph through a graph neural network, and output a summary of key information with timestamps. The alarm module is used to trigger alarm signals based on the key information summary and push them to external systems.

10. A computer-readable storage medium, characterized in that: The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 8.

Images