Foreign language simultaneous interpretation method and system based on real-time voice translation for foreign-related police

Abstract

The present application relates to the technical field of speech translation, and discloses a foreign-related police receiving and handling foreign language simultaneous interpretation method and system based on real-time speech translation, which comprises the following steps: deconstructing historical foreign-related police recording voiceprint features, constructing a triple training data set and an end-to-end simultaneous interpretation network; matching the language and scene information of the foreign police receiving end voice stream, generating a scene-aware feature spectrum, projecting the spectrum domain features of the first voice stream and the second voice stream of the police receiving end, inputting the first joint feature sequence and the second joint feature sequence obtained by projection into the end-to-end simultaneous interpretation network for cross-language attention mapping, obtaining the first simultaneous interpretation text and the second simultaneous interpretation text; through the display interface of the police receiving end, the second simultaneous interpretation text is echoed in real time, and the first simultaneous interpretation text is input into a speech synthesizer to generate a simultaneous interpretation voice stream, which is sent to the foreign police receiving end for broadcasting; the present application can improve the efficiency of foreign-related police receiving and handling foreign language simultaneous interpretation based on real-time speech translation.

Description

Technical Field

[0001] This invention relates to the field of speech translation technology, and in particular to a method and system for simultaneous interpretation in foreign languages ​​for handling foreign-related police calls based on real-time speech translation. Background Technology

[0002] Simultaneous interpretation in foreign languages ​​during international police response scenarios demands high real-time performance and accuracy. Existing speech translation technologies are not specifically optimized for international police scenarios. When processing foreign alarm voice streams, they are easily affected by background noise and differences in pronunciation characteristics of different languages. The accuracy of language recognition and judgment of police scenarios is low, which cannot effectively meet the actual business needs of international police response. The translation results often have semantic deviations, making it difficult for police officers to quickly grasp the core information of the police situation.

[0003] Simultaneous interpretation in foreign languages ​​during international police response scenarios demands stringent real-time performance and accuracy. Existing speech translation technologies are not specifically optimized for international police scenarios. When processing foreign alarm voice streams, they are easily affected by background noise and differences in pronunciation characteristics of different languages, resulting in low accuracy in language recognition and crime scene judgment. This makes it difficult to effectively meet the actual business needs of international police response, and the translation results often contain semantic deviations, making it difficult for dispatchers to quickly grasp the core information of the crime. Therefore, how to improve the efficiency of simultaneous interpretation in foreign languages ​​during international police response has become an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a method and system for simultaneous interpretation in foreign languages ​​for handling police calls and responding to incidents involving foreign nationals, based on real-time speech translation, to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, this invention provides a method for simultaneous interpretation in a foreign language for handling police calls based on real-time speech translation, comprising: S1. Deconstruct the voiceprint features of historical foreign-related police recording data to obtain the triplet training dataset of historical foreign-related police recording data. S2. Based on the triplet training dataset, construct an end-to-end simultaneous interpretation network for the triplet training dataset; S3. Perform similarity matching between the language features to be identified in the first voice stream of the foreign alarm terminal and the voice segments in the triplet training dataset to obtain the language information and scene information of the first voice stream. S4. Select a set of target speech segments that match both language and scene information from the triplet training dataset, and extract common features among the target speech segments to generate the scene-aware feature spectrum of the first speech stream. S5. Based on the scene-aware feature spectrum, perform spectral domain feature projection on the first speech stream and the second speech stream of the alarm receiving end, and input the first joint feature sequence and the second joint feature sequence obtained by the projection into the end-to-end simultaneous interpretation network for cross-language attention mapping to obtain the first simultaneous interpretation text of the first speech stream and the second simultaneous interpretation text of the second speech stream. S6. The second simultaneous interpretation text is displayed in real time through the display interface of the alarm receiving terminal, and the first simultaneous interpretation text is input into the speech synthesizer to generate a simultaneous interpretation speech stream with the same language as the first speech stream, which is then sent to the foreign alarm terminal for broadcast.

[0006] In a preferred embodiment, the step of deconstructing the voiceprint features of historical foreign-related police recording data to obtain a triplet training dataset of historical foreign-related police recording data includes: Silence and non-voice segments were removed from historical foreign-related police incident recordings to obtain a set of valid voice segments from the historical foreign-related police incident recordings. Each valid speech segment in the set of valid speech segments is subjected to frame-by-frame windowing, and the speech signal after frame-by-frame windowing is subjected to acoustic vector quantization to obtain the speaker feature vector of the valid speech segment. The voiceprint feature vector is associated with the alarm record corresponding to the effective voice segment to obtain the language label and scene label corresponding to the effective voice segment. The effective voice segment, language label and scene label are combined into a triplet sample of historical foreign-related police recording data. Data encapsulation of triplet samples yields a training dataset of triplet samples from historical foreign-related police incident recordings.

[0007] In a preferred embodiment, constructing an end-to-end simultaneous interpretation network for the triplet training dataset includes: The acoustic dimension of speech segments in the triplet training dataset is analyzed to construct the acoustic coding layer of the speech segments; Attention weights are assigned to the cross-lingual correspondence between language labels and scene labels in the triplet training dataset to construct a cross-lingual attention alignment layer between language labels and scene labels. Word embedding mapping is performed on the semantic structure and sequence length of bilingual simultaneous interpretation texts in the triplet training dataset to construct the text decoding layer of bilingual simultaneous interpretation texts; The acoustic coding layer, the cross-lingual attention alignment layer, and the text decoding layer are cascaded and stacked to obtain the initial end-to-end network architecture for the triplet training dataset; Using speech segments from the triplet training dataset as input and the corresponding bilingual simultaneous interpretation texts as supervision signals, the initial end-to-end network architecture is iteratively trained until the network converges, resulting in an end-to-end simultaneous interpretation network for the triplet training dataset.

[0008] In a preferred embodiment, the step of performing similarity matching between the language features to be identified in the first voice stream of the foreign alarm terminal and the voice segments in the triplet training dataset to obtain the language information and scene information of the first voice stream includes: Endpoint detection is performed on the first voice stream of the foreign alarm terminal, the silent segments and background noise segments at the beginning and end of the first voice stream are removed, and the remaining effective voice segments after removal are windowed for a short time to obtain the windowed voice frame sequence of the first voice stream. A short-time Fourier transform is performed on the windowed speech frame sequence, and the formant trajectory is traced on the transformed spectral amplitude to construct the feature vector to be identified for the first speech stream. Correlation analysis is performed between the feature vector to be identified and the voiceprint feature vector to obtain the similarity score between the speech segment in the triplet training dataset and the first speech stream. Based on the similarity score, the speech segments are ranked and aggregated to obtain the set of nearest matching segments for each speech segment; The frequency of language and scene tags carried by speech segments in the nearest neighbor matching segment set is counted. The language tag with the highest frequency is determined as the language information of the first speech stream, and the scene tag with the highest frequency is determined as the scene information of the first speech stream.

[0009] In a preferred embodiment, the similarity score is calculated using the following formula: ; In the formula, The similarity score is... For the feature vector to be identified in the th... Component values ​​in each dimension For the voiceprint feature vector at the th Component values ​​in each dimension This represents the total dimension of the feature vector to be identified and the voiceprint feature vector. To obtain the minimum value, To obtain the maximum value.

[0010] In a preferred embodiment, the step of selecting a set of target speech segments that match both language and scene information from the triplet training dataset, and extracting common features among the target speech segments to generate a scene-aware feature spectrum of the first speech stream, includes: Extract the language and scene labels carried by the speech segments in the triplet training dataset, and verify the language and scene labels with the language and scene information of the first speech stream item by item to obtain the target speech segment set of the triplet training dataset. Extract the voiceprint feature vectors corresponding to the target speech segment set from the triplet training dataset; Time-axis normalization is performed on the voiceprint feature vector to obtain the normalized feature vector set. The normalized feature vector set is aggregated dimension by dimension to obtain the aggregated feature vector of the normalized feature vector set; The aggregated feature vectors are reconstructed to obtain the scene-aware feature spectrum of the first speech stream.

[0011] In a preferred embodiment, the step of projecting spectral domain features onto the first speech stream and the second speech stream from the alarm receiving end based on the scene-aware feature spectrum, and then inputting the resulting first joint feature sequence and second joint feature sequence into an end-to-end simultaneous interpretation network for cross-language attention mapping, to obtain the first simultaneous interpretation text of the first speech stream and the second simultaneous interpretation text of the second speech stream, includes: The acoustic parameters of the first speech stream are analyzed to obtain the fundamental frequency trajectory and formant distribution of the first speech stream. The fundamental frequency trajectory and formant distribution are then concatenated with parameters to obtain the first original feature sequence of the first speech stream. The acoustic features of the second speech stream are deconstructed to obtain the fundamental frequency trajectory and formant distribution of the second speech stream. The feature dimensions of the fundamental frequency trajectory and formant distribution are then expanded to obtain the second original feature sequence of the second speech stream. Using the scene-aware feature spectrum as the projection basis, the first original feature sequence is subjected to spectral domain weighted projection to obtain the first joint feature sequence of the first speech stream; Using the scene-aware feature spectrum as a projection basis, feature modulation is performed on the second original feature sequence to obtain the second joint feature sequence of the second speech stream; The first joint feature sequence and the second joint feature sequence are input in parallel to the cross-language attention alignment layer of the end-to-end simultaneous interpretation network for attention convergence, to obtain the first context feature of the first joint feature sequence and the second context feature of the second joint feature sequence; The first and second context features are input into the text decoding layer of the end-to-end simultaneous interpretation network for autoregressive decoding to obtain the first simultaneous interpretation text of the first speech stream and the second simultaneous interpretation text of the second speech stream.

[0012] In a preferred embodiment, the step of inputting the first joint feature sequence and the second joint feature sequence in parallel to the cross-language attention alignment layer of the end-to-end simultaneous interpretation network for attention convergence to obtain the first contextual feature of the first joint feature sequence and the second contextual feature of the second joint feature sequence includes: In the cross-language attention alignment layer of the end-to-end simultaneous interpretation network, the first joint feature sequence and the second joint feature sequence are interacted to generate the association feature matrix between the first joint feature sequence and the second joint feature sequence. Attention weights are assigned to the associated feature matrix to obtain the attention query weight distribution of the first joint feature sequence to the second joint feature sequence, and the attention mapping weight distribution of the second joint feature sequence to the first joint feature sequence. Based on the attention mapping weight distribution, the first joint feature sequence is aggregated in context to generate the first context feature of the first joint feature sequence. Based on the attention query weight distribution, information fusion is performed on the second joint feature sequence to generate the second context feature of the second joint feature sequence.

[0013] In a preferred embodiment, the step of displaying the second simultaneous interpretation text in real time through the display interface of the alarm receiving terminal, and inputting the first simultaneous interpretation text into a speech synthesizer to generate a simultaneous interpretation speech stream consistent with the language of the first speech stream, and sending it to the foreign alarm terminal for broadcast, includes: Based on the text rendering specifications of the display interface in the alarm receiving terminal, the second simultaneous interpretation text is formatted to obtain the formatted simultaneous interpretation text of the second simultaneous interpretation text. The formatted simultaneous interpretation text is pushed to the text buffer of the display interface and a refresh command is triggered on the display interface, so that the formatted simultaneous interpretation text is displayed on the display interface in real time. Text normalization is performed on the first simultaneous interpretation text to obtain the normalized simultaneous interpretation text. The normalized simultaneous interpretation text is input into the speech synthesizer to synthesize speech parameters, resulting in the simultaneous interpretation speech stream of the normalized simultaneous interpretation text. The simultaneous interpretation voice stream is sent to the foreign alarm terminal through the real-time communication link between the alarm receiving terminal and the foreign alarm terminal, and the foreign alarm terminal's built-in speaker broadcasts the voice message.

[0014] To address the aforementioned problems, this invention also provides a foreign language simultaneous interpretation system for handling foreign-related police calls based on real-time speech translation, the system comprising: The triplet training dataset construction module is used to deconstruct the voiceprint features of historical foreign-related police recording data to obtain the triplet training dataset of historical foreign-related police recording data. The end-to-end simultaneous interpretation network construction module is used to construct an end-to-end simultaneous interpretation network for the triplet training dataset based on the triplet training dataset. The language and scene joint recognition module is used to perform similarity matching between the language features to be identified in the first voice stream of the foreign alarm terminal and the voice segments in the triplet training dataset to obtain the language information and scene information of the first voice stream. The scene-aware feature spectrum generation module is used to select a set of target speech segments that match both language information and scene information from the triple training dataset, and extract the common features among the target speech segments to generate the scene-aware feature spectrum of the first speech stream. The spectral domain feature projection and simultaneous interpretation mapping module is used to perform spectral domain feature projection on the first speech stream and the second speech stream of the alarm receiving end according to the scene-aware feature spectrum, and input the first joint feature sequence and the second joint feature sequence obtained by projection into the end-to-end simultaneous interpretation network for cross-language attention mapping to obtain the first simultaneous interpretation text of the first speech stream and the second simultaneous interpretation text of the second speech stream. The simultaneous interpretation result presentation and voice broadcast module is used to display the second simultaneous interpretation text in real time through the display interface of the alarm receiving terminal, and input the first simultaneous interpretation text into the speech synthesizer to generate a simultaneous interpretation voice stream that is consistent with the language of the first voice stream, and send it to the foreign alarm terminal for broadcast.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention constructs a triplet training dataset based on historical foreign-related police call recordings, enabling accurate joint recognition of the language of the foreign alarm terminal's voice stream and the police call scenario. Furthermore, it generates a scene-aware feature spectrum based on the matched target voice segment set, making the spectral domain feature projection of the voice stream more closely match the actual foreign-related police call scenarios. This makes the voice stream feature extraction more scenario-specific, effectively improving the accuracy of cross-language feature mapping, ensuring the semantic integrity of the simultaneous interpretation text, and ensuring that the simultaneous interpretation results accurately meet the business needs of foreign-related police call scenarios.

[0016] 2. The end-to-end simultaneous interpretation network constructed in this invention integrates acoustic coding, cross-language attention alignment, and text decoding layers, enabling efficient conversion of speech stream features into simultaneous interpretation text. The cross-language attention mapping mechanism strengthens the interactive fusion of speech stream features between Chinese and foreign parties, improving the efficiency and accuracy of simultaneous interpretation text generation. Simultaneously, the presentation and broadcasting process of bidirectional simultaneous interpretation results enables real-time text echoing at the alarm receiving end and real-time broadcasting of simultaneous interpretation voice at the foreign alarm receiving end, ensuring the real-time and smooth transmission of bilingual information in international emergency response, comprehensively improving the overall processing efficiency of foreign language simultaneous interpretation in international emergency response, and adapting to the practical operational requirements of international emergency response. Attached Figure Description

[0017] Figure 1 A flowchart illustrating a method for simultaneous interpretation in a foreign language for handling police calls based on real-time speech translation, provided as an embodiment of the present invention. Figure 2 A functional module diagram of a foreign language simultaneous interpretation system for handling foreign-related police calls based on real-time speech translation is provided in an embodiment of the present invention. The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] This application provides a method for simultaneous interpretation in a foreign language for handling police calls based on real-time speech translation. The executing entity of this method includes, but is not limited to, at least one electronic device that can be configured to execute the method provided in this application, such as a server or a terminal. In other words, this method can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cluster of cloud servers. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0020] Reference Figure 1 The diagram shown is a flowchart illustrating a method for simultaneous interpretation in a foreign language for handling police calls based on real-time speech translation, according to an embodiment of the present invention. In this embodiment, the method includes: S1. Deconstruct the voiceprint features of historical foreign-related police recording data to obtain the triplet training dataset of historical foreign-related police recording data. In this embodiment of the invention, the step of deconstructing the voiceprint features of historical foreign-related police recording data to obtain a triplet training dataset of historical foreign-related police recording data includes: Silence and non-voice segments were removed from historical foreign-related police incident recordings to obtain a set of valid voice segments from the historical foreign-related police incident recordings. Each valid speech segment in the set of valid speech segments is subjected to frame-by-frame windowing, and the speech signal after frame-by-frame windowing is subjected to acoustic vector quantization to obtain the speaker feature vector of the valid speech segment. The voiceprint feature vector is associated with the alarm record corresponding to the effective voice segment to obtain the language label and scene label corresponding to the effective voice segment. The effective voice segment, language label and scene label are combined into a triplet sample of historical foreign-related police recording data. Data encapsulation of triplet samples yields a training dataset of triplet samples from historical foreign-related police incident recordings.

[0021] Using the energy value of the voice signal as the criterion, a fixed voice energy threshold is preset, and the signal energy of historical foreign-related police recording data is detected frame by frame. When the detected voice signal energy value is consistently lower than the preset voice energy threshold, it is directly judged as a silent segment and removed. At the same time, the recording data is detected by extracting the spectral features of the signal, and only signal segments with voice spectral features are retained. All the signal segments with voice spectral features that have been detected and retained are integrated as a whole to form a set of effective voice segments of historical foreign-related police recording data.

[0022] For each valid speech segment in the set of valid speech segments, continuous framing processing is performed according to a fixed time length, so that each valid speech segment is divided into multiple continuous and non-overlapping speech frames. Then, a window function is applied to each segmented speech frame to achieve a smooth transition of the edge signal of the speech frame and avoid interference caused by signal abrupt changes. After completing the framing and windowing processing of the speech signal, acoustic vector quantization is performed on the processed speech signal to map the various acoustic features of the speech signal to a discrete vector space to form corresponding feature points. These feature points are integrated into a fixed-dimensional feature vector according to the temporal order of the signal. This feature vector is the voiceprint feature vector of the valid speech segment.

[0023] Retrieve the corresponding call records for each valid audio segment in the historical handling of foreign-related police incidents. Extract language-related information and incident-scene-related information that match the valid audio segment from the text information of the call records. Define the extracted language-related information as the language tag corresponding to the valid audio segment and the extracted incident-scene-related information as the scene tag corresponding to the valid audio segment. Take the valid audio segment as the core element and its corresponding language tag and scene tag as the correlation elements. Integrate the three elements into an independent single data unit according to a fixed element arrangement. This data unit is the triplet sample of historical foreign-related police incident recording data.

[0024] Collect triple samples corresponding to all valid speech segments from the valid speech segment set. Standardize all triple samples according to a preset unified data format to ensure that the storage structure, data field type, and field length of all triple samples are consistent. Then, batch integrate all triple samples after standardization. Perform data encapsulation operation on the integrated triple sample set to convert it into a structured data set that can be directly used for network model training. This structured data set is the triple training dataset of historical foreign-related police recording data.

[0025] The beneficial effects are as follows: by setting clear judgment criteria to remove silent segments and non-voice segments from historical foreign-related police recording data, the validity and purity of the effective voice segment set are guaranteed from the data source, providing a high-quality data foundation for subsequent extraction of voiceprint feature vectors. The effective voice segments are subjected to standardized frame segmentation, windowing, and acoustic vector quantization processing, so that the generated voiceprint feature vectors can accurately and uniquely represent the acoustic features of the corresponding effective voice segments, providing a reliable feature basis for subsequent language and scene recognition. The voiceprint feature vectors are associated with the alarm records to generate corresponding labels and combined into triplet samples, so that the samples contain three core information types: voice, language, and scene, enriching the feature dimensions of the samples. The standardized processing and data encapsulation of the unified format make the triplet training dataset form a structured and standardized whole, which can be directly adapted to the training input requirements of end-to-end simultaneous interpretation networks without additional format conversion, thereby improving the practical application value and training adaptability of the dataset.

[0026] S2. Based on the triplet training dataset, construct an end-to-end simultaneous interpretation network for the triplet training dataset; In this embodiment of the invention, constructing an end-to-end simultaneous interpretation network for the triplet training dataset includes: The acoustic dimension of speech segments in the triplet training dataset is analyzed to construct the acoustic coding layer of the speech segments; Attention weights are assigned to the cross-lingual correspondence between language labels and scene labels in the triplet training dataset to construct a cross-lingual attention alignment layer between language labels and scene labels. Word embedding mapping is performed on the semantic structure and sequence length of bilingual simultaneous interpretation texts in the triplet training dataset to construct the text decoding layer of bilingual simultaneous interpretation texts; The acoustic coding layer, the cross-lingual attention alignment layer, and the text decoding layer are cascaded and stacked to obtain the initial end-to-end network architecture for the triplet training dataset; Using speech segments from the triplet training dataset as input and the corresponding bilingual simultaneous interpretation texts as supervision signals, the initial end-to-end network architecture is iteratively trained until the network converges, resulting in an end-to-end simultaneous interpretation network for the triplet training dataset.

[0027] The acoustic feature types and feature dimension values ​​of all speech segments in the triplet training dataset are extracted. The common feature dimensions and dimension ranges of the acoustic dimensions of all speech segments are statistically analyzed. A unified standard for acoustic dimensions that is suitable for all speech segments is established. Based on this standard, a network layer structure containing an input layer, a hidden layer, and an output layer is constructed. The number of nodes in the input layer is consistent with the dimension value of the unified standard for acoustic dimensions. It is used to receive the acoustic feature data of speech segments. The hidden layer completes the encoding and transformation processing of the input acoustic features. The output layer outputs the encoded acoustic feature vector. The completed network layer structure is the acoustic coding layer of the speech segment.

[0028] The study analyzes the combined associations of all language labels and scene labels in the triplet training dataset, distinguishing between positive associations of different language labels corresponding to the same scene label and differential associations of the same language label corresponding to different scene labels. Each association is quantified based on its frequency in actual police incidents, and corresponding attention weights are assigned to associations with different frequencies. The higher the frequency of association, the higher the assigned attention weight. Based on the assigned attention weights, a network layer structure with feature association alignment function is constructed. This network layer structure enables cross-language feature alignment processing between language labels and scene labels. This network layer structure is the cross-language attention alignment layer for language labels and scene labels.

[0029] Semantic structural elements and text sequence length values ​​of all bilingual simultaneous interpretation texts in the triplet training dataset are extracted. A unified sequence length benchmark is set to adapt to all bilingual simultaneous interpretation texts. For bilingual simultaneous interpretation texts with sequence lengths shorter than the benchmark, meaningless semantic padding is performed to complete them. For bilingual simultaneous interpretation texts with sequence lengths longer than the benchmark, core semantic preservation is performed to truncate them. The various semantic structural elements of the bilingual simultaneous interpretation texts that have undergone length standardization are converted into low-dimensional dense vector forms to complete the word embedding mapping of the bilingual simultaneous interpretation texts. Based on the vector transformation rules of word embedding mapping and the requirements of semantic vector output, a network layer structure containing a feature receiving layer, a semantic decoding layer, and a text output layer is constructed. This network layer structure is the text decoding layer of the bilingual simultaneous interpretation texts.

[0030] A dedicated data transmission link is established between the output of the acoustic coding layer and the input of the cross-lingual attention alignment layer, allowing the encoded acoustic feature vector output by the acoustic coding layer to be directly transmitted to the cross-lingual attention alignment layer for subsequent processing. Similarly, a dedicated data transmission link is established between the output of the cross-lingual attention alignment layer and the input of the text decoding layer, allowing the aligned features output by the cross-lingual attention alignment layer to be directly transmitted to the text decoding layer for subsequent processing. This three-layer network structure is cascaded and stacked in the order of acoustic coding, cross-lingual attention alignment, and text decoding, forming a complete network architecture capable of handling the entire process of acoustic coding, cross-lingual alignment, and text decoding. This network architecture is the initial end-to-end network architecture for the triplet training dataset.

[0031] Speech segments from the triplet training dataset are sequentially input into the initial end-to-end network architecture as individual data points. The network architecture autonomously completes the entire process of acoustic feature encoding, cross-lingual feature alignment, and semantic text decoding, outputting the corresponding predicted simultaneous interpretation text. This predicted simultaneous interpretation text is then compared word-by-word with the corresponding bilingual simultaneous interpretation text in the triplet training dataset as a supervision signal. The differences in text content generated after the comparison are extracted and fed back into the initial end-to-end network architecture for network parameter adjustment and optimization. This process of speech segment input, predicted simultaneous interpretation text output, content comparison, difference feedback, and parameter adjustment is repeated until the predicted simultaneous interpretation text output by the initial end-to-end network architecture is completely consistent with the content of the bilingual simultaneous interpretation text used as a supervision signal. At this point, the network is considered to have reached convergence, and the converged network architecture is the end-to-end simultaneous interpretation network for the triplet training dataset.

[0032] The beneficial effects are as follows: the acoustic coding layer is constructed based on the actual acoustic dimension of the speech segments in the triplet training dataset, ensuring the adaptability and accuracy of speech acoustic feature encoding; the cross-language attention alignment layer assigns weights based on the frequency of label association, making the cross-language alignment of language and scene labels more in line with the actual scenario of handling foreign-related police incidents; the text decoding layer achieves accurate semantic parsing of bilingual simultaneous interpretation text through standardized sequence length and word embedding mapping; the cascading stacking of the three network layers forms a complete initial end-to-end network architecture, ensuring the continuity of data transmission and the integrity of processing; the iterative training with speech segments as input and bilingual simultaneous interpretation text as supervision signal until network convergence ensures that the output of the end-to-end simultaneous interpretation network is highly consistent with actual needs, ensuring that the network has efficient and accurate cross-language simultaneous interpretation processing capabilities, providing stable and adaptable network model support for real-time foreign language simultaneous interpretation for handling foreign-related police incidents, and ensuring the efficiency and accuracy of simultaneous interpretation processing from the model level.

[0033] S3. Perform similarity matching between the language features to be identified in the first voice stream of the foreign alarm terminal and the voice segments in the triplet training dataset to obtain the language information and scene information of the first voice stream. In this embodiment of the invention, the step of performing similarity matching between the language features to be identified in the first voice stream of the foreign alarm terminal and the voice segments in the triplet training dataset to obtain the language information and scene information of the first voice stream includes: Endpoint detection is performed on the first voice stream of the foreign alarm terminal, the silent segments and background noise segments at the beginning and end of the first voice stream are removed, and the remaining effective voice segments after removal are windowed for a short time to obtain the windowed voice frame sequence of the first voice stream. A short-time Fourier transform is performed on the windowed speech frame sequence, and the formant trajectory is traced on the transformed spectral amplitude to construct the feature vector to be identified for the first speech stream. Correlation analysis is performed between the feature vector to be identified and the voiceprint feature vector to obtain the similarity score between the speech segment in the triplet training dataset and the first speech stream. Based on the similarity score, the speech segments are ranked and aggregated to obtain the set of nearest matching segments for each speech segment; The frequency of language and scene tags carried by speech segments in the nearest neighbor matching segment set is counted. The language tag with the highest frequency is determined as the language information of the first speech stream, and the scene tag with the highest frequency is determined as the scene information of the first speech stream.

[0034] The formula for calculating the similarity score is as follows: ; In the formula, The similarity score is... For the feature vector to be identified in the th... Component values ​​in each dimension For the voiceprint feature vector at the th Component values ​​in each dimension This represents the total dimension of the feature vector to be identified and the voiceprint feature vector. To obtain the minimum value, To obtain the maximum value.

[0035] Based on preset speech energy thresholds and speech spectrum feature thresholds, frame-by-frame endpoint detection is performed on the first speech stream from the foreign alarm terminal. When the energy value of consecutively detected speech frames is lower than the speech energy threshold and does not exhibit speech spectrum features, it is determined to be a silent segment or background noise segment. Such segments at the beginning and end of the first speech stream are directly cut off. Speech segments that meet the judgment criteria after detection are retained as valid speech segments. Then, the valid speech segments are continuously and shortly slid-trimmed according to a preset fixed time window length. During the trimming process, the time windows remain non-overlapping. All the trimmed speech frames are then orderly integrated according to the trimming sequence to form the first speech stream. The process involves: First, a windowed speech frame sequence is generated. Then, a short-time Fourier transform is performed sequentially on each speech frame in the windowed speech frame sequence, converting the time-domain signal of a single speech frame into frequency-domain spectral amplitude data. After the transformation of all speech frames, formants are extracted from the full spectral amplitude data based on the peak characteristics of the spectral amplitude. The frequency change trajectory of the formants is continuously tracked according to the temporal sequence of the windowed speech frame sequence. The core feature dimensions that uniquely characterize the acoustic features of the speech are extracted from this trajectory. All core feature dimensions are then arranged in an ordered manner according to the extracted temporal sequence and integrated into a fixed-dimensional vector structure. This vector structure is the feature vector to be identified in the first speech stream. That is, the feature vector to be identified is at the th... The specific component values ​​extracted and quantized in each dimension are specific representation values ​​obtained after dimensionalizing and quantizing the acoustic features of the first speech stream.

[0036] The voiceprint feature vectors corresponding to all speech segments in the triplet training dataset are retrieved. These voiceprint feature vectors are obtained from the historical foreign-related police incident recording data by removing silent and non-speech segments to obtain the effective speech segment set. Each effective speech segment in the effective speech segment set is then subjected to frame-segmentation, windowing, and acoustic vector quantization operations to generate the vectors. Is the voiceprint feature vector at the th The specific component values ​​are extracted and quantized in each dimension. These component values ​​are specific representation values ​​obtained by dimensionalizing and quantizing the acoustic features of valid speech segments in historical foreign-related police incident recording data. The total number of dimensions of the feature vector to be identified and the voiceprint feature vector is preset to a uniform value after traversing all voiceprint feature vectors in the triplet training dataset and statistically analyzing their dimensional features. The extraction of the core feature dimensions of the feature vector to be identified strictly follows this uniform value standard to ensure that the number of dimensions of the two is completely consistent. The feature vector to be identified is directly compared with the feature information of each voiceprint feature vector in each dimension, and the degree of fit between the two feature information in each dimension is extracted. The degree of fit of all dimensions is summarized as a whole and quantified into a numerical value, and then combined with... , , The specific numerical values ​​are substituted into the formula to perform the calculation, resulting in a similarity score that objectively reflects the degree of correlation between the acoustic features of the two entities. This formula performs targeted quantitative calculations on the dimensional component values ​​of acoustic features, providing a clear and unique basis for the similarity matching process between the first speech stream and the speech segments in the triplet training dataset. The output similarity score also becomes the sole quantitative criterion for nearest neighbor ranking aggregation of all speech segments in the triplet training dataset.

[0037] All speech segments in the triplet training dataset are sorted in descending order of their similarity scores. If speech segments with the same similarity score exist, they are sorted according to their original storage order in the triplet training dataset. A fixed number of nearest neighbor segments is set according to the practical needs of handling foreign-related police incidents, and the top-ranked speech segments of that number are extracted. All extracted speech segments are then aggregated and integrated according to their sorting order to form an ordered set of speech segments, which is the nearest neighbor matching segment set. For each speech segment in the nearest neighbor matching segment set... Language and scene tags carried by each audio segment are extracted one by one. The audio segments are recorded in writing according to their order in the nearest matching segment set. The record includes the tag name and the corresponding audio segment number. Separate statistical ledgers are established for language tags and scene tags. The tags in the written record are entered into the corresponding ledgers one by one and the specific number of occurrences of each tag is accumulated. The language tag with the most occurrences in the language tag statistical ledger is determined as the language information of the first audio stream, and the scene tag with the most occurrences in the scene tag statistical ledger is determined as the scene information of the first audio stream.

[0038] The beneficial effects are as follows: Endpoint detection is performed from the source of data processing using a clear dual-threshold judgment standard. This ensures the validity of the acoustic features of effective speech segments while eliminating invalid segments without acoustic features, reducing redundant data in subsequent feature processing. Standardized truncation with a fixed time window length provides a unified operational standard for generating windowed speech frame sequences, ensuring consistent sequence format for easier subsequent processing. The combination of short-time Fourier transform and formant trajectory tracking, along with precise extraction of core feature dimensions, allows the feature vector to be identified to accurately and comprehensively represent the acoustic features of the first speech stream. , , The formula calculation allows the similarity score to objectively and quantitatively reflect the degree of matching between speech segments, freeing similarity matching from the interference of subjective judgment. The ranking aggregation method with a fixed number of nearest neighbor segments can accurately lock speech segments with a high degree of matching with the first speech stream, avoiding interference from irrelevant segments to subsequent recognition. The ledger-style statistics after tag extraction provide accurate recording basis for the number of occurrences of language tags and scene tags, and provide clear quantitative judgment standards for the determination of language information and scene information. The overall operation forms a standardized and reproducible language and scene recognition operation system. Different execution entities can obtain the same results by operating according to this system, ensuring the accuracy and uniqueness of the recognition results. The quantitative calculation of the formula also provides an accurate and reproducible quantitative reference standard for matching and screening work, giving the determination of language and scene information of the first speech stream a solid quantitative foundation. This provides accurate and unbiased basic information support for the subsequent screening of target speech segment sets and the generation of scene-aware feature spectra, improving the efficiency and accuracy of subsequent processing links.

[0039] S4. Select a set of target speech segments that match both language and scene information from the triplet training dataset, and extract common features among the target speech segments to generate the scene-aware feature spectrum of the first speech stream. In this embodiment of the invention, the step of selecting a set of target speech segments that match both language information and scene information from the triplet training dataset, and extracting common features among the target speech segments to generate a scene-aware feature spectrum of the first speech stream, includes: Extract the language and scene labels carried by the speech segments in the triplet training dataset, and verify the language and scene labels with the language and scene information of the first speech stream item by item to obtain the target speech segment set of the triplet training dataset. Extract the voiceprint feature vectors corresponding to the target speech segment set from the triplet training dataset; Time-axis normalization is performed on the voiceprint feature vector to obtain the normalized feature vector set. The normalized feature vector set is aggregated dimension by dimension to obtain the aggregated feature vector of the normalized feature vector set; The aggregated feature vectors are reconstructed to obtain the scene-aware feature spectrum of the first speech stream.

[0040] Each speech segment in the triplet training dataset is enumerated one by one. The language label and scene label carried by each speech segment are extracted and recorded in writing according to the speech segment number. The language label of each speech segment in the record is checked for consistency with the language information of the first speech stream. At the same time, the scene label of the speech segment is checked for consistency with the scene information of the first speech stream. Only speech segments whose language label and scene label are completely consistent with the corresponding information are included in the screening scope. All speech segments that meet the double matching requirements after verification are systematically integrated in order of number to form the target speech segment set of the triplet training dataset.

[0041] Based on the unique data identifier of each speech segment in the target speech segment set, a precise targeted retrieval is performed in the triplet training dataset to locate the exclusive storage location of the corresponding voiceprint feature vector for each speech segment in the dataset. According to the arrangement order of the target speech segment set, the voiceprint feature vector corresponding to each speech segment is completely extracted from its respective primary storage location without adding, deleting or modifying any feature values. All extracted voiceprint feature vectors are then orderly collected according to the retrieval order to form a set of voiceprint feature vectors that correspond one-to-one with the target speech segment set.

[0042] First, iterate through the time axis lengths of all voiceprint feature vectors in the triplet training dataset and determine their maximum values. Set this maximum value as the unified reference standard for time axis normalization of voiceprint feature vectors. Detect the time axis length of each extracted voiceprint feature vector. For voiceprint feature vectors with time axis lengths shorter than the reference standard, perform feature value completion at the end of their time axis. The completed feature value should be completely consistent with the last feature value at the end of the vector's time axis. For voiceprint feature vectors with time axis lengths longer than the reference standard, perform feature value truncation on the portion of their time axis that exceeds the reference standard, retaining only the time axis feature values ​​that conform to the reference standard length. Integrate all voiceprint feature vectors that have undergone time axis adjustment in an orderly manner to obtain the normalized feature vector set of voiceprint feature vectors.

[0043] All feature vectors in the normalized feature vector set are uniformly dimension-labeled, and a fixed and unique dimension identifier is assigned to each feature dimension to ensure that the dimension identifiers and arrangement order of all feature vectors remain completely consistent. Feature aggregation operation is performed on each dimension in sequence according to the dimension labeling order. Feature information is fused and extracted from the component values ​​of all feature vectors under the same dimension. Only the component information that can reflect the acoustic commonality of all feature vectors under that dimension is retained. The common component information obtained after aggregation of each dimension is arranged in an orderly manner according to the dimension labeling order to form a feature vector with a fixed dimension. This feature vector is the aggregated feature vector of the normalized feature vector set.

[0044] The dimensional structure of the scene-aware feature spectrum is determined. This system fully includes all feature dimensions of the aggregated feature vector. At the same time, each feature dimension is matched with a corresponding acoustic feature attribute identifier. The common component information of each dimension in the aggregated feature vector is accurately mapped to the corresponding dimension position of the feature spectrum. Each dimension position is assigned a pre-matched acoustic feature attribute identifier. All dimension positions in the feature spectrum are arranged in an orderly manner according to the logical association order of acoustic features. Then, the common component information of each dimension position is transformed into a spectral domain representation form that can be directly recognized by spectral domain feature processing. The complete feature spectrum reconstruction operation of the aggregated feature vector is completed. The resulting spectral domain structure with dimension, attribute, and common component information is the scene-aware feature spectrum of the first speech stream.

[0045] The beneficial effects are as follows: By screening the target speech segment set through dual consistency verification of language and scene labels, it is ensured that the segment set is highly adapted to the language and scene features of the first speech stream. This ensures the scene-specificity of subsequent feature extraction from the data source. Based on the unique data identifier, the voiceprint feature vector is accurately extracted, realizing a one-to-one correspondence between the feature vector and the target speech segment set, avoiding feature matching deviation. Time axis regularization is performed based on a unified standard of the maximum time axis length, eliminating the time axis dimension differences of different feature vectors. This gives the regularized feature vector set a unified processing benchmark, providing a standardized data foundation for subsequent feature aggregation. The dimension-wise feature aggregation method can accurately extract the acoustic common information of all feature vectors, allowing the aggregated feature vectors to fully reflect the core acoustic features of the target speech segment set. Based on the feature spectrum reconstruction composed of fixed dimensions, the aggregated feature vectors are transformed into a scene-aware feature spectrum represented by the spectral domain. This makes the feature information more in line with the scene-specific speech processing needs of foreign-related police calls, providing an accurate and adapted projection basis for the subsequent spectral domain feature projection of the first and second speech streams, ensuring the accuracy and scene adaptability of feature processing in the subsequent cross-language attention mapping stage.

[0046] S5. Based on the scene-aware feature spectrum, perform spectral domain feature projection on the first speech stream and the second speech stream of the alarm receiving end, and input the first joint feature sequence and the second joint feature sequence obtained by the projection into the end-to-end simultaneous interpretation network for cross-language attention mapping to obtain the first simultaneous interpretation text of the first speech stream and the second simultaneous interpretation text of the second speech stream. In this embodiment of the invention, the step of projecting spectral domain features onto the first speech stream and the second speech stream from the alarm receiving end based on the scene-aware feature spectrum, and inputting the resulting first joint feature sequence and second joint feature sequence into an end-to-end simultaneous interpretation network for cross-language attention mapping to obtain the first simultaneous interpretation text of the first speech stream and the second simultaneous interpretation text of the second speech stream includes: The acoustic parameters of the first speech stream are analyzed to obtain the fundamental frequency trajectory and formant distribution of the first speech stream. The fundamental frequency trajectory and formant distribution are then concatenated with parameters to obtain the first original feature sequence of the first speech stream. The acoustic features of the second speech stream are deconstructed to obtain the fundamental frequency trajectory and formant distribution of the second speech stream. The feature dimensions of the fundamental frequency trajectory and formant distribution are then expanded to obtain the second original feature sequence of the second speech stream. Using the scene-aware feature spectrum as the projection basis, the first original feature sequence is subjected to spectral domain weighted projection to obtain the first joint feature sequence of the first speech stream; Using the scene-aware feature spectrum as a projection basis, feature modulation is performed on the second original feature sequence to obtain the second joint feature sequence of the second speech stream; The first joint feature sequence and the second joint feature sequence are input in parallel to the cross-language attention alignment layer of the end-to-end simultaneous interpretation network for attention convergence, to obtain the first context feature of the first joint feature sequence and the second context feature of the second joint feature sequence; The first and second context features are input into the text decoding layer of the end-to-end simultaneous interpretation network for autoregressive decoding to obtain the first simultaneous interpretation text of the first speech stream and the second simultaneous interpretation text of the second speech stream.

[0047] The step of inputting the first joint feature sequence and the second joint feature sequence in parallel into the cross-language attention alignment layer of the end-to-end simultaneous interpretation network for attention convergence, to obtain the first contextual feature of the first joint feature sequence and the second contextual feature of the second joint feature sequence, includes: In the cross-language attention alignment layer of the end-to-end simultaneous interpretation network, the first joint feature sequence and the second joint feature sequence are interacted to generate the association feature matrix between the first joint feature sequence and the second joint feature sequence. Attention weights are assigned to the associated feature matrix to obtain the attention query weight distribution of the first joint feature sequence to the second joint feature sequence, and the attention mapping weight distribution of the second joint feature sequence to the first joint feature sequence. Based on the attention mapping weight distribution, context aggregation is performed on the first joint feature sequence to generate the first context feature of the first joint feature sequence; Based on the attention query weight distribution, information fusion is performed on the second joint feature sequence to generate the second context feature of the second joint feature sequence.

[0048] The acoustic parameters of the first speech stream are extracted frame by frame. Based on the temporal vibration characteristics of the speech signal, the fundamental frequency-related acoustic parameters are analyzed frame by frame. The fundamental frequency parameters of all frames are continuously connected in the temporal order of the speech frames to form a fundamental frequency trajectory that can completely characterize the fundamental frequency variation law of the first speech stream. At the same time, based on the frequency domain spectrum characteristics of the speech signal, the frequency, bandwidth and spatial location parameters of the formants are analyzed frame by frame. The formant-related parameters of all speech frames are integrated to form a formant distribution that can comprehensively reflect the formant characteristics of the first speech stream. The parameter sequence of the fundamental frequency trajectory and the parameter sequence of the formant distribution are linearly concatenated in a seamless manner to integrate the two independent parameter sequences into a single continuous parameter sequence, which is the first original feature sequence of the first speech stream.

[0049] A full-frame acoustic feature deconstruction operation is performed on the second speech stream. The fundamental frequency variation features of the speech signal in the time domain are extracted frame by frame. The fundamental frequency feature parameters of all frames are arranged according to the time sequence of the speech frames to form the fundamental frequency trajectory of the second speech stream. The formant feature information of the speech signal in the frequency domain is extracted frame by frame. The formant frequency, bandwidth and position parameters of all speech frames are integrated to form the formant distribution of the second speech stream. A fixed dimension expansion standard is set with reference to the number of feature dimensions of the scene perception feature spectrum of the first speech stream. Feature dimension completion operation is performed on the parameter sequences of the fundamental frequency trajectory and the formant distribution. The feature values ​​of the completed dimensions maintain the correlation and consistency of acoustic features with the feature values ​​of adjacent dimensions. The fundamental frequency trajectory parameter sequence and the formant distribution parameter sequence after dimension expansion are integrated into a continuous single parameter sequence. This parameter sequence is the second original feature sequence of the second speech stream.

[0050] Using the scene-aware feature spectrum of the generated first speech stream as the unique spectral projection basis, all parameters of the first original feature sequence are completely mapped to the spectral space corresponding to the projection basis. Based on the acoustic importance of each feature dimension in the scene-aware feature spectrum in the international emergency response scenario, corresponding weighting coefficients are assigned to each dimension parameter of the first original feature sequence. The parameters mapped to the spectral space are weighted and adjusted. During the adjustment process, the original correlation of the acoustic features of the first speech stream is preserved. All spectral parameters after weighting and adjustment are integrated into a continuous feature sequence according to the temporal and dimensional order. This feature sequence is the first joint feature sequence of the first speech stream.

[0051] Using the scene-aware feature spectrum of the first speech stream as the sole spectral domain projection basis, all parameters of the second original feature sequence are completely mapped to the spectral domain space corresponding to this projection basis. Based on the spectral domain feature attributes of the scene-aware feature spectrum, feature modulation processing is performed on the parameters of each dimension of the second original feature sequence so that the modulated parameters are fully adapted to the feature processing requirements of the spectral domain space. During the modulation process, the core information of the acoustic features of the second speech stream is preserved. All spectral domain parameters after feature modulation are integrated into a continuous feature sequence according to the dimensional and temporal order. This feature sequence is the second joint feature sequence of the second speech stream.

[0052] In the cross-language attention alignment layer of the end-to-end simultaneous interpretation network, the first joint feature sequence and the second joint feature sequence are processed in parallel input synchronously. The feature information interaction of the two feature sequences is carried out dimension by dimension, and the correlation relationship of the feature information of the two sequences under each dimension is extracted. The feature dimension of the first joint feature sequence is used as the row dimension of the matrix, and the feature dimension of the second joint feature sequence is used as the column dimension of the matrix. The feature correlation information extracted between each dimension is accurately filled into the corresponding row and column intersection positions of the matrix to form a complete matrix structure covering all dimensional correlation relationships. This matrix structure is the correlation feature matrix between the first joint feature sequence and the second joint feature sequence.

[0053] Based on the importance level of the speech acoustic features in the international emergency response scenario, attention weights are assigned to each matrix element in the associated feature matrix. The weight assignment result is positively correlated with the feature importance of the corresponding element. The matrix element with no associated features is assigned a weight of zero. All weights corresponding to each dimension of the first joint feature sequence are then orderly integrated according to the dimensional order of the first joint feature sequence to form the attention query weight distribution of the first joint feature sequence to the second joint feature sequence. At the same time, all weights corresponding to each dimension of the second joint feature sequence are then orderly integrated according to the dimensional order of the second joint feature sequence to form the attention mapping weight distribution of the second joint feature sequence to the first joint feature sequence.

[0054] Using the attention mapping weight distribution of the second joint feature sequence to the first joint feature sequence as the sole aggregation criterion, context aggregation is performed on the feature information of each dimension of the first joint feature sequence. The feature information of each dimension is filtered and integrated according to the magnitude of the weight distribution. Dimensions with high weight values ​​retain complete feature details, while dimensions with low weight values ​​retain only the core acoustic feature information. Irrelevant and redundant feature content is eliminated. All feature information after aggregation is integrated into a feature structure with complete context association. This feature structure is the first context feature of the first joint feature sequence.

[0055] Using the attention query weight distribution of the first joint feature sequence to the second joint feature sequence as the sole fusion basis, cross-sequence information fusion operation is performed on the feature information of each dimension of the second joint feature sequence. The associated feature information of the first joint feature sequence corresponding to the weight distribution is integrated into the corresponding dimension of the second joint feature sequence according to the weight value ratio. Dimensions with higher weight values ​​are integrated with more associated feature information, while dimensions with lower weight values ​​are integrated with a small amount of associated feature information. The core feature of the second joint feature sequence is retained. All feature information after information fusion is integrated into a feature structure with complete contextual association. This feature structure is the second context feature of the second joint feature sequence.

[0056] The first and second context features are simultaneously and in parallel input to the text decoding layer of the end-to-end simultaneous interpretation network. The text decoding layer processes the text using an autoregressive decoding method. Starting from the initial dimension of the context features, it generates corresponding text words sequentially based on the acoustic semantic information contained in the features. The previously generated word serves as the decoding reference for the next word, until the decoding operation of all context feature dimensions is completed. All generated text words are then integrated into a complete and coherent text according to the temporal order of decoding. The text generated by decoding the first context feature is the first simultaneous interpretation text of the first speech stream, and the text generated by decoding the second context feature is the second simultaneous interpretation text of the second speech stream.

[0057] The beneficial effects include the frame-by-frame processing approach for both acoustic parameter analysis of the first speech stream and acoustic feature deconstruction of the second speech stream, ensuring that the fundamental frequency trajectory and formant distribution can fully characterize the acoustic features of the speech streams. Parameter cascading and dimensional expansion allow the original feature sequences to form a standardized parameter structure, adapting to the requirements of subsequent spectral domain processing. Spectral domain weighted projection and feature modulation, using the scene-aware feature spectrum as the projection basis, ensure that the feature information of the joint feature sequence is highly adapted to the international emergency response scenario, improving the scene-specificity of the features. The associated feature matrix generated by feature interaction in the cross-language attention alignment layer achieves full-dimensional processing of the two joint feature sequences. Feature association, based on the importance of acoustic features in the police scene, assigns attention weights to make the weight distribution more in line with the actual needs of handling foreign-related police incidents. Context aggregation and information fusion based on the weight distribution strengthen the contextual relevance of features and eliminate redundant feature information. The autoregressive decoding method of the text decoding layer ensures the semantic coherence and accuracy of the simultaneous interpretation text generation. The entire process realizes the accurate conversion from acoustic features of the speech stream to simultaneous interpretation text, making the processing of cross-language attention mapping more efficient and the feature utilization more fully, which greatly improves the efficiency and accuracy of simultaneous interpretation text generation and fully adapts to the business requirements of real-time foreign language simultaneous interpretation in handling foreign-related police incidents.

[0058] S6. The second simultaneous interpretation text is displayed in real time through the display interface of the alarm receiving terminal, and the first simultaneous interpretation text is input into the speech synthesizer to generate a simultaneous interpretation speech stream with the same language as the first speech stream, which is then sent to the foreign alarm terminal for broadcast.

[0059] In this embodiment of the invention, the step of displaying the second simultaneous interpretation text in real time through the display interface of the alarm receiving terminal, and inputting the first simultaneous interpretation text into a speech synthesizer to generate a simultaneous interpretation speech stream consistent with the language of the first speech stream, and sending it to the foreign alarm terminal for broadcast, includes: Based on the text rendering specifications of the display interface in the alarm receiving terminal, the second simultaneous interpretation text is formatted to obtain the formatted simultaneous interpretation text of the second simultaneous interpretation text. The formatted simultaneous interpretation text is pushed to the text buffer of the display interface and a refresh command is triggered on the display interface, so that the formatted simultaneous interpretation text is displayed on the display interface in real time. Text normalization is performed on the first simultaneous interpretation text to obtain the normalized simultaneous interpretation text. The normalized simultaneous interpretation text is input into the speech synthesizer to synthesize speech parameters, resulting in the simultaneous interpretation speech stream of the normalized simultaneous interpretation text. The simultaneous interpretation voice stream is sent to the foreign alarm terminal through the real-time communication link between the alarm receiving terminal and the foreign alarm terminal, and the foreign alarm terminal's built-in speaker broadcasts the voice message.

[0060] A comprehensive analysis of the text rendering specifications for the police terminal display interface was conducted. These specifications include fixed font type, font size, line spacing standards, character encoding format, and line break rules. The font type is set to sans-serif, the font size is set to 10, the line spacing is a fixed value, the character encoding format is UTF-8, and line breaks are determined solely by complete semantic sentences. Following these specifications, the second simultaneous interpretation text underwent character-by-character encoding verification and format adjustment. This corrected garbled characters, removed redundant spaces between sentences, standardized punctuation, and divided text lines according to line break rules. All formatting indicators of the second simultaneous interpretation text fully met the rendering specifications. The resulting text is the formatted simultaneous interpretation text for the second simultaneous interpretation.

[0061] The alarm receiving terminal display interface is equipped with a dedicated text buffer. This area serves as the dedicated data storage area for the display interface driver module, and it has the functions of real-time data reception, caching, and overwriting. The formatted simultaneous interpretation text is written completely and continuously into the text buffer in the form of a character stream. During the writing process, the old text data stored in the buffer is directly overwritten. After the writing operation is completed, a refresh command with a fixed format is immediately generated and sent to the display interface driver module. After receiving the command, the driver module immediately retrieves the formatted simultaneous interpretation text in the text buffer and renders and presents the text line by line in the designated voice simultaneous interpretation display area of ​​the display interface according to the preset text rendering specifications, so as to realize the real-time echo of the formatted simultaneous interpretation text on the alarm receiving terminal display interface.

[0062] A text standardization standard adapted to the writing conventions of the target language is established. This standard includes rules for uniform character case, rules for deleting redundant characters, rules for standardizing punctuation, and rules for adjusting sentence order. Foreign language characters are uniformly cased according to the general writing conventions of the target language, while Chinese characters are not subject to case adjustment. Only redundant placeholder characters without semantic meaning are deleted from the text. Punctuation is corrected according to the punctuation usage conventions of the target language, and sentence order is adjusted according to the expression habits of the target language. Based on this standard, the first simultaneous interpretation text is checked and adjusted sentence by sentence to ensure that the text is standardized in character format, reasonable in word order, semantically complete, and free of redundant errors. The text obtained after this processing is the normalized simultaneous interpretation text of the first simultaneous interpretation text.

[0063] The normalized simultaneous interpretation text is completely input into the speech synthesizer. The speech synthesizer first performs word segmentation and syllable segmentation on the normalized simultaneous interpretation text according to the syllable rules of the first speech stream language. The segmented syllable information is converted into acoustic parameter instructions that the speech synthesizer can directly recognize. Then, the speech synthesizer retrieves a speech synthesis sound library that is completely consistent with the first speech stream language. This sound library contains all the basic pronunciation acoustic parameters of the target language. According to the acoustic parameter instructions, the corresponding acoustic parameters such as fundamental frequency, formants, and pronunciation duration are matched from the sound library. All the matched acoustic parameters are then continuously synthesized according to the sentence sequence of the normalized simultaneous interpretation text to generate a continuous speech signal stream that conforms to the pronunciation rules of the target language. This speech signal stream is the simultaneous interpretation speech stream of the normalized simultaneous interpretation text.

[0064] A dedicated real-time communication link is pre-established between the alarm receiving end and the foreign alarm end. This link features low-latency and high-stability voice data transmission and uses a fixed voice data frame format for data transmission. The simultaneous interpretation voice stream is encapsulated segment by segment according to this data frame format to form continuous voice data frames. All voice data frames are continuously transmitted to the foreign alarm end through this real-time communication link in the order of their generation. After receiving the voice data frames, the communication module of the foreign alarm end immediately decapsulates them to restore the original simultaneous interpretation voice stream. The restored simultaneous interpretation voice stream is then transmitted to the built-in speaker of the foreign alarm end. After receiving the voice stream, the speaker completes the conversion of electrical signals to sound signals according to standard audio playback parameters, realizing real-time voice broadcast of the simultaneous interpretation voice stream.

[0065] The beneficial effects include: formatting the second simultaneous interpretation text according to the exclusive text rendering specifications of the alarm receiver's display interface, ensuring the standardization and visual readability of the displayed text; and achieving real-time echoing of the simultaneous interpretation text at the alarm receiver through the triggering of text buffer write and refresh commands, allowing alarm receivers to obtain the voice information from foreign alarm receivers in a timely and clear manner. The first simultaneous interpretation text is normalized according to the target language writing specifications to obtain normalized simultaneous interpretation text, ensuring the semantic integrity and format standardization of the text, providing a high-quality text foundation for subsequent speech synthesis. The speech synthesizer generates speech through syllable segmentation, parameter matching, and continuous synthesis. The simultaneous interpretation voice stream, consistent with the first voice stream in language, ensures the accuracy of pronunciation and fluency of speech. Utilizing a dedicated real-time communication link, the voice stream is transmitted with low latency. Voice broadcasts are then delivered via the built-in speaker on the foreign alarm terminal, allowing foreign alarm personnel to receive simultaneous interpretation information from the receiving terminal in real time. This entire process achieves efficient, real-time presentation and broadcasting of bilingual simultaneous interpretation results between the receiving terminal and the foreign alarm terminal during international emergency response, ensuring the timeliness, accuracy, and fluency of bilingual information transmission. It fully adapts to the practical operational needs of international emergency response, effectively improving the overall service quality and execution efficiency of foreign language simultaneous interpretation in international emergency response.

[0066] like Figure 2 The diagram shown is a functional block diagram of a foreign language simultaneous interpretation system for handling foreign-related police calls based on real-time speech translation, provided by an embodiment of the present invention.

[0067] The foreign language simultaneous interpretation system 100 for handling foreign-related police calls based on real-time speech translation described in this invention can be installed in an electronic device. Depending on the functions implemented, the foreign language simultaneous interpretation system 100 may include a triplet training dataset construction module 101, an end-to-end simultaneous interpretation network construction module 102, a language and scene joint recognition module 103, a scene-aware feature spectrum generation module 104, a spectral domain feature projection and simultaneous interpretation mapping module 105, and a simultaneous interpretation result presentation and voice broadcast module 106. The modules described in this invention can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0068] In this embodiment, the functions of each module / unit are as follows: The triplet training dataset construction module 101 is used to deconstruct the voiceprint features of historical foreign-related police recording data to obtain the triplet training dataset of historical foreign-related police recording data. The end-to-end simultaneous interpretation network construction module 102 is used to construct an end-to-end simultaneous interpretation network for the triplet training dataset based on the triplet training dataset. The language and scene joint recognition module 103 is used to perform similarity matching between the language features to be identified in the first voice stream of the foreign alarm terminal and the voice segments in the triplet training dataset to obtain the language information and scene information of the first voice stream. The scene-aware feature spectrum generation module 104 is used to select a set of target speech segments that match both language information and scene information from the triplet training dataset, and extract common features among the target speech segments to generate the scene-aware feature spectrum of the first speech stream. The spectral domain feature projection and simultaneous interpretation mapping module 105 is used to perform spectral domain feature projection on the first speech stream and the second speech stream of the alarm receiving end according to the scene perception feature spectrum, and input the first joint feature sequence and the second joint feature sequence obtained by projection into the end-to-end simultaneous interpretation network for cross-language attention mapping to obtain the first simultaneous interpretation text of the first speech stream and the second simultaneous interpretation text of the second speech stream. The simultaneous interpretation result presentation and voice broadcast module 106 is used to display the second simultaneous interpretation text in real time through the display interface of the alarm receiving terminal, and input the first simultaneous interpretation text into the speech synthesizer to generate a simultaneous interpretation voice stream with the same language as the first voice stream, and send it to the foreign alarm terminal for broadcast.

[0069] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0070] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0071] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0072] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0073] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for simultaneous interpretation in a foreign language for handling police calls based on real-time speech translation, characterized in that, The method includes: S1. Deconstruct the voiceprint features of historical foreign-related police recording data to obtain the triplet training dataset of historical foreign-related police recording data. S2. Based on the triplet training dataset, construct an end-to-end simultaneous interpretation network for the triplet training dataset; S3. Perform similarity matching between the language features to be identified in the first voice stream of the foreign alarm terminal and the voice segments in the triplet training dataset to obtain the language information and scene information of the first voice stream. S4. Select a set of target speech segments that match both language and scene information from the triplet training dataset, and extract common features among the target speech segments to generate the scene-aware feature spectrum of the first speech stream. S5. Based on the scene-aware feature spectrum, perform spectral domain feature projection on the first speech stream and the second speech stream of the alarm receiving end, and input the first joint feature sequence and the second joint feature sequence obtained by the projection into the end-to-end simultaneous interpretation network for cross-language attention mapping to obtain the first simultaneous interpretation text of the first speech stream and the second simultaneous interpretation text of the second speech stream. S6. The second simultaneous interpretation text is displayed in real time through the display interface of the alarm receiving terminal, and the first simultaneous interpretation text is input into the speech synthesizer to generate a simultaneous interpretation speech stream with the same language as the first speech stream, which is then sent to the foreign alarm terminal for broadcast.

2. The method for simultaneous interpretation in a foreign language for handling police calls based on real-time speech translation as described in claim 1, characterized in that, The process of deconstructing the voiceprint features of historical foreign-related police incident recordings yields a triplet training dataset of historical foreign-related police incident recordings, including: Silence and non-voice segments were removed from historical foreign-related police incident recordings to obtain a set of valid voice segments from the historical foreign-related police incident recordings. Each valid speech segment in the set of valid speech segments is subjected to frame-by-frame windowing, and the speech signal after frame-by-frame windowing is subjected to acoustic vector quantization to obtain the speaker feature vector of the valid speech segment. The voiceprint feature vector is associated with the alarm record corresponding to the effective voice segment to obtain the language label and scene label corresponding to the effective voice segment. The effective voice segment, language label and scene label are combined into a triplet sample of historical foreign-related police recording data. Data encapsulation of triplet samples yields a training dataset of triplet samples from historical foreign-related police incident recordings.

3. The method for simultaneous interpretation in a foreign language for handling police calls based on real-time speech translation as described in claim 1, characterized in that, The step of constructing an end-to-end simultaneous interpretation network for the triplet training dataset includes: The acoustic dimension of speech segments in the triplet training dataset is analyzed to construct the acoustic coding layer of the speech segments; Attention weights are assigned to the cross-lingual correspondence between language labels and scene labels in the triplet training dataset to construct a cross-lingual attention alignment layer between language labels and scene labels. Word embedding mapping is performed on the semantic structure and sequence length of bilingual simultaneous interpretation texts in the triplet training dataset to construct the text decoding layer of bilingual simultaneous interpretation texts; The acoustic coding layer, the cross-lingual attention alignment layer, and the text decoding layer are cascaded and stacked to obtain the initial end-to-end network architecture for the triplet training dataset; Using speech segments from the triplet training dataset as input and the corresponding bilingual simultaneous interpretation texts as supervision signals, the initial end-to-end network architecture is iteratively trained until the network converges, resulting in an end-to-end simultaneous interpretation network for the triplet training dataset.

4. The method for simultaneous interpretation in a foreign language for handling police calls based on real-time speech translation as described in claim 2, characterized in that, The step of performing similarity matching between the language features to be identified in the first voice stream of the foreign alarm terminal and the voice segments in the triplet training dataset to obtain the language information and scene information of the first voice stream includes: Endpoint detection is performed on the first voice stream of the foreign alarm terminal, the silent segments and background noise segments at the beginning and end of the first voice stream are removed, and the remaining effective voice segments after removal are windowed for a short time to obtain the windowed voice frame sequence of the first voice stream. A short-time Fourier transform is performed on the windowed speech frame sequence, and the formant trajectory is traced on the transformed spectral amplitude to construct the feature vector to be identified for the first speech stream. Correlation analysis is performed between the feature vector to be identified and the voiceprint feature vector to obtain the similarity score between the speech segment in the triplet training dataset and the first speech stream. Based on the similarity score, the speech segments are ranked and aggregated to obtain the set of nearest matching segments for each speech segment; The frequency of language and scene tags carried by speech segments in the nearest neighbor matching segment set is counted. The language tag with the highest frequency is determined as the language information of the first speech stream, and the scene tag with the highest frequency is determined as the scene information of the first speech stream.

5. The method for simultaneous interpretation in a foreign language for handling police calls based on real-time speech translation as described in claim 4, characterized in that, The formula for calculating the similarity score is as follows: ； In the formula, The similarity score is... For the feature vector to be identified in the th... Component values ​​in each dimension For the voiceprint feature vector at the th Component values ​​in each dimension This represents the total dimension of the feature vector to be identified and the voiceprint feature vector. To obtain the minimum value, To obtain the maximum value.

6. The method for simultaneous interpretation in a foreign language for handling police calls based on real-time speech translation as described in claim 1, characterized in that, The step of selecting a set of target speech segments from the triplet training dataset that matches both language and scene information, and extracting common features among the target speech segments to generate a scene-aware feature spectrum of the first speech stream, includes: Extract the language and scene labels carried by the speech segments in the triplet training dataset, and verify the language and scene labels with the language and scene information of the first speech stream item by item to obtain the target speech segment set of the triplet training dataset. Extract the voiceprint feature vectors corresponding to the target speech segment set from the triplet training dataset; Time-axis normalization is performed on the voiceprint feature vector to obtain the normalized feature vector set. The normalized feature vector set is aggregated dimension by dimension to obtain the aggregated feature vector of the normalized feature vector set; The aggregated feature vectors are reconstructed to obtain the scene-aware feature spectrum of the first speech stream.

7. The method for simultaneous interpretation in a foreign language for handling police calls based on real-time speech translation as described in claim 1, characterized in that, The process involves projecting spectral domain features onto the first speech stream and the second speech stream from the alarm receiving end based on the scene-aware feature spectrum, and then inputting the resulting first and second joint feature sequences into an end-to-end simultaneous interpretation network for cross-language attention mapping to obtain the first simultaneous interpretation text of the first speech stream and the second simultaneous interpretation text of the second speech stream, including: The acoustic parameters of the first speech stream are analyzed to obtain the fundamental frequency trajectory and formant distribution of the first speech stream. The fundamental frequency trajectory and formant distribution are then concatenated with parameters to obtain the first original feature sequence of the first speech stream. The acoustic features of the second speech stream are deconstructed to obtain the fundamental frequency trajectory and formant distribution of the second speech stream. The feature dimensions of the fundamental frequency trajectory and formant distribution are then expanded to obtain the second original feature sequence of the second speech stream. Using the scene-aware feature spectrum as the projection basis, the first original feature sequence is subjected to spectral domain weighted projection to obtain the first joint feature sequence of the first speech stream; Using the scene-aware feature spectrum as a projection basis, feature modulation is performed on the second original feature sequence to obtain the second joint feature sequence of the second speech stream; The first joint feature sequence and the second joint feature sequence are input in parallel to the cross-language attention alignment layer of the end-to-end simultaneous interpretation network for attention convergence, to obtain the first context feature of the first joint feature sequence and the second context feature of the second joint feature sequence; The first and second context features are input into the text decoding layer of the end-to-end simultaneous interpretation network for autoregressive decoding to obtain the first simultaneous interpretation text of the first speech stream and the second simultaneous interpretation text of the second speech stream.

8. The method for simultaneous interpretation in a foreign language for handling police calls based on real-time speech translation as described in claim 7, characterized in that, The step of inputting the first joint feature sequence and the second joint feature sequence in parallel into the cross-language attention alignment layer of the end-to-end simultaneous interpretation network for attention convergence, to obtain the first contextual feature of the first joint feature sequence and the second contextual feature of the second joint feature sequence, includes: In the cross-language attention alignment layer of the end-to-end simultaneous interpretation network, the first joint feature sequence and the second joint feature sequence are interacted to generate the association feature matrix between the first joint feature sequence and the second joint feature sequence. Attention weights are assigned to the associated feature matrix to obtain the attention query weight distribution of the first joint feature sequence to the second joint feature sequence, and the attention mapping weight distribution of the second joint feature sequence to the first joint feature sequence. Based on the attention mapping weight distribution, the first joint feature sequence is aggregated in context to generate the first context feature of the first joint feature sequence. Based on the attention query weight distribution, information fusion is performed on the second joint feature sequence to generate the second context feature of the second joint feature sequence.

9. A method for simultaneous interpretation in a foreign language for handling police calls based on real-time speech translation as described in claim 1, characterized in that, The process of displaying the second simultaneous interpretation text in real time through the alarm receiving terminal's display interface, and inputting the first simultaneous interpretation text into a speech synthesizer to generate a simultaneous interpretation speech stream consistent with the language of the first speech stream, and sending it to the foreign alarm terminal for broadcast, includes: Based on the text rendering specifications of the display interface in the alarm receiving terminal, the second simultaneous interpretation text is formatted to obtain the formatted simultaneous interpretation text of the second simultaneous interpretation text. The formatted simultaneous interpretation text is pushed to the text buffer of the display interface and a refresh command is triggered on the display interface, so that the formatted simultaneous interpretation text is displayed on the display interface in real time. Text normalization is performed on the first simultaneous interpretation text to obtain the normalized simultaneous interpretation text. The normalized simultaneous interpretation text is input into the speech synthesizer to synthesize speech parameters, resulting in the simultaneous interpretation speech stream of the normalized simultaneous interpretation text. The simultaneous interpretation voice stream is sent to the foreign alarm terminal through the real-time communication link between the alarm receiving terminal and the foreign alarm terminal, and the foreign alarm terminal's built-in speaker broadcasts the voice message.

10. A foreign language simultaneous interpretation system for handling foreign-related police calls based on real-time speech translation, characterized in that: The system is used to implement the foreign language simultaneous interpretation method for handling foreign-related police calls based on real-time speech translation as described in claim 1, the system comprising: The triplet training dataset construction module is used to deconstruct the voiceprint features of historical foreign-related police recording data to obtain the triplet training dataset of historical foreign-related police recording data. The end-to-end simultaneous interpretation network construction module is used to construct an end-to-end simultaneous interpretation network for the triplet training dataset based on the triplet training dataset. The language and scene joint recognition module is used to perform similarity matching between the language features to be identified in the first voice stream of the foreign alarm terminal and the voice segments in the triplet training dataset to obtain the language information and scene information of the first voice stream. The scene-aware feature spectrum generation module is used to select a set of target speech segments that match both language information and scene information from the triple training dataset, and extract the common features among the target speech segments to generate the scene-aware feature spectrum of the first speech stream. The spectral domain feature projection and simultaneous interpretation mapping module is used to perform spectral domain feature projection on the first speech stream and the second speech stream of the alarm receiving end according to the scene-aware feature spectrum, and input the first joint feature sequence and the second joint feature sequence obtained by projection into the end-to-end simultaneous interpretation network for cross-language attention mapping to obtain the first simultaneous interpretation text of the first speech stream and the second simultaneous interpretation text of the second speech stream. The simultaneous interpretation result presentation and voice broadcast module is used to display the second simultaneous interpretation text in real time through the display interface of the alarm receiving terminal, and input the first simultaneous interpretation text into the speech synthesizer to generate a simultaneous interpretation voice stream that is consistent with the language of the first voice stream, and send it to the foreign alarm terminal for broadcast.

Images