Multilingual full-speech processing method and device based on speech recognition, and medium

Abstract

The invention discloses a multilingual full-speech processing method and device based on speech recognition and a medium, and relates to the technical field of speech recognition, and the method comprises the steps: calculating a language explicit trajectory based on a multilingual speech feature set, carrying out the association with a speech segment through language preference information in a historical session, and constructing a language implicit trajectory, integrating and generating a language weight track; dividing the language weight trajectory into voice trajectory nodes, recording multi-language voice trajectory composite features, connecting the multi-language voice trajectory composite features into a voice trajectory chain, calculating inter-node continuity indexes, and generating a voice trajectory node structure; constructing a track node continuity credibility field according to the voice track node structure, and adjusting multilingual voice recognition decision parameters to generate a multilingual transcription candidate set; and carrying out time sequence splicing and language mark arrangement on the language transcription candidate set to generate a multi-language full-voice transcription result set. According to the method, self-adaptive decoding of structure perception is realized, language switching is optimized, and the transcription precision is improved.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, and in particular to a multilingual full-speech processing method, device and medium based on speech recognition. Background Technology

[0002] In the development of speech recognition technology, multilingual speech processing has always been a key focus for both academia and industry. With the continuous evolution of deep neural networks, end-to-end modeling architectures, and large-scale multilingual corpora, the performance of speech recognition systems in monolingual scenarios has matured, and word error rates in several mainstream languages ​​have dropped to commercially viable levels. In recent years, research focus has gradually shifted from single-language recognition to multilingual mixed speech understanding. Related technical approaches include constructing multi-task acoustic models with shared underlying representations, introducing language embeddings to achieve cross-lingual feature decoupling, and developing language-adaptive decoders based on attention mechanisms.

[0003] Existing technologies have two main shortcomings: First, current multilingual speech recognition systems generally lack the ability to model the temporal evolution of languages ​​in a detailed manner. They often treat language recognition as a post-processing task independent of speech decoding, ignoring the strong coupling relationship between language state and acoustic features, making it difficult to achieve high-precision language discrimination at the frame or segment level. Second, most systems do not effectively integrate language usage habits from users' historical interaction data, resulting in a lack of context-aware language decision-making mechanisms when faced with ambiguous speech or noise interference, which in turn affects the overall transcription quality and consistency of language annotation. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a multilingual full-speech processing method based on speech recognition to solve the problems of low multilingual recognition accuracy and poor robustness in language switching.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a multilingual full-speech processing method based on speech recognition, which includes, Continuous speech signals are acquired and preprocessed. The preprocessed continuous speech signals are then framed and windowed. Basic acoustic features are extracted and aggregated into speech segments to generate a multilingual speech feature set. Based on multilingual speech feature sets, explicit language trajectories are calculated, and implicit language trajectories are constructed by associating language preference information with speech segments in historical conversations. These are then integrated to generate language weighted trajectories. The language weight trajectory is divided into speech trajectory nodes, and the composite features of multilingual speech trajectories are recorded and connected into a speech trajectory chain. At the same time, the continuity index between nodes is calculated to generate the speech trajectory node structure. Based on the structure of speech trajectory nodes, a continuity credibility field of trajectory nodes is constructed, and the decision parameters for multilingual speech recognition are adjusted to generate a multilingual transcription candidate set. The candidate language transcription set is sequentially spliced ​​and language-labeled to generate a multilingual full speech transcription result set.

[0007] As a preferred embodiment of the multilingual full-speech processing method based on speech recognition described in this invention, the steps of framing and windowing the preprocessed continuous speech signal, extracting basic acoustic features, and then aggregating them into speech segments to generate a multilingual speech feature set are as follows. The preprocessed continuous speech signal is framed and windowed weights are superimposed, and then organized into framed speech data in chronological order. Extract short-time energy, zero-crossing rate, frequency band energy distribution, spectral centroid, fundamental frequency trajectory and formant position from each frame of the framed speech data to generate frame-level basic acoustic features; Speech activity detection is performed on the frame-level basic acoustic features and aggregated into speech segments to generate a multilingual speech feature set.

[0008] As a preferred embodiment of the multilingual full-speech processing method based on speech recognition described in this invention, the step of calculating the explicit language trajectory based on the multilingual speech feature set includes the following steps. Based on the frame-level basic acoustic features in the multilingual speech feature set, the probability value of the candidate language is calculated at each time frame position and combined into an initial explicit language trajectory. Using a sliding time window, the local average probability and local change amplitude of the initial language explicit trajectory are calculated, and the language explicit probability is dynamically adjusted to generate the language explicit trajectory.

[0009] As a preferred embodiment of the multilingual full-speech processing method based on speech recognition described in this invention, the steps of constructing implicit language trajectories by associating language preference information from historical conversations with speech segments, and integrating them to generate language weight trajectories, are as follows: A correspondence is established between language preference information in historical conversations and the timestamp and semantic context of the current speech segment. Implicit preference weights are assigned based on the frequency of language occurrence in historical conversations to generate implicit language trajectories. The explicit and implicit language trajectories are weighted and integrated at each time frame position and then normalized to generate language weighted trajectories.

[0010] As a preferred embodiment of the multilingual full-speech processing method based on speech recognition described in this invention, the steps of dividing the language weight trajectory into speech trajectory nodes, recording the composite features of multilingual speech trajectories, connecting them into a speech trajectory chain, and simultaneously calculating the continuity index between nodes to generate a speech trajectory node structure are as follows. Extract the data range and speech segment boundaries from the language weight trajectory, divide it into multiple speech trajectory node intervals, and generate basic information of the speech trajectory nodes; Within the time range corresponding to the basic information of the speech trajectory nodes, frame-level basic acoustic features of each language are extracted from the multilingual speech feature set and combined into multilingual speech trajectory composite features. Multilingual speech trajectory composite features are added to the basic information of speech trajectory nodes, sorted in chronological order, and adjacent node connections are established to generate a speech trajectory chain. Based on the multilingual speech trajectory composite features of adjacent nodes in the speech trajectory chain, the continuity index between nodes is calculated to generate the speech trajectory node structure.

[0011] As a preferred embodiment of the multilingual full-speech processing method based on speech recognition described in this invention, the step of constructing a continuity reliability field of trajectory nodes according to the speech trajectory node structure is as follows: Multilingual speech trajectory composite features are extracted from the speech trajectory node structure. The continuity index between nodes is converted into initial continuity scores and sorted into the initial trajectory node continuity score distribution. Based on the initial continuity score distribution of trajectory nodes, a grid structure of the continuity confidence field of trajectory nodes is established and filled with the initial continuity scores to generate the continuity confidence field of trajectory nodes.

[0012] As a preferred embodiment of the multilingual full-speech processing method based on speech recognition described in this invention, the steps of adjusting the multilingual speech recognition decision parameters and generating a multilingual transcription candidate set are as follows: Based on the explicit and implicit language trajectories in the language weight trajectory, and combined with the semantic features in historical conversations, the decision factors are statistically estimated to generate multilingual speech recognition decision parameters. The initial continuity score in the continuity confidence field of trajectory nodes is converted into a confidence value, and the decision parameters for multilingual speech recognition are dynamically adjusted in combination with the confidence threshold. Based on the adjusted multilingual speech recognition decision parameters, the speech trajectory node structure is decoded node by node and candidate word sequences and language labels are integrated to generate a multilingual transcription candidate set.

[0013] As a preferred embodiment of the multilingual full-speech processing method based on speech recognition described in this invention, the steps for performing temporal concatenation and language tagging on the language transcription candidate set to generate a multilingual full-speech transcription result set are as follows: The speech trajectory nodes recorded in the multilingual transcription candidate set are organized node by node, and the candidate word sequences are sorted by time and overlapping segments are filled in to generate a language transcription candidate set. Based on the language transcription candidate set, and combined with the confidence value in the continuity confidence field of the trajectory node, the target language transcription result within the node is dynamically selected; Based on the speech trajectory chain, cross-node boundary checks and language switching marking are performed on the node-level language transcription results to generate a global language transcription sequence. The global language transcription sequence is merged into word sequences and time-stamped to generate language transcription paragraphs. The language-transcribed paragraphs, language weight trajectories, speech trajectory node structures, and trajectory node continuity credibility fields are integrated to generate a multilingual full speech transcription result set.

[0014] In a second aspect, the present invention provides a computer device, including a memory and a processor, wherein the memory stores a computer program, wherein when the computer program is executed by the processor, it implements any step of the multilingual full-speech processing method based on speech recognition as described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the multilingual full-speech processing method based on speech recognition as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: by fusing explicit and implicit language trajectories to generate language weight trajectories, dynamic contextual modeling of language states in speech recognition is realized, which improves the accuracy of multilingual discrimination and enhances the robustness of speech recognition; by constructing a continuity credibility field of speech trajectory nodes and dynamically adjusting decoding parameters, structure-aware adaptive decoding of speech recognition is realized, which optimizes language switching boundary processing and improves the accuracy of speech recognition transcription. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1This is a flowchart of a multilingual full-speech processing method based on speech recognition.

[0019] Figure 2 This is a flowchart for multilingual speech signal preprocessing and feature extraction.

[0020] Figure 3 A flowchart for language trajectory analysis and node structure modeling.

[0021] Figure 4 A flowchart for synthesizing and generating reports from multilingual transcription results. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a multilingual full-speech processing method based on speech recognition, including the following steps: S1: Acquire continuous speech signals and preprocess them. Then, frame and window the preprocessed continuous speech signals, extract basic acoustic features, and aggregate them into speech segments to generate a multilingual speech feature set.

[0026] S1.1: Continuous speech signals include speech waveform amplitude sequence, time stamp information, and vocal tract information.

[0027] Furthermore, the changes in the sound pressure of speech in the air are converted into electrical signals that vary over time using a microphone. The electrical signals are then periodically sampled by an analog-to-digital converter at a uniform sampling frequency. At each sampling moment, the corresponding amplitude value is recorded and arranged in the sampling order to form a speech waveform amplitude sequence. At the same time, a unique time stamp is assigned to each sampling point in the sampling clock according to the sampling start time and sampling interval (e.g., recording the sampling moment index in milliseconds). In multi-channel acquisition scenarios, channel information is added to each sampling point (e.g., marked as left channel, right channel, or other channel numbers). Through the above sampling and marking process, the continuous changes of speech in the time dimension are completely mapped into a data structure composed of three parts: speech waveform amplitude sequence, time stamp information, and channel information.

[0028] S1.2: Preprocessing includes bandpass filtering, ambient noise suppression, echo suppression, volume normalization, and dynamic range control.

[0029] Furthermore, in the frequency domain, for continuous speech signals, low-frequency interference components below 50Hz and high-frequency noise components above 8000Hz are suppressed by bandpass filtering, retaining only the speech signal portion within the effective speech frequency band. Silent segments without speech activity are selected from the bandpass-filtered speech signal to estimate environmental noise characteristics. Environmental noise suppression is performed on the entire speech signal through suppression processes such as spectral subtraction or noise masking. In scenarios where echoes exist at the other end, echo path estimation results are constructed based on the far-end reference signal. The estimated echo components are then extracted from the speech signal in the time or frequency domain. After removing echo suppression, the energy distribution of the entire speech signal is statistically analyzed on the speech signal after environmental noise suppression and echo suppression. The overall gain of the speech signal is adjusted according to the target loudness range to achieve volume normalization. A dynamic range control parameter set is set according to the instantaneous amplitude and overall energy distribution of the speech signal. High amplitude parts exceeding the dynamic range control threshold are compressed, and low amplitude parts are appropriately boosted to reduce the amplitude span. This ensures that the speech signal after continuous processing meets the requirements of subsequent processing in terms of frequency range, noise level, echo residue, overall volume, and amplitude dynamic range.

[0030] It should be noted that the dynamic range control threshold (example range: 60% to 90% of the maximum effective amplitude of the speech signal) is determined based on the long-term energy distribution of the continuous speech signal in a quiet speaking state, and the upper limit is determined based on the statistical results of the peak amplitude of the continuous speech signal in a scenario of intense speaking or superimposed environmental noise.

[0031] S1.3: The preprocessed continuous speech signal is framed and windowed weights are added, and the frames are organized into speech data in chronological order.

[0032] Furthermore, the frame length and frame shift are set according to the time resolution required for speech analysis (for example, the frame length is set to 25 milliseconds and the frame shift is set to 10 milliseconds, determined based on the rate of change of the speech signal on the time axis and the requirement for short-term stationarity). Based on the sampling point sequence of the continuous speech signal, several overlapping speech frames are formed by truncating the continuous speech signal according to the frame length. The next frame of speech signal is truncated by advancing forward on the time axis according to the frame shift. A windowing weight is applied to the sampling point sequence of each frame. The abrupt boundary effect at both ends of the speech frame is suppressed by multiplying the windowing function sequence with the sampling point sequence of the speech frame. The windowed speech frames are arranged in chronological order and the time stamp information corresponding to each frame is retained. In this way, the framed speech data organized in chronological order is formed in a way that continuously covers the entire preprocessed continuous speech signal.

[0033] It should be noted that the windowing function sequence is a set of fixed-length weight coefficients generated according to the frame length. It provides a smooth weight distribution between 0 and 1 for each sample point on the time axis from the start position to the end position of the speech frame. For example, the weight is larger at the center position of the frame and gradually decreases at the ends of the frame. It is used to apply different levels of contribution to different positions of the speech frame in subsequent calculations.

[0034] S1.4: Extract short-time energy, zero-crossing rate, frequency band energy distribution, spectral centroid, fundamental frequency trajectory and formant position from each frame of the framed speech data to generate frame-level basic acoustic features.

[0035] Furthermore, based on the windowed speech frame sampling point sequence, the amplitude values ​​are squared and accumulated point by point to obtain short-time energy, and the alternating amplitude changes in the speech frame are identified based on the number of positive and negative changes in the amplitude sign to obtain the zero-crossing rate; in frequency domain processing, a spectral transformation is performed on the speech frame sampling point sequence to obtain the spectral amplitude distribution, and the spectrum is accumulated and distributed to form a frequency band energy distribution. At the same time, the horizontal position of the spectral centroid is determined based on the distribution of spectral amplitude at different frequency points; based on the autocorrelation or cepstral shape of the speech frame, the position of periodic peaks is identified within the possible range of the fundamental frequency (e.g., 50Hz to 400Hz) to obtain the fundamental frequency trajectory; based on the spectral envelope of the smoothed speech frame, the position of the energy concentration region is identified to obtain the formant position. The short-time energy, zero-crossing rate, frequency band energy distribution, horizontal position of the spectral centroid, fundamental frequency trajectory, and formant position are combined according to the time sequence of the speech frame to form the frame-level basic acoustic features.

[0036] It should be noted that the speech frame spectral envelope is a continuous curve formed by smoothing the spectral amplitude distribution of the speech frame. It reflects the energy distribution trend of speech in different frequency ranges and is used to represent the overall resonant structure characteristics of speech.

[0037] S1.5: Perform speech activity detection on the frame-level basic acoustic features and aggregate them into speech segments to generate a multilingual speech feature set.

[0038] Furthermore, based on the overall distribution of short-time energy, zero-crossing rate, frequency band energy distribution, and the horizontal position of the spectral centroid in the frame-level basic acoustic features, a speech activity detection parameter set is constructed to distinguish between valid speech frames and silent frames by statistically analyzing the range of short-time energy values, the range of amplitude sign changes, and the concentration of frequency band energy in the main frequency bands across the entire segment of framed speech data. During speech activity detection, each frame is judged as a valid speech frame based on the energy range threshold, amplitude sign change range, and frequency band energy concentration in the speech activity detection parameter set. When several consecutive frames simultaneously exhibit short-time energy higher than the speech activity detection parameter set, zero-crossing rate falling within the speech activity detection parameter set, and frequency band energy distribution showing a concentrated speech energy pattern, these consecutive frames are marked as valid speech frames. On the time axis, consecutive valid speech frames are aggregated into speech segments. The start and end times of each speech segment are recorded, and all frame-level basic acoustic features within the speech segment are organized. All speech segments are combined in chronological order to form a speech segment set, and the frame-level basic acoustic features corresponding to the speech segment set are reorganized into a multilingual speech feature set according to the boundaries of the speech segments.

[0039] It should be noted that the energy range threshold (example range: 20% to 60% of the short-time energy statistical distribution) is determined based on the long-term stable range of silent frames and weak speech frames in the short-time energy distribution, and the upper limit is determined based on the peak range of high-energy speech frames in the short-time energy distribution.

[0040] The amplitude sign variation range (example range: 10% to 50% of the zero-crossing rate statistical distribution), where the lower limit is determined based on the characteristic that the zero-crossing rate in silent frames and background noise frames remains at a low variation level for a long time, and the upper limit is determined based on the periodic alternating frequency of vocal cord vibration during normal vocalization.

[0041] S2: Based on multilingual speech feature sets, calculate explicit language trajectories, and construct implicit language trajectories by associating language preference information with speech segments in historical conversations, and integrate to generate language weighted trajectories.

[0042] S2.1: Based on the frame-level basic acoustic features in the multilingual speech feature set, calculate the candidate language probability value at each time frame position and combine them into an initial language explicit trajectory.

[0043] Furthermore, based on the frame-level basic acoustic features corresponding to each time frame in the multilingual speech feature set, such as short-time energy, frequency band energy distribution, fundamental frequency trajectory, formant position, and horizontal position of the spectral centroid, acoustic pattern matching is performed on the frame-level basic acoustic features in combination with the preset multilingual acoustic modeling parameter set. By scoring the degree of matching between the acoustic feature patterns of different languages ​​and the current frame-level basic acoustic features, candidate language probability values ​​are generated at each time frame position to represent the possibility that the time frame belongs to different languages. The candidate language probability values ​​of all time frame positions are arranged in chronological order to form an initial explicit language trajectory composed of a continuous probability distribution sequence.

[0044] It should be noted that the preset multilingual acoustic modeling parameter set is a set of parameters that is statistically organized and fixed for each language in terms of the range of values, joint distribution relationship, and feature combination used to distinguish different languages ​​in terms of short-time energy, frequency band energy distribution, fundamental frequency trajectory, formant position and horizontal position of spectral centroid.

[0045] The formula for calculating the probability value of candidate languages ​​is: ; in, Represents time frame Language The probability value of candidate languages, This represents the smoothing adjustment coefficient for the matching score. Represents time frame The corresponding frame-level basic acoustic feature vector, Represents the multilingual acoustic modeling parameter set and language The corresponding parameter set, Represents the multilingual acoustic modeling parameter set and language The corresponding parameter set, This represents the set of languages ​​involved in the decision-making process.

[0046] It should be noted that the smoothing adjustment coefficient (example range: 0.5 to 2.0) is fixed after statistically analyzing the dispersion of matching score distributions in different languages ​​during multilingual acoustic modeling and determining the numerical range used to adjust the amplification or compression of matching scores based on the sensitivity requirements of the scores in the probability values ​​of candidate languages.

[0047] S2.2: Using a sliding time window, calculate the local average probability and local change amplitude of the initial language explicit trajectory, and dynamically adjust the language explicit probability to generate the language explicit trajectory.

[0048] Furthermore, a sliding time window (e.g., a sliding time window covering 200 milliseconds of speech duration) is set on the time axis for the initial explicit language trajectory. Within the time range covered by each sliding time window, the probability values ​​of candidate languages ​​at all time frame positions are read sequentially along the time axis. The probability values ​​of candidate languages ​​at all time frame positions within the window for the same language are summed and normalized according to the number of time frames to obtain the local average probability. At the same time, the local variation amplitude is obtained based on the degree of difference in the probability values ​​of candidate languages ​​at adjacent time frame positions. The probability of languages ​​with higher local average probabilities and stable local variation amplitudes is appropriately increased at the corresponding time frame positions, while the probability of languages ​​with lower local average probabilities or abrupt changes in local variation amplitudes is decreased at the corresponding time frame positions. The above processing is repeated as the sliding time window moves frame by frame along the time axis, so that the probability of candidate languages ​​at each time frame position is dynamically adjusted under the constraints of the preceding and following time context. The language probability sequence at all time frame positions after the sliding time window processing is recombined in chronological order to form an explicit language trajectory that changes continuously on the time axis and reflects the stable and switching intervals of languages.

[0049] The formula for calculating the explicit trajectory of a language is: ; in, Indicates explicit language trajectory Indicates time frame The normalization factor on, This represents the local average probability adjustment coefficient. Indicates time frame The set of time frame indices for the set sliding time window on the time axis. Indicates the sliding time window Internal targeting language For all time frames The local cumulative probability, obtained by summing the candidate language probabilities for a given location, is used to determine the language. In time frame Local average probability within the neighborhood. This represents the adjustment coefficient for the local variation amplitude. Indicates the sliding time window Internal targeting language For all time frames The summation of the probability differences of candidate languages ​​in adjacent time frames is used to reflect the language. In time frame The overall magnitude of the change in the probability of languages ​​within the neighborhood over time. Represents time frame Language The probability value of candidate languages, Represents time frame Language The probability value of candidate languages.

[0050] It should be noted that the local average probability adjustment coefficient (example range: 0.2 to 0.6) is fixed after statistically analyzing the stability level of different languages ​​in the time neighborhood of the multilingual speech feature set and determining its influence ratio on the stationary component of the candidate language probability value based on the requirement of the local average probability to smooth the explicit trajectory of the language. The local variation amplitude adjustment coefficient (example range: 0.1 to 0.4) is determined and fixed after determining the adjustment ratio used to control the influence intensity of the change trend term, based on the distribution of the change rate of candidate language probability values ​​during multilingual speech switching, combined with the need to balance the suppression of abrupt segments and the maintenance of sensitivity to stable segments.

[0051] S2.3: Establish a correspondence between language preference information in historical conversations and the timestamp and semantic scene of the current speech segment, and assign implicit preference weights according to the frequency of language occurrence in historical conversations to generate implicit language trajectories.

[0052] Furthermore, the preference intensity, usage frequency, and corresponding contextual probability of each language in a specific semantic scenario are read. The timestamp of the current speech segment is matched with the preference intensity records of the same time period in the historical conversation, and the semantic scenario of the current speech segment is matched with the preference intensity records of the same semantic scenario in the historical conversation. The time-matching and semantic-matching results are weighted and combined to obtain the implicit preference weight allocation result of the current speech segment in multiple languages. The implicit preference weight allocation result is then expanded frame by frame according to the time frame covered by the speech segment to form a language implicit preference weight sequence, so that each time frame position corresponds to an implicit preference weight value. The implicit preference weight allocation result is arranged in chronological order and continuously spliced ​​for each speech segment to form a language implicit trajectory covering the time range of all speech segments.

[0053] S2.4: The explicit and implicit language trajectories are weighted and integrated at each time frame position and normalized to generate the language weighted trajectory.

[0054] Furthermore, on the timeline, the language probability sequence and the language implicit preference weight sequence corresponding to each time frame position in the explicit language trajectory and the implicit language preference weight sequence in the implicit language trajectory are read sequentially according to the time frame order. For each language in the language set, the explicit language probability value is extracted from the explicit language trajectory and the implicit language preference weight value is extracted from the implicit language trajectory. The explicit language probability value and the implicit language preference weight value are weighted and superimposed to form an unnormalized language fusion score sequence. At each time frame position, the unnormalized language fusion score is normalized to ensure that all language fusion scores fall within a unified probability range and satisfy the normalization constraint in the language dimension. The normalized language fusion scores are arranged sequentially according to the time frame order to form a language weight trajectory covering all time frame positions.

[0055] S3: Divide the language weight trajectory into speech trajectory nodes, record the composite features of multilingual speech trajectories, connect them into a speech trajectory chain, and calculate the continuity index between nodes to generate the speech trajectory node structure.

[0056] S3.1: Extract the data range and speech segment boundaries from the language weight trajectory, divide them into multiple speech trajectory node intervals, and generate basic information of the speech trajectory nodes.

[0057] Furthermore, the timestamps corresponding to each time frame position in the language weight trajectory are read along the time axis to determine the start and end times covered by the language weight trajectory as the data range of the language weight trajectory. At the same time, the start and end times of each speech segment in the speech segment set are read from the multilingual speech feature set. The data range of the language weight trajectory is compared with the time boundaries in the speech segment set, and silent time intervals that do not fall within the time range of the speech segments are eliminated. Within the time range covered by the speech segments, the target duration of the speech trajectory nodes (e.g., from 300 milliseconds to 800 milliseconds in the example, based on the duration range of a single continuous semantic unit in the multilingual speech segment in the time dimension) and the target number of frames are set. Within each speech segment, the continuous time frames are divided into multiple speech trajectory node intervals according to the time order. The corresponding start time, end time, covered time frame index range, and associated speech segment number are recorded for the speech trajectory node intervals. They are arranged in chronological order and assigned unique speech trajectory node numbers to generate the basic information of the speech trajectory nodes.

[0058] It should be noted that the basic information of the speech trajectory nodes includes structural content used to describe each speech trajectory node interval, such as the start time, end time, covered time frame index range, associated speech segment number, and corresponding speech trajectory node number of the speech trajectory node interval.

[0059] S3.2: Within the time range corresponding to the basic information of the speech trajectory nodes, extract the frame-level basic acoustic features of each language from the multilingual speech feature set and combine them into multilingual speech trajectory composite features.

[0060] Furthermore, following the order of the voice trajectory node numbers recorded in the basic information of the voice trajectory nodes, the start time, end time, and covered time frame index range of each voice trajectory node interval are read sequentially. Data records whose timestamps fall within the voice trajectory node interval are retrieved from the multilingual voice feature set based on the start and end times. Then, frame-level basic acoustic features at the corresponding time frame positions are filtered from the multilingual voice feature set according to the time frame index range. During the retrieval process, the frame-level basic acoustic feature field is read for each time frame position. All frame-level basic acoustic features belonging to the same voice trajectory node interval are rearranged in chronological order. Based on the sequential arrangement, the frame-level basic acoustic features are structured and combined: on the one hand, a corresponding multilingual speech trajectory composite feature record is established for each speech trajectory node interval, and all frame-level basic acoustic features within the time frame index range covered by the speech trajectory node interval are stored in the record; on the other hand, the one-to-one correspondence between the speech trajectory node number, start time, end time and frame-level basic acoustic features is maintained, so that the frame-level basic acoustic features stored in a centralized and dispersed manner are aggregated and recombined according to the speech trajectory node interval within the time range limited by the basic information of the speech trajectory node, and combined into multilingual speech trajectory composite features.

[0061] S3.3: Add multilingual speech trajectory composite features to the basic information of speech trajectory nodes, sort them in chronological order and establish the connection relationship between adjacent nodes to generate a speech trajectory chain.

[0062] Furthermore, the voice trajectory node number, start time, end time, and covered time frame index range recorded in the basic information of the voice trajectory nodes are used as additional positions. The corresponding multilingual voice trajectory composite feature records are read one by one according to the voice trajectory node number, and the multilingual voice trajectory composite features are appended to the structure of the basic information of the voice trajectory nodes as a whole. This ensures that each basic information of the voice trajectory nodes contains multilingual voice trajectory composite features formed by combining them in chronological order within the covered time range. After appending, the basic information of all voice trajectory nodes is sorted according to the start time field in the basic information of the voice trajectory nodes. A time sorting sequence is generated by sorting the start time values ​​from earliest to latest, so that all the basic information of the voice trajectory nodes is arranged in natural order on the time axis. The voice trajectory node numbers of adjacent basic information of the voice trajectory nodes are read from the time sorting sequence. By establishing forward connections between the numbers, the chronological order of adjacent basic information of the voice trajectory nodes is recorded. Based on all the basic information of the voice trajectory nodes covered by the time sorting sequence, continuous connections are established between adjacent numbers, so that the sorted basic information of the voice trajectory nodes forms a continuous connection node by node on the time axis, generating a voice trajectory chain.

[0063] S3.4: Based on the multilingual speech trajectory composite features of adjacent nodes in the speech trajectory chain, calculate the continuity index between nodes and generate the speech trajectory node structure.

[0064] Furthermore, the multilingual speech trajectory composite features of adjacent speech trajectory nodes are read sequentially in the speech trajectory chain. By aligning the time frame index ranges covered by the two speech trajectory nodes in the time frame dimension, the difference and continuity of change between adjacent speech trajectory nodes are calculated for short-time energy, zero-crossing rate, frequency band energy distribution, horizontal position of spectral centroid, fundamental frequency trajectory, and formant position. The difference and continuity of change are then weighted and synthesized to generate an inter-node continuity index that reflects the overall continuity between the two speech trajectory nodes. The inter-node continuity index is repeatedly calculated for all pairs of adjacent speech trajectory nodes in the speech trajectory chain, and each inter-node continuity index is attached to the corresponding adjacent speech trajectory node connection relationship record. This makes the speech trajectory node number, basic information of the speech trajectory node, multilingual speech trajectory composite features, and inter-node continuity index form an ordered association structure, thus forming the speech trajectory node structure.

[0065] The formula for calculating the continuity index between nodes is: ; in, Indicators representing continuity between nodes. The weighting coefficients represent the balance between the similarity of static features and the consistency of dynamic changes. Represents speech trajectory nodes With voice trajectory nodes The set of time frame indices that are jointly covered after alignment on the timeline. Indicates the time frame index is Time belongs to the speech trajectory node Multilingual speech trajectory composite feature vector Indicates the time frame index is Time belongs to the speech trajectory node Multilingual speech trajectory composite feature vector Indicates the time frame index is Time belongs to the speech trajectory node Multilingual speech trajectory composite feature vector Indicates the time frame index is Time belongs to the speech trajectory node Multilingual speech trajectory composite feature vector.

[0066] It should be noted that the weighting coefficient for balancing static feature similarity and dynamic change consistency (example range: 0.3-0.7) is an adjustment coefficient that is fixed after statistically comparing the influence of static feature differences and dynamic change differences on the subjective judgment of continuity in a large number of multilingual speech samples and determining the relative contribution ratio of the two types of differences in the continuity index between nodes.

[0067] S4: Based on the speech trajectory node structure, construct the trajectory node continuity confidence field, adjust the multilingual speech recognition decision parameters, and generate a multilingual transcription candidate set.

[0068] S4.1: Extract the multilingual speech trajectory composite features from the speech trajectory node structure, convert the continuity index between nodes into initial continuity scores, and sort them into the initial trajectory node continuity score distribution.

[0069] Furthermore, in the speech trajectory node structure, each adjacent speech trajectory node connection record is read in chronological order of the speech trajectory chain. For each adjacent speech trajectory node connection record, multilingual speech trajectory composite features corresponding to the preceding and subsequent speech trajectory nodes, as well as the inter-node continuity index, are extracted. The inter-node continuity index is then subjected to numerical normalization and interval compression, mapping it to a unified continuous numerical range (e.g., the interval from Example 0 to Example 1) according to a monotonic correspondence. During the mapping process, a quantization function for the speech trajectory node connection is generated for each adjacent speech trajectory node connection record. An initial continuity score for stability is determined, and this initial continuity score is bound and stored in conjunction with the speech trajectory node number, the preceding speech trajectory node, and the multilingual speech trajectory composite features of the subsequent speech trajectory nodes. Following the temporal or node order of the connection relationships between adjacent speech trajectory nodes in the speech trajectory chain, all initial continuity scores are arranged into a numerical sequence arranged in connection order. Simultaneously, a continuity strength distribution perspective is established based on numerical magnitude as needed, ensuring that the initial continuity scores corresponding to each adjacent speech trajectory node connection in the speech trajectory node structure are continuously arranged in the time dimension to form the initial trajectory node continuity score distribution.

[0070] It should be noted that the continuity strength distribution perspective refers to the sorting view formed by rearranging the initial continuity scores in descending or ascending order according to their values ​​after obtaining all initial continuity scores, in order to observe the differences in the stability of node connections. The method of establishing this view is to sort the initial continuity scores and generate the corresponding ordered distribution, which is used to intuitively present the relative relationship between the continuity strength of different speech trajectory node connections.

[0071] S4.2: Based on the initial continuity score distribution of trajectory nodes, establish a grid structure for the continuity confidence field of trajectory nodes, fill in the initial continuity scores, and generate the continuity confidence field of trajectory nodes.

[0072] Furthermore, based on the time range covered by the speech trajectory node structure and the number of connections between adjacent speech trajectory nodes, the number of raster divisions in the time dimension is determined according to the total time axis duration and the time resolution required for continuity analysis. The number of raster layers in the continuity dimension is also set based on the number of connections between adjacent speech trajectory nodes, ensuring the raster structure can fully cover the time distribution range and variation range of all initial continuity scores. This results in a two-dimensional raster with a fixed time step and a fixed number of continuity layers. The mapping position of adjacent speech trajectory node connections is determined within the raster structure by reading the corresponding time range, and then... The initial continuity score is filled into the corresponding grid node in a temporal sequence, so that the initial continuity scores in adjacent time intervals form a continuous arrangement in the grid structure. During the filling process, the initial continuity scores are directly assigned according to the time axis resolution and continuity credibility range of the grid structure, so that the values ​​in each grid node are consistent with the distribution of the initial trajectory node continuity scores. After all grid nodes are filled, the grid structure forms a continuous credibility expression region from the start time to the end time of the speech trajectory node structure in the time dimension. The trajectory node continuity credibility field is generated by the set including the grid structure and all initial continuity scores.

[0073] S4.3: Based on the explicit and implicit language trajectories in the language weight trajectory, combined with the semantic features in historical conversations, the decision factors are statistically estimated to generate multilingual speech recognition decision parameters. Furthermore, based on the explicit and implicit language trajectories in the language weight trajectory, the language probability distribution corresponding to each time frame position is read along the time axis. The local stable intervals and language switching intervals recorded in the explicit language trajectory are time-aligned with the historical preference trends recorded in the implicit language trajectory. Semantic scene labels, context change positions, and semantic span information are read from the semantic features in historical conversations. The direction of change, magnitude of change, and stability delay of the explicit and implicit language trajectories at the semantic scene boundary are statistically summarized. Based on the statistical results, interval estimation and weighted updates are performed on the decision factor set to which the decision factor belongs, so that each decision factor tends to take a stable value in the continuous language interval and increases its sensitivity in the language switching interval. After completing the statistical estimation at all time frame positions, the updated decision factor set is reorganized according to the usage order of the language recognition process to generate multilingual speech recognition decision parameters.

[0074] S4.4: Convert the initial continuity score in the continuity confidence field of trajectory nodes into confidence values, and dynamically adjust the multilingual speech recognition decision parameters in combination with the confidence threshold.

[0075] Furthermore, the initial continuity scores in the trajectory node continuity confidence field are converted into confidence values. The confidence value corresponding to each grid node is read along the time axis in the trajectory node continuity confidence field, and interval comparisons are performed on the confidence values ​​based on a confidence threshold. Within the time range where the confidence value is higher than the confidence threshold, enhancement processing is performed on the multilingual speech recognition decision parameters, adjusting the decision parameters related to language stability towards stable decoding. Within the time range where the confidence value is lower than the confidence threshold, suppression processing is performed on the multilingual speech recognition decision parameters, adjusting the decision parameters related to language switching or unstable regions towards conservative decoding. Within the time range where the confidence value is within the confidence threshold transition range, smooth adjustments are made based on the difference between the confidence value and the confidence threshold using a continuous mapping method. This allows the multilingual speech recognition decision parameters to form a dynamic adaptive process along the time axis that changes with the continuity confidence of the trajectory nodes. The adjusted multilingual speech recognition decision parameters are then aligned with the speech trajectory node structure in the time dimension.

[0076] It should be noted that the credibility threshold (example range: 40% to 70% of the numerical range of the continuity credibility field of trajectory nodes) is set with the lower limit based on the long-term lower bound of the continuity credibility of regions with obviously unstable languages ​​in the speech trajectory node structure, and the upper limit based on the statistical upper bound of the continuity credibility of regions with basically stable languages ​​but still with a small number of switching risks in the speech trajectory node structure.

[0077] S4.5: Based on the adjusted multilingual speech recognition decision parameters, decode the speech trajectory node structure node by node and integrate the candidate word sequence and language label to generate a multilingual transcription candidate set.

[0078] Furthermore, the speech trajectory node numbers, basic information, and multilingual speech trajectory composite features are sequentially read along the time sequence of the speech trajectory node structure. Based on the adjusted multilingual speech recognition decision parameters, acoustic pattern matching and language discrimination are performed on the time frame range covered by each speech trajectory node. Acoustic features such as short-time energy, zero-crossing rate, frequency band energy distribution, fundamental frequency trajectory, and formant positions are read. The acoustic patterns of candidate languages ​​are matched and scored according to the adjusted multilingual speech recognition decision parameters, and the language of the current speech trajectory node is determined based on the scoring results. The search range of tags and candidate words establishes a direct correspondence between acoustic pattern matching results and subsequent candidate word generation, generating candidate word sequences and corresponding language tags belonging to the current speech trajectory node. After completing node-by-node decoding on all speech trajectory nodes, the candidate word sequences and language tags of each speech trajectory node are recombined according to the speech trajectory node number order, maintaining the continuity of the speech trajectory node structure on the time axis, so that the candidate word sequences are seamlessly spliced ​​in time order at the boundaries of speech trajectory nodes, and the sequentially spliced ​​candidate word sequences and language tags form a multilingual transcription candidate set.

[0079] S5: Perform temporal splicing and language tagging on the language transcription candidate set to generate a multilingual full speech transcription result set.

[0080] S5.1: Organize the speech trajectory nodes recorded in the multilingual transcription candidate set node by node, and sort the candidate word sequences by time and fill in overlapping segments to generate a language transcription candidate set.

[0081] Furthermore, in the multilingual transcription candidate set, speech trajectory node records are read one by one in order of their numbers. For each speech trajectory node, the start time, end time, corresponding candidate word sequence, and time stamp information are read. Within each candidate word sequence, time sorting is performed according to the time stamp information, ensuring that the candidate word sequences within the same speech trajectory node maintain a monotonically increasing order from start to end time. Based on the time range covered by the speech trajectory node, it is checked whether there are time gaps or overlaps in the candidate word sequences near the start and end times of the speech trajectory node. Within the time gap intervals, the speech trajectory node is sorted according to the speech trajectory node's time coverage. The coverage of the speech trajectory nodes extends the start and end times of candidate words or shortens the intervals between adjacent candidate words to complete the temporal continuity within the speech trajectory nodes. In the time overlap interval, the start and end times of overlapping candidate words are adjusted according to the candidate word confidence parameter set, or redundant overlapping candidate words are deleted to eliminate duplicate coverage. After the candidate word sequence is sorted within all speech trajectory nodes, the candidate word sequences of each speech trajectory node are spliced ​​in chronological order according to the speech trajectory node number order to form a continuous candidate word sequence covering the entire speech trajectory node structure, while keeping the language label and speech trajectory node number corresponding to each candidate word unchanged, generating a language transcription candidate set.

[0082] S5.2: Based on the language transcription candidate set, and combined with the credibility value in the continuity credibility field of the trajectory node, dynamically select the target language transcription result within the node.

[0083] Furthermore, in the language transcription candidate set, candidate word records and language tag records corresponding to each speech trajectory node are read according to the speech trajectory node number. Based on the time range covered by the speech trajectory node, the corresponding confidence value is extracted from the continuity confidence field of the trajectory node. The time stamp of the candidate word record is aligned with the confidence value on the time axis. The confidence weight of each candidate word record is applied by confidence threshold. In the time interval where the confidence value is higher than the confidence threshold, the comprehensive score of candidate word records with higher language stability is increased. In the time interval where the confidence value is lower than the confidence threshold, the comprehensive score of candidate word records with unstable language changes is decreased. In the interval where the confidence value is close to the confidence threshold, a smooth adjustment is performed based on the difference between the confidence value and the confidence threshold. The candidate word record with the highest comprehensive score is selected as the target language transcription result within the node. The target language transcription results within all speech trajectory nodes are combined in chronological order to form the target language transcription result within the node.

[0084] S5.3: Based on the speech trajectory chain, perform cross-node boundary checks and language switching marking on the node-level language transcription results to generate a global language transcription sequence.

[0085] Furthermore, the target language transcription results within each node are read sequentially according to the node number in the speech trajectory chain. The node-level language transcription results are arranged chronologically based on the start and end times recorded in the basic information of the speech trajectory nodes. By comparing the time markers at the boundaries of adjacent speech trajectory nodes, the overlapping time segments are processed to truncate word sequences. For positions with time gaps, transitional filling is performed based on the time markers before and after. After completing the cross-node time boundary processing, the language labels of adjacent speech trajectory nodes are compared. If the language labels remain consistent, the node-level language transcription results are directly concatenated. If the language labels change, a language switching marker is inserted at the boundary position according to the language switching marker parameter set, so that all speech trajectory nodes form a continuous global language transcription sequence containing language switching information.

[0086] It should be noted that the language switching marker parameter set is a set of rules used to determine how to insert language switching markers when the language of adjacent speech trajectory nodes changes. Its value is statistically set based on the time and position of the language switching, the difference in language weight between the preceding and following nodes, and typical language switching patterns in historical conversations, so that the switching markers are semantically reasonable, temporally aligned, and have stable triggering conditions in the recognition process.

[0087] S5.4: Merge word sequences and organize time stamps on the global language transcription sequence to generate language transcription paragraphs.

[0088] Furthermore, in the global language transcription sequence, each word record, its corresponding timestamp, and language tag are read sequentially. Word records are merged based on adjacent word records, and adjacent word records that meet the conditions of continuous pronunciation and language consistency are concatenated into a longer word sequence at the text level. At the time level, the earliest start time is used as the start time of the word sequence, and the latest end time is used as the end time. Word records that do not meet the merging conditions are retained as independent word sequences. After merging the word sequences within the global language transcription sequence, the timestamps of all word sequences are uniformly organized. By performing deduplication, sorting, and continuity checks on the start and end times, the word sequences are presented in an ordered distribution from the start time to the end time of the global language transcription sequence on the timeline. Based on continuous semantics and language switching positions, several adjacent word sequences are aggregated into semantically complete language transcription paragraphs, and the language transcription paragraphs arranged in chronological order are output.

[0089] S5.5: Integrate the language transcription paragraphs, language weight trajectories, speech trajectory node structure, and trajectory node continuity credibility field to generate a multilingual full speech transcription result set.

[0090] Furthermore, language-weighted trajectory segments overlapping with the start and end times of the language-transcribed segments are retrieved on a unified timeline and appended as language confidence information. Within the same time range, the corresponding speech trajectory node numbers and multilingual speech trajectory composite feature summaries are extracted based on the speech trajectory node structure and bound to the language-transcribed segments. Then, the confidence value sequence falling within the time range of the language-transcribed segments is extracted from the trajectory node continuity confidence field and added to the language-transcribed segments as continuity evaluation information. By integrating and formatting the language-transcribed segments and time correspondence information, a multilingual full speech-transcription result set is generated.

[0091] It should be noted that the multilingual speech trajectory composite feature summary refers to a simplified feature set extracted from the time range covered by the speech trajectory node, which is composed of acoustic features such as short-time energy, frequency band energy distribution, fundamental frequency trajectory, and formant position. It is used to summarize the key information of multilingual acoustics and language discrimination of the node.

[0092] This embodiment also provides a computer device applicable to the multilingual full-speech processing method based on speech recognition, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the multilingual full-speech processing method based on speech recognition as proposed in the above embodiment.

[0093] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0094] This embodiment also provides a storage medium storing a computer program that, when executed by a processor, implements the multilingual full-speech processing method based on speech recognition as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0095] In summary, this invention achieves dynamic contextual modeling of language states in speech recognition by fusing explicit and implicit language trajectories to generate language weight trajectories, thereby improving the accuracy of multilingual discrimination and enhancing the robustness of speech recognition. Furthermore, by constructing a continuity credibility field for speech trajectory nodes and dynamically adjusting decoding parameters, it achieves structure-aware adaptive decoding for speech recognition, optimizing language switching boundary processing and improving the accuracy of speech recognition transcription.

[0096] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A multilingual full-speech processing method based on speech recognition, characterized in that: include, Continuous speech signals are acquired and preprocessed. The preprocessed continuous speech signals are then framed and windowed. Basic acoustic features are extracted and aggregated into speech segments to generate a multilingual speech feature set. Based on multilingual speech feature sets, explicit language trajectories are calculated, and implicit language trajectories are constructed by associating language preference information with speech segments in historical conversations. These are then integrated to generate language weighted trajectories. The language weight trajectory is divided into speech trajectory nodes, and the composite features of multilingual speech trajectories are recorded and connected into a speech trajectory chain. At the same time, the continuity index between nodes is calculated to generate the speech trajectory node structure. Based on the structure of speech trajectory nodes, a continuity credibility field of trajectory nodes is constructed, and the decision parameters for multilingual speech recognition are adjusted to generate a multilingual transcription candidate set. The candidate language transcription set is sequentially spliced ​​and language-labeled to generate a multilingual full speech transcription result set.

2. The multilingual full-speech processing method based on speech recognition as described in claim 1, characterized in that: The steps for segmenting and windowing the preprocessed continuous speech signal, extracting basic acoustic features, and aggregating them into speech segments to generate a multilingual speech feature set are as follows. The preprocessed continuous speech signal is framed and windowed weights are superimposed, and then organized into framed speech data in chronological order. Extract short-time energy, zero-crossing rate, frequency band energy distribution, spectral centroid, fundamental frequency trajectory and formant position from each frame of the framed speech data to generate frame-level basic acoustic features; Speech activity detection is performed on the frame-level basic acoustic features and aggregated into speech segments to generate a multilingual speech feature set.

3. The multilingual full-speech processing method based on speech recognition as described in claim 2, characterized in that: The steps for calculating explicit language trajectories based on multilingual speech feature sets are as follows. Based on the frame-level basic acoustic features in the multilingual speech feature set, the probability value of the candidate language is calculated at each time frame position and combined into an initial explicit language trajectory. Using a sliding time window, the local average probability and local change amplitude of the initial language explicit trajectory are calculated, and the language explicit probability is dynamically adjusted to generate the language explicit trajectory.

4. The multilingual full-speech processing method based on speech recognition as described in claim 3, characterized in that: The process of constructing implicit language trajectories by associating language preference information with speech segments from historical conversations and then integrating these trajectories to generate language weighted trajectories involves the following steps: A correspondence is established between language preference information in historical conversations and the timestamp and semantic context of the current speech segment. Implicit preference weights are assigned based on the frequency of language occurrence in historical conversations to generate implicit language trajectories. The explicit and implicit language trajectories are weighted and integrated at each time frame position and then normalized to generate the language weighted trajectory.

5. The multilingual full-speech processing method based on speech recognition as described in claim 4, characterized in that: The steps for dividing the language weight trajectory into speech trajectory nodes, recording the composite features of multilingual speech trajectories, connecting them into a speech trajectory chain, and simultaneously calculating the continuity index between nodes to generate the speech trajectory node structure are as follows. Extract the data range and speech segment boundaries from the language weight trajectory, divide it into multiple speech trajectory node intervals, and generate basic information of the speech trajectory nodes; Within the time range corresponding to the basic information of the speech trajectory nodes, frame-level basic acoustic features of each language are extracted from the multilingual speech feature set and combined into multilingual speech trajectory composite features. Multilingual speech trajectory composite features are added to the basic information of speech trajectory nodes, sorted in chronological order, and adjacent node connections are established to generate a speech trajectory chain. Based on the multilingual speech trajectory composite features of adjacent nodes in the speech trajectory chain, the continuity index between nodes is calculated to generate the speech trajectory node structure.

6. The multilingual full-speech processing method based on speech recognition as described in claim 5, characterized in that: The steps for constructing a continuity reliability field for trajectory nodes based on the speech trajectory node structure are as follows: Multilingual speech trajectory composite features are extracted from the speech trajectory node structure. The continuity index between nodes is converted into initial continuity scores and sorted into the initial trajectory node continuity score distribution. Based on the initial continuity score distribution of trajectory nodes, a grid structure of the continuity confidence field of trajectory nodes is established and filled with the initial continuity scores to generate the continuity confidence field of trajectory nodes.

7. The multilingual full-speech processing method based on speech recognition as described in claim 6, characterized in that: The steps for adjusting the multilingual speech recognition decision parameters and generating a multilingual transcription candidate set are as follows: Based on the explicit and implicit language trajectories in the language weight trajectory, and combined with the semantic features in historical conversations, the decision factors are statistically estimated to generate multilingual speech recognition decision parameters. The initial continuity score in the continuity confidence field of trajectory nodes is converted into a confidence value, and the decision parameters for multilingual speech recognition are dynamically adjusted in combination with the confidence threshold. Based on the adjusted multilingual speech recognition decision parameters, the speech trajectory node structure is decoded node by node and candidate word sequences and language labels are integrated to generate a multilingual transcription candidate set.

8. The multilingual full-speech processing method based on speech recognition as described in claim 7, characterized in that: The steps for temporal concatenation and language tagging of the language transcription candidate set to generate a multilingual full speech transcription result set are as follows: The speech trajectory nodes recorded in the multilingual transcription candidate set are organized node by node, and the candidate word sequences are sorted by time and overlapping segments are filled in to generate a language transcription candidate set. Based on the language transcription candidate set, and combined with the confidence value in the continuity confidence field of the trajectory node, the target language transcription result within the node is dynamically selected; Based on the speech trajectory chain, cross-node boundary checks and language switching marking are performed on the node-level language transcription results to generate a global language transcription sequence. The global language transcription sequence is merged into word sequences and time-stamped to generate language transcription paragraphs. The language-transcribed paragraphs, language weight trajectories, speech trajectory node structures, and trajectory node continuity credibility fields are integrated to generate a multilingual full speech transcription result set.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the multilingual full-speech processing method based on speech recognition as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the multilingual full-speech processing method based on speech recognition as described in any one of claims 1 to 8.

Images