Voice simultaneous transmission method and device, medium and equipment

Abstract

The embodiment of the invention provides a simultaneous voice transmission method and device, a computer readable storage medium and electronic equipment, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring an ith voice feature vector corresponding to an ith voice segment in a to-be-processed audio; the i-th voice feature vector is input into a trained large language model, semantic integrity analysis is carried out on the i-th voice feature vector through the large language model, output information of the large language model is used for determining a semantic unit about the i-th voice segment, and the semantic integrity of the semantic unit meets a first preset condition; background semantic information corresponding to the current semantic unit is obtained, and the background semantic information comprises a translation result of a previous semantic unit; and determining a translation result corresponding to the current semantic unit according to the semantic feature vector and the background semantic information of the current semantic unit. According to the embodiment of the invention, the accuracy and continuity of translation results can be improved, and the translation quality can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a simultaneous speech interpretation method, a simultaneous speech interpretation device, a computer-readable storage medium, and an electronic device. Background Art

[0002] Speech simultaneous interpretation technology, also known as speech simultaneous translation technology (Simultaneous Interpretation Technology), is a system that combines artificial intelligence technologies such as speech recognition, natural language processing and machine translation. It can convert speech content into another language almost in real time (such as a delay of about 3-6 seconds) while the speaker is speaking, and output it to the audience in the form of audio or text. This technology greatly improves the efficiency of cross-language communication and is particularly suitable for scenarios such as international conferences, online education, and multinational corporate communication.

[0003] In related technologies, simultaneous voice interpretation technology relies on voice activity detection (VAD) technology to identify the start and end positions of voice activities, i.e., "endpoints", from an audio stream, thereby effectively intercepting the part containing actual voice. Based on the results of VAD, the continuous voice stream can be divided into multiple relatively independent audio segments. Furthermore, automatic voice recognition is performed on each audio segment and converted into text form. After semantic analysis and optimization, the text will be sent to the machine translation engine and translated into the target language.

[0004] However, the quality of the translation results provided by the related technologies needs to be improved. Summary of the invention

[0005] The embodiments of the present application provide a method for simultaneous speech interpretation, a device for simultaneous speech interpretation, a computer-readable storage medium, and an electronic device, which can improve the accuracy and coherence of translation results and help improve the quality of translation.

[0006] In a first aspect, an embodiment of the present application provides a method for simultaneous speech interpretation, the method comprising: obtaining an i-th speech feature vector corresponding to the i-th speech segment in the audio to be processed, wherein i is a positive integer; inputting the i-th speech feature vector into a trained large language model to perform a semantic integrity analysis on the i-th speech feature vector through the large language model, wherein the output information of the large language model is used to determine a semantic unit about the i-th speech segment, and the semantic integrity of the semantic unit satisfies a first preset condition; obtaining background semantic information corresponding to a current semantic unit, wherein the background semantic information includes a translation result of a previous semantic unit; and determining a translation result corresponding to the current semantic unit based on the semantic feature vector of the current semantic unit and the background semantic information.

[0007] In a second aspect, an embodiment of the present application provides a voice simultaneous interpretation device, which includes: a first acquisition module, a processing module, a second acquisition module and a determination module.

[0008] Among them, the above-mentioned first acquisition module is used to obtain the i-th speech feature vector corresponding to the i-th speech segment in the audio to be processed, wherein i is a positive integer; the above-mentioned processing module is used to input the above-mentioned i-th speech feature vector into a trained large language model, so as to perform semantic integrity analysis on the above-mentioned i-th speech feature vector through the above-mentioned large language model, wherein the output information of the above-mentioned large language model is used to determine the semantic unit about the above-mentioned i-th speech segment, and the semantic integrity of the above-mentioned semantic unit meets the first preset condition; the above-mentioned second acquisition module is used to obtain the background semantic information corresponding to the current semantic unit, wherein the above-mentioned background semantic information includes the translation result of the previous semantic unit; and the above-mentioned determination module is used to determine the translation result corresponding to the above-mentioned current semantic unit according to the semantic feature vector of the above-mentioned current semantic unit and the above-mentioned background semantic information.

[0009] In an exemplary embodiment, based on the above solution, the above first acquisition module includes: a detection unit, an acquisition unit, a connection unit and a determination unit;

[0010] Among them, the above-mentioned detection unit is used to: perform voice endpoint detection on the audio to be processed to obtain multiple voice endpoints; the above-mentioned acquisition unit is used to: for the j-th audio segment determined according to the above-mentioned voice endpoint, obtain the feature vector corresponding to the j-th audio segment, wherein j is a positive integer; the above-mentioned connection unit is used to: if the feature vector of the j-th audio segment does not meet the second preset condition, then connect the feature vector of the j-th audio segment with the feature vector of the j+1-th audio segment to obtain a voice feature vector corresponding to a voice segment; the above-mentioned determination unit is used to: if the feature vector of the j-th audio segment meets the second preset condition, then use the features of the j-th audio segment as a voice feature vector corresponding to a voice segment.

[0011] In an exemplary embodiment, based on the above solution, the above first acquisition module further includes: a judgment unit;

[0012] Among them, the above-mentioned judgment unit is used to: determine the j-th feature length according to the parameters of the feature vector corresponding to the above-mentioned j-th audio segment; if the sum of the above-mentioned j-th feature length and the first threshold is less than the second threshold, it is determined that the feature vector of the above-mentioned j-th audio segment does not meet the second preset condition; or, if the above-mentioned j-th feature length is greater than or equal to the second threshold, or the sum of the above-mentioned j-th feature length and the first threshold is greater than or equal to the second threshold, it is determined that the feature vector of the above-mentioned j-th audio segment meets the second preset condition; wherein the above-mentioned second threshold is related to the length of the input feature of the above-mentioned large language model, and the above-mentioned first threshold is related to the parameters of the feature vector corresponding to the j-th audio segment.

[0013] In an exemplary embodiment, based on the above scheme, the above acquisition unit is specifically used to: determine a sliding window function according to the attribute characteristics of the audio segment; determine the window length corresponding to the j-th audio segment according to the above sliding window function; and, based on the above window length, perform feature extraction on the j-th audio segment to obtain a feature vector corresponding to the j-th audio segment; wherein the above attribute characteristics include at least one of the following: fluctuations in spectral components within a first preset time length, frequency of occurrence of transient components within a second preset time length, and frequency of occurrence of steady-state components within the second preset time length.

[0014] In an exemplary embodiment, based on the above solution, the above second acquisition module includes: a first determination unit, an acquisition unit, a second determination unit and a third determination unit;

[0015] Among them, the above-mentioned first determination unit is used to: determine the audio frame range of the previous semantic unit for the above-mentioned current semantic unit; the above-mentioned acquisition unit is used to: acquire the semantic feature vector and the previous translation result corresponding to the above-mentioned previous semantic unit according to the above-mentioned audio frame range; the above-mentioned second determination unit is used to: determine the translation feature vector corresponding to the above-mentioned previous translation result; the above-mentioned third determination unit is used to: determine the background semantic information corresponding to the above-mentioned current semantic unit according to the semantic feature vector corresponding to the above-mentioned previous semantic unit and the above-mentioned translation feature vector.

[0016] In an exemplary embodiment, based on the above scheme, the above determination module is specifically used to: perform feature splicing on the semantic feature vector corresponding to the above current semantic unit, the semantic feature vector corresponding to the above previous semantic unit and the above translation feature vector to obtain the splicing vector corresponding to the above current semantic unit; and, input the above splicing vector into a trained translation model to determine the translation result corresponding to the above current semantic unit according to the output of the above translation model.

[0017] In an exemplary embodiment, based on the above solution, the output information of the above large language model includes an end mark of the kth semantic unit, where k is a positive integer;

[0018] The device further includes: a tag inserting module, which is used to: add a target tag at the end frame of the kth semantic unit in the audio frame of the audio to be processed, wherein the end frame of the kth semantic unit corresponds to the end mark of the kth semantic unit;

[0019] The first determination unit is specifically used to determine the audio frame range according to the N target tags that are closest to the current semantic unit, where N is an integer greater than 1.

[0020] In an exemplary embodiment, based on the above scheme, the above-mentioned label insertion module is specifically used to: for the audio frame after the target label corresponding to the k-1th semantic unit in the above-mentioned audio to be processed, decode each audio frame frame by frame based on the trained hidden Markov model to obtain the hidden state corresponding to each audio frame, and obtain a decoded state sequence, wherein the above-mentioned hidden state is the end state or the non-end state of the speech unit; and in the above-mentioned decoded state sequence, if the end state of the above-mentioned semantic unit is detected, insert the above-mentioned target label at the position of the corresponding audio frame.

[0021] In an exemplary embodiment, based on the above scheme, the above-mentioned label insertion module is also specifically used for: after inserting the above-mentioned target label at the position of the corresponding audio frame, determining the kth audio sequence according to the target label corresponding to the above-mentioned kth semantic unit and the target label corresponding to the k-1th semantic unit, wherein k is an integer greater than 1; constructing a distance matrix according to the above-mentioned kth audio sequence and a reference audio sequence, wherein the above-mentioned reference audio sequence is the audio corresponding to a labeled semantic unit; traversing the distance matrix using a dynamic programming algorithm to determine a target path with the minimum cumulative distance, wherein the above-mentioned target path reflects the matching method between the above-mentioned kth audio sequence and the above-mentioned reference audio sequence; and, according to the above-mentioned target path, adjusting the insertion position of the target label of the above-mentioned kth semantic unit.

[0022] In an exemplary embodiment, based on the above scheme, the speech feature vector corresponding to the above speech segment is stored in the first memory by means of a ring buffer;

[0023] The second acquisition module is also used to: acquire the semantic feature vector corresponding to the previous semantic unit from the first memory; input the semantic feature vectors corresponding to the current semantic unit and the previous semantic unit respectively into a trained recursive neural network model to determine the generalized context features corresponding to the current semantic unit through the recursive neural network model; acquire the translation result corresponding to the previous semantic unit from the second memory, and determine the translation feature vector corresponding to the previous translation result; and determine the background semantic information corresponding to the current semantic unit based on the generalized context features corresponding to the current semantic unit and the translation feature vector.

[0024] In an exemplary embodiment, based on the above scheme, the above determination module is also specifically used to: splice the generalized context feature corresponding to the above current semantic unit and the above translation feature vector to obtain the splicing vector corresponding to the above current semantic unit; and input the above splicing vector into a trained translation model to determine the translation result corresponding to the above current semantic unit according to the output of the above translation model.

[0025] In an exemplary embodiment, based on the above scheme, the above processing module is specifically used to: input the above i-th speech feature vector into a trained large language model to determine whether the semantic integrity of the above i-th speech segment meets the first preset condition through the above large language model; wherein, if the above i-th speech segment meets the above first preset condition, the last frame of the above i-th speech segment is used as the end frame of the semantic unit, and the end frame of the above semantic unit is output; if the above i-th speech segment does not meet the above first preset condition, then receive the i+1-th speech feature vector, and determine whether the semantic integrity of the above i-th speech segment and the above i+1-th speech segment meets the first preset condition through the above large language model.

[0026] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the speech simultaneous interpretation method provided in the first aspect.

[0027] In a sixth aspect, an embodiment of the present application provides a chip for implementing the voice simultaneous interpretation method provided in the first aspect or the second aspect. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes the voice simultaneous interpretation method provided in the first aspect.

[0028] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the simultaneous speech interpretation method provided in the first aspect.

[0029] In an eighth aspect, an embodiment of the present application provides a computer program product, including computer program instructions, and the above-mentioned computer program instructions enable a computer to execute the voice simultaneous interpretation method provided in the above-mentioned first aspect.

[0030] In a ninth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, enables the computer to execute the voice simultaneous interpretation method provided in the first aspect above.

[0031] In summary, in the speech simultaneous interpretation solution provided by the embodiment of the present application, for any speech segment in the audio to be processed, the electronic device obtains the speech feature vector corresponding to the speech segment. Taking the i-th speech segment as an example, the electronic device inputs the i-th speech feature vector into the trained large language model to use the large language model to capture the inherent semantic structure features in the audio, thereby realizing the semantic integrity analysis of the i-th speech feature vector. Further, the semantic unit about the above-mentioned i-th speech segment is determined according to the output information of the large language model, and specifically, the semantic integrity of the semantic unit meets the first preset condition. In the related art, speech recognition and translation are performed based on speech segments such as after VAD. Since the above-mentioned speech segments may have the problem of semantic fragmentation, the translation results output in the related art have the problem of incoherence and inaccuracy; and in the embodiment of the present application, the semantic integrity of the speech segment is first determined, and the semantic unit whose semantic integrity meets the preset conditions is specifically operated on in the next step, which is conducive to the accuracy and coherence of the translation results. When translating a semantic unit whose semantic integrity meets preset conditions, the embodiment of the present application also obtains its context information (specifically, the background semantic information includes the translation result of the previous semantic unit), and determines the translation result of the current semantic unit in combination with its context information, thereby further improving the accuracy and coherence of the translation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Figure 1

[0033] Figure 1 A schematic diagram of the system architecture of an application environment of a voice simultaneous interpretation solution provided in an embodiment of the present application;

[0034] Figure 2 A flowchart of a method for simultaneous voice interpretation provided in an embodiment of the present application;

[0035] Figure 3 A flowchart of a method for determining a semantic unit provided in an embodiment of the present application;

[0036] Figure 4 A flowchart of a method for determining a speech feature vector provided in an embodiment of the present application;

[0037] Figure 5 A flowchart of a method for translating a current semantic unit provided in an embodiment of the present application;

[0038] Figure 6 A flowchart of a method for translating a current semantic unit provided in an embodiment of the present application;

[0039] Figure 7 A flowchart of a method for translating a current semantic unit provided in an embodiment of the present application;

[0040] Figure 8 A schematic block diagram of a voice simultaneous interpretation device provided in an embodiment of the present application;

[0041] Fig. 9 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here can be implemented in a sequence other than those illustrated or described here. In the present application embodiment of the present invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined according to A. However, it should also be understood that determining B according to A does not mean determining B only according to A, but can also determine B according to A and / or other information. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. In the description of the present application, unless otherwise specified, "multiple" refers to two or more than two.

[0044] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0045] With the acceleration of globalization and the frequent cross-border exchanges, the application of speech simultaneous interpretation technology in conferences, seminars, press conferences and other occasions has become more and more extensive, and users' requirements for the efficiency and accuracy of speech simultaneous interpretation systems have also been continuously improved. In this context, semantic integrity, as a key indicator of the simultaneous interpretation system, directly affects the translation effect of the entire system, and has therefore become one of the hot spots for technology improvement.

[0046] The inventors found that the speech simultaneous interpretation scheme provided by the related art relies on VAD to accurately detect the beginning and end of speech. The current VAD system identifies speech segments based on energy, spectral features or machine learning models. Specifically, the VAD system can accurately identify the beginning and end of speech, but it is difficult to handle natural pauses or semantic connections within the speech, that is, it is unable to detect semantic integrity, resulting in the sentence cutting position (i.e., "endpoint") may not be a semantically complete position. Since the machine translation system relies on contextual information to provide accurate translation, however, due to the limitations of speech endpoint detection, the complete semantic information may be interrupted, that is, there is a problem of semantic fragmentation. Therefore, when the translation system processes some speech fragments, it is unable to obtain the preceding context information, and the input fragments lacking complete context information often lead to semantic loss or errors, which ultimately leads to a decrease in the accuracy of the translation result. For example, in a multilingual simultaneous interpretation scenario, the incomplete semantics of a sentence may lead to the loss of background semantics, thereby affecting the translation quality. For another example, when the system cannot obtain full sentence information, both the translation quality and the context coherence are affected. This problem is particularly prominent in multilingual talks, real-time live broadcasts and other occasions.

[0047] In addition, existing simultaneous interpretation systems usually use a segment-by-segment translation method, which focuses on the current audio segment and ignores the correlation between consecutive segments. This limitation not only affects the accuracy and fluency of the translation, but also limits the system's performance in complex contexts.

[0048] In view of the above technical problems existing in the related art, the embodiment of the present application provides a speech simultaneous interpretation solution. In view of the problem of semantic fragmentation in the related art, the electronic device first determines the semantic integrity of the speech segment, and then performs the next operation on the semantic unit (Semantic Unit) whose semantic integrity meets the preset conditions. Specifically, the i-th speech feature vector corresponding to the speech segment is input into the trained large language model to capture the inherent semantic structure features in the audio using the large language model, so as to realize the semantic integrity analysis of the i-th speech feature vector. Further, the semantic unit about the above-mentioned i-th speech segment is determined according to the output information of the large language model, and specifically, the semantic integrity of the semantic unit meets the first preset condition. The embodiment of the present application performs translation operations based on semantically complete semantic units, which is conducive to improving the accuracy and coherence of the translation results. Further, when translating the semantic unit whose semantic integrity meets the preset conditions, the embodiment of the present application also obtains its context information (specifically, the background semantic information includes the translation result of the previous semantic unit), and determines the translation result of the current semantic unit in combination with its context information. Thus, the correlation between continuous semantic units can be paid attention to, which is conducive to further improving the accuracy and coherence of the translation results.

[0049] Below through Figure 1 An exemplary system architecture of an embodiment of the present application is introduced.

[0050] For example, Figure 1 Schematic diagram of a system architecture 100 of an application environment of a speech simultaneous interpretation solution provided in an embodiment of the present application. Figure 1 As shown, the system architecture of the implementation environment of the embodiment of the present application may include: a terminal 102 and a server 104 that executes a voice simultaneous interpretation solution.

[0051] Exemplarily, the terminal 102 can be used to collect audio and / or play audio. Exemplarily, the terminal 102 includes a computer, a smart phone, a tablet, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, a wearable smart device, a medical device, etc., but is not limited thereto. The server 104 can be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers.

[0052] In an exemplary embodiment, reference Figure 1 , the terminal 102 with audio collection function collects audio. For example, a microphone device. Specifically, the microphone device is installed in the simultaneous interpretation environment and can fully cover all sound source positions to clearly collect voice signals. It adopts advanced acoustic design, optimizes the sound pickup effect in noisy vehicles, and can effectively reduce background noise interference, such as environmental noise and audience conversations, to ensure accurate recognition of specific user's voice commands. In addition, these microphones have excellent directionality and can electronically adjust and optimize the receiving direction to enhance the signal in a specific direction and suppress noise and interference in other directions.

[0053] The collected audio is input to the server 104. Specifically, the server 104 provides a central processing unit (CPU) and a general-purpose computing graphics processor (GPU). In the solution provided in the embodiment of the present application, the CPU, as a key component of the simultaneous interpretation system, is responsible for the scheduling of the modules of the entire system and also handles complex computing tasks. Specifically, the CPU assumes the core responsibilities of data processing and analysis, including speech recognition, natural language processing, and computationally intensive tasks such as real-time translation. In addition, the CPU coordinates the allocation of resources and manages the task flow to ensure the collaborative work of various functional modules, thereby ensuring the stability and efficient operation of the simultaneous interpretation system and improving the accuracy and fluency of real-time translation. The powerful parallel processing capability of the GPU is used in the simultaneous interpretation system to give full play to its advantages in computationally intensive tasks, significantly improving the efficiency of the system in speech signal processing and large data set analysis. In the embodiment of the present application, the GPU assists the CPU in performing deep learning model reasoning operations, such as semantic integrity analysis through a large language model, using the translation model to output the translation results of the unit semantic unit, etc., to achieve more accurate speech recognition and natural language processing. Through efficient computing power and batch data processing features, the response speed and performance of the entire system are improved, making simultaneous interpretation smoother and more reliable.

[0054] Exemplarily, the translation result obtained after being processed by the server 104 may be transmitted to the terminal 102 having an audio playback function to output the translation result.

[0055] It should be noted that Figure 1 The system architecture of the implementation environment of the embodiment of the present application is exemplified, but the system architecture of the implementation environment of the embodiment of the present application is not limited to Figure 1 shown.

[0056] The following describes in detail the speech simultaneous interpretation method of the present application through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0057] Figure 2 The flowchart of a method P200 for simultaneous voice interpretation provided in an embodiment of the present application is shown in FIG. Figure 1 Server 104 is shown.

[0058] In step S210, the i-th speech feature vector corresponding to the i-th speech segment in the audio to be processed is obtained, where i is a positive integer.

[0059] The above-mentioned audio to be processed is the input information of the speech simultaneous interpretation system, which is the semantics to be translated. Exemplarily, the above-mentioned audio to be processed is collected by a microphone device. Exemplarily, the audio collected by the microphone of the simultaneous interpretation sound source can be processed by echo cancellation technology to obtain real-time multi-channel audio data so as to extract pure speech signals from a complex acoustic environment, thereby providing accurate and reliable input information for the subsequent simultaneous interpretation model.

[0060] refer to Figure 3 In an exemplary embodiment, the i-th voice segment may be an audio segment 34 directly obtained after VAD is performed on the processed audio 32. In this case, the method for obtaining the i-th language feature vector may refer to the embodiments corresponding to steps S210-1 to S210-2.

[0061] refer to Figure 3 In another exemplary embodiment, the above-mentioned i-th voice segment can be obtained by connecting the audio segment 34 after VAD. Specifically, if the adjacent voice endpoints are relatively close after VAD, the audio segment duration is relatively short. In this case, it is easy to cause misrecognition due to lack of context, and further lead to translation errors or "auditory hallucinations", especially affecting accurate communication during short conversations or rapid conversions. In this case, the method for obtaining the i-th language feature vector can refer to the embodiments corresponding to steps S210-1 to S210-5.

[0062] refer to Figure 4 In step S210-1, voice endpoint detection is performed on the audio to be processed to obtain multiple voice endpoints.

[0063] In step S210-2, for the j-th audio segment determined according to the voice endpoint, a feature vector corresponding to the j-th audio segment is obtained.

[0064] The audio feature vector is a quantitative description of the internal structure and properties of the audio signal. Obtaining the audio feature vector can more effectively perform audio tasks, such as simultaneous speech interpretation tasks. Exemplarily, the type of the above feature vector can be a short-time Fourier transform (STFT) feature, Mel Frequency Cepstral Coefficients (MFCCs), Linear Predictive Coding (LPC), etc., which is not limited in the embodiments of the present application.

[0065] In the following embodiments, the STFT feature is used as an example for explanation. Specifically, the j-th audio signal is analyzed in the time-frequency domain by short-time Fourier transform, and the feature vector of the audio segment can be extracted. In an exemplary embodiment, for example, the resolution and time localization of the characteristic spectrum can be optimized, and the window length of the STFT can be adjusted by a sliding window function. Specifically, the embodiments corresponding to step S21-step S23 can be referred to.

[0066] In step S21, a sliding window function is determined according to the attribute characteristics of the audio segment.

[0067] According to the attribute characteristics of different audio segments, the window length corresponding to each audio segment is determined in a targeted manner. For example, if the content of an audio segment is a rapidly changing speech event, a shorter window length is suitable. A shorter window helps capture rapidly changing speech features and improve the accuracy of endpoint detection. If the content of an audio segment is relatively stable, a longer window length is suitable. A longer window helps distinguish between background noise and speech signals and reduce misjudgment.

[0068] Exemplary attribute features include at least one of the following: fluctuation of spectral components within a first preset time length, frequency of occurrence of transient components within a second preset time length, and frequency of occurrence of steady-state components within a second preset time length.

[0069] For example, audio with a shorter window length is suitable for the following situations: the spectral components fluctuate greatly in a short period of time and are difficult to capture with a long window; the audio signal changes significantly in a short period of time (such as within a few milliseconds); it contains more transient components, such as plosives, etc. For example, audio with a longer window length is suitable for the following situations: the signal remains relatively stable for a long period of time (such as tens of milliseconds or even longer); the audio segment contains more steady-state components, such as vowels and voiced consonants; the spectral components do not change much over a long period of time, which is suitable for capture with a long window.

[0070] In step S22, the window length corresponding to the jth audio segment is determined according to the sliding window function. And in step S23, based on the determined window length, feature extraction is performed on the jth audio segment to obtain a feature vector corresponding to the jth audio segment.

[0071] Among them, the window length is the length of the audio segment processed each time, which determines how many sample points or time units are contained in each window. It is also necessary to determine the sliding step size, which refers to the moving distance between adjacent windows to ensure that there is a certain overlap between windows. Exemplarily, the sliding step size can be 30% of the response window length. Perform a fast Fourier transform on the audio within each window length to obtain a frequency domain representation; further calculate the amplitude spectrum or power spectrum as needed; combine the frequency domain representations of all windows in the jth audio segment into a two-dimensional matrix to obtain the eigenvector corresponding to the jth audio segment.

[0072] In the above process of determining the feature vector (such as STFT feature) corresponding to the audio segment, by dynamically adjusting the sliding window function according to the attributes of the audio segment, the details in the audio can be captured more accurately, thereby improving the accuracy of subsequent tasks.

[0073] Continue to refer Figure 4 In step S210-3, it is determined whether the feature vector of the jth audio segment meets the second preset condition. For specific implementation methods, reference may be made to the embodiments corresponding to steps S31-S33.

[0074] In step S31, the j-th feature length is determined according to the parameters of the feature vector corresponding to the j-th audio segment.

[0075] Specifically, the electronic device can count the length of the specified dimension of the feature corresponding to the j-th audio segment as the j-th feature length, which can be recorded as len j In addition, a first threshold may be set as the minimum length of the feature corresponding to the audio segment after VAD cutting, which may be recorded as min. Also, a second threshold may be set, which may be the maximum length of the input feature of the large language model used for semantic integrity analysis, which may be recorded as max.

[0076] Furthermore, according to len j , the relationship between the first threshold and the second threshold, to determine whether it is necessary to perform feature connection on the j-th audio segment.

[0077] In step S32, if the jth feature length len j The sum of the first threshold min is less than the second threshold max: len j +min<max, it is determined that the feature vector of the jth audio segment does not meet the second preset condition. It can be seen that when the jth feature length does not meet the second preset condition, it means that the feature vector corresponding to the jth audio segment does not meet the minimum feature information required by the large language model.

[0078] In step S33, if the j-th characteristic length len j Greater than or equal to the second threshold max:len j ≥max, or the jth characteristic length len j The sum of the first threshold min is greater than or equal to the second threshold max: len j +min≥max, it is determined that the feature vector of the j-th audio segment meets the second preset condition. It can be seen that when the j-th feature length meets the second preset condition, it means that the feature vector corresponding to the j-th audio segment meets the minimum feature information required by the large language model.

[0079] In this embodiment, a dynamic feature information amount judgment mechanism is used to make a refined judgment and process the input features according to the feature information amount threshold, and the final connected audio features are given. This ensures that the large language model can avoid the reduction of translation effect caused by premature decoding when the input data is insufficient, and timely process when the data amount reaches or exceeds the requirement, so as to achieve the improvement of the overall translation effect while ensuring fluency.

[0080] Continue to refer Figure 4 If the feature vector of the jth audio segment does not meet the second preset condition, then execute step S210-4: concatenate the feature vector of the jth audio segment with the feature vector of the j+1th audio segment to obtain a speech feature vector corresponding to a speech segment. Figure 3 The concatenated speech feature vector 38 is obtained.

[0081] Since the j-th feature length does not meet the second preset condition, the feature vector corresponding to the j-th audio segment does not meet the minimum feature information required by the large language model, feature connection is required to ensure the accuracy of the large language model's semantic integrity analysis.

[0082] Continue to refer Figure 4 If the feature vector of the j-th audio segment meets the second preset condition, step S210-5 is executed: the feature of the j-th audio segment is used as a speech feature vector corresponding to a speech segment. Figure 3 , without performing feature connection, the speech feature vector 36 is directly used as the input information of the large language model 300.

[0083] Since the j-th feature length meets the second preset condition, the feature vector corresponding to the j-th audio segment meets the minimum feature information required by the large language model, there is no need to perform feature connection, and the feature vector corresponding to the j-th audio segment can be directly input into the large language model for semantic integrity analysis.

[0084] Continue to refer Figure 2 In step S220, the i-th speech feature vector is input into a trained large language model to perform a semantic integrity analysis on the i-th speech feature vector through the large language model, wherein the output information of the large language model is used to determine a semantic unit about the i-th speech segment, and the semantic integrity of the semantic unit meets a first preset condition.

[0085] refer to Figure 3 , the trained large language model 300 is able to capture the intrinsic semantic structural features in the audio, enabling it to demonstrate superior accuracy and robustness in identifying and understanding semantically complete points, thus enabling the large language model to have the ability to identify and understand semantically complete information.

[0086] Exemplarily, the i-th speech feature vector is input into a trained large language model to determine whether the semantic integrity of the i-th speech segment meets the first preset condition through the large language model. Specifically, if the i-th speech segment meets the first preset condition, the last frame of the i-th speech segment is used as the end frame of the semantic unit, and the end frame of the semantic unit is output. If the i-th speech segment does not meet the first preset condition, the i+1-th speech feature vector is received, and the large language model is used to determine whether the semantic integrity of the i-th speech segment and the i+1-th speech segment meets the first preset condition, until the semantic integrity of the combination of multiple speech segments meets the above-mentioned first preset condition, then it belongs to the end frame of the semantic unit.

[0087] The first preset condition may be that the semantic integrity is greater than a preset value, and when it is greater than the preset value, the first preset condition is considered to be satisfied; or multiple levels of semantic integrity are preset, and the first preset condition is considered to be satisfied when the level is the highest. Specifically, a semantic unit that satisfies the first preset condition is semantically complete within the unit, and if interrupted, the semantics of each interrupted audio segment is incomplete.

[0088] In order to enable subsequent processing to clearly identify and distinguish each semantic unit, the embodiment of the present application also sets the end mark 310 of the semantic unit according to the output information of the large language model, that is, inserting a target tag at the end frame of a unit, and using the inserted target tag to help identify the starting position and end position of the semantic unit with complete semantics. The target tag can be recorded as the semantic unit end tag ( <end>Label).

[0089] Suppose the output information of the large language model includes the end mark X of the kth (k is a positive integer) semantic unit. For example, for a speech "The weather is very good today", when the model recognizes that this sentence expresses a complete concept, it will output information X, indicating the end mark of the current semantic unit. Furthermore, the embodiment of the present application will add a target label at the end frame of the kth semantic unit in the audio frame of the audio to be processed, wherein the end frame of the kth semantic unit corresponds to the end mark of the kth semantic unit.

[0090] Specifically, in this embodiment, the position is accumulated and calculated in frames in each detected semantic unit. When the semantic completeness mark is reached, the corresponding position is inserted. <end>Label. This label embedding process is implemented based on a trained hidden Markov model to achieve accurate labeling of semantic integrity points, thereby improving the accuracy of speech recognition and the effectiveness of semantic analysis. Specifically, the target label insertion process is provided through the embodiment corresponding to step S41-step S42.

[0091] In step S41, for the audio frames after the target label corresponding to the k-1th semantic unit in the audio to be processed, the hidden state corresponding to each audio frame is obtained by decoding frame by frame based on the trained hidden Markov model, and a decoded state sequence is obtained, wherein the hidden state is the end state or non-end state of the speech unit.

[0092] In step S42, in the decoded state sequence, if the semantic unit end state is detected, the target tag is inserted at the position of the corresponding audio frame.

[0093] In order to further improve the accuracy of target tag insertion, the insertion position of the corresponding target tag can also be adjusted in the following manner.

[0094] In step S51, the kth audio sequence is determined according to the target tag corresponding to the kth semantic unit and the target tag corresponding to the k-1th semantic unit.

[0095] In step S52, a distance matrix is ​​constructed according to the kth audio sequence and a reference audio sequence, wherein the reference audio sequence is audio corresponding to a labeled semantic unit.

[0096] In step S53, a dynamic programming algorithm is used to traverse the distance matrix to determine a target path with the minimum cumulative distance, wherein the target path reflects the matching mode between the kth audio sequence and the reference audio sequence.

[0097] In step S54, according to the target path, the target frame corresponding to the end frame of the kth semantic unit is adjusted, so as to insert the end frame of the kth semantic unit. <end>Label.

[0098] In the solution provided by steps S51 to S54, the insertion position of the target label is fine-tuned in combination with the dynamic time warping algorithm, which can further ensure the synchronization and accuracy of the frame label and the actual semantic unit.

[0099] In the embodiment of the present application, the position of each detected semantic unit is accumulated in units of frames, and when a semantic completeness mark is detected, a corresponding position is inserted. <end>Label. The label embedding process relies on frame number calculation and is combined with a post-processing algorithm to ensure that the label is synchronized and accurate with the actual semantic unit, and can accurately mark the semantic integrity point, thereby improving the accuracy of speech recognition and the effectiveness of semantic parsing.

[0100] Continue to see Figure 2 In step S230, background semantic information corresponding to the current semantic unit is obtained, wherein the background semantic information includes the translation result of the previous semantic unit.

[0101] Since the existing simultaneous voice interpretation system does not include the previous and following contextual information when processing each independent voice segment, the translation model is prone to misunderstand the semantics due to the lack of necessary context and the lack of an effective segment connection mechanism, resulting in discontinuous translation results and incoherent semantics. Figure 5 In the embodiment of the present application, when translating a semantic unit 52 with semantic integrity, the information related to the previous semantic unit of the semantic unit is also obtained as the background semantic information 54, so that the context information of the current semantic unit can be combined for translation, providing the necessary context for the translation process of the translation model 500 and providing effective connection, which is beneficial to the coherence and accuracy of the translation result 56. It is particularly suitable for scenes where the audio to be processed contains polysemous words, context-related phrases and subtle intonation differences. Based on the background semantic information, it can provide coherent connection between semantic units, thereby ensuring a natural transition in the translation result and enhancing the coherence of the translation result.

[0102] refer to Figure 6 , through the embodiments corresponding to steps S230 - 1 to S230 - 3, the background semantic information corresponding to the current semantic unit is obtained.

[0103] In step S230 - 1 , for the current semantic unit, the audio frame range of the previous semantic unit is determined.

[0104] The above audio frame range is used to determine the amount of information of the acquired background semantic information.

[0105] Exemplarily, the audio frame range may be determined according to N target tags that are closest to the current semantic unit, where N is an integer greater than 1.

[0106] Specifically, the embodiment of the present application supports the dynamic change of the amount of background semantic information. Since simultaneous interpretation scenarios are quite diverse, such as meetings, interviews, and live broadcasts, the understanding of context information by the large language model will change in different scenarios and different speech speeds. To adapt to such changes, this case uses backtracking <end>The number of tags N can be changed dynamically to achieve dynamic adjustment of the amount of background information. For example, in the case of fast speech and complex language, by using a larger value <end>The number of tags N, the large language model can obtain more semantic background information so as to always maintain the integrity of each semantic segment during the processing process. The final output translation result is more accurate, improving the translation quality and the user experience in the simultaneous interpretation scenario.

[0107] In step S230-2, the semantic feature vector and the previous translation result corresponding to the previous semantic unit are obtained according to the audio frame range. And in step S230-3, the translation feature vector corresponding to the previous translation result is determined.

[0108] In the embodiment of the present application, the background semantic information of the current semantic unit includes two aspects: one is the feature vector corresponding to the previous semantic unit (denoted as the semantic feature vector); the other is the feature vector corresponding to the translation result (decoded tokens) corresponding to the previous semantic unit (denoted as the translation feature vector).

[0109] Exemplarily, in order to efficiently process and realize real-time processing of large-volume audio streams, the acquisition of two aspects of data can be realized in parallel. Specifically, on the one hand, the semantic feature vector corresponding to the relevant background semantic unit is obtained according to the audio frame range. On the other hand, the relevant translation results are obtained from the memory cache according to the audio frame range, and further, the translation results are word embedded to obtain the Embeddings vector, which is recorded as the translation feature vector. Exemplarily, the above-mentioned semantic feature vectors and translation results are distributed and stored in the memory cache, and the memory copy technology can be used to efficiently integrate the memory cache information.

[0110] In step S230 - 4 , background semantic information corresponding to the current semantic unit is determined based on the semantic feature vector corresponding to the previous semantic unit and the translation feature vector.

[0111] Specifically, the semantic feature vector corresponding to the current semantic unit, the semantic feature vector corresponding to the previous semantic unit and the translation feature vector may be used as the background semantic information corresponding to the current semantic unit.

[0112] Continue to refer Figure 2 In step S240, a translation result corresponding to the current semantic unit is determined according to the semantic feature vector of the current semantic unit and the background semantic information.

[0113] Continue to refer Figure 6 In step S240-1, the semantic feature vector corresponding to the current semantic unit, the semantic feature vector corresponding to the previous semantic unit and the translation feature vector are concatenated to obtain a concatenated vector corresponding to the current semantic unit.

[0114] Exemplarily, in order to efficiently integrate audio features (semantic feature vectors) and Embeddings vectors (translation feature vectors) to form a unified tensor as input information of a translation model, the concatenation operation of feature axis tensors may be implemented by the following steps.

[0115] Step S61, ensure that the audio features and Embeddings vectors have the same meaning in some dimensions. For example, the time dimension should correspond to the same time step, and the feature dimension should have similar semantics. If the time dimensions of the audio features and Embeddings vectors are inconsistent, the time dimensions can be aligned by interpolation (such as linear interpolation) or padding. In addition, ensure that the shapes of the two tensors match in other dimensions, such as feature dimensions, batch dimensions, etc.

[0116] Step S62, using an asynchronous data transmission mechanism to perform data transmission, thereby reducing the synchronization waiting time between the CPU and the GPU.

[0117] Step S63, using a tensor concatenation operation (such as tf.concat of TensorFlow) to concatenate the audio features and the Embeddings vector in a specified dimension (such as a feature dimension or a time dimension).

[0118] In step S240 - 2 , the concatenated vector is input into a trained translation model to be decoded by the translation model in conjunction with background semantic information, and a translation result corresponding to the current semantic unit is determined according to an output of the translation model.

[0119] Exemplarily, the above translation model can adopt a large language model. Specifically, through the multi-head attention mechanism and feature fusion, the above translation model can accurately capture the important semantic information and background contextual relationship of the current semantic unit. The above translation model can use a multi-layer Transformer structure and a memory mechanism to enhance the model's understanding of long-term context to enhance the capture of semantic background information. Exemplarily, the above translation model can generate high-quality real-time translation results through the Beam Search decoding algorithm, improve the performance of the entire system in complex dialogue scenarios, and provide high-precision real-time translation.

[0120] In the embodiment of the present application, for each semantic feature vector to be synchronously translated, the features of the semantically complete audio segment preceding it and its translation result are extracted, and this information is synchronously input into the large language model as background information together with the translation result of the preceding audio to obtain the translation result. This process ensures that during translation, the second and subsequent segments can be combined with important historical semantic background for auxiliary translation, thereby avoiding translation errors caused by semantic incoherence in related technologies.

[0121] The above embodiment includes a specific implementation of step S230 and step S240. The following embodiment provides another specific implementation of step S230 and step S240.

[0122] In an exemplary embodiment, the speech feature vector corresponding to the speech segment is stored in the first memory by means of a circular buffer. The circular buffer is an efficient cache mechanism that can store data in a fixed-size buffer in a loop to ensure that the most recent historical information is always available. Exemplarily, the semantic feature vector (STFT feature) is cached in the cache memory of the translation model by means of a circular buffer technology, and for the current semantic unit to be translated, the feature of its preceding semantic unit (e.g., can be recorded as his_stft) is also stored in the memory.

[0123] In this case, as a specific implementation of step S230, the following steps are performed.

[0124] In step S230 - 1 ′, a semantic feature vector corresponding to the previous semantic unit is obtained from the first memory.

[0125] Since the STFT features are stored in the buffer memory of the translation model in a circular buffer, the translation model can be calculated based on his_stft, so that although the audio speech is cut by VAD, the feature vector corresponding to the current semantic unit can be collected during the audio feature calculation process, while the feature vector corresponding to the previous semantic unit can also be collected, thereby achieving a result equivalent to the uncut result. For each semantic unit, the translation model can combine the context information provided by the previous semantic unit according to his_stft, so as to ensure that the subsequent translation language model works under conditions with higher context relevance, which helps to maintain the consistency and coherence of the context and improve the accuracy of translation.

[0126] Furthermore, in order to achieve effective translation for each semantic unit, the contextual background information of the generalized Sang can also be determined based on his_stft. Exemplarily, the semantic feature vectors his_stft corresponding to the current semantic unit and the previous semantic unit are respectively input into a trained recursive neural network model to determine the generalized context features corresponding to the current semantic unit through the recursive neural network model. For example, the generalized contextual relationship is captured by a long short-term memory network (LSTM) to obtain the generalized context features corresponding to the current semantic unit.

[0127] Broadly speaking, contextual relationships include not only relationships between directly adjacent audio clips, but also broader semantic and temporal dependencies. For example, words in a sentence may affect the understanding of multiple subsequent sentences, or the tone and emotion in a conversation may affect the understanding of the entire conversation.

[0128] In step S230-2', the translation result corresponding to the previous semantic unit is obtained from the second memory; and the translation feature vector (Embedding vector) corresponding to the previous translation result is determined.

[0129] The specific implementation of step S230 - 2 ′ is similar to the specific implementation of step S230 - 3 , and will not be described in detail here.

[0130] In step S230 - 3 ′, background semantic information corresponding to the current semantic unit is determined based on the generalized context feature corresponding to the current semantic unit and the translation feature vector.

[0131] As a specific implementation of step S240, the following steps are performed.

[0132] In step S240 - 1 ′, the context feature corresponding to the current semantic unit and the translation feature vector are concatenated to obtain a concatenated vector corresponding to the current semantic unit.

[0133] The specific implementation of step S240 - 1 ′ is similar to the specific implementation of step S240 - 1 , and will not be repeated here.

[0134] In step S240-2', the concatenated vector is input into a trained translation model to determine a translation result corresponding to the current semantic unit according to an output of the translation model.

[0135] The specific implementation of step S240 - 2 ′ is similar to the specific implementation of step S240 - 2 and will not be described in detail here.

[0136] The speech interpretation solution provided in the embodiment of the present application can improve the ability to process semantic coherence. Specifically, when the existing speech interpretation system performs speech endpoint detection, the semantics of the entire sentence are often segmented due to frequent speech cutting, thereby affecting the accuracy of simultaneous interpretation. The proposal of this application can identify semantic units with complete semantics and insert a semantic unit at the end frame of each semantic unit. <end>Tags provide a richer context for the subsequent semantic segments, which is beneficial to speech continuity processing and provides more coherent translation results.

[0137] The speech simultaneous interpretation scheme provided in the embodiment of the present application can reduce the loss of translation information. Specifically, due to the lack of an effective context compensation mechanism, the traditional speech simultaneous interpretation translation system cannot fully retain and transmit the semantic background information when processing the cut segments, which often causes the translation results to deviate from the original meaning. The scheme proposed in the embodiment of the present application significantly reduces the phenomenon of information loss through audio feature caching and connection, ensuring that the translation results are more accurate and more contextually consistent.

[0138] The speech simultaneous interpretation solution provided in the embodiment of the present application can reduce the delay of simultaneous interpretation. Specifically, the embodiment of the present application introduces intelligent speech recognition and processing optimization algorithms, so that the speech simultaneous interpretation system can effectively reduce delays and improve the speed and accuracy of real-time translation. The system adopts a parallel processing architecture, and the translation process can be started without waiting for the complete completion of the speech input. It can provide nearly instant translation feedback in specific scenarios, effectively improving the responsiveness of the overall translation experience, especially in situations that require quick decision-making and discussion.

[0139] The language simultaneous interpretation solution provided in the embodiment of the present application not only enhances the user's trust in the speech translation results, but also greatly optimizes the ability to meet multilingual communication needs, and improves the user's interactive experience and convenience in scenarios such as international conferences and real-time speeches.

[0140] Combination of the above Figures 1 to 7 , describes the method embodiment of the present application, and the following is combined with Figure 8 An embodiment of the speech simultaneous interpretation device of the present application is described.

[0141] Figure 8 A schematic block diagram of a voice simultaneous interpretation device 800 provided in an embodiment of the present application.

[0142] refer to Figure 8 The speech simultaneous interpretation device 800 provided in the embodiment of the present application includes: a first acquisition module 810, a processing module 820, a second acquisition module 830 and a determination module 840.

[0143] Among them, the above-mentioned first acquisition module 810 is used to obtain the i-th speech feature vector corresponding to the i-th speech segment in the audio to be processed, wherein i is a positive integer; the above-mentioned processing module 820 is used to input the above-mentioned i-th speech feature vector into a trained large language model, so as to perform semantic integrity analysis on the above-mentioned i-th speech feature vector through the above-mentioned large language model, wherein the output information of the above-mentioned large language model is used to determine the semantic unit about the above-mentioned i-th speech segment, and the semantic integrity of the above-mentioned semantic unit meets the first preset condition; the above-mentioned second acquisition module 830 is used to obtain the background semantic information corresponding to the current semantic unit, wherein the above-mentioned background semantic information includes the translation result of the previous semantic unit; and the above-mentioned determination module 840 is used to determine the translation result corresponding to the above-mentioned current semantic unit according to the semantic feature vector of the above-mentioned current semantic unit and the above-mentioned background semantic information.

[0144] In an exemplary embodiment, based on the above solution, the above first acquisition module 810 includes: a detection unit, an acquisition unit, a connection unit and a determination unit;

[0145] Among them, the above-mentioned detection unit is used to: perform voice endpoint detection on the audio to be processed to obtain multiple voice endpoints; the above-mentioned acquisition unit is used to: for the j-th audio segment determined according to the above-mentioned voice endpoint, obtain the feature vector corresponding to the j-th audio segment, wherein j is a positive integer; the above-mentioned connection unit is used to: if the feature vector of the j-th audio segment does not meet the second preset condition, then connect the feature vector of the j-th audio segment with the feature vector of the j+1-th audio segment to obtain a voice feature vector corresponding to a voice segment; the above-mentioned determination unit is used to: if the feature vector of the j-th audio segment meets the second preset condition, then use the features of the j-th audio segment as a voice feature vector corresponding to a voice segment.

[0146] In an exemplary embodiment, based on the above solution, the first acquisition module 810 further includes: a judgment unit;

[0147] Among them, the above-mentioned judgment unit is used to: determine the j-th feature length according to the parameters of the feature vector corresponding to the above-mentioned j-th audio segment; if the sum of the above-mentioned j-th feature length and the first threshold is less than the second threshold, it is determined that the feature vector of the above-mentioned j-th audio segment does not meet the second preset condition; or, if the above-mentioned j-th feature length is greater than or equal to the second threshold, or the sum of the above-mentioned j-th feature length and the first threshold is greater than or equal to the second threshold, it is determined that the feature vector of the above-mentioned j-th audio segment meets the second preset condition; wherein the above-mentioned second threshold is related to the length of the input feature of the above-mentioned large language model, and the above-mentioned first threshold is related to the parameters of the feature vector corresponding to the j-th audio segment.

[0148] In an exemplary embodiment, based on the above scheme, the above acquisition unit is specifically used to: determine a sliding window function according to the attribute characteristics of the audio segment; determine the window length corresponding to the j-th audio segment according to the above sliding window function; and, based on the above window length, perform feature extraction on the j-th audio segment to obtain a feature vector corresponding to the j-th audio segment; wherein the above attribute characteristics include at least one of the following: fluctuations in spectral components within a first preset time length, frequency of occurrence of transient components within a second preset time length, and frequency of occurrence of steady-state components within the second preset time length.

[0149] In an exemplary embodiment, based on the above solution, the second acquisition module 830 includes: a first determination unit, an acquisition unit, a second determination unit and a third determination unit;

[0150] Among them, the above-mentioned first determination unit is used to: determine the audio frame range of the previous semantic unit for the above-mentioned current semantic unit; the above-mentioned acquisition unit is used to: acquire the semantic feature vector and the previous translation result corresponding to the above-mentioned previous semantic unit according to the above-mentioned audio frame range; the above-mentioned second determination unit is used to: determine the translation feature vector corresponding to the above-mentioned previous translation result; the above-mentioned third determination unit is used to: determine the background semantic information corresponding to the above-mentioned current semantic unit according to the semantic feature vector corresponding to the above-mentioned previous semantic unit and the above-mentioned translation feature vector.

[0151] In an exemplary embodiment, based on the above scheme, the determination module 840 is specifically used to: perform feature splicing on the semantic feature vector corresponding to the current semantic unit, the semantic feature vector corresponding to the previous semantic unit and the translation feature vector to obtain a splicing vector corresponding to the current semantic unit; and input the splicing vector into a trained translation model to determine the translation result corresponding to the current semantic unit according to the output of the translation model.

[0152] In an exemplary embodiment, based on the above solution, the output information of the above large language model includes an end mark of the kth semantic unit, where k is a positive integer;

[0153] The device further includes: a tag inserting module, which is used to: add a target tag at the end frame of the kth semantic unit in the audio frame of the audio to be processed, wherein the end frame of the kth semantic unit corresponds to the end mark of the kth semantic unit;

[0154] The first determination unit is specifically used to determine the audio frame range according to the N target tags that are closest to the current semantic unit, where N is an integer greater than 1.

[0155] In an exemplary embodiment, based on the above scheme, the above-mentioned label insertion module is specifically used to: for the audio frame after the target label corresponding to the k-1th semantic unit in the above-mentioned audio to be processed, decode each audio frame frame by frame based on the trained hidden Markov model to obtain the hidden state corresponding to each audio frame, and obtain a decoded state sequence, wherein the above-mentioned hidden state is the end state or the non-end state of the speech unit; and in the above-mentioned decoded state sequence, if the end state of the above-mentioned semantic unit is detected, insert the above-mentioned target label at the position of the corresponding audio frame.

[0156] In an exemplary embodiment, based on the above scheme, the above-mentioned label insertion module is also specifically used for: after inserting the above-mentioned target label at the position of the corresponding audio frame, determining the kth audio sequence according to the target label corresponding to the above-mentioned kth semantic unit and the target label corresponding to the k-1th semantic unit, wherein k is an integer greater than 1; constructing a distance matrix according to the above-mentioned kth audio sequence and a reference audio sequence, wherein the above-mentioned reference audio sequence is the audio corresponding to a labeled semantic unit; traversing the distance matrix using a dynamic programming algorithm to determine a target path with the minimum cumulative distance, wherein the above-mentioned target path reflects the matching method between the above-mentioned kth audio sequence and the above-mentioned reference audio sequence; and, according to the above-mentioned target path, adjusting the insertion position of the target label of the above-mentioned kth semantic unit.

[0157] In an exemplary embodiment, based on the above scheme, the speech feature vector corresponding to the above speech segment is stored in the first memory by means of a ring buffer;

[0158] The second acquisition module 830 is also used to: acquire the semantic feature vector corresponding to the previous semantic unit from the first memory; input the semantic feature vectors corresponding to the current semantic unit and the previous semantic unit respectively into a trained recursive neural network model to determine the generalized context features corresponding to the current semantic unit through the recursive neural network model; acquire the translation result corresponding to the previous semantic unit from the second memory, and determine the translation feature vector corresponding to the previous translation result; and determine the background semantic information corresponding to the current semantic unit based on the generalized context features corresponding to the current semantic unit and the translation feature vector.

[0159] In an exemplary embodiment, based on the above scheme, the above determination module 840 is also specifically used to: splice the generalized context feature corresponding to the above current semantic unit and the above translation feature vector to obtain the splicing vector corresponding to the above current semantic unit; and input the above splicing vector into a trained translation model to determine the translation result corresponding to the above current semantic unit according to the output of the above translation model.

[0160] In an exemplary embodiment, based on the above scheme, the processing module 820 is specifically used to: input the i-th speech feature vector into a trained large language model to determine whether the semantic integrity of the i-th speech segment meets the first preset condition through the large language model; wherein, if the i-th speech segment meets the first preset condition, the last frame of the i-th speech segment is used as the end frame of the semantic unit, and the end frame of the semantic unit is output; if the i-th speech segment does not meet the first preset condition, the i+1-th speech feature vector is received, and the large language model is used to determine whether the semantic integrity of the i-th speech segment and the i+1-th speech segment meets the first preset condition.

[0161] It should be understood that if Figure 8 The embodiment of the speech simultaneous interpretation device shown in the figure can correspond to the embodiment of the speech simultaneous interpretation method described above, and similar descriptions can refer to the method embodiment. To avoid repetition, it will not be repeated here. Specifically, by Figure 8 The information interaction between the modules in the voice simultaneous interpretation device shown in FIG. 1 can execute the above-mentioned embodiment of the voice simultaneous interpretation method, by Figure 8 The information interaction between the various modules in the voice simultaneous interpretation device shown can execute the above-mentioned embodiment of the voice simultaneous interpretation method, and the method embodiments corresponding to the above-mentioned and other operations and / or functions of each module in the device are not repeated here for the sake of brevity.

[0162] The operation and maintenance related devices of the software agent of the embodiment of the present application are described above from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or a combination of hardware and software modules in the decoding processor to execute. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0163] An embodiment of the present application also provides an electronic device.

[0164] Fig. 9 Schematic block diagram of an electronic device 900 provided in an embodiment of the present application. As mentioned above, the operation and maintenance related devices of the software agent can be deployed in Fig. 9 The electronic device shown can be used to perform the above-mentioned simultaneous voice interpretation method.

[0165] like Fig. 9 As shown, the electronic device 900 may include:

[0166] The memory 910 and the processor 920, the memory 910 is used to store the computer program 930 and transmit the program code 930 to the processor 920. In other words, the processor 920 can call and run the computer program 930 from the memory 910 to implement the method in the embodiment of the present application.

[0167] For example, the processor 920 may be configured to execute the steps of the above-mentioned method for simultaneous speech interpretation according to the instructions in the computer program 930, or to execute the steps of the above-mentioned method for simultaneous speech interpretation.

[0168] In some embodiments of the present application, the processor 920 may include but is not limited to:

[0169] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0170] In some embodiments of the present application, the memory 910 includes but is not limited to:

[0171] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0172] In some embodiments of the present application, the computer program 930 may be divided into one or more modules, which are stored in the memory 910 and executed by the processor 920 to complete the speech simultaneous interpretation method provided by the present application, or to complete the steps in the above-mentioned speech simultaneous interpretation method. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 930 in the electronic device.

[0173] like Fig. 9 As shown, the electronic device 900 may further include:

[0174] The transceiver 940 may be connected to the processor 920 or the memory 910 .

[0175] The processor 920 may control the transceiver 940 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 940 may include a transmitter and a receiver. The transceiver 940 may further include an antenna, and the number of antennas may be one or more.

[0176] It should be understood that the various components in the electronic device 900 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0177] According to one aspect of the present application, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0178] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of the above method embodiment.

[0179] In other words, when software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instruction is loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instruction can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instruction can be transmitted from a website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integration. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a digital video disc (Digital Video Disc, DVD)), or a semiconductor medium (for example, a solid-state drive (Solid State Disk, SSD)), etc.

[0180] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0181] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0182] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0183] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.< / end> < / end> < / end> < / end> < / end> < / end> < / end>

Claims

1. A method for simultaneous speech interpretation, characterized in that: The method comprises: Obtaining the i-th speech feature vector corresponding to the i-th speech segment in the audio to be processed, where i is a positive integer; Inputting the i-th speech feature vector into a trained large language model to perform semantic integrity analysis on the i-th speech feature vector through the large language model, wherein output information of the large language model is used to determine a semantic unit about the i-th speech segment, and the semantic integrity of the semantic unit meets a first preset condition; Acquire background semantic information corresponding to the current semantic unit, wherein the background semantic information includes the translation result of the previous semantic unit; A translation result corresponding to the current semantic unit is determined according to the semantic feature vector of the current semantic unit and the background semantic information.

2. The method according to claim 1, characterized in that The step of obtaining the i-th speech feature vector corresponding to the i-th speech segment in the audio to be processed includes: Perform voice endpoint detection on the audio to be processed to obtain multiple voice endpoints; For the j-th audio segment determined according to the voice endpoint, obtaining a feature vector corresponding to the j-th audio segment, where j is a positive integer; If the feature vector of the j-th audio segment does not meet the second preset condition, the feature vector of the j-th audio segment is concatenated with the feature vector of the j+1-th audio segment to obtain a speech feature vector corresponding to a speech segment; or If the feature vector of the j-th audio segment meets the second preset condition, the feature of the j-th audio segment is used as a speech feature vector corresponding to a speech segment.

3. The method according to claim 2, characterized in that The method further comprises: Determining the jth feature length according to the parameters of the feature vector corresponding to the jth audio segment; If the sum of the j-th feature length and the first threshold is less than the second threshold, it is determined that the feature vector of the j-th audio segment does not meet the second preset condition; or, If the j-th feature length is greater than or equal to the second threshold, or the sum of the j-th feature length and the first threshold is greater than or equal to the second threshold, it is determined that the feature vector of the j-th audio segment meets the second preset condition; The second threshold is related to the length of the input feature of the large language model, and the first threshold is related to the parameter of the feature vector corresponding to the j-th audio segment.

4. The method according to claim 2, characterized in that: The obtaining the feature vector corresponding to the j-th audio segment includes: Determine the sliding window function according to the attribute characteristics of the audio segment; Determine the window length corresponding to the j-th audio segment according to the sliding window function; Based on the window length, extract features of the j-th audio segment to obtain a feature vector corresponding to the j-th audio segment; The attribute feature includes at least one of the following: fluctuation of spectral components within a first preset time length, frequency of occurrence of transient components within a second preset time length, and frequency of occurrence of steady-state components within a second preset time length.

5. The method according to any one of claims 1 to 4, characterized in that: The obtaining of background semantic information corresponding to the current semantic unit includes: For the current semantic unit, determining the audio frame range of the previous semantic unit; According to the audio frame range, obtaining a semantic feature vector and a prior translation result corresponding to the prior semantic unit; Determining a translation feature vector corresponding to the previous translation result; The background semantic information corresponding to the current semantic unit is determined according to the semantic feature vector corresponding to the previous semantic unit and the translation feature vector.

6. The method according to claim 5, characterized in that The step of determining the translation result corresponding to the current semantic unit according to the semantic feature vector of the current semantic unit and the background semantic information includes: Perform feature concatenation on the semantic feature vector corresponding to the current semantic unit, the semantic feature vector corresponding to the previous semantic unit, and the translation feature vector to obtain a concatenation vector corresponding to the current semantic unit; The concatenated vector is input into a trained translation model to determine a translation result corresponding to the current semantic unit according to an output of the translation model.

7. The method according to claim 5, characterized in that The output information of the large language model includes an end mark of the kth semantic unit, where k is a positive integer; The method further comprises: Adding a target tag at the end frame of the kth semantic unit in the audio frame of the audio to be processed, wherein the end frame of the kth semantic unit corresponds to the end mark of the kth semantic unit; The step of determining the audio frame range of the previous semantic unit for the current semantic unit includes: The audio frame range is determined according to the N target tags that are closest to the current semantic unit, where N is an integer greater than 1.

8. The method according to claim 7, characterized in that The step of adding a target tag at the end frame of the kth semantic unit in the audio frame of the audio to be processed includes: For the audio frame after the target label corresponding to the k-1th semantic unit in the audio to be processed, decoding the hidden state corresponding to each audio frame frame by frame based on the trained hidden Markov model to obtain a decoded state sequence, wherein the hidden state is an end state or a non-end state of the speech unit; In the decoded state sequence, if the semantic unit end state is detected, the target tag is inserted at the position of the corresponding audio frame.

9. The method according to claim 8, characterized in that After inserting the target tag at the position of the corresponding audio frame, the method further includes: Determine a kth audio sequence according to the target label corresponding to the kth semantic unit and the target label corresponding to the k-1th semantic unit, where k is an integer greater than 1; Constructing a distance matrix according to the kth audio sequence and a reference audio sequence, wherein the reference audio sequence is an audio sequence corresponding to a labeled semantic unit; Using a dynamic programming algorithm to traverse the distance matrix to determine a target path with the minimum cumulative distance, wherein the target path reflects the matching mode between the kth audio sequence and the reference audio sequence; The target tag insertion position of the kth semantic unit is adjusted according to the target path.

10. The method according to any one of claims 1 to 4, characterized in that: The speech feature vector corresponding to the speech segment is stored in the first memory by means of a ring buffer; The obtaining of background semantic information corresponding to the current semantic unit includes: Acquire, from the first memory, a semantic feature vector corresponding to the previous semantic unit; Inputting the semantic feature vectors corresponding to the current semantic unit and the previous semantic unit into a trained recursive neural network model to determine the generalized context feature corresponding to the current semantic unit through the recursive neural network model; Obtaining, from the second memory, a translation result corresponding to the previous semantic unit, and determining a translation feature vector corresponding to the previous translation result; The background semantic information corresponding to the current semantic unit is determined according to the generalized context feature corresponding to the current semantic unit and the translation feature vector.

11. The method according to claim 10, characterized in that The step of determining the translation result corresponding to the current semantic unit according to the semantic feature vector of the current semantic unit and the background semantic information includes: Concatenating the generalized context feature corresponding to the current semantic unit and the translation feature vector to obtain a concatenated vector corresponding to the current semantic unit; The concatenated vector is input into a trained translation model to determine a translation result corresponding to the current semantic unit according to an output of the translation model.

12. The method according to any one of claims 1 to 4, characterized in that: The step of inputting the i-th speech feature vector into a trained large language model to perform semantic integrity analysis on the i-th speech feature vector through the large language model comprises: Inputting the i-th speech feature vector into a trained large language model to determine whether the semantic integrity of the i-th speech segment meets a first preset condition through the large language model; Wherein, if the i-th voice segment satisfies the first preset condition, the last frame of the i-th voice segment is used as the end frame of the semantic unit, and the end frame of the semantic unit is output; If the i-th speech segment does not meet the first preset condition, the i+1-th speech feature vector is received, and the large language model is used to determine whether the semantic integrity of the i-th speech segment and the i+1-th speech segment meets the first preset condition.

13. A voice simultaneous interpretation device, characterized in that: The device comprises: A first acquisition module is used to acquire an i-th speech feature vector corresponding to an i-th speech segment in the audio to be processed, where i is a positive integer; a processing module, configured to input the i-th speech feature vector into a trained large language model, so as to perform a semantic integrity analysis on the i-th speech feature vector through the large language model, wherein output information of the large language model is used to determine a semantic unit about the i-th speech segment, and the semantic integrity of the semantic unit satisfies a first preset condition; A second acquisition module is used to acquire background semantic information corresponding to the current semantic unit, wherein the background semantic information includes the translation result of the previous semantic unit; The determination module is used to determine the translation result corresponding to the current semantic unit according to the semantic feature vector of the current semantic unit and the background semantic information.

14. A computer-readable storage medium, characterized in that: For storing computer programs; The computer program enables a computer to execute the speech simultaneous interpretation method as described in any one of claims 1 to 12.

15. An electronic device, wherein: including a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program to implement the speech simultaneous interpretation method as described in any one of claims 1 to 12.

Images