Method, device, medium and equipment for simultaneous voice interpretation

Abstract

Embodiments of the present application provide a speech simultaneous interpretation method and device, a computer readable storage medium and an electronic device, and relate to the technical field of artificial intelligence. The method comprises: obtaining an i-th speech feature vector corresponding to an i-th speech segment in the audio to be processed; inputting the i-th speech feature vector into a trained large language model to analyze the semantic integrity of the i-th speech feature vector by the large language model, wherein the output information of the large language model is used to determine a semantic unit about the i-th speech segment, and the semantic integrity of the semantic unit satisfies a first preset condition; obtaining background semantic information corresponding to the current semantic unit, wherein the background semantic information comprises a translation result of a previous semantic unit; and determining a translation result corresponding to the current semantic unit according to the semantic feature vector of the current semantic unit and the background semantic information. The embodiments of the present application can improve the accuracy and coherence of the translation result, and are beneficial to improving the translation quality.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of artificial intelligence, and in particular, to a voice simultaneous interpretation method, a voice simultaneous interpretation device, a computer readable storage medium, and an electronic device. BACKGROUND

[0002] Voice simultaneous interpretation technology, also known as simultaneous interpretation technology, is a system that combines speech recognition, natural language processing, and machine translation, and other artificial intelligence technologies. It can convert speech content into another language almost in real time (with a delay of about 3-6 seconds) while the speaker is speaking, and output it to the audience in the form of audio or text. This technology greatly improves the efficiency of cross-language communication, and is particularly suitable for international conferences, online education, cross-border enterprise communication, and other scenarios.

[0003] In related technologies, voice simultaneous interpretation technology relies on voice activity detection (VAD) technology to identify the start and end positions of speech activity, i.e., "endpoints", from an audio stream, thereby effectively intercepting the part containing the actual speech. According to the results of VAD, the continuous speech stream can be divided into multiple relatively independent audio segments. Further, automatic speech recognition is performed on each audio segment to convert it into text form. After semantic analysis and optimization, the text is sent to a machine translation engine to be translated into a target language.

[0004] However, the translation results of the solutions provided by the related technologies need to be improved in quality. SUMMARY

[0005] Embodiments of the present application provide a voice simultaneous interpretation method, a voice simultaneous interpretation device, a computer readable storage medium, and an electronic device, which can improve the accuracy and coherence of the translation results, and are beneficial to improving the translation quality.

[0006] In a first aspect, an embodiment of the present application provides a speech simultaneous translation method, which comprises: obtaining an i-th speech feature vector corresponding to an i-th speech segment in to-be-processed audio, wherein i is a positive integer; inputting the i-th speech feature vector into a trained large language model to perform semantic integrity analysis on the i-th speech feature vector through the large language model, wherein output information of the large language model is used to determine a semantic unit about the i-th speech segment, and semantic integrity of the semantic unit satisfies a first preset condition; obtaining background semantic information corresponding to a current semantic unit, wherein the background semantic information comprises a translation result of a previous semantic unit; and determining a translation result corresponding to the current semantic unit according to a semantic feature vector of the current semantic unit and the background semantic information.

[0007] In a second aspect, an embodiment of the present application provides a speech simultaneous translation device, which comprises: a first obtaining module, a processing module, a second obtaining module, and a determining module.

[0008] The first obtaining module is configured to obtain an i-th speech feature vector corresponding to an i-th speech segment in to-be-processed audio, wherein i is a positive integer; the processing module is configured to input the i-th speech feature vector into a trained large language model to perform semantic integrity analysis on the i-th speech feature vector through the large language model, wherein output information of the large language model is used to determine a semantic unit about the i-th speech segment, and semantic integrity of the semantic unit satisfies a first preset condition; the second obtaining module is configured to obtain background semantic information corresponding to a current semantic unit, wherein the background semantic information comprises a translation result of a previous semantic unit; and the determining module is configured to determine a translation result corresponding to the current semantic unit according to a semantic feature vector of the current semantic unit and the background semantic information.

[0009] In an exemplary embodiment, based on the above scheme, the first obtaining module comprises: a detecting unit, an obtaining unit, a connecting unit, and a determining unit.

[0010] The detecting unit is configured to perform speech endpoint detection on to-be-processed audio to obtain a plurality of speech endpoints; the obtaining unit is configured to obtain a feature vector corresponding to a j-th segment of audio determined according to the speech endpoints, wherein j is a positive integer; the connecting unit is configured to, if the feature vector of the j-th segment of audio does not satisfy a second preset condition, connect the feature vector of the j-th segment of audio with a feature vector of a j+1-th segment of audio to obtain a speech feature vector corresponding to one speech segment; and the determining unit is configured to, if the feature vector of the j-th segment of audio satisfies the second preset condition, take the feature vector of the j-th segment of audio as a speech feature vector corresponding to one speech segment.

[0011] In an example embodiment, based on the above scheme, the first obtaining module further includes a judging unit.

[0012] The judging unit is configured to determine a jth feature length according to the parameter of the feature vector corresponding to the jth audio segment, determine that the feature vector of the jth audio segment does not satisfy the second preset condition if the sum of the jth feature length and a first threshold value is less than a second threshold value, or determine that the feature vector of the jth audio segment satisfies the second preset condition if the jth feature length is greater than or equal to the second threshold value or the sum of the jth feature length and the first threshold value is greater than or equal to the second threshold value, wherein the second threshold value is related to the length of the input feature of the large language model, and the first threshold value is related to the parameter of the feature vector corresponding to the jth audio segment.

[0013] In an example embodiment, based on the above scheme, the obtaining unit is specifically configured to determine a sliding window function according to the attribute feature of the audio segment, determine a window length corresponding to the jth audio segment according to the sliding window function, and perform feature extraction on the jth audio segment based on the window length to obtain the feature vector corresponding to the jth audio segment, wherein the attribute feature includes at least one of the following: a spectrum component fluctuation condition within a first preset time length, a transient component occurrence frequency within a second preset time length, and a steady-state component occurrence frequency within the second preset time length.

[0014] In an example embodiment, based on the above scheme, the second obtaining module includes a first determining unit, an obtaining unit, a second determining unit, and a third determining unit.

[0015] The first determining unit is configured to determine an audio frame range of a previous semantic unit for the current semantic unit, the obtaining unit is configured to obtain a semantic feature vector corresponding to the previous semantic unit and a previous translation result according to the audio frame range, the second determining unit is configured to determine a translation feature vector corresponding to the previous translation result, and the third determining unit is configured to determine background semantic information corresponding to the current semantic unit according to the semantic feature vector corresponding to the previous semantic unit and the translation feature vector.

[0016] In an example embodiment, based on the above scheme, the determining module is specifically configured to perform feature splicing on the semantic feature vector corresponding to the current semantic unit, the semantic feature vector corresponding to the previous semantic unit, and the translation feature vector to obtain a spliced vector corresponding to the current semantic unit, and input the spliced vector into a trained translation model to determine the translation result corresponding to the current semantic unit according to the output of the translation model.

[0017] In an example embodiment, based on the above scheme, the output information of the large language model includes an end mark of the kth semantic unit, k being a positive integer;

[0018] The device further includes a label insertion module configured to add a target label at an end frame of the kth semantic unit in the audio frame of the audio to be processed, wherein the end frame of the kth semantic unit corresponds to the end mark of the kth semantic unit.

[0019] The first determination unit is specifically configured to determine the audio frame range according to N target labels closest to the current semantic unit, N being an integer greater than 1.

[0020] In an example embodiment, based on the above scheme, the label insertion module is specifically configured to, for an audio frame after the target label corresponding to the k-1th semantic unit in the audio to be processed, obtain a hidden state corresponding to each audio frame by frame decoding based on the trained hidden Markov model to obtain a decoded state sequence, wherein the hidden state is a speech unit end state or a non-end state; and when the semantic unit end state is detected in the decoded state sequence, the target label is inserted at a position corresponding to the audio frame.

[0021] In an example embodiment, based on the above scheme, the label insertion module is further specifically configured to, after the target label is inserted at the position corresponding to the audio frame, determine a kth audio sequence according to the target label corresponding to the kth semantic unit and the target label corresponding to the k-1th semantic unit, k being an integer greater than 1; construct a distance matrix according to the kth audio sequence and a reference audio sequence, wherein the reference audio sequence is an audio corresponding to an annotated semantic unit; use a dynamic programming algorithm to traverse the distance matrix to determine a target path with the minimum cumulative distance, wherein the target path reflects a matching manner between the kth audio sequence and the reference audio sequence; and adjust the target label insertion position of the kth semantic unit according to the target path.

[0022] In an example embodiment, based on the above scheme, the speech feature vectors corresponding to the speech segments are stored in the first memory in a ring buffer manner.

[0023] The second obtaining module is further configured to: obtain the semantic feature vector corresponding to the previous semantic unit from the first memory; input the semantic feature vectors corresponding to the current semantic unit and the previous semantic unit into the trained recurrent neural network model to determine the generalized context feature corresponding to the current semantic unit through the recurrent neural network model; obtain the translation result corresponding to the previous semantic unit from the second memory, determine the translation feature vector corresponding to the previous translation result, and determine the background semantic information corresponding to the current semantic unit according to the generalized context feature corresponding to the current semantic unit and the translation feature vector.

[0024] In an example embodiment, based on the above scheme, the determining module is further configured to: concatenate the generalized context feature corresponding to the current semantic unit and the translation feature vector to obtain a concatenated vector corresponding to the current semantic unit; and input the concatenated vector into the trained translation model to determine the translation result corresponding to the current semantic unit according to the output of the translation model.

[0025] In an example embodiment, based on the above scheme, the processing module is configured to: input the ith speech feature vector into the trained large language model to determine whether the semantic integrity of the ith speech segment meets a first preset condition through the large language model; if the ith speech segment meets the first preset condition, take the last frame of the ith speech segment as the end frame of the semantic unit, and output the end frame of the semantic unit; if the ith speech segment does not meet the first preset condition, receive an (i+1)th speech feature vector, and determine whether the semantic integrity of the ith speech segment and the (i+1)th speech segment meets the first preset condition through the large language model.

[0026] In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory. The memory is configured to store a computer program, and the processor is configured to call and run the computer program stored in the memory to execute the speech simultaneous interpretation method provided in the first aspect.

[0027] In a sixth aspect, an embodiment of the present application provides a chip for implementing the speech simultaneous interpretation method provided in the first aspect or the second aspect. Specifically, the chip includes a processor configured to call and run a computer program from a memory, so that a device installed with the chip executes the speech simultaneous interpretation method provided in the first aspect.

[0028] In a seventh aspect, an embodiment of the present application provides a computer readable storage medium for storing a computer program, and the computer program causes a computer to execute the speech simultaneous interpretation method provided in the first aspect.

[0029] In an eighth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which cause a computer to execute the speech simultaneous interpretation method provided in the first aspect.

[0030] In a ninth aspect, an embodiment of the present application provides a computer program, which, when running on a computer, causes the computer to execute the speech simultaneous interpretation method provided in the first aspect.

[0031] To sum up, in the speech simultaneous interpretation scheme provided by the embodiments of the present application, for any one speech segment in the audio to be processed, the electronic device obtains a speech feature vector corresponding to the speech segment. Taking the i th speech segment as an example, the electronic device inputs the i th speech feature vector into the trained large language model, so as to capture the internal semantic structure features in the audio by using the large language model, thereby realizing semantic integrity analysis on the i th speech feature vector. Further, the semantic unit about the i th speech segment is determined according to the output information of the large language model. Specifically, the semantic integrity of the semantic unit meets the first preset condition. In the related art, speech recognition and translation are performed based on the speech segment after VAD. Since the semantic of the speech segment may be split, the translation result output in the related art has the problems of incoherence and inaccuracy. In the embodiments of the present application, the semantic integrity of the speech segment is determined first, and specifically, the semantic unit whose semantic integrity meets the preset condition is operated next, thereby being conducive to the accuracy and coherence of the translation result. When the semantic unit whose semantic integrity meets the preset condition is translated, the context information (specifically, the background semantic information includes the translation result of the previous semantic unit) thereof is also obtained, and the translation result of the current semantic unit is determined in combination with the context information, thereby being able to further improve the accuracy and coherence of the translation result. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Figure 1

[0033] Figure 1 The system architecture diagram of the application environment of the speech simultaneous interpretation scheme provided by an embodiment of the present application;

[0034] Figure 2 The flowchart of the speech simultaneous interpretation method provided by an embodiment of the present application;

[0035] Figure 3 A flowchart of a method for determining a semantic unit according to an embodiment of the present application is provided;

[0036] Figure 4 A flowchart of a method for determining a semantic unit according to an embodiment of the present application is provided;

[0037] Figure 5 A flowchart of a method for determining a semantic unit according to an embodiment of the present application is provided;

[0038] Figure 6 A flowchart of a method for determining a semantic unit according to an embodiment of the present application is provided;

[0039] Figure 7 A flowchart of a method for determining a semantic unit according to an embodiment of the present application is provided;

[0040] Figure 8 A schematic block diagram of a speech simultaneous interpretation device according to an embodiment of the present application is provided;

[0041] Figure 9 A schematic block diagram of an electronic device according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In the embodiments of the present application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined according to A. However, it should also be understood that determining B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices. In the description of the present application, "a plurality of" means two or more than two, unless otherwise specified.

[0044] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0045] With the acceleration of globalization and frequent cross-border exchanges, the application of voice simultaneous interpretation technology in conferences, seminars, press conferences and other occasions has become increasingly widespread, and users' requirements for the efficiency and accuracy of voice simultaneous interpretation systems have also been continuously improved. Under this background, semantic integrity as a key indicator of simultaneous interpretation systems directly affects the translation effect of the entire system, and thus has become one of the hotspots for technical improvement.

[0046] The inventors find that in the speech simultaneous interpretation solution provided by the related art, the accurate detection of the start and end of speech by VAD is relied on. The current VAD system identifies the speech segment based on energy, spectral features or machine learning models. Specifically, the VAD system can accurately identify the beginning and end of speech, but it is difficult to handle natural pauses or semantic connections within the speech, that is, it cannot detect the semantic integrity, resulting in the sentence cutting position (i.e., "endpoint") may not be the semantic integrity position. Since the machine translation system relies on context information to provide accurate translation, however, due to the limitations of speech endpoint detection, the complete semantic information may be interrupted, that is, the semantic is fragmented, so the translation system cannot obtain the previous context information when processing partial speech segments, and the input segment lacking complete context information often leads to semantic loss or error, ultimately resulting in a decrease in the accuracy of the translation result. For example, in a multi-language simultaneous interpretation scenario, incomplete semantic of a sentence may result in loss of background semantics, thereby affecting the translation quality. For another example, when the system cannot obtain the full sentence information, both the translation quality and the context coherence are affected, and this problem is particularly prominent in multi-lingual conversations, live broadcasts and the like.

[0047] In addition, the existing simultaneous interpretation system usually adopts a segment-by-segment translation manner, which mainly focuses on the current intercepted speech segment, while ignoring the relevance between consecutive segments. This limitation not only affects the accuracy and fluency of translation, but also limits the performance of the system in complex contexts.

[0048] To solve the above technical problems in the related art, the embodiments of the present application provide a speech simultaneous interpretation solution, which determines the semantic integrity of a speech segment first, and then performs the next operation on a semantic unit (Semantic Unit) whose semantic integrity meets a preset condition. Specifically, the i-th speech feature vector corresponding to the speech segment is input into a large language model trained to capture the intrinsic semantic structure features in the audio by using the large language model, thereby realizing semantic integrity analysis on the i-th speech feature vector. Further, the semantic unit related to the i-th speech segment is determined according to the output information of the large language model, specifically, the semantic integrity of the semantic unit meets a first preset condition. The embodiments of the present application perform translation operation based on the semantic unit with semantic integrity, thereby facilitating to improve the accuracy and coherence of the translation result. Further, when the translation is performed on the semantic unit whose semantic integrity meets the preset condition, its context information (specifically, the background semantic information is included in the translation result of the previous semantic unit) is also obtained, and the translation result of the current semantic unit is determined in combination with the context information. Thus, the relevance between consecutive semantic units can be focused on, which is conducive to further improving the accuracy and coherence of the translation result.

[0049] The following will be describedFigure 1 An exemplary system architecture of an embodiment of the present application is introduced.

[0050] An exemplary, Figure 1 A system architecture 100 of an application environment of a voice simultaneous interpretation scheme provided by an embodiment of the present application is shown. As Figure 1 shown, the system architecture of the implementation environment of the embodiment of the present application can include a terminal 102 and a server 104 for implementing the voice simultaneous interpretation scheme.

[0051] An exemplary, the terminal 102 can be used for collecting audio and / or for playing audio. An exemplary, the terminal 102 includes a computer, a smart phone, a tablet, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, a wearable smart device, a medical device, and the like, but is not limited thereto. The server 104 can be a standalone physical server, or a server cluster or distributed system composed of multiple physical servers.

[0052] In an exemplary embodiment, referring to Figure 1 , the terminal 102 with audio collection function collects audio. For example, a microphone device. Specifically, the microphone device is installed in a simultaneous interpretation environment and can fully cover all sound source positions to clearly collect voice signals. It adopts advanced acoustic design and optimizes the sound pickup effect in noisy vehicles, which can effectively reduce background noise interference such as environmental noise and audience conversation, and ensure accurate recognition of specific user voice commands. In addition, these microphones have excellent directivity, which can adjust and optimize the receiving direction through electronics to enhance the signal in a specific direction and suppress noise and interference in other directions.

[0053] The collected audio is input to the server 104. Specifically, the server 104 provides a central processing unit (CPU) and a general computing graphics processing unit (GPU). In the scheme provided by the embodiments of the present application, the CPU is a key component of the simultaneous interpretation system and is responsible for scheduling of all system modules and also handles complex computing tasks. Specifically, the CPU undertakes the core responsibilities of data processing and analysis, including speech recognition, natural language processing, and real-time translation, and other computationally intensive tasks. In addition, the CPU ensures the collaborative work of various functional modules by coordinating resource allocation and managing task flow, thereby ensuring the stable and efficient operation of the simultaneous interpretation system and improving the accuracy and fluency of real-time translation. The powerful parallel processing capability of the GPU is utilized in the simultaneous interpretation system to take advantage of it in computationally intensive tasks, significantly improving the efficiency of the system in speech signal processing and large data set analysis. In the embodiments of the present application, the GPU assists the CPU in deep learning model inference operations, such as semantic integrity analysis through large language models and translation of unit semantics using translation model output units, to achieve more accurate speech recognition and natural language processing. Through efficient computing power and batch data processing characteristics, the response speed and performance of the entire system are improved, making the simultaneous interpretation more smooth and reliable.

[0054] For example, the translation result obtained after processing by the server 104 can be transmitted to the terminal 102 with audio playback function to realize output of the translation result.

[0055] It should be noted that Figure 1 The system architecture of the implementation environment of the embodiments of the present application is exemplarily shown, but the system architecture of the implementation environment of the embodiments of the present application is not limited to Figure 1 as shown.

[0056] The speech simultaneous interpretation method of the embodiments of the present application will be described in detail below through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments.

[0057] Figure 2 A flowchart of a speech simultaneous interpretation method P200 provided by the embodiments of the present application is shown. The execution subject of the method P200 can be an electronic device, such as the server 104 shown. Figure 1 as shown.

[0058] In step S210, the i-th speech feature vector corresponding to the i-th speech segment in the audio to be processed is obtained, where i is a positive integer.

[0059] The to-be-processed audio is input information of a speech simultaneous interpretation system and is semantics to be translated. The to-be-processed audio is collected by a microphone device, for example. The microphone collected audio of the simultaneous interpretation sound source can be processed by an echo cancellation technology, real-time multi-channel audio data is obtained, a pure speech signal is extracted from a complex acoustic environment, and accurate and reliable input information is provided for a subsequent simultaneous interpretation model.

[0060] Reference Figure 3 In an example embodiment, the i-th speech segment can be an audio segment 34 directly obtained after VAD is performed on the to-be-processed audio 32. In this case, the i-th language feature vector can be obtained according to the corresponding embodiments of steps S210-1 to S210-2.

[0061] Reference Figure 3 In another example embodiment, the i-th speech segment can be obtained by connecting the audio segments 34 after VAD is performed. If the adjacent speech endpoints are close to each other after VAD is performed, the audio segment is short in length, and this case is prone to misrecognition due to lack of context, which further leads to translation errors or "phantom hearing", especially when short conversations or rapid switching affect accurate communication. In this case, the i-th language feature vector can be obtained according to the corresponding embodiments of steps S210-1 to S210-5.

[0062] Reference Figure 4 In step S210-1, speech endpoint detection is performed on the to-be-processed audio to obtain a plurality of speech endpoints.

[0063] In step S210-2, for a j-th audio segment determined according to the speech endpoints, a feature vector corresponding to the j-th audio segment is obtained.

[0064] The audio feature vector is a quantitative description of the internal structure and properties of the audio signal, and the feature vector of the audio can be obtained more effectively to perform an audio task, such as a speech simultaneous interpretation task. The type of the feature vector can be a short-time Fourier transform (STFT) feature, a mel frequency cepstral coefficient (MFCC), a linear predictive coding (LPC), and the like, and the embodiments of the present application are not limited thereto.

[0065] The following embodiments illustrate the STFT feature as an example. Specifically, a short-time Fourier transform is used to perform time-frequency analysis on the jth audio signal segment to extract the feature vector of the audio segment. In an exemplary embodiment, a sliding window function can be used to adjust the STFT window length to optimize the resolution and time localization of the feature spectrum. For details, see the embodiments corresponding to steps S21-S23.

[0066] In step S21, a sliding window function is determined according to the attribute characteristics of the audio segment.

[0067] The window length for each audio segment is determined based on the attributes of each audio segment. For example, if an audio segment contains rapidly changing speech events, a shorter window length is suitable. A shorter window helps capture rapidly changing speech features and improves endpoint detection accuracy. If an audio segment contains relatively stable content, a longer window length is suitable. A longer window length helps distinguish background noise from speech signals and reduces false positives.

[0068] Exemplary attribute features include at least one of the following: fluctuation of spectral components within a first preset time length, frequency of occurrence of transient components within a second preset time length, and frequency of occurrence of steady-state components within a second preset time length.

[0069] For example, audio with a shorter window length is suitable for the following situations: the spectral components fluctuate greatly in a short period of time, making it difficult to capture with a long window; the audio signal changes significantly in a short period of time (such as within a few milliseconds); and it contains many transient components, such as plosives. For example, audio with a longer window length is suitable for the following situations: the signal remains relatively stable for a long period of time (such as tens of milliseconds or even longer); the audio segment contains many steady-state components, such as vowels and voiced consonants; and the spectral components do not change much over a long period of time, which are suitable for capture with a long window length.

[0070] In step S22, the window length corresponding to the jth audio segment is determined according to the sliding window function. And in step S23, based on the determined window length, feature extraction is performed on the jth audio segment to obtain a feature vector corresponding to the jth audio segment.

[0071] The window length is the length of the audio segment processed each time, which determines the number of sample points or time units contained in each window. It is also necessary to determine the sliding step size, which refers to the moving distance between adjacent windows to ensure a certain overlap between windows. For example, the sliding step size can be 30% of the response window length. The audio within each window length is fast Fourier transformed to obtain a frequency domain representation; further, the amplitude spectrum or power spectrum is calculated as needed; the frequency domain representations of all windows within the j-th audio segment are combined into a two-dimensional matrix to obtain the eigenvector corresponding to the j-th audio segment.

[0072] In the above process of determining the feature vector (such as STFT feature) corresponding to the audio segment, by dynamically adjusting the sliding window function according to the attributes of the audio segment, the details in the audio can be captured more accurately, thereby improving the accuracy of subsequent tasks.

[0073] Continue to refer Figure 4 In step S210-3, it is determined whether the feature vector of the jth audio segment meets the second preset condition. For specific implementation, please refer to the embodiment corresponding to steps S31-S33.

[0074] In step S31, the j-th feature length is determined according to the parameters of the feature vector corresponding to the j-th audio segment.

[0075] Specifically, the electronic device can count the length of the specified dimension of the feature corresponding to the j-th audio segment as the j-th feature length, which can be recorded as len j In addition, a first threshold value may be set as the minimum length of the feature corresponding to the audio segment after VAD segmentation, which may be denoted as min. Furthermore, a second threshold value may be set, which may be the maximum length of the input feature of the large language model used for semantic integrity analysis, which may be denoted as max.

[0076] Furthermore, according to len j , the relationship between the first threshold and the second threshold, to determine whether it is necessary to perform feature connection on the j-th audio segment.

[0077] In step S32, if the jth characteristic length len j The sum of the first threshold min is less than the second threshold max: len j +min<max, it is determined that the feature vector of the jth audio segment does not meet the second preset condition. It can be seen that when the jth feature length does not meet the second preset condition, it means that the feature vector corresponding to the jth audio segment does not meet the minimum feature information required by the large language model.

[0078] In step S33, if the j-th characteristic length len j Greater than or equal to the second threshold max:len j ≥max, or the jth characteristic length len j The sum of the first threshold min is greater than or equal to the second threshold max: len j +min≥max, it is determined that the feature vector of the j-th audio segment meets the second preset condition. It can be seen that when the j-th feature length meets the second preset condition, it means that the feature vector corresponding to the j-th audio segment meets the minimum feature information required by the large language model.

[0079] In this embodiment, the dynamic feature information amount judgment mechanism is used for fine judgment and processing of the input features according to the threshold of the feature information amount, and the final joint audio features are given. Thus, the large language model can avoid the reduction of translation effect caused by early decoding in the case of insufficient input data, and timely process when the data amount reaches or exceeds the demand, thereby ensuring fluency while improving the overall translation effect.

[0080] With reference to Figure 4 , if the feature vector of the jth audio segment does not satisfy the second preset condition, step S210-4 is performed: the feature vector of the jth audio segment is jointed with the feature vector of the j+1th audio segment to obtain a speech feature vector corresponding to a speech segment. With reference to Figure 3 The jointed speech feature vector 38 is obtained.

[0081] Since the jth feature length does not satisfy the second preset condition, the feature vector corresponding to the jth audio segment does not satisfy the minimum feature information amount required by the large language model, and therefore feature connection is needed to ensure the accuracy of semantic integrity analysis by the large language model.

[0082] With reference to Figure 4 , if the feature vector of the jth audio segment satisfies the second preset condition, step S210-5 is performed: the feature of the jth audio segment is taken as a speech feature vector corresponding to a speech segment. With reference to Figure 3 No feature jointing is performed, and the speech feature vector 36 is directly taken as the input information of the large language model 300.

[0083] Since the jth feature length satisfies the second preset condition, the feature vector corresponding to the jth audio segment satisfies the minimum feature information amount required by the large language model, and therefore no feature connection is needed, and the feature vector corresponding to the jth audio segment can be directly input to the large language model for semantic integrity analysis.

[0084] With reference to Figure 2 In step S220, the ith speech feature vector is input to the trained large language model to perform semantic integrity analysis on the ith speech feature vector by the large language model, wherein the output information of the large language model is used to determine a semantic unit about the ith speech segment, and the semantic integrity of the semantic unit satisfies the first preset condition.

[0085] With reference to Figure 3 The trained large language model 300 can capture the inherent semantic structure features in the audio, and has excellent accuracy and robustness in identifying and understanding the semantic integrity points, thereby enabling the large language model to have the ability to identify and understand the semantic integrity information.

[0086] An exemplary, the i-th speech feature vector is input into the trained large language model to determine whether the semantic integrity of the i-th speech segment meets the first preset condition through the large language model. Specifically, if the i-th speech segment meets the first preset condition, the last frame of the i-th speech segment is taken as the end frame of the semantic unit, and the end frame of the semantic unit is output. If the i-th speech segment does not meet the first preset condition, the i+1-th speech feature vector is received, and the semantic integrity of the i-th speech segment and the i+1-th speech segment is determined through the large language model to meet the first preset condition, until the semantic integrity of the combined multiple speech segments meets the first preset condition, which is the end frame of the semantic unit.

[0087] The first preset condition can be that the semantic integrity is greater than a preset value, and when it is greater than the preset value, it is considered to meet the first preset condition; or a plurality of preset levels of semantic integrity, and the largest level is considered to meet the first preset condition. Specifically, the semantic unit that meets the first preset condition has complete semantics within the unit, and if it is interrupted, the semantics of each interrupted audio segment is incomplete.

[0088] In order to enable subsequent processing to clearly identify and distinguish each semantic unit, the embodiment of the present application also sets the end mark 310 of the semantic unit according to the output information of the large language model, that is, inserts a target label at the end frame of a unit, and helps to identify the starting position and end position of the semantic unit with complete semantics through the inserted target label. The target label can be denoted as semantic unit end label ( <end>Tag).

[0089] If the output information of the large language model includes an end tag X of the kth (k is a positive integer) semantic unit. For example, for a voice "today the weather is good", when the model identifies that the sentence expresses a complete concept, it will output information X, indicating the end tag of the current semantic unit. Further, the embodiment of the present application adds a target tag at the end frame of the kth semantic unit in the audio frame of the audio to be processed, wherein the end frame of the kth semantic unit corresponds to the end tag of the kth semantic unit.

[0090] Specifically, in the embodiment, the position is accumulated and calculated in each detected semantic unit in units of frames. When the semantic complete tag is reached, the <end>The tag insertion process is implemented based on a trained hidden Markov model to achieve accurate marking of semantic complete points, thereby improving the accuracy of speech recognition and the effectiveness of semantic analysis. The target tag insertion process is provided by the corresponding embodiments of steps S41-S42.

[0091] In step S41, for the audio frame after the target tag corresponding to the k-1th semantic unit in the audio to be processed, the trained hidden Markov model is used to decode each audio frame to obtain the hidden state corresponding to each audio frame, and a decoded state sequence is obtained, wherein the hidden state is an end state or a non-end state of a speech unit.

[0092] In step S42, if the end state of the semantic unit is detected in the decoded state sequence, the target tag is inserted at the position of the corresponding audio frame.

[0093] To further improve the accuracy of target tag insertion, the insertion position of the target tag can also be adjusted in the following manner.

[0094] In step S51, the kth audio sequence is determined according to the target tag corresponding to the kth semantic unit and the target tag corresponding to the k-1th semantic unit.

[0095] In step S52, a distance matrix is constructed according to the kth audio sequence and a reference audio sequence, wherein the reference audio sequence is an audio corresponding to an annotated semantic unit.

[0096] In step S53, a dynamic programming algorithm is used to traverse the distance matrix to determine a target path with the minimum cumulative distance, wherein the target path reflects the matching manner between the kth audio sequence and the reference audio sequence.

[0097] In step S54, the target frame corresponding to the end frame of the kth semantic unit is adjusted according to the target path, and the kth semantic unit is inserted. <end>Tag.

[0098] In the scheme provided by the step S51 to step S54, the dynamic time warping algorithm is combined to fine-tune the insertion position of the target tag, which can further ensure the synchronization and accuracy of the frame tag and the actual semantic unit.

[0099] In the embodiment of the application, the position of each detected semantic unit is accumulated in units of frames, and when the semantic complete flag is detected, the corresponding tag is inserted at the position <end>The embedding process of the label depends on frame number calculation and combines a post-processing algorithm to ensure that the label is synchronized and accurate with the actual semantic unit, can accurately mark the semantic complete point, and thus improve the accuracy of speech recognition and the effectiveness of semantic analysis.

[0100] With reference to the foregoing Figure 2 In step S230, background semantic information corresponding to the current semantic unit is obtained, wherein the background semantic information includes translation results of previous semantic units.

[0101] Since the existing simultaneous interpretation system does not include the context information before and after processing each independent speech segment, the translation model is prone to misinterpret the semantics due to the lack of necessary context, lacks an effective segment connection mechanism, and thus leads to discontinuous translation results and incoherent semantics. Figure 5 In the embodiments of the present application, when translating the semantic unit 52 with semantic integrity, the related information of the previous semantic unit of the semantic unit is also obtained as the background semantic information 54, so that the translation can be combined with the context information of the current semantic unit, providing the necessary context for the translation process of the translation model 500 and providing effective connection to facilitate the coherence and accuracy of the translation results 56. Especially suitable for scenarios where the audio to be processed contains ambiguous words, context-related phrases, and subtle tone differences, based on the background semantic information, the coherent connection between semantic units can be provided, thus ensuring natural transition in the translation results and enhancing the coherence of the translation results.

[0102] With reference to the foregoing Figure 6 The background semantic information corresponding to the current semantic unit is obtained through the embodiments of steps S230-1 to S230-3.

[0103] In step S230-1, for the current semantic unit, the audio frame range of the previous semantic unit is determined.

[0104] The audio frame range is used to determine the amount of information of the obtained background semantic information.

[0105] For example, the audio frame range can be determined according to the N target labels closest to the current semantic unit, wherein N is an integer greater than 1.

[0106] Specifically, the embodiments of the present application support dynamic variability of the amount of information of the background semantic information. Since the simultaneous interpretation scene is relatively diverse, such as meetings, interviews, and live broadcasts, the understanding of context information by large language models will change under different scenarios and different speech rates. To adapt to such changes, the present case traces back <end>The number N of tags is dynamically variable, so as to dynamically adjust the amount of background information. For example, in the case of fast speech and complex language, a larger value is taken <end>The number of tags N, the large language model can obtain more semantic background information to maintain the integrity of each semantic segment during the processing process, and the final output translation result is more accurate, improving the translation quality and user experience in simultaneous interpretation scenarios.

[0107] In step S230-2, the semantic feature vector corresponding to the previous semantic unit and the previous translation result are obtained according to the audio frame range. In step S230-3, the translation feature vector corresponding to the previous translation result is determined.

[0108] In the embodiments of the present application, the background semantic information of the current semantic unit includes two aspects: one is the feature vector corresponding to the previous semantic unit (denoted as semantic feature vector), and the other is the feature vector corresponding to the translation result (decoded tokens) of the previous semantic unit (denoted as translation feature vector).

[0109] For example, in order to efficiently process to realize real-time processing of large amount of audio stream, the acquisition of the two aspects of data can be realized in parallel. Specifically, on the one hand, the semantic feature vector corresponding to the relevant background semantic unit is obtained according to the audio frame range. On the other hand, the relevant translation result is obtained from the memory cache according to the audio frame range, and further, the translation result is processed by word embedding to obtain the Embeddings vector, denoted as translation feature vector. For example, the above-mentioned semantic feature vector and translation result are distributed stored in the memory cache, and the memory copy technology can be used to efficiently integrate the memory cache information.

[0110] In step S230-4, the background semantic information corresponding to the current semantic unit is determined according to the semantic feature vector corresponding to the previous semantic unit and the translation feature vector.

[0111] Specifically, the semantic feature vector corresponding to the current semantic unit, the semantic feature vector corresponding to the previous semantic unit and the translation feature vector can be taken as the background semantic information corresponding to the current semantic unit.

[0112] Continuing to refer to Figure 2 In step S240, the translation result corresponding to the current semantic unit is determined according to the semantic feature vector of the current semantic unit and the background semantic information.

[0113] Continuing to refer to Figure 6 In step S240-1, the semantic feature vector corresponding to the current semantic unit, the semantic feature vector corresponding to the previous semantic unit and the translation feature vector are spliced to obtain the splicing vector corresponding to the current semantic unit.

[0114] To efficiently integrate the audio features (semantic feature vectors) and Embeddings vectors (translation feature vectors) to form a unified tensor as the input information of the translation model, the splicing operation of the feature axis tensor can be implemented through the following steps.

[0115] In step S61, it is ensured that the audio features and the Embeddings vectors have the same meaning in certain dimensions. For example, the time dimension should correspond to the same time step, and the feature dimension should have similar semantics. If the time dimensions of the audio features and the Embeddings vectors are inconsistent, the time dimensions can be aligned through interpolation (such as linear interpolation) or padding. In addition, it is ensured that the shapes of the two tensors match in other dimensions, such as the feature dimension, the batch dimension, etc.

[0116] In step S62, data transmission is performed using an asynchronous data transmission mechanism to reduce the synchronization latency between the CPU and the GPU.

[0117] In step S63, the audio features and the Embeddings vectors are spliced in a specified dimension (such as the feature dimension or the time dimension) using a tensor splicing operation (such as TensorFlow's tf.concat).

[0118] In step S240-2, the spliced vector is input into the trained translation model to decode the background semantic information through the translation model, and the translation result corresponding to the current semantic unit is determined according to the output of the translation model.

[0119] The translation model can be a large language model. Through the multi-head attention mechanism and feature fusion, the translation model can accurately capture important semantic information of the current semantic unit and background context relationships. The multi-layer Transformer structure and memory mechanism can be used in the translation model to enhance the understanding of long-term context and capture semantic background information. The translation model can generate high-quality real-time translation results through the Beam Search decoding algorithm, improve the performance of the entire system in complex conversation scenarios, and provide high-precision real-time translation.

[0120] In the embodiments of the present application, for each semantic feature vector to be synchronized and translated, the features of the previous semantic complete audio segment and the translation result thereof are extracted, and this information is input into the large language model as background information together with the translation result of the previous audio to obtain the translation result. This process ensures that the second segment and subsequent segments can be assisted in translation by combining important historical semantic background during translation, thereby avoiding translation errors caused by discontinuous semantics in related technologies.

[0121] The above-provided embodiment includes one specific implementation of steps S230 and S240. Next, the following embodiment provides another specific implementation of steps S230 and S240.

[0122] In an exemplary embodiment, the speech feature vectors corresponding to the speech segments are stored in the first memory in a circular buffer manner. The circular buffer is a high-efficiency caching mechanism that can cyclically store data in a fixed-size buffer area to ensure that the most recent historical information is always available. For example, the semantic feature vectors (STFT features) are cached in the cache memory of the translation model by the circular buffer technology. For the current semantic unit to be translated, the features of the previous semantic unit (which can be denoted as his_stft) are also stored in the memory.

[0123] In this case, as one specific implementation of step S230, the following steps are performed.

[0124] In step S230-1', the semantic feature vector corresponding to the previous semantic unit is obtained from the first memory.

[0125] Since the STFT features are stored in the buffer memory of the translation model in a circular buffer manner, the translation model can perform calculations based on his_stft, thereby achieving the result equivalent to that without cutting, even though the audio speech is cut by VAD. For each semantic unit, the translation model can combine the context information provided by the previous semantic unit according to his_stft, thereby ensuring that the subsequent translation large language model works under the condition of higher context relevance, which helps to maintain the consistency and coherence of the context and improve the translation accuracy.

[0126] Further, to achieve effective translation for each semantic unit, the context background information of the general context can also be determined based on his_stft. For example, the semantic feature vectors his_stft corresponding to the current semantic unit and the previous semantic unit are input into a trained recurrent neural network model to determine the general context features corresponding to the current semantic unit by the recurrent neural network model. For example, the general context relationship is captured by a long short-term memory (LSTM) network to obtain the general context features corresponding to the current semantic unit.

[0127] Among them, the general context relationship not only includes the relationship between the directly adjacent audio segments, but also includes the more extensive semantic and temporal dependence relationship. For example, the words in a sentence can affect the understanding of multiple subsequent sentences, or the tone and emotion in a dialogue can affect the understanding of the entire dialogue.

[0128] In step S230-2', the translation result corresponding to the previous semantic unit is obtained from the second memory, and a translation feature vector (Embedding vector) corresponding to the previous translation result is determined.

[0129] Among them, the specific implementation of step S230-2' is similar to that of step S230-3, which will not be repeated here.

[0130] In step S230-3', the background semantic information corresponding to the current semantic unit is determined according to the general context feature corresponding to the current semantic unit and the translation feature vector.

[0131] As a specific implementation of step S240, the following steps are performed.

[0132] In step S240-1', the context feature corresponding to the current semantic unit and the translation feature vector are spliced to obtain a spliced vector corresponding to the current semantic unit.

[0133] Among them, the specific implementation of step S240-1' is similar to that of step S240-1, which will not be repeated here.

[0134] In step S240-2', the spliced vector is input into the trained translation model to determine the translation result corresponding to the current semantic unit according to the output of the translation model.

[0135] Among them, the specific implementation of step S240-2' is similar to that of step S240-2, which will not be repeated here.

[0136] The speech simultaneous translation scheme provided by the embodiments of the present application can improve the semantic coherence processing capability. Specifically, the existing speech simultaneous translation system often causes the whole sentence semantic to be segmented due to frequent speech cutting when performing speech endpoint detection, thereby affecting the accuracy of simultaneous translation. The present application can identify semantic units with complete semantics, and insert <end>The tags provide richer context background for the subsequent semantic segments, thereby facilitating speech continuity processing and providing a translation result with better coherence.

[0137] The speech simultaneous interpretation scheme provided by the embodiments of the present application can reduce the loss of translation information. Specifically, due to the lack of an effective context compensation mechanism, the conventional speech simultaneous interpretation translation system cannot completely retain and transmit semantic background information when processing cut segments, which often causes the translation result to deviate from the original meaning. The scheme proposed by the embodiments of the present application significantly reduces the phenomenon of information loss through audio feature caching and connection, ensuring that the translation result is more accurate and has more context consistency.

[0138] The speech simultaneous interpretation scheme provided by the embodiments of the present application can reduce the simultaneous interpretation delay. Specifically, the embodiments of the present application introduce intelligent speech recognition and processing optimization algorithms, so that the speech simultaneous interpretation system can effectively reduce the delay and improve the speed and accuracy of real-time translation. The system uses a parallel processing architecture and can start the translation process without waiting for the complete end of the speech input, which can provide almost instant translation feedback in specific scenarios, effectively improving the responsiveness of the overall translation experience, especially in situations that require quick decision-making and discussion.

[0139] The language simultaneous interpretation scheme provided by the embodiments of the present application not only improves the user's trust in the speech translation result, but also greatly optimizes the satisfaction of multilingual communication needs, improving the user's interactive experience and convenience in international conferences, real-time speeches and other scenarios.

[0140] The above describes the method embodiments of the present application in combination with Figures 1 to 7 The following describes the speech simultaneous interpretation device embodiments of the present application in combination with Figure 8 The following describes the speech simultaneous interpretation device embodiments of the present application in combination with

[0141] Figure 8 A schematic block diagram of a speech simultaneous interpretation device 800 provided by the embodiments of the present application.

[0142] With reference to Figure 8 The speech simultaneous interpretation device 800 provided by the embodiments of the present application includes a first acquisition module 810, a processing module 820, a second acquisition module 830, and a determination module 840.

[0143] The first obtaining module 810 is configured to obtain an ith speech feature vector corresponding to an ith speech segment in the audio to be processed, where i is a positive integer. The processing module 820 is configured to input the ith speech feature vector into the large language model that has been trained, so as to perform semantic integrity analysis on the ith speech feature vector by using the large language model. The output information of the large language model is used to determine a semantic unit related to the ith speech segment. The semantic integrity of the semantic unit satisfies a first preset condition. The second obtaining module 830 is configured to obtain background semantic information corresponding to a current semantic unit, where the background semantic information includes a translation result of a previous semantic unit. The determining module 840 is configured to determine a translation result corresponding to the current semantic unit according to a semantic feature vector of the current semantic unit and the background semantic information.

[0144] In an exemplary embodiment, based on the above scheme, the first obtaining module 810 includes a detection unit, an obtaining unit, a connection unit, and a determining unit.

[0145] The detection unit is configured to perform speech endpoint detection on the audio to be processed to obtain a plurality of speech endpoints. The obtaining unit is configured to obtain a feature vector corresponding to a jth segment of audio determined according to the speech endpoints, where j is a positive integer. The connection unit is configured to, if the feature vector of the jth segment of audio does not satisfy a second preset condition, connect the feature vector of the jth segment of audio with a feature vector of a j+1th segment of audio to obtain a speech feature vector corresponding to one speech segment. The determining unit is configured to, if the feature vector of the jth segment of audio satisfies the second preset condition, take the feature of the jth segment of audio as the speech feature vector corresponding to one speech segment.

[0146] In an exemplary embodiment, based on the above scheme, the first obtaining module 810 further includes a judgment unit.

[0147] The judgment unit is configured to determine a jth feature length according to a parameter of the feature vector corresponding to the jth segment of audio. If a sum of the jth feature length and a first threshold value is less than a second threshold value, it is determined that the feature vector of the jth segment of audio does not satisfy the second preset condition. Or, if the jth feature length is greater than or equal to the second threshold value, or the sum of the jth feature length and the first threshold value is greater than or equal to the second threshold value, it is determined that the feature vector of the jth segment of audio satisfies the second preset condition. The second threshold value is related to the length of the input feature of the large language model, and the first threshold value is related to the parameter of the feature vector corresponding to the jth segment of audio.

[0148] In an example embodiment, based on the above scheme, the obtaining unit is specifically configured to: determine a sliding window function according to the attribute feature of the audio segment; determine a window length corresponding to the jth audio segment according to the sliding window function; and perform feature extraction on the jth audio segment based on the window length to obtain a feature vector corresponding to the jth audio segment; wherein the attribute feature includes at least one of the following: a spectrum component fluctuation condition within a first preset time length, a transient component occurrence frequency within a second preset time length, and a steady-state component occurrence frequency within the second preset time length.

[0149] In an example embodiment, based on the above scheme, the second obtaining module 830 includes a first determining unit, an obtaining unit, a second determining unit, and a third determining unit.

[0150] The first determining unit is configured to determine an audio frame range of a previous semantic unit for the current semantic unit; the obtaining unit is configured to obtain a semantic feature vector corresponding to the previous semantic unit and a previous translation result according to the audio frame range; the second determining unit is configured to determine a translation feature vector corresponding to the previous translation result; and the third determining unit is configured to determine background semantic information corresponding to the current semantic unit according to the semantic feature vector corresponding to the previous semantic unit and the translation feature vector.

[0151] In an example embodiment, based on the above scheme, the determining module 840 is specifically configured to: perform feature splicing on the semantic feature vector corresponding to the current semantic unit, the semantic feature vector corresponding to the previous semantic unit, and the translation feature vector to obtain a spliced vector corresponding to the current semantic unit; and input the spliced vector into a trained translation model to determine a translation result corresponding to the current semantic unit according to an output of the translation model.

[0152] In an example embodiment, based on the above scheme, the output information of the large language model includes an end marker of a kth semantic unit, k being a positive integer.

[0153] The device further includes a label insertion module configured to add a target label at an end frame of the kth semantic unit in an audio frame of the audio to be processed, wherein the end frame of the kth semantic unit corresponds to the end marker of the kth semantic unit.

[0154] The first determining unit is specifically configured to determine the audio frame range according to N target labels closest to the current semantic unit, wherein N is an integer greater than 1.

[0155] In an example embodiment, based on the above scheme, the label insertion module is specifically configured to: for an audio frame after a target label corresponding to a (k-1)-th semantic unit in the to-be-processed audio, obtain a hidden state corresponding to each audio frame by frame decoding based on a trained hidden Markov model, to obtain a decoded state sequence, wherein the hidden state is an end state or a non-end state of a speech unit; and in the decoded state sequence, if the end state of the semantic unit is detected, insert the target label at a position corresponding to the audio frame.

[0156] In an example embodiment, based on the above scheme, the label insertion module is further specifically configured to: after the target label is inserted at the position corresponding to the audio frame, determine a k-th audio sequence according to the target label corresponding to the k-th semantic unit and the target label corresponding to the (k-1)-th semantic unit, wherein k is an integer greater than 1; construct a distance matrix according to the k-th audio sequence and a reference audio sequence, wherein the reference audio sequence is an audio corresponding to an annotated semantic unit; use a dynamic programming algorithm to traverse the distance matrix to determine a target path with a minimum cumulative distance, wherein the target path reflects a matching manner between the k-th audio sequence and the reference audio sequence; and adjust a target label insertion position of the k-th semantic unit according to the target path.

[0157] In an example embodiment, based on the above scheme, the speech feature vectors corresponding to the speech segments are stored in the first memory in a ring buffer manner.

[0158] The second obtaining module 830 is further configured to: obtain a semantic feature vector corresponding to the previous semantic unit from the first memory; input the semantic feature vectors corresponding to the current semantic unit and the previous semantic unit into a trained recurrent neural network model to determine a generalized context feature corresponding to the current semantic unit through the recurrent neural network model; obtain a translation result corresponding to the previous semantic unit from the second memory, and determine a translation feature vector corresponding to the previous translation result; and determine background semantic information corresponding to the current semantic unit according to the generalized context feature corresponding to the current semantic unit and the translation feature vector.

[0159] In an example embodiment, based on the above scheme, the determining module 840 is further specifically configured to: concatenate the generalized context feature corresponding to the current semantic unit and the translation feature vector to obtain a concatenated vector corresponding to the current semantic unit; and input the concatenated vector into a trained translation model to determine a translation result corresponding to the current semantic unit according to an output of the translation model.

[0160] In an example embodiment, based on the above scheme, the processing module 820 is specifically configured to: input the i th< voice feature vector into the trained large language model to determine whether the semantic integrity of the i th< voice segment meets a first preset condition through the large language model; if the i th< voice segment meets the first preset condition, the last frame of the i th< voice segment is taken as an end frame of a semantic unit, and the end frame of the semantic unit is output; if the i th< voice segment does not meet the first preset condition, the i+1 th< voice feature vector is received, and whether the semantic integrity of the i th< voice segment and the i+1 th< voice segment meets the first preset condition is determined through the large language model.

[0161] It should be understood that the embodiments of the speech simultaneous interpretation device as shown in Figure 8 correspond to the above-mentioned embodiments of the speech simultaneous interpretation method, and similar descriptions can be referred to the method embodiments. To avoid repetition, it will not be described here. Specifically, through the information interaction between the various modules in the speech simultaneous interpretation device as shown in Figure 8 , the above-mentioned embodiments of the speech simultaneous interpretation method can be executed, through the information interaction between the various modules in the speech simultaneous interpretation device as shown in Figure 8 , the above-mentioned embodiments of the speech simultaneous interpretation method can be executed, and the aforementioned and other operations and / or functions of each module in the device correspond to the method embodiments. For the sake of brevity, they will not be described here.

[0162] The software agent operation and maintenance related device of the embodiments of the present application is described above in conjunction with the accompanying drawings from the perspective of functional modules. It should be understood that the functional modules can be realized by hardware, or by software instructions, or by a combination of hardware and software modules. Specifically, each step of the method embodiments in the embodiments of the present application can be completed by integrated logic circuits and / or software instructions in the processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as hardware code processor execution completion, or executed by a combination of hardware and software modules in the code processor. Alternatively, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and combines its hardware to complete the steps in the above method embodiments.

[0163] The embodiments of the present application also provide an electronic device.

[0164] Figure 9 A schematic block diagram of an electronic device 900 provided by the embodiments of the present application is shown. As described above, the software agent operation and maintenance related device can be deployed in the electronic device as shown in Figure 9 The electronic device is shown, and thus the electronic device can be used to perform the voice simultaneous interpretation method described above.

[0165] As shown in Figure 9 The electronic device 900 can include:

[0166] The memory 910 is configured to store a computer program 930 and transmit the program code 930 to the processor 920. In other words, the processor 920 can call and run the computer program 930 from the memory 910 to implement the method in the embodiments of the present application.

[0167] For example, the processor 920 can be configured to perform the steps in the voice simultaneous interpretation method described above according to the instructions in the computer program 930, or perform the steps in the voice simultaneous interpretation method described above.

[0168] In some embodiments of the present application, the processor 920 can include but is not limited to:

[0169] A general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0170] In some embodiments of the present application, the memory 910 includes but is not limited to:

[0171] The non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0172] In some embodiments of the present application, the computer program 930 can be divided into one or more modules stored in the memory 910 and executed by the processor 920 to complete the voice communication method provided by the present application or to complete the steps in the voice communication method described above. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 930 in the electronic device.

[0173] As shown in Figure 9 The electronic device 900 can further include:

[0174] The transceiver 940 can be connected to the processor 920 or the memory 910.

[0175] The processor 920 can control the transceiver 940 to communicate with other devices, specifically, can send information or data to other devices, or receive information or data sent by other devices. The transceiver 940 can include a transmitter and a receiver. The transceiver 940 can further include an antenna, and the number of antennas can be one or more.

[0176] It should be understood that various components of the electronic device 900 are connected by a bus system, wherein the bus system includes, in addition to a data bus, a power bus, a control bus, and a state signal bus.

[0177] According to an aspect of the present application, a computer storage medium is provided, which stores a computer program. The computer program is executed by a computer to enable the computer to perform the method of the above method embodiments. Alternatively, the embodiments of the present application also provide a computer program product containing instructions. The instructions are executed by a computer to perform the method of the above method embodiments.

[0178] According to another aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium. The processor executes the computer instructions to enable the computer device to perform the method of the above method embodiments.

[0179] In other words, when implemented using software, the embodiments of the present application can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer program instructions generate, wholly or partially, the flow or function according to the embodiments of the present application. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center through a wired (for example, coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (for example, floppy disk, hard disk, magnetic tape), an optical medium (for example, Digital Video Disc (DVD)), or a semiconductor medium (for example, Solid State Disk (SSD)), etc.

[0180] Those skilled in the art can understand that the modules and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0181] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are merely illustrative, for example, the division of the modules is only a logical function division, and actual implementation can have another division manner, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed modules can be indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0182] The modules described as separate components can or can not be physically separated, and the components shown as modules can or can not be physical modules, i.e. can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. For example, the functional modules in each embodiment of the present application can be integrated in one processing module, or each module can be physically present separately, or two or more modules can be integrated in one module.

[0183] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.< / end> < / end> < / end> < / end> < / end> < / end> < / end>

Claims

1. A method for simultaneous speech interpretation, characterized in that: The method comprises: Obtain the i-th speech feature vector corresponding to the i-th speech segment in the audio to be processed, where i is a positive integer; Inputting the i-th speech feature vector into a trained large language model to perform a semantic integrity analysis on the i-th speech feature vector using the large language model, wherein output information of the large language model is used to determine a semantic unit related to the i-th speech segment, and the semantic integrity of the semantic unit satisfies a first preset condition; Obtaining background semantic information corresponding to the current semantic unit, wherein the background semantic information includes a translation result of a previous semantic unit; Determining a translation result corresponding to the current semantic unit according to the semantic feature vector of the current semantic unit and the background semantic information; The step of obtaining background semantic information corresponding to the current semantic unit includes: For the current semantic unit, determining the audio frame range of the previous semantic unit; Obtaining, according to the audio frame range, a semantic feature vector and a prior translation result corresponding to the prior semantic unit; Determining a translation feature vector corresponding to the previous translation result; The background semantic information corresponding to the current semantic unit is determined according to the semantic feature vector corresponding to the previous semantic unit and the translation feature vector.

2. The method according to claim 1, characterized in that The step of obtaining the i-th speech feature vector corresponding to the i-th speech segment in the audio to be processed includes: Perform voice endpoint detection on the audio to be processed to obtain multiple voice endpoints; For the j-th audio segment determined according to the voice endpoint, obtaining a feature vector corresponding to the j-th audio segment, where j is a positive integer; If the feature vector of the j-th audio segment does not meet the second preset condition, then concatenate the feature vector of the j-th audio segment with the feature vector of the j+1-th audio segment to obtain a speech feature vector corresponding to a speech segment; or If the feature vector of the j-th audio segment meets the second preset condition, the feature of the j-th audio segment is used as the speech feature vector corresponding to a speech segment.

3. The method according to claim 2, characterized in that The method further comprises: Determining a j-th feature length according to parameters of a feature vector corresponding to the j-th audio segment; If the sum of the j-th feature length and the first threshold is less than the second threshold, it is determined that the feature vector of the j-th audio segment does not meet the second preset condition; or, If the j-th feature length is greater than or equal to the second threshold, or the sum of the j-th feature length and the first threshold is greater than or equal to the second threshold, determining that the feature vector of the j-th audio segment meets the second preset condition; The second threshold is related to the length of the input feature of the large language model, and the first threshold is related to the parameters of the feature vector corresponding to the j-th audio segment.

4. The method according to claim 2, characterized in that The obtaining of the feature vector corresponding to the j-th audio segment includes: Determine the sliding window function according to the attribute characteristics of the audio segment; Determine the window length corresponding to the j-th audio segment according to the sliding window function; Based on the window length, feature extraction is performed on the j-th audio segment to obtain a feature vector corresponding to the j-th audio segment; The attribute characteristics include at least one of the following: fluctuation of spectral components within a first preset time length, frequency of occurrence of transient components within a second preset time length, and frequency of occurrence of steady-state components within a second preset time length.

5. The method according to claim 4, characterized in that The determining, based on the semantic feature vector of the current semantic unit and the background semantic information, a translation result corresponding to the current semantic unit includes: Perform feature splicing on the semantic feature vector corresponding to the current semantic unit, the semantic feature vector corresponding to the previous semantic unit, and the translation feature vector to obtain a splicing vector corresponding to the current semantic unit; The concatenated vector is input into a trained translation model to determine a translation result corresponding to the current semantic unit according to an output of the translation model.

6. The method according to claim 4, characterized in that The output information of the large language model includes an end mark of the kth semantic unit, where k is a positive integer; The method further comprises: Adding a target tag at the end frame of the k-th semantic unit in the audio frame of the audio to be processed, wherein the end frame of the k-th semantic unit corresponds to the end mark of the k-th semantic unit; The step of determining, for the current semantic unit, an audio frame range of a previous semantic unit includes: The audio frame range is determined according to the N target tags that are closest to the current semantic unit, where N is an integer greater than 1.

7. The method according to claim 6, characterized in that The adding a target label at the end frame of the k-th semantic unit in the audio frame of the audio to be processed includes: For the audio frame after the target label corresponding to the k-1th semantic unit in the audio to be processed, decoding the hidden state corresponding to each audio frame frame by frame based on the trained hidden Markov model to obtain a decoded state sequence, wherein the hidden state is the end state or non-end state of the speech unit; In the decoded state sequence, if the semantic unit end state is detected, the target tag is inserted at the position of the corresponding audio frame.

8. The method according to claim 7, characterized in that After inserting the target tag at the position of the corresponding audio frame, the method further includes: Determine a kth audio sequence according to the target label corresponding to the kth semantic unit and the target label corresponding to the k-1th semantic unit, where k is an integer greater than 1; Constructing a distance matrix based on the kth audio sequence and a reference audio sequence, wherein the reference audio sequence is audio corresponding to a labeled semantic unit; traversing a distance matrix using a dynamic programming algorithm to determine a target path with a minimum cumulative distance, wherein the target path reflects a matching manner between the kth audio sequence and the reference audio sequence; The target tag insertion position of the k-th semantic unit is adjusted according to the target path.

9. The method according to any one of claims 1 to 4, characterized in that Storing the speech feature vector corresponding to the speech segment in a first memory in a ring buffer manner; The obtaining of background semantic information corresponding to the current semantic unit includes: Obtaining, from the first memory, a semantic feature vector corresponding to the previous semantic unit; Inputting the semantic feature vectors corresponding to the current semantic unit and the previous semantic unit into a trained recursive neural network model to determine the generalized context feature corresponding to the current semantic unit through the recursive neural network model; Obtaining a translation result corresponding to the previous semantic unit from the second memory, and determining a translation feature vector corresponding to the previous translation result; The background semantic information corresponding to the current semantic unit is determined according to the generalized context feature corresponding to the current semantic unit and the translation feature vector.

10. The method according to claim 9, characterized in that The determining, based on the semantic feature vector of the current semantic unit and the background semantic information, a translation result corresponding to the current semantic unit includes: splicing the generalized context feature corresponding to the current semantic unit and the translation feature vector to obtain a splicing vector corresponding to the current semantic unit; The concatenated vector is input into a trained translation model to determine a translation result corresponding to the current semantic unit according to an output of the translation model.

11. The method according to any one of claims 1 to 4, characterized in that Inputting the i-th speech feature vector into a trained large language model to perform semantic integrity analysis on the i-th speech feature vector through the large language model includes: Inputting the i-th speech feature vector into a trained large language model to determine whether the semantic integrity of the i-th speech segment meets a first preset condition through the large language model; If the i-th speech segment satisfies the first preset condition, the last frame of the i-th speech segment is used as the end frame of the semantic unit, and the end frame of the semantic unit is output; If the i-th speech segment does not meet the first preset condition, the i+1-th speech feature vector is received, and whether the semantic integrity of the i-th speech segment and the i+1-th speech segment meets the first preset condition is determined through the large language model.

12. A simultaneous voice interpretation device, characterized in that: The device comprises: A first acquisition module is configured to acquire an i-th speech feature vector corresponding to an i-th speech segment in the audio to be processed, where i is a positive integer; a processing module, configured to input the i-th speech feature vector into a trained large language model to perform a semantic integrity analysis on the i-th speech feature vector using the large language model, wherein output information of the large language model is used to determine a semantic unit regarding the i-th speech segment, and the semantic integrity of the semantic unit satisfies a first preset condition; A second acquisition module is used to acquire background semantic information corresponding to the current semantic unit, wherein the background semantic information includes the translation result of the previous semantic unit; a determination module, configured to determine a translation result corresponding to the current semantic unit based on the semantic feature vector of the current semantic unit and the background semantic information; Among them, the second acquisition module is specifically used to: determine the audio frame range of the previous semantic unit for the current semantic unit; obtain the semantic feature vector and the previous translation result corresponding to the previous semantic unit based on the audio frame range; determine the translation feature vector corresponding to the previous translation result; and determine the background semantic information corresponding to the current semantic unit based on the semantic feature vector corresponding to the previous semantic unit and the translation feature vector.

13. A computer-readable storage medium, characterized in that For storing computer programs; The computer program enables a computer to execute the speech simultaneous interpretation method according to any one of claims 1 to 11.

14. An electronic device, wherein: including processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the speech simultaneous interpretation method according to any one of claims 1 to 11.

Images