An online conference translation method and system

Abstract

The application discloses an online conference translation method and system, and belongs to the technical field of speech recognition. The language space is constrained by the interactive side signal, and the recognition uncertainty is reduced. In the recording stage, a target language label generated by a sliding gesture is introduced. The label reflects the language of the speaker to whom the speech is directed. The language label of the speaker's mother tongue is combined, the candidate language set is optimally converged into a two-language set of "mother tongue + target language", and the search space of speech recognition in the language dimension is significantly reduced from the "complete language set of the conference" to the "two-language set". Based on this, the misrecognition probability caused by multi-language competition is reduced, especially the probability of misrecognizing foreign language terms as mother tongue homophonic words is reduced, and it can be quickly determined which languages are involved in each sentence of speech, so as to quickly determine which mixed language model is used for speech recognition, and the speech recognition efficiency of mixed language speech is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech recognition, in particular to an online conference translation method and system. BACKGROUND

[0002] In a cross-country social software, it is necessary to provide services for users of multiple languages, especially in the scene of online meeting (i.e., online conference), specifically when discussing a meeting in a group chat, a language translation function is provided, which is to translate the text message in a first language input by other conference participants into a text message in a second language used by the user. When the voice message input by other conference participants is a voice message, it is necessary to first convert the voice message into a text message through speech recognition, and then perform the task of translating the text message in the first language input by other conference participants into the text message in the second language used by the user. When an online meeting is conducted in a group chat, a conference participant whose native language is the first language may mix the second language in a voice message, for example, a conference participant whose native language is Chinese may mix some French words in a voice message when replying to a speech made by a French user, that is, it involves multi-language mixed speech recognition.

[0003] Referring to the speech recognition method provided in the Chinese invention patent with publication number CN115440217A, in the scene of multi-language mixed speech recognition task, it is generally necessary to first collect training samples (i.e., mixed multi-language audio) for multi-languages that need to be mixedly recognized, and then train a corresponding neural network model using the mixed multi-language audio, so as to be applicable to the mixed language speech recognition task.

[0004] On the one hand, in a cross-country social software, there are many languages involved, and different combinations of mixed speech recognition tasks correspond to different mixed language models, for example, the combination of languages includes Chinese-French, Chinese-Korean, French-Korean, etc., therefore, how to quickly determine which mixed language model is used for recognizing each voice is a problem to be solved.

[0005] On the other hand, multilingual social media applications involve numerous languages, and the speech recognition tasks involving mixed language combinations mean that corresponding training audio needs to be collected in advance for each language combination. This presents a resource constraint for training neural network models, making multilingual speech recognition in multilingual social media applications a pressing issue. For example, language combinations include Chinese-French, Chinese-Korean, and French-Korean, each requiring a corresponding speech recognition model to be trained before the multilingual social media application can be adapted to the mixed language speech recognition task. In other words, to achieve good results in multilingual speech recognition in multilingual social media applications, corresponding neural network models need to be trained in advance for each language combination task. Clearly, integrating various mixed language speech recognition models into multilingual social media systems leads to excessive computer resource consumption. Summary of the Invention

[0006] The purpose of this invention is to at least solve one of the technical problems existing in the prior art, and to provide an online conference translation method and system that can quickly determine which languages ​​each speech sentence involves, thereby quickly determining which language hybrid model to use for speech recognition, and improving the speech recognition efficiency of mixed language speech.

[0007] Firstly, an online meeting translation method is provided, characterized in that the method includes:

[0008] The conference client receives the recording operation from the first user to collect audio.

[0009] During the process of acquiring voice audio, the system receives a sliding gesture from a first user on a preset recording control and generates a target language tag based on the sliding direction of the gesture. The target language tag is used to characterize the voice audio as a response to a meeting participant using the target language.

[0010] Obtain the native language marker of the first user; determine the candidate language set of the speech audio based on the native language marker and the target language marker, and select a bilingual mixed speech recognition model corresponding to the candidate language set from a preset speech recognition model set to perform speech recognition on the speech audio to obtain the recognized text;

[0011] Machine translation is performed on the identified text to obtain translated text in the target language, and a conference message is generated and output based on the translated text. The conference message includes at least text messages in the target language and / or synthesized speech messages in the target language.

[0012] As a further improvement, the method also includes:

[0013] At the beginning of the conference or during the conference, the language category set of the current conference participant is counted, and a recording gesture operation prompt is generated based on the language category set, the recording gesture operation prompt being used to indicate the mapping relationship between different sliding directions and different target languages, and the recording gesture operation prompt being displayed on the recording interface.

[0014] As a further improvement, the target language mark is generated according to the sliding direction of the sliding gesture, comprising:

[0015] A mapping table corresponding to at least one sliding direction is pre-configured for each target language;

[0016] When it is detected that the first user presses the recording control and completes the sliding in the pressed state, the target language mark is determined in the mapping table according to the sliding direction, or the target language mark is determined according to the sliding direction and the language category set of the conference participant.

[0017] As a further improvement, the method further comprises:

[0018] When it is determined that there is no bilingual mixed speech recognition model corresponding to the mother tongue language mark and the target language mark in the speech recognition model set, a split timing mark mode is entered;

[0019] In the split timing mark mode, at least one split mark operation triggered by the first user during the recording process is received to record at least one time split point of the speech audio;

[0020] The speech audio is split into multiple audio segments based on the time split point, and for each audio segment, a monolingual speech recognition model is selected to perform speech recognition to obtain a segment recognition text, and multiple segment recognition texts are spliced and translated to obtain the translation text.

[0021] In the embodiment, when there is no bilingual mixed speech recognition model corresponding to the mother tongue language mark and the target language mark, the speech recognition task of mixed languages in various combinations can be applied by splitting the speech segments of different languages and selecting a monolingual speech recognition model to perform speech recognition to obtain a segment recognition text, using less computer resources.

[0022] As a further improvement, entering the split timing mark mode comprises at least one of:

[0023] Receiving inaccurate feedback of the first user on the output translation result and entering the split timing mark mode accordingly;

[0024] Or automatically entering the split timing mark mode based on the confidence output during the speech recognition process being lower than a threshold.

[0025] or entering the split opportunity marker mode when a multi-lingual switching feature is detected in the voice audio.

[0026] As a further improvement, to distinguish the target language marker from the split opportunity marker, the same type of gesture is used but distinguished by using different trigger areas, specifically including:

[0027] a swipe gesture detected within a first trigger area is used to generate the target language marker;

[0028] a swipe gesture or tap / swipe gesture detected within a second trigger area is used to generate the split marker operation; wherein the first trigger area and the second trigger area do not overlap with each other in the recording interface.

[0029] As a further improvement, the method further includes:

[0030] In the split opportunity marker mode, a segment language marker is determined for at least part of the audio segments, the determination of the segment language marker including: determining according to the direction, trigger area, trigger times and / or trigger timing of the split marker operation; and selecting a corresponding monolingual speech recognition model for the corresponding audio segment according to the segment language marker.

[0031] In a second aspect, the present application further provides an online conference translation system, including:

[0032] a recording collection module for collecting conference voice audio at a conference client;

[0033] a gesture detection module for detecting a swipe gesture during recording and generating a target language marker, and / or generating a split marker operation in a split opportunity marker mode;

[0034] a language determination module for obtaining a user's native language marker and jointly determining a candidate language set with the target language marker;

[0035] a model selection module for selecting a bilingual mixed speech recognition model from a preset speech recognition model set, or selecting a monolingual speech recognition model when the corresponding bilingual mixed speech recognition model is missing;

[0036] a speech recognition module for performing speech recognition on the voice audio or audio segments obtained by splitting the voice audio to obtain recognized text;

[0037] a translation module for performing machine translation on the recognized text to obtain translated text;

[0038] a speech synthesis and output module for generating synthesized speech messages and / or text messages based on the translated text and outputting to a conference group chat.

[0039] In a third aspect, the present application provides a computer readable storage medium, having stored thereon a computer program, which when executed by a processor implements the method as described above.

[0040] In a fourth aspect, the present application provides an online conference translation device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the steps of the online conference translation method as described above.

[0041] Beneficial effects: Compared with the prior art, the online conference translation method and system provided by the present application reduces the recognition uncertainty by constraining the language space through interactive side signals. The present application introduces a target language label generated by a sliding gesture in the recording phase. The label represents the language of the speaker to whom the current speech is directed. In combination with the speaker's native language label, the candidate language set is optimally converged into a two-language set of "native language + target language", thereby significantly narrowing the search space of speech recognition in the language dimension from the "entire conference language set" to the "two-language set". Based on this, the probability of misrecognition caused by multi-language competition is reduced, especially the probability of misrecognizing foreign terms as native homophonic words is reduced. The language involved in each sentence of speech can be quickly determined, and thus the mixed language model used for speech recognition can be quickly determined, thereby improving the speech recognition efficiency of mixed language speech.

[0042] Additional aspects and advantages of the present application will be given in part in the following description, become apparent from the following description, or be understood through practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0043] The present application will be further described below in conjunction with the accompanying drawings and embodiments;

[0044] Figure 1 The flowchart of an online conference translation method in one embodiment of the present application is shown;

[0045] Figure 2 The flowchart of an online conference translation method in another embodiment of the present application is shown;

[0046] Figure 3 The flowchart of generating a target language label in one embodiment of the present application is shown;

[0047] Figure 4 The flowchart of an online conference translation method in another embodiment of the present application is shown. DETAILED DESCRIPTION

[0048] This part will describe the specific embodiments of the present application in detail, the preferred embodiments of the present application are shown in the drawings, the role of the drawings is to supplement the description of the text part with figures, so that people can intuitively and visually understand each technical feature and the overall technical scheme of the present application, but it cannot be understood as a limitation on the protection scope of the present application.

[0049] Reference Figure 1 In one embodiment, an online conference translation method is provided, the method comprising:

[0050] Step S101, receiving a recording operation of a first user at a conference client to collect voice audio.

[0051] The conference client provides a recording control in the group chat interface. When the first user presses and holds the recording control, a recording start event is triggered, the audio collection module collects audio frames and writes them into a buffer.

[0052] For example, user A (native Chinese) long presses the "record key" in the group chat, and the client starts collecting audio at a sampling rate of 16 kHz and optionally performs noise reduction. The system records the recording start time t0, and continuously receives the audio stream until the user releases the record key.

[0053] Step S102, during the collection of voice audio, receiving a sliding gesture of the first user on the preset recording control, and generating a target language mark according to the sliding direction of the sliding gesture, the target language mark being used to represent that the voice audio is a reply to conference participants using a target language.

[0054] During the user's pressing of the record key, the gesture detection module continuously monitors the touch point movement trajectory. When the condition of "pressing state + moving distance exceeding threshold d_th" is met, it is determined that sliding occurs, and the main direction is calculated. If Δx> Δy and Δx> 0, it is determined that right sliding; if Δx> Δy and Δx< 0, it is determined that left sliding; if Δy> Δx and Δy> 0, it is determined that downward sliding; if Δy> Δx and Δy< 0, it is determined that upward sliding; then the language corresponding to the direction is looked up in the mapping table, and the target language mark is generated.

[0055] Specifically, there is a French user B in the conference. The system prompts "right slide = French". User A slides to the right by about 50 pixels after pressing the record key, and the gesture module determines that right sliding is established, generating target_lang=fr (French). Further, to prevent accidental touch, the system can set parameters such as minimum sliding distance threshold d_th (such as 20 pixels), minimum pressing time t_hold (such as 150 ms); it can also allow "direction + sliding distance" to encode more languages.

[0056] Step S103, obtaining a native language mark of the first user.

[0057] The native language marker can be obtained in one of the following ways: reading from a user profile field, such as profile.lang=zh; historical statistics: identifying the language distribution of the user's historical speech and selecting the language with the highest proportion as the native language; real-time inference: using light language identification (LID) to determine the first few seconds of audio as a candidate for the native language. For example, if the user A profile shows that the native language is Chinese, then speaker_lang=zh.

[0058] Step S104, determining a candidate language set of the voice audio according to the native language marker and the target language marker, and selecting a bilingual mixed speech recognition model corresponding to the candidate language set from a preset speech recognition model set to perform speech recognition on the voice audio to obtain a recognized text.

[0059] The language marker management module generates candidate_langs={speaker_lang, target_lang}. The model selection module uses the two-language key (such as zh-fr) to search for a bilingual mixed model. For example, candidate_langs={zh, fr}, and there is a “zh-fr mixed recognition model” in the model repository, the system loads the model and prepares for inference. If the model needs to be downloaded on demand from the cloud, the system can first download the model file and then load it.

[0060] Step S105, performing machine translation on the recognized text to obtain a translation text corresponding to the target language.

[0061] The speech recognition module performs decoding on the audio stream, which can use end-to-end ASR (CTC / Transducer / Attention, etc.) or a hybrid acoustic + language model structure. The output recognized text can contain bilingual mixed words. Example speech: A says: “I think this proposal needs to be reviewed again, especially the lebudget part.” Recognized output: I think this proposal needs to be reviewed again, especially the lebudget part. Compared with a multi-language model without two-language restriction, this solution is more likely to recognize “lebudget” as the correct French word rather than a Chinese homophone.

[0062] Step S106, generating and outputting a conference message based on the translation text, the conference message including at least a text message in the target language and / or a synthesized voice message in the target language.

[0063] The translation module will identify the text and translate it into the target language (in this case, French), outputting the translated text; the output module will display the translated text as a group chat message and optionally generate TTS speech in the target language. Example translation: Jepensequeceplandoitêtrerevu, surtoutlapartiebudget. The output module can display a "translation bubble" on the B side, or show the "original text + translation" side by side on the A side.

[0064] The present embodiment is applicable to the online meeting group chat scenario of cross-border social software or conference software. The conference participants come from different countries and regions, and the languages they use may include Chinese, English, French, Korean, Spanish, etc. In the conference, the expression mode of "quoting the original words / terminology of the other party" is common, for example, a Chinese native speaker will intersperse French words or short sentences in a Chinese sentence when replying to a French user, resulting in a mixed (code-switch) feature of multi-language in the voice content. The present embodiment introduces the target language label generated by the gesture in the recording stage, narrows down the "languages that may be contained in the voice" from the full language set of the conference to the two-language set of "the speaker's native language + the language of the person being replied to (target language)", so that a more specialized bilingual mixed speech recognition model can be selected for recognition, and then the translation and output of the translation and / or synthesized speech.

[0065] Specifically, the conference client includes a conference / social application running on a terminal such as a mobile phone, tablet, PC, etc., for collecting audio, displaying interfaces, uploading / downloading data. The first user refers to the speaker who initiates the recording and sends voice messages. The recording control refers to the interactive control for recording in the client interface, such as the "hold to speak" button. The sliding gesture refers to the sliding trajectory performed by the user during the pressing of the recording control, which can be determined as left slide / right slide / up slide / down slide, etc.; the starting point, ending point, distance, speed, etc. of the sliding can also be recorded. The target language label refers to the language label mapped by the sliding gesture (especially the sliding direction), which is used to indicate that the current speech is mainly directed at (replying to) a language user, and is not equivalent to "the audio only contains this language", but indicates that this language may appear in the audio and needs to be accurately recognized. The native language label refers to the main language label of the first user, which can be inferred from the user's registration information, system statistics or language recognition module. The candidate language set is the set of languages that the system infers are most likely to appear in the voice. In this embodiment, it is usually {native language, target language}. The bilingual mixed speech recognition model refers to a speech recognition model (ASR) trained or adapted for the possible alternation of two languages, which can be an end-side model or a cloud-side model. The recognized text is the text sequence output by the ASR, which can contain words or subwords of two languages. Machine translation is the process of translating the recognized text into the target language text. The conference message is the message object displayed in the group chat, which can include the fields of original voice, recognized text, translated text, synthesized speech, etc.

[0066] The core principle of the embodiment is to introduce "gesture-generated target language markers" as strong priors, which, together with the native language markers, jointly constrain the candidate language set to be bilingual, thereby narrowing the search space of speech recognition from "meeting full language" to "native language + target language". Based on this, the misrecognition probability caused by multilingual competition is reduced, especially the probability of misrecognizing foreign language terms as native language homophonic words is reduced, and it is possible to quickly determine which languages are involved in each sentence of speech, thereby quickly determining which mixed language model to use for speech recognition, and improving the speech recognition efficiency of mixed language speech.

[0067] Reference Figure 2 In one embodiment, the method further comprises:

[0068] Step S201, count the language set of the conference participants.

[0069] The client can obtain the participant list and its language label from the conference server, or pull the language field from the group chat member profile. L_meet is obtained. Example: member A Chinese, B French, C English, D Korean, L_meet={zh, fr, en, ko} is obtained.

[0070] Step S202, generate a direction map according to the language set.

[0071] When |L_meet|≤4, each language can be directly mapped to a direction; when |L_meet|>4, an extension strategy can be used. Example: the system generates: left=en, right=fr, up=ko, and down=zh (or the default native language does not occupy the direction). The mapping is written into Map_dir_lang and is sent to the gesture detection module.

[0072] Step S203, display a recording gesture operation prompt.

[0073] When the user triggers the recording control button, the client displays a prompt UI near the recording key, such as "left swipe English / right swipe French / upper swipe Korean". Example: A holds down the recording key, and four sectors appear around the recording key, marked EN / FR / KO / ZH. The user can select FR by swiping right.

[0074] Specifically, the language set L_meet is the language label set of the current participants of the conference, such as {zh, en, fr, ko}. The mapping relationship Map_dir_lang is the mapping of the sliding direction set D={left, right, up, down} to the language set L_meet. The gesture prompt UI can be a floating layer, a prompt bar, a ring sector, a bubble prompt, etc.

[0075] In this embodiment, the "target language label generation" is moved to before the user operation, the probability of user selecting the wrong language is reduced through the visual prompt of the language set in the meeting, so as to improve the correctness of the subsequent model selection. Especially in a multi-lingual meeting, the prompt can place the commonly used language in a more easily operated direction (such as right slide), and avoid showing the language direction that does not exist in the meeting, so as to improve the interaction efficiency and learning consistency. Thus, the accuracy and stability of the whole speech recognition link are improved.

[0076] Specifically, in a multi-lingual meeting, the language set of participants in different conference rooms is different. If a fixed "left = English, right = French" mapping is used, it may cause user confusion or mapping waste (for example, there is no French user in the meeting). Therefore, in this embodiment, the language set of the conference participants is also automatically counted and a recording gesture operation prompt for the conference is generated, so as to improve the usability and label accuracy.

[0077] Specifically, referring to Figure 3 , the target language label is generated according to the sliding direction of the sliding gesture, comprising:

[0078] Step S301, for each target language, a mapping table corresponding to at least one sliding direction is pre-configured.

[0079] Step S302, when it is detected that the first user presses the recording control and completes the sliding in the pressing state, the target language label is determined in the mapping table according to the sliding direction, or the target language label is determined according to the sliding direction and the language set of the conference participants.

[0080] The target language label is generated by the mapping table when the sliding is completed in the pressing state, and the target language can be determined according to the conference language set, so that "when to trigger, how to determine the effective sliding, and how to map the language" are clearly implemented at the implementation level.

[0081] Referring to Figure 4 , in one embodiment, the method further comprises:

[0082] Step 401, when it is determined that there is no bilingual mixed speech recognition model corresponding to the mother tongue language label and the target language label in the speech recognition model set, enter the split timing label mode.

[0083] Specifically, the non-existence in the model set means that the corresponding two-language key model cannot be found in the current available model list on the local / cloud; it also includes version mismatch, cannot be downloaded, cannot be used, etc. The split timing label mode refers to an interaction mode that allows the user to insert a "switch point" during recording.

[0084] In one example, the model selection module queries the model warehouse with the candidate language set {speaker_lang, target_lang} as the key. If the query fails, it is determined that the model is missing. Example: A native speaker of Chinese (zh), target language Korean (ko). The model warehouse has no zh-ko mixed model → trigger degradation. The client prompts the user to enable segmentation markers and switches the UI display (e.g., a waveform bar, segmentation button, or segmentation trigger area appears). Example: A pop-up prompt appears: "No Chinese-Korean mixed recognition model is currently available. Enable segmentation markers to improve recognition accuracy." The user selects "Enable".

[0085] Step 402, in the segmentation marker mode, receiving at least one segmentation marker operation triggered by the first user during the recording process to record at least one time segmentation point of the speech audio.

[0086] Specifically, the segmentation marker operation refers to an event triggered by the user at a specific moment through a specified gesture / click, and the system records the timestamp of this moment on the audio timeline. The time segmentation point refers to t1, t2, …, which is used to cut the audio into [0, t1), [t1, t2), … segments.

[0087] In one example, when the user triggers the segmentation marker operation, the system records the "current recording relative time" as the segmentation point. Example speech: A says: "I agree with this requirement, but I need to change it." After saying "but", A is about to say the Korean word " ", and clicks on the segmentation trigger area once to record t = 2.10s; after saying " ", click again to record t = 2.65s. The final segmentation point set T = {2.10, 2.65}. To avoid errors, the system can make "forward and backward fine-tuning" (e.g., back 80ms) to the segmentation point when recording to cover the transition sound before language switching.

[0088] Step S403, based on the time segmentation points, the speech audio is segmented into multiple audio segments; and for each audio segment, a monolingual speech recognition model is selected to perform speech recognition to obtain a segment recognition text, and the multiple segment recognition texts are spliced and translated to obtain the translation text.

[0089] The audio collection module cuts the cached audio into segments according to the timestamp. Example: Segment P1: [0, 2.10) mainly in Chinese; Segment P2: [2.10, 2.65) mainly in Korean; Segment P3: [2.65, end) mainly in Chinese. The system calls the corresponding monolingual ASR for each segment. Example: P1 uses Chinese ASR → "I agree with this requirement, but", P2 uses Korean ASR → " P3: "I need to change it", the recognized text is obtained by splicing: "This requirement I agree, but I need to change it".

[0090] In actual products, it is impossible to cover all small language combinations of bilingual mixed models. For example, the combinations of "Chinese-Icelandic" and "French-Thai" have low occurrence probability, and training mixed models for each combination will lead to uncontrollable training costs and model repository size. Therefore, the embodiment provides a degradation but high accuracy path when a corresponding two-language mixed model is missing: entering a split timing marking mode, using user marking of switching points to split the mixed audio into multiple segments, and then using monolingual models to recognize and splice the segments.

[0091] When a two-language mixed model is missing, directly using a monolingual model to recognize the entire segment will misrecognize foreign words as noise or near-phonetic, resulting in a serious deviation of the translation. The principle of the split marking mode is to use the user-provided switching points to convert the multilingual mixed problem into multiple "approximately monolingual" sub-problems, so that each sub-problem falls on the distribution that the monolingual ASR is best at, thereby obtaining recognition quality close to or better than the mixed model without training rare two-language mixed models, and significantly reducing system model coverage costs and storage / computing power consumption.

[0092] Specifically, entering the split timing marking mode includes at least one of the following:

[0093] receiving inaccurate feedback of the first user on the output translation result and entering the split timing marking mode accordingly;

[0094] or automatically entering the split timing marking mode based on the confidence output in the speech recognition process being lower than a threshold;

[0095] or entering the split timing marking mode when detecting that there is a multilingual switching feature in the speech audio.

[0096] Inaccurate feedback refers to the user clicking on the "recognition inaccurate / translation inaccurate / correction" entry for the displayed recognition / translation result. Confidence refers to the probability index output by ASR, which can be evaluated at the sentence level (average confidence) or the word level (minimum confidence). The multilingual switching feature includes but is not limited to: a large number of OOV appearing in a short time, frequent jumps in language recognizer output, acoustic distribution mutation, and multiple language competition in decoding beam.

[0097] Specifically, in the first trigger mode, the user feedback trigger step is as follows:

[0098] Step F1: The user views the translation and finds that the key term is wrong.

[0099] Step F2: User clicks "Not recognized / Not translated".

[0100] Step F3: System prompts "Segmentation marker enabled for mixed language recognition" and enters segmentation mode automatically or provides a switch for next recording.

[0101] Example: A finds "lebudget" is recognized as "le budget", clicks feedback, and the system suggests to enable segmentation marker.

[0102] In the second triggering mode, the steps triggered by the confidence threshold are as follows:

[0103] Step C1: ASR outputs sentence-level confidence conf_sent.

[0104] Step C2: If conf_sent<θ (such as 0.60) or the minimum word confidence of the key segment is <θ_word (such as 0.35), it is determined that there is a high-risk misrecognition.

[0105] Step C3: The system automatically pops up a prompt or automatically switches to segmentation mode.

[0106] Example: Sentence-level confidence is 0.48, and the system prompts "Mixed language may cause misrecognition, do you want to enable segmentation marker".

[0107] In the third triggering mode, the steps triggered by the multilingual switching feature are as follows:

[0108] Step L1: The system performs light language recognition on the audio stream to obtain a language sequence that changes over time.

[0109] Step L2: If the number of language switching times is >k (such as 2 times) or the switching interval is too short (such as <1.5s), it is determined that the mixed language is obvious.

[0110] Step L3: Trigger to enter segmentation marker mode or prompt the user.

[0111] Example: Detecting Chinese→French→Chinese jumps, the system suggests segmentation.

[0112] In this embodiment, "subjective feedback" and "objective risk indicators" are used to identify failed samples and guide them into a stronger constraint processing path (segmentation marker). In this way, most ordinary samples can still be processed by the two-language mixed model (or automatic strategy), and only in high-risk scenarios, the segmentation mode is called to improve the consistency of overall experience and system robustness, while reducing the learning burden of users.

[0113] In one embodiment, to distinguish the target language marker from the segmentation timing marker, the same type of gesture is used but different trigger areas are used for distinction, specifically including:

[0114] The sliding gesture detected in the first trigger area is used to generate the target language mark;

[0115] The sliding gesture or click / slide gesture detected in the second trigger area is used to generate the segmentation mark operation; wherein the first trigger area and the second trigger area do not overlap with each other in the recording interface.

[0116] The first trigger area is an area for generating a target language mark, such as a recording button area or a sector area around it. The second trigger area is an area for generating a segmentation mark operation, such as a waveform bar area, a time axis area, a screen bottom dedicated mark bar area, etc. Non-overlapping means that the first and second trigger areas do not overlap with each other in the screen coordinate range, avoiding double interpretation of the same touch event.

[0117] In one example, to distinguish the target language mark from the segmentation timing mark, the same type of gesture is used but different trigger areas are used to distinguish, which specifically includes:

[0118] Step S601, interface division trigger area.

[0119] The client divides the recording interface into area R1 (first trigger area): recording key and its slidable sector; and area R2 (second trigger area): waveform bar / time axis bar displayed during recording. Example: R1 is a central button below the screen; R2 is a horizontal waveform bar above the button.

[0120] Step S602, identify a sliding gesture in R1 to generate a target language mark.

[0121] The user holds down the recording key and slides in R1, and the system interprets the sliding direction as target language selection. Example: right slide→French; UI displays "FR selected".

[0122] Step S603, use the same type of gesture to trigger segmentation marks in R2.

[0123] During recording, the user moves his finger to the waveform bar R2 and swipes (also a sliding gesture), which is interpreted by the system as "inserting a segmentation point". Example: swipe right once in R2→insert segmentation point t=3.20s; vertical line mark appears on the waveform.

[0124] Step S604, conflict processing.

[0125] If the user's touch trajectory crosses R1 and R2, the system can determine the semantics according to the "area where the starting point is located", or prioritize "pressing the control binding" to determine R1 semantics. Example: if the starting point is in R1, it is processed as target language selection to avoid inserting segmentation points during recording.

[0126] If the "target language mark" and "segment mark" are designed with completely different gestures, the user's learning cost is high and it is easy to be confused. In the embodiment, the same type of gesture is used, but the semantics are distinguished by different trigger areas, so that more functions are completed with fewer gestures. In the embodiment, the "space area" is used as a semantic distinction signal, so that the same gesture has different meanings in different areas, thereby reducing the size of the gesture set. The effect is that the user only needs to understand the "slide" operation, and can complete the target language selection and segment mark functions; the interaction is more consistent, the misoperation is less, and the overall usability is improved.

[0127] In one embodiment, the method further comprises:

[0128] In the segment opportunity mark mode, a segment language mark is determined for at least part of the audio segments, the determination of the segment language mark comprising: determining according to the direction, trigger area, trigger times and / or trigger timing of the segment mark operation; and selecting a monolingual speech recognition model of a corresponding language for the corresponding audio segment according to the segment language mark.

[0129] For example, the specific process includes the following:

[0130] Step S701: generating segments and segment points.

[0131] The user inserts a set of segment points T = {t1, t2,...}, and obtains segments P1, P2,..., Pn.

[0132] Step S702: determining a segment language mark.

[0133] When the segment mark is triggered in R2, the sliding direction direction_mark is recorded: right sliding indicates "entering the target language segment from here"; left sliding indicates "entering the mother language segment from here". For example: A speaks Chinese as the mother language, and French as the target language. Before starting to speak French, A right-slides once in R2 after t1, and the segment mark after t1 is French; after the French is finished, A left-slides once in R2 after t2, and the segment mark after t2 is Chinese. The system obtains P1 = Chinese, P2 = French, and P3 = Chinese accordingly.

[0134] Step S703: determining a segment language mark.

[0135] R2 is divided into an upper half area and a lower half area, the upper half area trigger = target language, and the lower half area trigger = mother language. For example, the user clicks to insert t1 in the upper half area, which indicates that the subsequent segment is the target language; and clicks to insert t2 in the lower half area, which indicates that the subsequent segment is the mother language.

[0136] Step S704: determining a segment language mark.

[0137] When only two languages are involved, an "alternation rule" can be adopted: start with the native language by default, switch to the target language for the first marking, switch back to the native language for the second marking, and so on. Example: default P1 = Chinese; P2 = French after the first marking; P3 = Chinese after the second marking, without additional direction or region coding, and the operation is the simplest.

[0138] Step S705: Single-language model recognition according to segment language marking.

[0139] The model selection module selects the corresponding single-language ASR for each segment Pi according to seg_lang[i]: seg_lang[i] = zh → Chinese ASR; seg_lang[i] = fr → French ASR. Example: P2 is a French segment, and the French single-language model is used to recognize "le budget"; this avoids misrecognition as a homophone by a Chinese model. Here, seg_lang[i] refers to the language label of the ith audio segment (such as zh and fr).

[0140] Step S706: Splicing and translation output.

[0141] The recognized texts of each segment are spliced in chronological order and input into the translation module to output the target language translation. Example: spliced into "I think le budget needs to be discussed again", and the French translation is more accurate.

[0142] In this embodiment, "multi-language mixed recognition" is further divided into "segment-level single-language recognition", and the correct single-language model is selected for each segment through segment language marking, so that each segment is decoded on the optimal distribution of the corresponding language model. Compared with the scheme where only the split point is provided but the language is unknown, this embodiment significantly reduces the risk of "model selection error leading to entire segment garbled code", especially for small language terms, personal names, and place names, thereby improving recognition accuracy and translation quality, and enhancing the versatility of the system for rare language combinations.

[0143] In one embodiment, an online conference translation system is also provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to perform the steps of the online conference translation method described above. The steps of the online conference translation method can be the steps in the above embodiments.

[0144] Specifically, the present embodiment can be implemented by the following functional modules (the modules can be deployed on the terminal side, the cloud side, or in a distributed manner between the terminal and the cloud): an audio acquisition module: responsible for acquiring a microphone audio stream, performing noise reduction, echo cancellation, and mute detection, etc. pre-processing. A gesture detection module: listens to the touch events of the recording control area, identifies the sliding direction and generates a target language label. A language label management module: obtains a native language label, and fuses a target language label to form a candidate language set. A model selection module: selects a corresponding bilingual mixed model from a model repository according to the candidate language set. A speech recognition module (ASR): recognizes the audio and outputs the recognized text and optional confidence. A translation module (MT): translates the recognized text into a target language and outputs the translated text. An output module: outputs the translated text in the form of text and / or TTS speech to the target user or all members in the conference group chat.

[0145] In one embodiment, a computer readable storage medium is also provided, which stores computer executable instructions for causing a computer to perform the steps of the online conference translation method described above. The steps of the online conference translation method described above can be the steps of the online conference translation method of each of the embodiments described above.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRA), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0147] Any technical features in the above embodiments can be combined, and for the sake of brevity, not all possible combinations are described, however, any combination of the technical features is considered to be within the scope of the present specification.

Claims

1. An online meeting translation method, characterized in that, The method comprises: receiving a recording operation of a first user at a conference client to collect voice audio; during the collection of the voice audio, receiving a sliding gesture of the first user on a preset recording control, and generating a target language mark according to a sliding direction of the sliding gesture, the target language mark being used to represent that the voice audio is a reply to a conference participant using a target language; obtaining a native language mark of the first user; determining a candidate language set of the voice audio according to the native language mark and the target language mark, and selecting a bilingual mixed speech recognition model corresponding to the candidate language set from a preset speech recognition model set to perform speech recognition on the voice audio to obtain recognized text; performing machine translation on the recognized text to obtain a translation text corresponding to the target language, and generating and outputting a conference message based on the translation text, the conference message at least including a text message in the target language and / or a synthesized speech message in the target language; The method further comprises: when it is determined that there is no bilingual mixed speech recognition model corresponding to the native language mark and the target language mark in the speech recognition model set, entering a split timing mark mode; in the split timing mark mode, receiving at least one split mark operation triggered by the first user during the recording process to record at least one time split point of the voice audio; based on the time split point, splitting the voice audio into a plurality of audio segments; and for each audio segment, selecting a monolingual speech recognition model to perform speech recognition to obtain a segment recognized text, and then splicing and translating a plurality of segment recognized texts to obtain the translation text; entering the split timing mark mode comprises at least one of the following: receiving inaccurate feedback of the first user on the output translation result and entering the split timing mark mode accordingly; or automatically entering the split timing mark mode based on a confidence level output during the speech recognition process being lower than a threshold; or entering the split timing mark mode when it is detected that there is a multi-language switching feature in the voice audio; To distinguish the target language mark from the split timing mark, the same type of gesture is used but different trigger areas are used for distinction, specifically including: the sliding gesture detected in the first trigger area is used to generate the target language mark; the sliding gesture or click / slide gesture detected in the second trigger area is used to generate the split mark operation; wherein the first trigger area and the second trigger area do not overlap with each other in the recording interface.

2. The method of claim 1, wherein, Further comprising: at the beginning of the conference or during the conference, counting a language category set of current conference participants, and generating a recording gesture operation prompt based on the language category set, the recording gesture operation prompt being used to indicate a mapping relationship between different sliding directions and different target languages, and displaying the recording gesture operation prompt in a recording interface.

3. The method according to claim 1 or 2, characterized in that, The target language mark is generated according to the sliding direction of the sliding gesture, comprising: for each target language, a mapping table corresponding to at least one sliding direction is pre-configured; When detecting that the first user presses the recording control and completes the sliding in the pressing state, a target language mark is determined in the mapping table according to the sliding direction, or the target language mark is determined together with the set of conference participant language categories according to the sliding direction.

4. The method of claim 1, wherein, Also comprising: In the segmentation opportunity marking mode, a segment language mark is determined for at least part of the audio segments, and the determination manner of the segment language mark comprises: determining according to the direction, trigger area, trigger times and / or trigger timing of the segmentation marking operation; And a monolingual speech recognition model of a corresponding language is selected for the corresponding audio segment according to the segment language mark.

5. An online meeting translation system, characterized by, Comprising: A recording collection module, configured to collect conference speech audio at a conference client; A gesture detection module, configured to detect a sliding gesture in a recording process and generate a target language mark, and / or generate a segmentation marking operation in a segmentation opportunity marking mode; A language determination module, configured to obtain a user's mother tongue language mark, and determine a set of candidate language categories together with the target language mark; A model selection module, configured to select a bilingual mixed speech recognition model from a preset speech recognition model set, or select a monolingual speech recognition model when a corresponding bilingual mixed speech recognition model is lacking; A speech recognition module, configured to perform speech recognition on the speech audio or audio segments segmented therefrom to obtain recognized text; A translation module, configured to perform machine translation on the recognized text to obtain translated text; A speech synthesis and output module, configured to generate synthesized speech messages and / or text messages based on the translated text and output to a conference group chat; The online conference translation system is also configured to: When it is determined that there is no bilingual mixed speech recognition model corresponding to the mother tongue language mark and the target language mark in the set of speech recognition models, enter a segmentation opportunity marking mode; In the segmentation opportunity marking mode, receive at least one segmentation marking operation triggered by the first user in the recording process to record at least one time segmentation point of the speech audio; Segment the speech audio into a plurality of audio segments based on the time segmentation point; and for each audio segment, select a monolingual speech recognition model to perform speech recognition to obtain segment recognized text, and then splice and translate a plurality of segment recognized texts to obtain the translated text; Entering the segmentation opportunity marking mode comprises at least one of the following: Receiving inaccurate feedback of the first user on the output translated result and entering the segmentation opportunity marking mode accordingly; Or automatically entering the segmentation opportunity marking mode based on the confidence output in the speech recognition process being lower than a threshold; Or entering the segmentation opportunity marking mode when detecting that there is a multi-language switching feature in the speech audio; To distinguish the target language mark from the segmentation opportunity mark, the same type of gesture is used but different trigger areas are used for the distinction, specifically comprising: The sliding gesture detected in the first trigger area is used to generate the target language mark; The sliding gesture or click / slide gesture detected in the second trigger area is used to generate the segmentation marking operation; wherein the first trigger area and the second trigger area do not overlap with each other in the recording interface.

6. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, which is executed by a processor, implements the method according to any one of claims 1 to 4.

7. An online meeting translation apparatus comprising: Memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that the processor, when executing the program, executes the steps of the online conference translation method according to any one of claims 1 to 4.

Images