Remote conference real-time voice recognition optimization method and system based on visual context

Abstract

The invention provides a remote conference real-time voice recognition optimization method and system based on a visual context, and the method comprises the steps: collecting the visual context information according to a projection screen image; acquiring a voice audio and inputting the voice audio into the voice recognition model to obtain a voice recognition result; and adjusting the voice recognition result by using the context information. On the basis of a traditional remote conference voice real-time recognition function, word and image content information of a current projection screen is analyzed, and the information is fused into a voice recognition model, so that words with low recognition probability or ambiguous words are optimized. According to the method, the error recognition rate can be effectively reduced, the vocabulary recognition problem in the professional field is solved, and the overall performance of the real-time speech translation function is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, system, terminal and medium for optimizing real-time speech recognition in a remote conference based on visual context. Background Art

[0002] In current remote conference systems (such as Teams, Tencent Conference, Zoom, etc.), real-time voice translation has been integrated. This function relies on the speech recognition model and can convert the voices of conference participants into text information in real time, such as Figure 1 At present, the development of technology mainly relies on various speech recognition algorithm models to improve the accuracy of recognition.

[0003] However, in the specific environment of remote meetings, the use of professional terms and differences in speaker accents often reduce the accuracy of existing speech recognition models, thereby affecting recognition results. Existing real-time speech translation systems mainly rely on the speech signals of meeting participants, and fail to make full use of more contextual information to assist speech recognition. In contrast, when humans perform speech translation and understanding, they combine rich environmental information and knowledge background to better assist in understanding the speaker's speech.

[0004] Currently, the main strategy to solve this problem is to improve the speech recognition model itself. In particular, large models are trained using large amounts of data (including professional domain data, data on different accents and dialects, etc.) in order to solve the above problems. However, the data collection, model training and inference process are costly. Summary of the invention

[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a method, system, terminal and medium for optimizing real-time speech recognition in remote conferences based on visual context.

[0006] According to one aspect of the present invention, a method for optimizing real-time speech recognition in a remote conference based on visual context is provided, comprising:

[0007] Collect visual context information based on the projection screen;

[0008] Acquire speech audio and input it into the speech recognition model to obtain speech recognition results;

[0009] The speech recognition result is adjusted using the context information.

[0010] Preferably, collecting visual context information according to the projection screen includes:

[0011] Real-time acquisition of the current projection screen;

[0012] Performing image recognition on the projection screen to obtain semantic information of the text and pictures in the projection screen;

[0013] Adding the text and the semantic information to a context information library;

[0014] Determine whether the projection screen has changed. If so, repeat the above image recognition and adding process.

[0015] Preferably, the context information library stores all text and semantic information of the projection screen with a fixed number of frames.

[0016] Preferably, the adjusting the speech recognition result by using the context information includes:

[0017] Obtaining words at all positions and corresponding probability values ​​from the speech recognition result;

[0018] For a word whose probability value is lower than the threshold, obtain the top K candidate words with the possible probability values ​​corresponding to the word at that position in the speech recognition model, and the probability value of the candidate word is expressed as: P_origin(j), j=1:K;

[0019] Obtain the correlation probability P_context_corr(j) between the candidate word and all text and semantic information in the context information library, j=1:K;

[0020] The two probabilities are weighted to obtain the final probability P(j)=P_origin(j)*P_context_corr(j), j=1:K;

[0021] According to the final probability, the word with the highest probability is re-sorted and selected as the word after the word position is optimized.

[0022] Preferably, the adjusting the speech recognition result by using the context information includes:

[0023] From the speech recognition results, obtaining words whose prior probability is lower than a threshold;

[0024] Find out whether there are homophones in all texts and semantic information of the context information base; if yes, replace them with homophones; if no, do not process them.

[0025] Preferably, the speech recognition model is a speech recognition model based on a large model.

[0026] Preferably, the adjusting the speech recognition result by using the context information includes:

[0027] Using the context information and the original speech signal as input prompt words for the large model;

[0028] The large model optimizes and adjusts the speech recognition results according to the input prompt words.

[0029] According to a second aspect of the present invention, there is provided a remote conference real-time speech recognition optimization system based on visual context, comprising:

[0030] Context information collection module: collects visual context information based on the projection screen;

[0031] Speech recognition module: obtains speech audio and inputs it into the speech recognition model to obtain speech recognition results;

[0032] Adjustment and optimization module: using the context information to adjust the speech recognition result.

[0033] According to a third aspect of the present invention, a terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any one of the methods described above, or to run the system described above when executing the program.

[0034] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any of the methods described above, or to run the system described above.

[0035] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0036] The visual context-based remote conference real-time speech recognition optimization method involved in the embodiment of the present invention, based on the traditional remote conference real-time speech recognition function, analyzes the text and image content information of the current projection screen and integrates this information into the speech recognition model to optimize words with low recognition probability or ambiguity. This method can effectively reduce the error recognition rate, solve the problem of vocabulary recognition in professional fields, and thus improve the overall performance of the real-time speech translation function. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0038] Figure 1 A flowchart of a remote conference speech recognition method in the prior art;

[0039] Figure 2 The figure is a flow chart of a method for optimizing real-time speech recognition in a remote conference based on visual context in one embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0041] In a teleconference, participants usually explain documents (PPT / Word) and the contents of the explanations are often related to the contents of the current remote screen. In order to improve the accuracy of real-time speech translation in teleconference scenarios, an embodiment of the present invention provides a teleconference real-time speech recognition optimization method based on visual context, such as Figure 2 As shown, the specific process is as follows:

[0042] Step 1: Collect visual context information based on the projection screen;

[0043] Step 2, obtain the speech audio and input it into the speech recognition model to obtain the speech recognition result;

[0044] In this step, the voice audio corresponds to the voice audio of the screen projection in the above step 1, that is, step 1 collects the visual information corresponding to the screen projection, and step 2 collects the voice audio corresponding to the screen projection. For example, during a meeting, the speaker projects the content of his speech to form a screen projection, and at the same time, the speaker will have a corresponding voice to explain the content corresponding to the screen projection. This application obtains the visual context information of the screen projection and the corresponding voice through the above steps.

[0045] Step 3, using the context information of step 1, adjust the speech recognition result obtained in step 2.

[0046] The above-mentioned embodiment makes full use of the fact that the content of the document material explained in a remote conference is highly relevant to the document material. The text and semantic information obtained by visual content analysis of the projected document in the remote conference is used to enhance the results of real-time speech recognition, which can effectively reduce the problems of erroneous recognition and professional vocabulary recognition.

[0047] In some embodiments, for some projection images, the speech content of the conference speaker is relatively small, so the collection of the projection images and the speech recognition process can be performed simultaneously. Therefore, in this embodiment, the execution order of step 1 and step 2 can also be parallel.

[0048] In a preferred embodiment of the present invention, a preferred process of step 1 is provided, and visual context information is collected according to the projection screen, including:

[0049] S101, real-time acquisition of the current projection screen;

[0050] S102, performing image recognition on the projection screen captured in S101 to obtain semantic information of text and pictures in the projection screen;

[0051] In some specific embodiments, image recognition is mainly used to analyze and understand the visual information of the current projection screen. The text information therein can be extracted through OCR, and the semantic information therein can be obtained through target detection and recognition.

[0052] S103, adding the text and semantic information extracted in S102 to the context information library;

[0053] S104, determine whether the projected screen image has changed. If it has changed, repeat the above image recognition and adding process, i.e. S101-S103.

[0054] In further elaboration, a more preferred embodiment involves storing the context information library as all text and semantic information in the projection screen of a fixed number of frames. This configuration helps to save memory usage during the operation of the entire method without adversely affecting the effect of conference communication.

[0055] Generally, the main reasons for speech recognition errors are inaccurate dialect recognition and insufficient prior knowledge of professional terms. Therefore, in a preferred embodiment of the present invention, in order to solve the problem of inaccurate dialect recognition, a set of optimization and adjustment schemes is proposed, which are as follows:

[0056] S301, obtaining the words at all positions and the probability values ​​corresponding to the words from the speech recognition results obtained in step 2;

[0057] S302, for a word whose probability value is lower than a threshold, obtain the top K candidate words with the possible probability values ​​corresponding to the word at the position in the speech recognition model, and express the probability value of the candidate word as: P_origin(j), j=1:K;

[0058] S303, obtaining the correlation probability P_context_corr(j) of all text and semantic information of the candidate word in the context information library, j=1:K

[0059] S304, performing weighted calculation on the two probabilities to obtain a final probability P(j)=P_origin(j)*P_context_corr(j), j=1:K;

[0060] S305, re-ranking according to the final probability to select the word with the highest probability as the word after the word position is optimized.

[0061] Another preferred embodiment optimizes the situation where the professional terminology recognition is not accurate enough, specifically:

[0062] S31, obtaining words whose prior probability is lower than a threshold from the speech recognition result;

[0063] Each word has a prior probability, which expresses the probability of the word being used. It can be obtained by counting the frequency of these words in a large number of documents or IoT corpora. The lower the prior probability, the lower the frequency of use of the word. For example, uncommon words and professional vocabulary generally have a low probability of use.

[0064] S32, searching whether there are homophones in the text and semantic information in the context information library; if yes, use the homophones for replacement. If no, no operation is required.

[0065] In order to better recognize speech, in some preferred embodiments, speech recognition uses a large model. A large model refers to a machine learning model with large-scale parameters and complex computing structure, usually constructed by a deep neural network, with billions or even hundreds of billions of parameters. These models can process massive amounts of data and complete various complex tasks, such as natural language processing, computer vision, speech recognition, etc.

[0066] In order to obtain more accurate speech recognition results, a preferred embodiment of the present invention directly modifies the input of the large model, that is, the prompt word of the large model is modified to the original speech signal and context information, and the output result is adjusted and optimized by the large model.

[0067] Furthermore, in a specific embodiment, text and semantic information are input as context information in the prompt. For example, in addition to the original voice signal, a description such as "the content you are currently recognizing is related to X" is added to the prompt, where X represents the text and semantic information extracted from image recognition.

[0068] According to the same technical concept, in another embodiment of the present invention, a remote conference real-time speech recognition optimization system based on visual context is provided, comprising:

[0069] Context information collection module: collects visual context information according to the projection screen;

[0070] Speech recognition module: obtains speech audio and inputs it into the speech recognition model to obtain speech recognition results;

[0071] Adjustment and optimization module: using the context information to adjust the speech recognition result.

[0072] The modules / units in the above-mentioned examples of the present invention may specifically refer to the implementation techniques of the corresponding steps of the remote conference real-time speech recognition optimization method based on visual context in the above-mentioned embodiment, which will not be described in detail here.

[0073] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the above-mentioned method or to run the above-mentioned system when executing the program.

[0074] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories.

[0075] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0076] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.

[0077] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the above method, or run the above system.

[0078] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.

[0079] It will be appreciated by those skilled in the art that embodiments of the present invention may provide methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0081] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0083] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A remote conference real-time speech recognition optimization method based on visual context, characterized in that: include: Collect visual context information based on the projection screen; Obtaining the voice audio corresponding to the projection screen, and inputting it into the voice recognition model to obtain the voice recognition result; The speech recognition result is adjusted using the context information.

2. According to claim 1, a remote conference real-time speech recognition optimization method based on visual context is characterized in that: The collecting visual context information according to the projection screen includes: Real-time acquisition of the current projection screen; Performing image recognition on the projection screen to obtain semantic information of the text and pictures in the projection screen; Adding the text and the semantic information to a context information library; Determine whether the projection screen has changed. If so, repeat the above image recognition and adding process.

3. The method for optimizing real-time speech recognition in remote conferences based on visual context according to claim 2 is characterized in that: The context information library stores all text and semantic information of the projection screen with a fixed number of frames.

4. The method for optimizing real-time speech recognition in remote conferences based on visual context according to claim 2, characterized in that: The using the context information to adjust the speech recognition result includes: Obtaining words at all positions and corresponding probability values ​​from the speech recognition result; For a word whose probability value is lower than the threshold, obtain the top K candidate words with the possible probability values ​​corresponding to the word at that position in the speech recognition model, and the probability value of the candidate word is expressed as: P_origin(j), j=1:K; Obtain the correlation probability P_context_corr(j) between the candidate word and all text and semantic information in the context information library, j=1:K; The two probabilities are weighted to obtain the final probability P(j)=P_origin(j)*P_context_corr(j), j=1:K; According to the final probability, the word with the highest probability is re-sorted and selected as the word after the word position is optimized.

5. The method for optimizing remote conference real-time speech recognition based on visual context according to claim 2 is characterized in that: The using the context information to adjust the speech recognition result includes: From the speech recognition results, obtaining words whose prior probability is lower than a threshold; Find out whether there are homophones in all texts and semantic information of the context information base; if yes, replace them with homophones; if no, do not process them.

6. The method for optimizing real-time speech recognition in remote conferences based on visual context according to claim 1, characterized in that: The speech recognition model is a speech recognition model based on a large model.

7. The method for optimizing real-time speech recognition in remote conferences based on visual context according to claim 6, characterized in that: The using the context information to adjust the speech recognition result includes: Using the context information and the original speech signal as input prompt words for the large model; The large model optimizes and adjusts the speech recognition results according to the input prompt words.

8. A remote conference real-time speech recognition optimization system based on visual context, characterized in that: include: Context information collection module: collects visual context information based on the projection screen; Speech recognition module: obtains speech audio and inputs it into the speech recognition model to obtain speech recognition results; Adjustment and optimization module: using the context information to adjust the speech recognition result.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the method described in any one of claims 1 to 7, or to run the system described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 7, or to run the system described in claim 8.

Images