Method and system for remote meeting real-time speech recognition optimization based on visual context

Abstract

The application provides a remote conference real-time voice recognition optimization method and system based on visual context, comprising: collecting visual context information according to the screen projection picture; obtaining voice audio and inputting into a voice recognition model to obtain a voice recognition result; and adjusting the voice recognition result by using the context information. On the basis of the traditional remote conference voice real-time recognition function, the application analyzes the text and image content information of the current screen projection picture, and integrates the information into the voice recognition model, so as to optimize the words with lower recognition probability or ambiguity. The method can effectively reduce the error recognition rate, solve the recognition problem of professional field vocabulary, and further improve the overall performance of the real-time voice translation function.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a remote conference real-time speech recognition optimization method and system based on visual context, a terminal and a medium. BACKGROUND

[0002] In the current remote conference system (such as Teams, Tencent Conference, Zoom, etc.), a real-time speech translation function has been integrated. This function relies on a speech recognition model and can convert the sound of conference participants into text information in real time, as shown in the following figure. Figure 1 Currently, the development of technology mainly relies on various speech recognition algorithm models to improve the accuracy of recognition.

[0003] However, in the specific environment of a remote conference, the use of professional terms and the difference in the accent of the speaker often leads to a decrease in the accuracy of existing speech recognition models, thereby affecting the recognition effect. The existing real-time speech translation system mainly relies on the speech signal of the conference participants and fails to fully utilize more context information to assist speech recognition. In contrast, humans combine rich environmental information and knowledge background when performing speech translation understanding to better assist the understanding of the speaker's voice.

[0004] Currently, the main strategy to solve this problem is to improve the speech recognition model itself. In particular, a large model is trained by using a large amount of data (including professional field data, data of different accents and dialects, etc.) in order to solve the above-mentioned problems. However, the collection of data, the training of the model and the inference process are costly. SUMMARY

[0005] In view of the defects in the prior art, the present application aims to provide a remote conference real-time speech recognition optimization method and system based on visual context, a terminal and a medium.

[0006] According to one aspect of the present application, a remote conference real-time speech recognition optimization method based on visual context is provided, comprising:

[0007] According to the screen projection picture, visual context information is collected;

[0008] The speech audio is obtained and input into a speech recognition model to obtain a speech recognition result;

[0009] The context information is used to adjust the speech recognition result.

[0010] Preferably, the visual context information is collected according to the screen projection picture, comprising:

[0011] The current screen projection picture is collected in real time;

[0012] performing image recognition on the projection picture to obtain text and semantic information of pictures in the projection picture;

[0013] adding the text and the semantic information to a context information library;

[0014] judging whether the projection picture changes, and if the projection picture changes, repeating the image recognition and the adding.

[0015] Preferably, the context information library stores all text and semantic information of a fixed number of projection pictures.

[0016] Preferably, the adjusting the speech recognition result by using the context information comprises:

[0017] obtaining words in all positions and corresponding probability values from the speech recognition result;

[0018] for a word whose probability value is lower than a threshold value, obtaining K candidate words corresponding to the word in a speech recognition model, and a probability value of the candidate word is represented as P_origin(j), j=1:K;

[0019] obtaining a correlation probability P_context_corr(j) of the candidate word and all text and semantic information in the context information library, j=1:K;

[0020] performing weighted calculation on the two probabilities to obtain a final probability P(j)=P_origin(j)*P_context_corr(j), j=1:K;

[0021] reordering according to the final probability to select a word with the maximum probability as an optimized word in the position.

[0022] Preferably, the adjusting the speech recognition result by using the context information comprises:

[0023] obtaining a word whose prior probability is lower than a threshold value from the speech recognition result;

[0024] checking whether there is a homonym in all text and semantic information in the context information library, and if there is, replacing the word with the homonym, and if there is not, not processing.

[0025] Preferably, the speech recognition model is a speech recognition model based on a large model.

[0026] Preferably, the adjusting the speech recognition result by using the context information comprises:

[0027] The context information is used as an input prompt word of a large model together with the original voice signal.

[0028] The large model optimizes and adjusts the voice recognition result according to the input prompt word.

[0029] According to a second aspect of the present application, a visual context-based remote conference real-time voice recognition optimization system is provided, comprising:

[0030] A context information acquisition module acquires visual context information according to a projection screen image.

[0031] A voice recognition module acquires voice audio and inputs the voice audio into a voice recognition model to obtain a voice recognition result.

[0032] An adjustment and optimization module adjusts the voice recognition result by using the context information.

[0033] According to a third aspect of the present application, a terminal comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform any of the methods or run the system.

[0034] According to a fourth aspect of the present application, a computer readable storage medium is provided, which stores a computer program executable by a processor to perform any of the methods or run the system.

[0035] Compared with the prior art, the embodiments of the present application have at least one of the following beneficial effects:

[0036] The visual context-based remote conference real-time voice recognition optimization method of the embodiments of the present application, on the basis of the traditional remote conference voice real-time recognition function, analyzes the text and image content information of the current projection screen image, and integrates the information into the voice recognition model to optimize the words with lower recognition probability or ambiguity. The method can effectively reduce the error recognition rate, solve the problem of professional field vocabulary recognition, and further improve the overall performance of the real-time voice translation function. BRIEF DESCRIPTION OF DRAWINGS

[0037] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments, read in conjunction with the accompanying drawings:

[0038] Figure 1 A flowchart of a remote conference voice recognition method in the prior art is shown.

[0039] Figure 2 A flowchart of a visual context-based remote conference real-time voice recognition optimization method in an embodiment of the present application is shown. Detailed Implementation

[0040] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0041] During remote meetings, participants typically present documents (PPT / Word) materials, and the content of their presentations is often related to the content projected onto the remote screen. To improve the accuracy of real-time speech translation in remote meeting scenarios, one embodiment of this invention provides a method for optimizing real-time speech recognition in remote meetings based on visual context, such as... Figure 2 As shown, the specific process is as follows:

[0042] Step 1: Collect visual context information based on the projected screen;

[0043] Step 2: Acquire the audio and input it into the speech recognition model to obtain the speech recognition result;

[0044] In this step, the audio refers to the audio corresponding to the screen projection in step 1 above. That is, step 1 collects the visual context information of the screen projection, and step 2 collects the audio corresponding to that screen projection. For example, during a meeting, the speaker projects the content of their presentation onto the screen, and simultaneously, the speaker provides corresponding audio explanations for that screen projection. This application obtains the visual context information of the screen projection and the corresponding audio through the above steps.

[0045] Step 3: Adjust the speech recognition results obtained in Step 2 using the context information from Step 1.

[0046] The above embodiments fully utilize the strong correlation between the content being explained and the document material when it is presented in a remote meeting. By analyzing the visual content of the document projected in the remote meeting to obtain text and semantic information, the results of real-time speech recognition are enhanced, which can effectively reduce the problems of misidentification and recognition of professional domain terms.

[0047] In some embodiments, for certain projected screens, the speaker's audio content is relatively short; therefore, the acquisition of the projected screen and the speech recognition process can be performed simultaneously. Thus, in this embodiment, steps 1 and 2 can also be executed in parallel.

[0048] In a preferred embodiment of the present application, the preferred process of step 1 is provided, according to the screen projection picture, the visual context information is collected, including:

[0049] S101, real-time collection of the current screen projection picture;

[0050] S102, image recognition is performed on the screen projection picture collected in S101 to obtain the text and semantic information of the picture in the screen projection picture;

[0051] In some specific embodiments, image recognition is mainly used to analyze and understand the visual information of the current screen projection picture, and the text information therein can be extracted by OCR, and the semantic information therein can be obtained by target detection and recognition.

[0052] S103, the text and semantic information extracted in S102 are added to the context information library;

[0053] S104, it is judged whether the screen projection picture changes, if it changes, the above-mentioned image recognition and addition process is repeated, that is, S101-S103.

[0054] In further elaboration, a more preferred embodiment relates to storing all the text and semantic information in the screen projection picture of a fixed number of frames in the context information library. This configuration helps to save the memory usage during the entire method operation process, and at the same time does not adversely affect the effect of the conference communication.

[0055] Generally, the main reason for the error of speech recognition is the inaccuracy of dialect recognition and the lack of prior knowledge of professional terms. Therefore, in a preferred embodiment of the present application, in order to solve the problem of inaccurate dialect recognition, an optimization adjustment scheme is proposed, which is as follows:

[0056] S301, from the speech recognition result obtained in step 2, all words in the position and the probability value corresponding to the word are obtained;

[0057] S302, for the words whose recognition probability value is lower than the threshold value, the candidate words corresponding to the possible probability value of the position word in the speech recognition model are obtained, and the probability value of the candidate word is represented as: P_origin(j), j=1:K;

[0058] S303, the correlation probability P_context_corr(j) of the candidate word in all the text and semantic information of the context information library is obtained, j=1:K

[0059] S304, the two probabilities are weighted to obtain the final probability P(j)=P_origin(j)*P_context_corr(j), j=1:K;

[0060] S305, according to the final probability, reordering selects the word with the maximum probability as the optimized word in the word position.

[0061] Another preferred embodiment, for the case of inaccurate professional term identification, is optimized, specifically as follows:

[0062] S31, from the speech recognition result, obtain a word with a prior probability lower than a threshold value;

[0063] Each word has a prior probability, which represents the probability of using the word, which can be obtained by counting the frequency of the appearance of these words in a large number of literature or Internet of Things corpus. The lower the prior probability, the lower the frequency of expressing the use of the word, for example, rare words and professional terms generally have a lower probability of use.

[0064] S32, check whether there is a homonym in the text and semantic information in the context information library; if there is, replace the homonym; if not, no operation is needed.

[0065] In order to better recognize speech, in some preferred embodiments, speech recognition uses a large model. A large model refers to a machine learning model with large-scale parameters and complex computational structure, usually constructed by a deep neural network, with tens of billions or even hundreds of billions of parameters. These models can process massive data and complete various complex tasks such as natural language processing, computer vision, speech recognition, etc.

[0066] In order to obtain more accurate speech recognition results, one preferred embodiment of the present application directly modifies the input of the large model. That is, the prompt word prompt of the large model is modified to the original speech signal and the context information. The output result is adjusted and optimized by the large model.

[0067] Further, in a specific embodiment, the text and semantic information are input as context information in the prompt, for example, in addition to the original speech signal, the prompt is increased. "The content you currently recognize is related to X" such description, where X represents the text and semantic information extracted in image recognition.

[0068] According to the same technical concept, in other embodiments of the present application, a remote conference real-time speech recognition optimization system based on visual context is provided, comprising:

[0069] The context information acquisition module acquires visual context information according to the projection screen picture;

[0070] The speech recognition module acquires speech audio and inputs it into a speech recognition model to obtain a speech recognition result;

[0071] An adjustment optimization module adjusts the speech recognition result based on the context information.

[0072] The modules / units in the above examples can be implemented by referring to the implementation of the steps in the method for optimizing real-time speech recognition in a remote conference based on visual context.

[0073] Based on the same inventive concept, in other embodiments of the present application, a terminal is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor is configured to execute the program to perform the above method or run the above system.

[0074] Optionally, the memory is configured to store the program, and the memory can comprise a volatile memory, such as a random-access memory (RAM), e.g., a static random-access memory (SRAM), a Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), etc., or a non-volatile memory, such as a flash memory. The memory is configured to store computer programs (e.g., application programs, functional modules, etc. for implementing the above method), computer instructions, etc., and the above computer programs, computer instructions, etc. can be stored in one or more memories.

[0075] The processor is configured to execute the computer program stored in the memory to implement each step in the method according to the above embodiments. For details, refer to the related description in the method embodiments.

[0076] The processor and the memory can be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor can be coupled and connected through a bus.

[0077] Based on the same inventive concept, in other embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, and the program is executable by a processor to perform the above method or run the above system.

[0078] Computer readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired computer program code means in the form of computer readable instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a combination thereof. Also, functional

[0079] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code means.

[0080] The present application is described in reference to the flowchart illustrations and / or block diagrams of the methods, apparatus (systems) and computer program products according to the present application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams. Figure 1 means for (an apparatus with) one or more functions specified in one or more of the flowchart illustrations and / or block diagrams.

[0081] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in one or more of the flowchart illustrations and / or block diagrams. Figure 1 means for (an apparatus with) one or more functions specified in one or more of the flowchart illustrations and / or block diagrams.

[0082] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams block or blocks. Figure 1one or more processes and / or blocks Figure 1 the steps of a function specified in one or more blocks.

[0083] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A method for visual context based remote meeting real-time speech recognition optimization, the method comprising: The method comprises the following steps: According to the screen projection picture, the visual context information is collected; The voice audio corresponding to the screen projection picture is obtained and input into a voice recognition model to obtain a voice recognition result; The voice recognition result is adjusted by using the context information; The method of collecting visual context information according to the screen projection picture comprises the following steps: The current screen projection picture is collected in real time; The screen projection picture is subjected to image recognition to obtain the semantic information of the text and pictures in the screen projection picture; The text and the semantic information are added to a context information library; It is judged whether the screen projection picture has changed, and if so, the above-mentioned image recognition and addition process is repeated; The method of adjusting the voice recognition result by using the context information comprises the following steps: All words in positions and corresponding probability values are obtained from the voice recognition result; For the words whose probability values are lower than a threshold value, the candidate words of the position in the voice recognition model are obtained, and the probability values of the candidate words are represented as P_origin(j), j=1:K; The correlation probability P_context_corr(j), j=1:K of the candidate words and all text and semantic information in the context information library is obtained; The two probabilities are weighted to obtain the final probability P(j)=P_origin(j)*P_context_corr(j), j=1:K; The words with the maximum probability are selected as the optimized words in the position according to the final probability.

2. The method of claim 1, wherein, The context information library stores all text and semantic information of a fixed number of screen projection pictures.

3. The method of claim 1, wherein, The method of adjusting the voice recognition result by using the context information comprises the following steps: Words with prior probability lower than a threshold value are obtained from the voice recognition result; It is judged whether there are homophonic words in all text and semantic information in the context information library, and if so, the homophonic words are replaced; if not, no processing is performed.

4. The method of claim 1, wherein, The voice recognition model is a voice recognition model based on a large model.

5. The method of claim 4, wherein, The method of adjusting the voice recognition result by using the context information comprises the following steps: The context information and the original voice signal are used as input prompts of the large model; The large model optimizes and adjusts the voice recognition result according to the input prompts.

6. A visual context based remote meeting real-time speech recognition optimization system for implementing the visual context based remote meeting real-time speech recognition optimization method of any one of claims 1-5, characterized in that, The method comprises the following steps: A context information collection module collects visual context information according to a screen projection picture; A voice recognition module obtains voice audio and inputs it into a voice recognition model to obtain a voice recognition result; An adjustment and optimization module adjusts the voice recognition result by using the context information.

7. A terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to execute the method of any one of claims 1-5, or run the system of claim 6.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to execute the method of any one of claims 1-5, or run the system of claim 6.

Images