Speech recognition method and system for translation machine with double-sided transparent screen

Abstract

The invention relates to the technical field of speech recognition, and provides a speech recognition method and system for a double-sided transparent screen translation machine. The method comprises the following steps: analyzing voice information based on an accent analysis model to obtain an accent category of a translator user; matching target model parameters in a database based on the accent category, updating model parameters of a preset speech recognition model based on the target model parameters to obtain an updated speech recognition model, and inputting speech information into the updated speech recognition model to obtain an initial speech recognition text; acquiring a second lip change video corresponding to the initial speech recognition text; judging whether the initial speech recognition text is qualified or not based on the first lip change video and the second lip change video; and if not, optimizing the initial speech recognition text based on the lip comparison video and the lip change video to obtain a target speech recognition text. The method can improve the accuracy of a speech recognition result in a complex environment.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method and system for a double-sided transparent screen translation machine. Background Art

[0002] As the globalization process accelerates, the demand for multilingual communication is growing. In cross-language communication, language translation devices have become an important tool. When language translation devices translate languages, the accuracy of speech recognition is crucial.

[0003] Traditional speech recognition technology is usually based on matching the acoustic features of speech signals with language models. This method has high recognition accuracy under ideal conditions, but the recognition effect will be significantly reduced in complex environments (for example, when there is background noise or the speech signal is interfered). Summary of the invention

[0004] The present application provides a speech recognition method and system for a double-sided transparent screen translation machine to solve the problems raised by the above background technology.

[0005] In a first aspect, the present application provides a speech recognition method for a double-sided transparent screen translation machine, comprising: Acquire voice information of a translation machine user and a first lip change video corresponding to the voice information; Analyze the voice information based on an accent analysis model to obtain the accent category of the translator user; Matching target model parameters in a database based on the accent category, and updating model parameters of a preset speech recognition model based on the target model parameters to obtain an updated speech recognition model, and inputting the speech information into the updated speech recognition model to obtain an initial speech recognition text; Obtaining a second lip change video corresponding to the initial speech recognition text; Determining whether the initial speech recognition text is qualified based on the first lip change video and the second lip change video; If qualified, determining the initial speech recognition text as the target speech recognition text; If it fails, the initial speech recognition text is optimized based on the lip comparison video and the lip change video to obtain the target speech recognition text.

[0006] In a possible implementation, the obtaining a second lip change video corresponding to the initial speech recognition text includes: Acquiring facial image information of the user, and updating a preset virtual robot based on the facial image information and the accent category to obtain an updated virtual robot; The updated virtual robot is controlled to read the initial voice recognition text with the voice features corresponding to the voice information, and in the process of the updated virtual robot reading the initial voice recognition text, the lip image information of the updated virtual robot is obtained to obtain the second lip change video.

[0007] In a possible implementation, controlling the updated virtual robot to read the initial voice recognition text with the voice feature corresponding to the voice information includes: The voice information is played to the update virtual robot so that the update virtual robot learns the voice features of the voice information; the update virtual robot reads the initial voice recognition text based on the learned voice features of the voice information.

[0008] In a possible implementation, judging whether the initial speech recognition text is qualified based on the first lip change video and the second lip change video includes: Sequentially segmenting the first lip change video based on a preset video segmentation method to obtain a first video clip sequence, and segmenting the second lip change video based on the video segmentation method to obtain a second video clip sequence; Combining the first video segment and the second video segment with the same sequence to obtain a plurality of video segment combinations; For each of the video clip combinations, determining whether the lip shape of the first video clip matches the lip shape of the second video clip in the video clip combination; If the lip shape of the first video segment in any of the video segment combinations does not match the lip shape of the second video segment, determining that the initial speech recognition text is unqualified; If the lip shapes of the first video segment and the second video segment in each of the video segment combinations match, the initial speech recognition text is determined to be qualified.

[0009] In a possible implementation, the determining whether the lip shape of the first video segment in the video segment combination matches the lip shape of the second video segment includes: Performing frame processing on the first video segment based on a preset video frame segmentation method to obtain a first video frame sequence, and performing frame processing on the second video segment based on the video frame segmentation method to obtain a second video frame sequence; Obtaining the similarity between the lip shape in the first video frame and the lip shape in the second video frame of the same sequence; Comparing each of the similarities with a preset similarity respectively, and generating a target number based on the comparison result; the target number is the number of similarities less than the preset similarity; Determining whether the target quantity is greater than a preset quantity; If it is greater than, it is determined that the lip shape of the first video segment and the lip shape of the second video segment in the video segment combination do not match; If not, it is determined that the lip shape of the first video segment in the video segment combination matches the lip shape of the second video segment.

[0010] In a possible implementation, the optimizing the initial speech recognition text based on the lip comparison video and the lip change video to obtain the target speech recognition text includes: Determine a target video segment combination in each of the video segment combinations, and determine the first video segment in the target video segment combination as the target first video segment; wherein the lip shape of the first video segment in the target video segment combination does not match the lip shape of the second video segment; matching lip shape data text in a database based on the accent category; For each of the target first video segments, a lip reading text sequence corresponding to the target first video segment is generated based on the lip shape data text, and the lip reading text sequence is used to replace the text sequence corresponding to the speech segment corresponding to the target first video segment in the initial speech recognition text.

[0011] In a possible implementation, generating a lip reading text sequence corresponding to the target first video clip based on the lip shape data text includes: Sequentially extracting target lip shapes corresponding to each lip change in the target first video segment to obtain a target lip shape sequence; Based on the lip shape data text, lip reading characters are matched in sequence for each target lip shape in the target lip shape sequence to obtain the lip reading character sequence.

[0012] In a second aspect, the present application provides a speech recognition system for a double-sided transparent screen translation machine, comprising: A first acquisition module, used to acquire voice information of a translation machine user and a first lip change video corresponding to the voice information; An analysis module, configured to analyze the speech information based on an accent analysis model to obtain an accent category of the translation machine user; A speech recognition module, used for matching target model parameters in a database based on the accent category, updating model parameters of a preset speech recognition model based on the target model parameters to obtain an updated speech recognition model, and inputting the speech information into the updated speech recognition model to obtain an initial speech recognition text; A second acquisition module, used to acquire a second lip change video corresponding to the initial speech recognition text; A judging module, configured to judge whether the initial speech recognition text is qualified based on the first lip change video and the second lip change video; A determination module, configured to determine the initial speech recognition text as a target speech recognition text if the initial speech recognition text is qualified; The text optimization module is used to optimize the initial speech recognition text based on the lip comparison video and the lip change video to obtain a target speech recognition text if the initial speech recognition text is unqualified.

[0013] The present application provides a speech recognition method and system for a double-sided transparent screen translation machine, the method comprising: obtaining speech information of a translation machine user and a first lip change video corresponding to the speech information; analyzing the speech information based on an accent analysis model to obtain an accent category of the translation machine user; matching target model parameters in a database based on the accent category, and updating model parameters of a preset speech recognition model based on the target model parameters to obtain an updated speech recognition model, and inputting the speech information into the updated speech recognition model to obtain an initial speech recognition text; obtaining a second lip change video corresponding to the initial speech recognition text; judging whether the initial speech recognition text is qualified based on the first lip change video and the second lip change video; if qualified, determining that the initial speech recognition text is a target speech recognition text; if unqualified, optimizing the initial speech recognition text based on the lip comparison video and the lip change video to obtain a target speech recognition text. On the one hand, the method realizes the synchronous collection of voice and video by simultaneously collecting the voice information of the translation machine user and the first lip change video corresponding to the voice information. The first lip change video can provide additional visual information support for the voice recognition method, especially when the voice signal is affected by noise or interference, the lip shape features can be used for auxiliary verification and optimization, which helps to improve the accuracy of voice recognition. On the other hand, the present invention uses a pre-trained accent analysis model to analyze the voice information and accurately identify the user's accent category, which effectively solves the problem of insufficient adaptability of traditional voice recognition methods when dealing with diverse accents. On the other hand, by obtaining the first lip change video and the second lip change video corresponding to the initial voice recognition text, and comparing and analyzing the two segment by segment, the initial voice recognition text is accurately verified and optimized. Compared with the traditional voice recognition method that only relies on acoustic features for judgment, the video verification mechanism can effectively discover potential errors in the text, thereby improving the reliability of the recognition results. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0015] Figure 1 A schematic flow chart of a speech recognition method for a double-sided transparent screen translator provided in an embodiment of the present application; Figure 2 A schematic block diagram of the structure of the speech recognition system of the double-sided transparent screen translator provided in the embodiment of the present application; Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.

[0018] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0019] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0020] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0021] See also Figure 1 , Figure 1 A flow chart of a speech recognition method for a double-sided transparent screen translator provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the speech recognition method of the double-sided transparent screen translation machine provided in the embodiment of the present application includes steps S1 to S7.

[0022] Step S1: Acquire voice information of a translation machine user and a first lip change video corresponding to the voice information.

[0023] Specifically, the double-sided transparent screen translator is provided with a voice acquisition device and an image acquisition device. When the double-sided transparent screen translator receives a translation request instruction from the translator user, the translator user is prompted to align his lips with the image acquisition device, so as to obtain the voice information and the first lip change video of the translator user at the same time while the translator user is speaking.

[0024] It can be understood that step S1 synchronously acquires voice information and the first lip change video through the voice acquisition device and the image acquisition device, which provides accurate and rich basic data for the subsequent verification of the initial voice recognition text and improves the applicability of the system in complex environments.

[0025] Step S2: Analyze the voice information based on the accent analysis model to obtain the accent category of the translation machine user.

[0026] Wherein, the accent analysis model is a pre-trained neural network model.

[0027] It can be understood that step S2 utilizes the pre-trained accent analysis model to accurately classify the voice information, which can quickly determine the user's accent category, thereby effectively solving the problem of poor adaptability of traditional voice recognition systems.

[0028] Step S3, matching target model parameters in a database based on the accent category, and updating model parameters of a preset speech recognition model based on the target model parameters to obtain an updated speech recognition model, and inputting the speech information into the updated speech recognition model to obtain an initial speech recognition text.

[0029] Specifically, a matching relationship between various accent categories and their corresponding model parameters is set in the database, and a search is performed in the database based on the accent category to obtain the target model parameters corresponding to the accent category.

[0030] It can be understood that step S3 updates the model parameters of the preset speech recognition model based on the target model parameters to obtain an updated speech recognition model, which helps to improve the accuracy of the speech recognition result.

[0031] Step S4: Obtain a second lip change video corresponding to the initial speech recognition text.

[0032] Specifically, step S4 includes the following steps: Acquire facial image information of the user, and update a preset virtual robot based on the facial image information and the accent category to obtain an updated virtual robot; it can be understood that the facial feature information of the updated virtual robot is consistent with the facial image information, and the accent category of the updated virtual robot is consistent with the accent category; Control the updated virtual robot to read the initial voice recognition text with the voice features corresponding to the voice information, and in the process of the updated virtual robot reading the initial voice recognition text, obtain the lip image information of the updated virtual robot to obtain the second lip change video; specifically, play the voice information to the updated virtual robot so that the updated virtual robot learns the voice features of the voice information; the updated virtual robot reads the initial voice recognition text based on the learned voice features of the voice information; in the process of the updated virtual robot reading the initial voice recognition text, obtain the lip image information of the updated virtual robot through a preset camera device to obtain the second lip change video.

[0033] It can be understood that the second lip change video obtained in step S4 provides a reliable data basis for the subsequent determination of the eligibility of the initial speech recognition text and the optimization of the initial speech recognition text.

[0034] Step S5: Determine whether the initial speech recognition text is qualified based on the first lip change video and the second lip change video.

[0035] Specifically, step S5 includes the following steps: Sequentially segmenting the first lip change video based on a preset video segmentation method to obtain a first video clip sequence, and segmenting the second lip change video based on the video segmentation method to obtain a second video clip sequence; Combining the first video segment and the second video segment with the same sequence to obtain a plurality of video segment combinations; For each of the video clip combinations, determining whether the lip shape of the first video clip matches the lip shape of the second video clip in the video clip combination; If the lip shape of the first video segment in any of the video segment combinations does not match the lip shape of the second video segment, determining that the initial speech recognition text is unqualified; If the lip shapes of the first video segment and the second video segment in each of the video segment combinations match, the initial speech recognition text is determined to be qualified.

[0036] It can be understood that the above-mentioned method of judging whether the initial speech recognition text is qualified based on the first lip change video and the second lip change video, firstly, realizes the fine-grained verification of the initial speech recognition text through the method of video segmentation and fragment matching, thereby avoiding the problem of subtle differences that may be ignored by traditional speech recognition methods, and helps to improve the judgment accuracy; secondly, the lip shape feature matching in the lip change video effectively makes up for the shortcomings of relying solely on speech signal verification, especially when the speech signal is interfered by noise or the user's pronunciation is unclear, the introduction of video information helps to significantly improve the accuracy of speech recognition results; in addition, the present invention adopts fragment-level lip shape matching judgment logic, which can quickly discover local errors in the initial speech recognition text, thereby providing accurate positioning basis for subsequent optimization.

[0037] The step of determining whether the lip shape of the first video segment in the video segment combination matches the lip shape of the second video segment includes the following steps: Performing frame processing on the first video segment based on a preset video frame segmentation method to obtain a first video frame sequence, and performing frame processing on the second video segment based on the video frame segmentation method to obtain a second video frame sequence; Obtaining the similarity of the lip shape in the first video frame and the lip shape in the second video frame of the same sequence; specifically, inputting the lip shape in the first video frame into a preset lip shape feature extraction model to obtain a first feature vector, and inputting the lip shape in the second video frame into a preset lip shape feature extraction model to obtain a second feature vector, and determining the cosine value of the first feature vector and the second feature vector as the similarity; Comparing each of the similarities with a preset similarity respectively, and generating a target number based on the comparison result; the target number is the number of similarities less than the preset similarity; Determining whether the target quantity is greater than a preset quantity; If it is greater than, it is determined that the lip shape of the first video segment and the lip shape of the second video segment in the video segment combination do not match; If not, it is determined that the lip shape of the first video segment in the video segment combination matches the lip shape of the second video segment.

[0038] It can be understood that the above-mentioned method for determining whether the lip shape of the first video clip in the video clip combination matches the lip shape of the second video clip is designed by comparing the similarity of each frame with the preset similarity and counting the number of frames below the preset similarity. A judgment mechanism based on the target number is designed. This judgment mechanism helps to avoid the situation where a single frame error leads to an overall misjudgment, and effectively improves the robustness and anti-interference ability of the judgment logic. In particular, when some frames in the video may have errors due to image acquisition angle or resolution problems, the system can still maintain the accuracy of the overall judgment.

[0039] Step S6: If qualified, determine the initial speech recognition text as the target speech recognition text.

[0040] Step S7: If unqualified, the initial speech recognition text is optimized based on the lip comparison video and the lip change video to obtain a target speech recognition text.

[0041] Specifically, step S7 includes the following steps: Determine a target video segment combination in each of the video segment combinations, and determine the first video segment in the target video segment combination as the target first video segment; wherein the lip shape of the first video segment in the target video segment combination does not match the lip shape of the second video segment; Matching lip shape data text in a database based on the accent category; wherein the database is provided with lip shape data text corresponding to various accent categories, and the lip shape data text includes a matching relationship between various lip shapes and their corresponding lip reading texts; For each of the target first video clips, a lip reading text sequence corresponding to the target first video clip is generated based on the lip shape data text, and the lip reading text sequence is used to replace the text sequence corresponding to the voice clip corresponding to the target first video clip in the initial speech recognition text; it can be understood that the voice clip corresponding to the target first video clip refers to the voice clip corresponding to the target first video clip in the voice information.

[0042] It can be understood that step S7 achieves efficient optimization of the recognition text by generating a lip reading text sequence corresponding to the target first video segment based on the lip shape data text for each of the target first video segments, and replacing the text sequence corresponding to the speech segment corresponding to the target first video segment in the initial speech recognition text with the lip reading text sequence, thereby ensuring the consistency and accuracy of the target speech recognition text with the actual pronunciation of the user, thereby helping to improve the quality of the translation result.

[0043] The step of generating a lip reading text sequence corresponding to the target first video clip based on the lip shape data text comprises the following steps: Sequentially extracting target lip shapes corresponding to each lip change in the target first video segment to obtain a target lip shape sequence; Based on the lip shape data text, lip reading text is matched in sequence for each target lip shape in the target lip shape sequence to obtain the lip reading text sequence; specifically, for each target lip shape, the lip reading text corresponding to the lip shape with the highest similarity to the target lip shape in the lip shape data text is determined as the lip reading text of the target lip shape.

[0044] It can be understood that the above-mentioned method of generating the lip reading text sequence corresponding to the target first video clip based on the lip shape data text helps to effectively avoid text deviation caused by factors such as accent and noise in traditional speech recognition methods.

[0045] The method provided in this embodiment, on the one hand, realizes the synchronous collection of voice and video by simultaneously collecting the voice information of the translation machine user and the first lip change video corresponding to the voice information. The first lip change video can provide additional visual information support for the voice recognition method, especially when the voice signal is affected by noise or interference, the lip shape features can be used for auxiliary verification and optimization, which helps to improve the accuracy of voice recognition. On the other hand, the present invention uses a pre-trained accent analysis model to analyze the voice information and accurately identify the user's accent category, which effectively solves the problem of insufficient adaptability of traditional voice recognition methods when dealing with diverse accents. On the other hand, by obtaining the first lip change video and the second lip change video corresponding to the initial voice recognition text, and comparing and analyzing the two segment by segment, accurate verification and optimization of the initial voice recognition text is achieved. Compared with the traditional voice recognition method that only relies on acoustic features for judgment, the video verification mechanism can effectively discover potential errors in the text, thereby improving the reliability of the recognition results.

[0046] See also Figure 2 , Figure 2 The structure schematic block diagram of the speech recognition system 100 of the double-sided transparent screen translation machine provided in the embodiment of the present application is as follows: Figure 2 As shown, the speech recognition system 100 of the double-sided transparent screen translator provided in the embodiment of the present application includes: The first acquisition module 110 is used to acquire voice information of a translation machine user and a first lip change video corresponding to the voice information.

[0047] The analysis module 120 is used to analyze the speech information based on the accent analysis model to obtain the accent category of the translation machine user.

[0048] The speech recognition module 130 is used to match the target model parameters in the database based on the accent category, and update the model parameters of the preset speech recognition model based on the target model parameters to obtain an updated speech recognition model, and input the speech information into the updated speech recognition model to obtain an initial speech recognition text.

[0049] The second acquisition module 140 is used to acquire a second lip change video corresponding to the initial speech recognition text.

[0050] The judgment module 150 is used to judge whether the initial speech recognition text is qualified based on the first lip change video and the second lip change video.

[0051] The determination module 160 is configured to determine that the initial speech recognition text is a target speech recognition text if the initial speech recognition text is qualified.

[0052] The text optimization module 170 is used to optimize the initial speech recognition text based on the lip comparison video and the lip change video to obtain a target speech recognition text if the initial speech recognition text is unqualified.

[0053] It should be noted that technicians in the relevant technical field can clearly understand that for the convenience and conciseness of description, the specific working process of the system and each module described above can refer to the process in the aforementioned double-sided transparent screen translation machine speech recognition method embodiment, and will not be repeated here.

[0054] The speech recognition system 100 of the double-sided transparent screen translator provided in the above embodiment can be implemented in the form of a computer program. The computer program can be used in Figure 3 The system is run on the terminal device 200 shown.

[0055] See also Figure 3 , Figure 3 The present invention provides a schematic block diagram of the structure of a terminal device 200 according to an embodiment of the present application. The terminal device 200 includes a processor 201 and a memory 202. The processor 201 and the memory 202 are connected via a device bus 203, wherein the memory 202 may include a non-volatile storage medium and an internal memory.

[0056] The non-volatile storage medium can store a computer program. The computer program includes program instructions, and when the program instructions are executed by the processor 201, the processor 201 can execute any of the above-mentioned speech recognition methods of the double-sided transparent screen translator.

[0057] The processor 201 is used to provide computing and control capabilities to support the operation of the entire terminal device 200.

[0058] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor 201, the processor 201 can execute any of the above-mentioned speech recognition methods of the double-sided transparent screen translator.

[0059] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the terminal device 200 involved in the scheme of the present application. The specific terminal device 200 may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0060] It should be understood that the processor 201 may be a central processing unit (CPU), and the processor 201 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0061] In some embodiments, the processor 201 is used to run a computer program stored in the memory to implement the following steps: Acquire voice information of a translation machine user and a first lip change video corresponding to the voice information; Analyze the voice information based on an accent analysis model to obtain the accent category of the translator user; Matching target model parameters in a database based on the accent category, and updating model parameters of a preset speech recognition model based on the target model parameters to obtain an updated speech recognition model, and inputting the speech information into the updated speech recognition model to obtain an initial speech recognition text; Obtaining a second lip change video corresponding to the initial speech recognition text; Determining whether the initial speech recognition text is qualified based on the first lip change video and the second lip change video; If qualified, determining the initial speech recognition text as the target speech recognition text; If it fails, the initial speech recognition text is optimized based on the lip comparison video and the lip change video to obtain the target speech recognition text.

[0062] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working process of the terminal device 200 described above can refer to the process of the speech recognition method of the aforementioned double-sided transparent screen translator, and will not be repeated here.

[0063] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by one or more processors, the one or more processors implement the speech recognition method of the double-sided transparent screen translator provided in the embodiment of the present application.

[0064] The computer-readable storage medium may be an internal storage unit of the terminal device 200 in the aforementioned embodiment, such as a hard disk or memory of the terminal device 200. The computer-readable storage medium may also be an external storage device of the terminal device 200, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped with the terminal device 200.

[0065] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A speech recognition method for a double-sided transparent screen translator, characterized in that: include: Acquire voice information of a translation machine user and a first lip change video corresponding to the voice information; Analyze the voice information based on an accent analysis model to obtain the accent category of the translator user; Matching target model parameters in a database based on the accent category, and updating model parameters of a preset speech recognition model based on the target model parameters to obtain an updated speech recognition model, and inputting the speech information into the updated speech recognition model to obtain an initial speech recognition text; Obtaining a second lip change video corresponding to the initial speech recognition text; Determining whether the initial speech recognition text is qualified based on the first lip change video and the second lip change video; If qualified, determining the initial speech recognition text as the target speech recognition text; If it fails, the initial speech recognition text is optimized based on the lip comparison video and the lip change video to obtain the target speech recognition text.

2. The speech recognition method of the double-sided transparent screen translator according to claim 1 is characterized in that: The obtaining of a second lip change video corresponding to the initial speech recognition text includes: Acquiring facial image information of the user, and updating a preset virtual robot based on the facial image information and the accent category to obtain an updated virtual robot; The updated virtual robot is controlled to read the initial voice recognition text with the voice features corresponding to the voice information, and in the process of the updated virtual robot reading the initial voice recognition text, the lip image information of the updated virtual robot is obtained to obtain the second lip change video.

3. The speech recognition method of the double-sided transparent screen translator according to claim 2 is characterized in that: The controlling the updated virtual robot to read the initial voice recognition text with the voice feature corresponding to the voice information includes: The voice information is played to the update virtual robot so that the update virtual robot learns the voice features of the voice information; the update virtual robot reads the initial voice recognition text based on the learned voice features of the voice information.

4. The speech recognition method of the double-sided transparent screen translator according to claim 1 is characterized in that: The determining whether the initial speech recognition text is qualified based on the first lip change video and the second lip change video includes: Sequentially segmenting the first lip change video based on a preset video segmentation method to obtain a first video clip sequence, and segmenting the second lip change video based on the video segmentation method to obtain a second video clip sequence; Combining the first video segment and the second video segment with the same sequence to obtain a plurality of video segment combinations; For each of the video clip combinations, determining whether the lip shape of the first video clip matches the lip shape of the second video clip in the video clip combination; If the lip shape of the first video segment in any of the video segment combinations does not match the lip shape of the second video segment, determining that the initial speech recognition text is unqualified; If the lip shapes of the first video segment and the second video segment in each of the video segment combinations match, the initial speech recognition text is determined to be qualified.

5. The speech recognition method of the double-sided transparent screen translator according to claim 4 is characterized in that: The determining whether the lip shape of the first video segment in the video segment combination matches the lip shape of the second video segment includes: Performing frame processing on the first video segment based on a preset video frame segmentation method to obtain a first video frame sequence, and performing frame processing on the second video segment based on the video frame segmentation method to obtain a second video frame sequence; Obtaining the similarity between the lip shape in the first video frame and the lip shape in the second video frame of the same sequence; Comparing each of the similarities with a preset similarity respectively, and generating a target number based on the comparison result; the target number is the number of similarities less than the preset similarity; Determining whether the target quantity is greater than a preset quantity; If it is greater than, determining that the lip shape of the first video segment in the video segment combination does not match the lip shape of the second video segment; If not, it is determined that the lip shape of the first video segment in the video segment combination matches the lip shape of the second video segment.

6. The speech recognition method of the double-sided transparent screen translator according to claim 4 is characterized in that: The step of optimizing the initial speech recognition text based on the lip comparison video and the lip change video to obtain a target speech recognition text includes: Determine a target video segment combination in each of the video segment combinations, and determine the first video segment in the target video segment combination as the target first video segment; wherein the lip shape of the first video segment in the target video segment combination does not match the lip shape of the second video segment; matching lip shape data text in a database based on the accent category; For each of the target first video segments, a lip reading text sequence corresponding to the target first video segment is generated based on the lip shape data text, and the lip reading text sequence is used to replace the text sequence corresponding to the speech segment corresponding to the target first video segment in the initial speech recognition text.

7. The speech recognition method of the double-sided transparent screen translator according to claim 6 is characterized in that: The step of generating a lip reading text sequence corresponding to the target first video clip based on the lip shape data text includes: Sequentially extracting target lip shapes corresponding to each lip change in the target first video segment to obtain a target lip shape sequence; Based on the lip shape data text, lip reading characters are matched in sequence for each target lip shape in the target lip shape sequence to obtain the lip reading character sequence.

8. A speech recognition system for a double-sided transparent screen translator, characterized in that: include: A first acquisition module, used to acquire voice information of a translation machine user and a first lip change video corresponding to the voice information; An analysis module, configured to analyze the speech information based on an accent analysis model to obtain an accent category of the translation machine user; A speech recognition module, used for matching target model parameters in a database based on the accent category, updating model parameters of a preset speech recognition model based on the target model parameters to obtain an updated speech recognition model, and inputting the speech information into the updated speech recognition model to obtain an initial speech recognition text; A second acquisition module, used to acquire a second lip change video corresponding to the initial speech recognition text; A judging module, configured to judge whether the initial speech recognition text is qualified based on the first lip change video and the second lip change video; A determination module, configured to determine the initial speech recognition text as a target speech recognition text if the initial speech recognition text is qualified; The text optimization module is used to optimize the initial speech recognition text based on the lip comparison video and the lip change video to obtain a target speech recognition text if the initial speech recognition text is unqualified.

Images