Model training, natural language translation method and device, equipment and storage medium

Abstract

The present disclosure relates to a model training method, a natural language translation method, a device, equipment and a storage medium. The present disclosure obtains a standard first sign language text corresponding to a natural language sample text and at least one non-standard second sign language text, and outputs a predicted sign language text corresponding to the natural language sample text through a translation model to be trained. According to the predicted sign language text and the first sign language text, a first loss value of the translation model is calculated, and according to the predicted sign language text and at least one second sign language text, at least one second loss value of the translation model is calculated. Since the smaller the first loss value is, the better, and the larger the second loss value is, the better, according to the first loss value and the at least one second loss value, the model parameters of the translation model can be accurately controlled, so that the trained translation model is more accurate. Thus, the trained translation model can obtain accurate sign language text when translating the natural language text to be translated.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and in particular to a model training, natural language translation method, apparatus, device, and storage medium. Background Technology

[0002] In the world of hearing people, information is usually conveyed through natural language. However, for hearing-impaired people, sign language is their primary language. Therefore, it is necessary to translate natural language into sign language.

[0003] However, the inventors of this application have discovered that since sign language is an independent language, its unique vocabulary and grammar are very different from those of natural language. Therefore, how to accurately translate natural language into sign language has become an urgent problem to be solved. Summary of the Invention

[0004] To address, or at least partially address, the aforementioned technical problems, this disclosure provides a model training, natural language translation method, apparatus, device, and storage medium, enabling the trained translation model to produce accurate sign language text when translating natural language text to be translated.

[0005] In a first aspect, embodiments of this disclosure provide a model training method, including:

[0006] Obtain a first sign language text and at least one second sign language text corresponding to a natural language sample text, wherein the accuracy of the second sign language text is lower than the accuracy of the first sign language text;

[0007] The natural language sample text is input into the translation model to be trained, and the translation model outputs the predicted sign language text corresponding to the natural language sample text.

[0008] Calculate the first loss value of the translation model based on the predicted sign language text and the first sign language text;

[0009] Based on the predicted sign language text and the at least one second sign language text, calculate at least one second loss value for the translation model;

[0010] The translation model is trained based on the first loss value and the at least one second loss value.

[0011] Secondly, embodiments of this disclosure provide a natural language translation method, including:

[0012] Obtain the target natural language text to be translated;

[0013] The target natural language text is input into a pre-trained translation model, and the translation model outputs the target sign language text corresponding to the target natural language text. The translation model is obtained according to the model training method described above.

[0014] Thirdly, embodiments of this disclosure provide a model training apparatus, comprising:

[0015] The acquisition module is used to acquire a first sign language text and at least one second sign language text corresponding to a natural language sample text, wherein the accuracy of the second sign language text is lower than that of the first sign language text.

[0016] The input module is used to input the natural language sample text into the translation model to be trained, and output the predicted sign language text corresponding to the natural language sample text through the translation model;

[0017] The calculation module is configured to calculate a first loss value of the translation model based on the predicted sign language text and the first sign language text; and to calculate at least one second loss value of the translation model based on the predicted sign language text and the at least one second sign language text.

[0018] The training module is used to train the translation model based on the first loss value and the at least one second loss value.

[0019] Fourthly, embodiments of this disclosure provide a natural language translation apparatus, comprising:

[0020] The acquisition module is used to acquire the target natural language text to be translated;

[0021] The input module is used to input the target natural language text into a pre-trained translation model, and output the target sign language text corresponding to the target natural language text through the translation model. The translation model is obtained according to the model training method described above.

[0022] Fifthly, embodiments of this disclosure provide an electronic device, including:

[0023] Memory;

[0024] Processor; and

[0025] Computer programs;

[0026] The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first or second aspect.

[0027] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described in the first or second aspect.

[0028] The model training, natural language translation method, apparatus, device, and storage medium provided in this disclosure acquire a standard first sign language text and at least one non-standard second sign language text corresponding to a natural language sample text, and output a predicted sign language text corresponding to the natural language sample text through a translation model to be trained. Further, based on the predicted sign language text and the first sign language text, a first loss value of the translation model is calculated, and based on the predicted sign language text and the at least one second sign language text, at least one second loss value of the translation model is calculated. Since a smaller first loss value is better and a larger second loss value is better, the model parameters of the translation model can be precisely controlled based on the first loss value and the at least one second loss value, making the trained translation model more accurate. Therefore, the trained translation model can obtain accurate sign language text when translating the natural language text to be translated. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0030] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart of the model training method provided in this embodiment of the disclosure;

[0032] Figure 2 A schematic diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0033] Figure 3 A flowchart of a natural language translation method provided in another embodiment of this disclosure;

[0034] Figure 4 A schematic diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0035] Figure 5 A flowchart of a natural language translation method provided in another embodiment of this disclosure;

[0036] Figure 6A flowchart of a natural language translation method provided in another embodiment of this disclosure;

[0037] Figure 7 A flowchart of a natural language translation method provided in another embodiment of this disclosure;

[0038] Figure 8 A flowchart of a natural language translation method provided in another embodiment of this disclosure;

[0039] Figure 9 A flowchart of a natural language translation method provided in another embodiment of this disclosure;

[0040] Figure 10 A flowchart of a natural language translation method provided in another embodiment of this disclosure;

[0041] Figure 11 A schematic diagram illustrating intent classification provided for embodiments of this disclosure;

[0042] Figure 12 A flowchart of a natural language translation method provided in another embodiment of this disclosure;

[0043] Figure 13 A flowchart of a natural language translation method provided in another embodiment of this disclosure;

[0044] Figure 14 A schematic diagram of the structure of the translation model provided in this embodiment of the disclosure;

[0045] Figure 15 A schematic diagram of a user interface provided for an embodiment of this disclosure;

[0046] Figure 16 This is a schematic diagram of the structure of the model training device provided in the embodiments of this disclosure;

[0047] Figure 17 This is a schematic diagram of the structure of the natural language translation device provided in the embodiments of this disclosure;

[0048] Figure 18 A schematic diagram of the structure of an electronic device embodiment provided in this disclosure. Detailed Implementation

[0049] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0050] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0051] Typically, since sign language is a distinct language with vocabulary and grammar significantly different from those of natural language, accurately translating natural language into sign language has become a pressing issue. To address this problem, this disclosure provides a model training and natural language translation method, which will be described below with reference to specific embodiments.

[0052] Figure 1 This is a flowchart illustrating a model training method provided in an embodiment of this disclosure. The method can be executed by a model training device, which can be implemented in software and / or hardware. This device can be configured in an electronic device, such as a server or terminal, where the terminal specifically includes a mobile phone, computer, or tablet computer. Furthermore, the model training method described in this embodiment is applicable to... Figure 2 The application scenarios shown are as follows. Figure 2 As shown, this application scenario includes a terminal 21 and a server 22. The server 22 can train a translation model to be trained, which can translate natural language text into sign language text. The following section combines... Figure 2 This method will be described in detail, such as Figure 1 As shown, the specific steps of this method are as follows:

[0053] S101. Obtain a first sign language text and at least one second sign language text corresponding to the natural language sample text, wherein the accuracy of the second sign language text is lower than the accuracy of the first sign language text.

[0054] For example, server 22 can obtain natural language text and the corresponding standard sign language text from terminal 21. Server 22 can train the translation model to be trained based on the natural language text and the corresponding standard sign language text. That is, the natural language text and the corresponding standard sign language text are training samples. Therefore, in this embodiment, the natural language text used to train the translation model can be denoted as the natural language sample text, and the corresponding standard sign language text can be denoted as the first sign language text. Alternatively, server 22 can also obtain the natural language sample text from terminal 21 and generate the corresponding standard sign language text, i.e., the first sign language text. In addition, server 22 can also obtain at least one second sign language text corresponding to the natural language sample text, where the accuracy of the second sign language text is lower than that of the first sign language text. That is, the first sign language text is the standard sign language text corresponding to the natural language sample text, while the second sign language text is a non-standard sign language text corresponding to the natural language sample text.

[0055] Optionally, the first sign language text includes multiple sign language words; the second sign language text is obtained by adjusting the order of at least some of the multiple sign language words; or the second sign language text is obtained by replacing at least some of the multiple sign language words; or the second sign language text is a historical sign language text output by the translation model during historical training.

[0056] For example, the natural language sample text is "I believe the athletes will bring you a wonderful basketball feast," and the corresponding standard sign language text is "believe / athlete / person / bring / J / color / basketball / match / will." Here, "believe," "athlete," "person," "bring," "come," "J," "color," "basketball," "match," and "will" are all sign language words. In other words, the standard sign language text includes multiple sign language words.

[0057] In one feasible implementation, server 22 can adjust the order of at least some sign language words in the standard sign language text to obtain a second sign language text. For example, adjusting the order of "believe," "sports," "member," "bring," "come," "J," "color," and "basketball" yields the second sign language text "sports / believe / bring / member / come / basketball / color / J / match / will." It is understood that there are multiple ways to adjust the order of at least some sign language words; this is merely illustrative and not specifically limited. Therefore, the number of second sign language texts is not limited and can be one or more.

[0058] In another feasible implementation, server 22 can replace at least some of the sign language words in the standard sign language text to obtain a second sign language text. For example, in the phrase "believe / athlete / bring / J / color / basketball / match / meeting", "J / color" can be replaced with "wonderful" to obtain the second sign language text "believe / athlete / bring / wonderful / basketball / match / meeting".

[0059] In another feasible implementation, the second sign language text can also be the historical sign language text output by the translation model during its historical training process. For example, the second sign language text could be the historical sign language text output by the translation model during its first training process. It is understood that the accuracy of the translation model's output results in the first few training processes is not accurate enough. Therefore, the historical sign language text output by the model in each of the first few training processes can be used as a second sign language text.

[0060] S102. Input the natural language sample text into the translation model to be trained, and output the predicted sign language text corresponding to the natural language sample text through the translation model.

[0061] For example, the translation model to be trained may be a model that has undergone several training sessions but is not yet fully trained, meaning that the model parameters have not yet stabilized. In this case, the natural language sample text can be input into the translation model to be trained, so that the translation model outputs the predicted sign language text corresponding to the natural language sample text. In other words, the predicted sign language text is the sign language text corresponding to the natural language sample text predicted by the translation model.

[0062] S103. Calculate the first loss value of the translation model based on the predicted sign language text and the first sign language text.

[0063] For example, the first loss value of the translation model is obtained by calculating the difference between the predicted sign language text and the first sign language text.

[0064] S104. Calculate at least one second loss value for the translation model based on the predicted sign language text and the at least one second sign language text.

[0065] For example, a second loss value for the translation model can be obtained by calculating the difference between the predicted sign language text and any second sign language text. When there are n second sign language texts, there are also n second loss values. Here, n is an integer greater than or equal to 1.

[0066] S105. The translation model is trained based on the first loss value and the at least one second loss value.

[0067] Since the first loss value represents the difference between the predicted sign language text and the standard sign language text, a smaller first loss value is better during the training of the translation model. Conversely, since the second loss value represents the difference between the predicted sign language text and the non-standard sign language text, a larger second loss value is better during the training of the translation model. Therefore, based on the calculated first loss value and at least one second loss value, by controlling the decrease of the first loss value while simultaneously controlling the increase of each second loss value, the adjustment direction or trend of the model parameters of the translation model can be determined, thereby obtaining new model parameters, which achieves one iterative training of the translation model. It is understandable that as the natural language sample text continuously changes, the translation model can be continuously iterated and trained. When the number of iterations reaches a preset number, or when the model parameters tend to stabilize or converge, it can be determined that the translation model has been successfully trained.

[0068] This embodiment of the disclosure obtains a standard first sign language text and at least one non-standard second sign language text corresponding to a natural language sample text, and outputs a predicted sign language text corresponding to the natural language sample text through a translation model to be trained. Further, based on the predicted sign language text and the first sign language text, a first loss value of the translation model is calculated, and based on the predicted sign language text and the at least one second sign language text, at least one second loss value of the translation model is calculated. Since a smaller first loss value is better and a larger second loss value is better, the model parameters of the translation model can be precisely controlled based on the first loss value and the at least one second loss value, making the trained translation model more accurate. Therefore, the trained translation model can obtain accurate sign language text when translating the natural language text to be translated.

[0069] Figure 3 A flowchart of a natural language translation method provided in another embodiment of this disclosure. In this embodiment, the specific steps of the method are as follows:

[0070] S301. Obtain the target natural language text to be translated.

[0071] S302. Input the target natural language text into a pre-trained translation model, and output the target sign language text corresponding to the target natural language text through the translation model. The translation model is obtained according to the model training method described above.

[0072] The natural language translation method described in this embodiment can be applied to, for example, Figure 2The application scenarios shown, or, can be applied to, such as Figure 4 The application scenarios shown. For example, in situations such as Figure 2 In the application scenario shown, terminal 21 can be a terminal used by hearing-impaired individuals. Since server 22 can be equipped with the trained translation model described above (the specific training process will not be elaborated here), server 22 can use this translation model to translate the target natural language text into accurate target sign language text. Furthermore, server 22 can generate a video of a digital human performing sign language actions based on the target sign language text. Further, server 22 can send the target sign language text and the video, or the video itself, to terminal 21, enabling hearing-impaired individuals to access more online information more conveniently, just like hearing individuals.

[0073] In addition, it is understood that the training process of the translation model is not limited to being performed by server 22. For example, the translation model can also be trained by other servers and the trained translation model can be deployed on server 22.

[0074] In addition, people with hearing impairments and hearing people can also access services such as... Figure 4 The scenario illustrates communication. For example, terminal 23 could be a terminal for a hearing person, and terminal 21 could be a terminal for a hearing-impaired person. Terminal 23 can send audio information from the hearing person or video information including audio information to server 22. Further, server 22 can convert the audio information into text information using Automatic Speech Recognition (ASR) technology, which can be the target natural language text to be translated. Further, server 22 can translate the target natural language text into accurate target sign language text using a trained translation model, and generate a video of a digital human performing sign language actions based on the target sign language text. Further, server 22 can send the target sign language text and the video, or the video alone, to terminal 21, enabling the hearing-impaired person to understand the meaning expressed by the hearing person, thus providing a feasible foundation for smooth communication between the hearing-impaired and hearing people.

[0075] This embodiment acquires the target natural language text to be translated and inputs it into a pre-trained translation model. The translation model then outputs the corresponding target sign language text. Since the translation model is trained using a first loss value and at least one second loss value, these values ​​allow for precise control of the model's parameters, resulting in a more accurate translation model. Therefore, when the trained translation model translates the target natural language text, it can produce an accurate target sign language text.

[0076] Based on the above embodiments, input the target natural language text into a pre-trained translation model, and output the target sign language text corresponding to the target natural language text through the translation model, including: input the target natural language text into the pre-trained translation model, and output the target sign language text corresponding to the target natural language text and the target action identifier corresponding to the polysemous word in the target sign language text through the translation model.

[0077] For example, the server 22 inputs the target natural language text to be translated into a pre-trained translation model, and the translation model can translate the target natural language text into a target sign language text. Since the target sign language text includes at least one target word, and some target words may be polysemous words. For example, the target word is "press", which has the meaning of "in accordance with" and also the meaning of "press or click". Therefore, during the translation process, if the translation model determines that a certain target word translated is a polysemous word, the translation model can select a target action identifier that best conforms to the context from multiple action identifiers corresponding to the polysemous word according to the context of the polysemous word. Therefore, the translation model can also output the target action identifier corresponding to the polysemous word in the target sign language text while outputting the target sign language text, so that the digital human can make the most context-conforming actions when signing. That is to say, the same sign language word "press" may cause the digital human to make different sign language actions when it appears in different sign language texts.

[0078] In this embodiment, the translation model determines the target action identifier that best conforms to the context corresponding to the polysemous word in the target sign language text according to the context of the polysemous word, so that the translation model can also output the target action identifier corresponding to the polysemous word in the target sign language text while outputting the target sign language text, making the target action corresponding to the target action identifier accurately express the meaning of the polysemous word in the target sign language text. Thus, during the action synthesis process, the most accurate actions of each sign language word in the target sign language text can be synthesized, ensuring that the hearing-impaired person can understand the finally synthesized sign language video.

[0079] Figure 5 It is a flowchart of a natural language translation method provided by another embodiment of the present disclosure. In this embodiment, the specific steps of the method are as follows:

[0080] S501. Obtain the target natural language text to be translated.

[0081] S502. Input the target natural language text into a pre-trained translation model, and output the target sign language text corresponding to the target natural language text through the translation model.

[0082] In this embodiment of the disclosure, natural language text can be denoted as Text, and sign language text as Gloss. For example... Figure 6 As shown, Text can specifically be the target natural language text as described above. Server 22 inputs Text into a pre-trained translation model, and the translation model can output the target sign language text corresponding to the target natural language text. The target sign language text includes at least one target word. Figure 6 The Gloss shown can be the sign language text obtained by processing the target sign language text by the translation rewriting correction module and the polysemous action selection module.

[0083] S503, modify the target vocabulary in the target sign language text that does not belong to the sign language vocabulary into a sign language vocabulary or a combination of at least two sign language vocabulary.

[0084] Since the number of sign language words is much smaller than the number of words in natural language, the target sign language text output by the translation model, i.e., the translation result, may contain some non-sign language words. In this embodiment, the translation rewriting correction module in server 22 can correct these non-sign language words. Specifically, the translation rewriting correction module can correct the non-sign language words in the translation result based on an expert-aligned dictionary. For example, if the translation result contains the non-sign language word "map," and the expert-aligned dictionary includes the correspondence between "map" and "land / drawing," then the translation rewriting correction module can correct the non-sign language word "map" in the translation result to a combination of the sign language words "land" and "drawing," i.e., "map" can be corrected to "land / drawing." In other embodiments, a non-sign language word can also be replaced with a sign language word, and is not limited to a combination of at least two sign language words.

[0085] S504. If a sign language word or a combination of at least two sign language words includes a polysemous word, then a target action sign is selected from the multiple action signs corresponding to the polysemous word.

[0086] In this embodiment, if the translation rewriting and correction module replaces a non-sign language word in the translation result with a sign language word or a combination of at least two sign language words, and the server 22 determines that the replaced sign language word or the combination of at least two sign language words includes a polysemous word, then the server 22 can further select the target action identifier from the multiple action identifiers corresponding to the polysemous word through the polysemous word action selection module.

[0087] S505. Based on the context corresponding to the polysemous words in the target sign language text, select the target action identifier from the multiple action identifiers corresponding to the polysemous words.

[0088] For example, after the translation model outputs the target sign language text, server 22 can also determine whether there are polysemous words in at least one target word in the target sign language text. If polysemous words exist, the polysemous word action selection module in server 22 selects the target action identifier from multiple action identifiers corresponding to each polysemous word. That is, as... Figure 6 The translation rewriting and correction module and the polysemous word action selection module shown can execute their functions in parallel or sequentially. For example, if the target sign language text output by the translation model includes the non-sign language word "map" and the polysemous word "press," the translation rewriting and correction module can correct "map" to "ground / draw," while the polysemous word action selection module can select the target action identifier from the multiple action identifiers corresponding to "press." As another example, if the target sign language text output by the translation model includes the non-sign language word "wonderful," the translation rewriting and correction module first corrects "wonderful" to "J / color," where "J" is a polysemous word. Then, the polysemous word action selection module selects the target action identifier from the multiple action identifiers corresponding to "J."

[0089] This embodiment corrects the target sign language text by replacing non-sign language words in the target sign language text output by the translation model with a single sign language word or a combination of at least two sign language words. This makes the corrected target sign language text more accurate and ensures that hearing-impaired people can understand the final generated sign language video.

[0090] Figure 7 A flowchart of a natural language translation method provided in another embodiment of this disclosure. In this embodiment, the specific steps of the method are as follows:

[0091] S701. Obtain the target natural language text to be translated.

[0092] S702. Determine whether there exists a preset natural language text that matches the target natural language text in the correspondence between the preset natural language text and the preset sign language text.

[0093] like Figure 8 As shown, server 22 also includes a knowledge base matching module, which can query a knowledge base containing a mapping between preset natural language text and preset sign language text. The preset sign language text configured in this knowledge base is manually translated sign language text; that is, the knowledge base contains manually translated sign language text.<text,gloss> In this context, "text" refers to the preset natural language text, and "gloss" refers to the preset sign language text. After obtaining the target natural language text "Text" to be translated, server 22 can first use the knowledge base matching module to determine whether there is a preset natural language text in the knowledge base that matches "Text".

[0094] S703. If there is no preset natural language text matching the target natural language text in the correspondence, then the target natural language text is input into the pre-trained translation model.

[0095] If no pre-defined natural language text matching the target sign language text exists in the knowledge base, the target sign language text can be input into the translation model. The translation model will then translate the target sign language text to obtain the target sign language text. Subsequent processing of the target sign language text can be referred to the above embodiments, and will not be repeated here.

[0096] This embodiment queries a correspondence between preset natural language text and preset sign language text to determine if a preset natural language text matching the target natural language text exists within this correspondence. If it does, the corresponding preset sign language text can be directly used as the target sign language text. Since the preset sign language text is pre-set, its accuracy is guaranteed, further improving the accuracy of the target sign language text. Furthermore, knowledge base matching enhances the maneuverability of the translation effect, replacing the complex process of a translation model and allowing for rapid correction of errors generated online. Moreover, for new application scenarios, a high-frequency knowledge base can be configured to quickly cover the new application scenario and ensure the accuracy of the translation results in the new application scenario.

[0097] Figure 9 A flowchart of a natural language translation method provided in another embodiment of this disclosure. In this embodiment, the specific steps of the method are as follows:

[0098] S901. Obtain the target natural language text to be translated.

[0099] S902. Determine whether there exists a preset natural language text that matches the target natural language text in the correspondence between the preset natural language text and the preset sign language text.

[0100] S903. If there is a preset natural language text that matches the target natural language text in the correspondence, then the preset sign language text corresponding to the matched preset natural language text shall be used as the target sign language text corresponding to the target natural language text.

[0101] like Figure 10As shown, based on the above embodiment, this embodiment can also add an intent analysis module to the server 22. After the server 22 obtains the target natural language text (Text) to be translated, it can first determine whether there is a preset natural language text matching Text in the knowledge base through the knowledge base matching module. If there is, the preset sign language text corresponding to the preset natural language text can be directly used as the target sign language text corresponding to the target natural language text.

[0102] S904. If there is no preset natural language text matching the target natural language text in the correspondence, then determine whether the intent of the target natural language text is a preset intent.

[0103] If the knowledge base matching module does not find a preset natural language text matching Text in the knowledge base, then Text can be input as follows: Figure 10 The intent analysis module is shown. At this point, the intent analysis module can determine whether the intent of the Text is a preset intent.

[0104] S905. If the intent of the target natural language text is not a preset intent, then the target natural language text is input into a pre-trained translation model.

[0105] If the intent analysis module determines that the intent of the text is not a preset intent, then the text can be input to the translation model for translation. In other words, in this embodiment, if the text does not match a preset natural language text in the knowledge base and does not have a preset intent, it will flow into the translation model.

[0106] S906. If the intent of the target natural language text is a preset intent, then extract keywords from the target natural language text according to the preset slot template corresponding to the preset intent.

[0107] If the intent analysis module determines that the intent of the text is a preset intent, the intent analysis module can call the preset slot template corresponding to the preset intent to extract keywords from the text. For example... Figure 11As shown, the target natural language text is "There are 2 tickets left for the basketball game at 3 PM." After intent classification, the intent of the target natural language text is determined to be "selling tickets." Then, keywords are extracted from the target natural language text based on the preset slot template corresponding to "selling tickets." For example, the preset slot template includes fields such as quantity, time, item, price, and team. Keywords are extracted from the target natural language text based on these fields. For instance, the keyword matching quantity is 2, the keyword matching time is "3 PM," and the keyword matching item is "basketball." It is understandable that different intents may correspond to different preset slot templates.

[0108] S907. Generate the target sign language text corresponding to the target natural language text based on the keywords.

[0109] like Figure 11 As shown, after extracting keywords based on the preset slot template, these keywords can be used to construct the target sign language text, i.e., Gloss, corresponding to the target natural language text.

[0110] This embodiment determines the intent of the target natural language text, and when the intent is a preset intent, extracts keywords from the target natural language text according to the preset slot template corresponding to the preset intent, and generates the target sign language text corresponding to the target natural language text based on the keywords, thereby enhancing the controllability and accuracy of sign language translation in specific scenarios.

[0111] like Figure 12 The diagram illustrates the conversion process from a hearing person's speech to sign language video, including the translation process from natural language text (Text) to sign language text (Gloss), which is a crucial step in the conversion. Figure 12 As shown, this translation process can be a full-link translation solution. Since the Gloss includes multiple sign language words, and each sign language word can correspond to a movement, the sign language video can be obtained by synthesizing the movements corresponding to each sign language word.

[0112] Although many sign language digital humans have emerged in the market, they are still in their early stages, and no product can provide stable and reliable services for the hearing impaired. The core reason for this is insufficient translation capabilities, resulting in low intelligibility for the hearing impaired. This disclosure aims to ensure high translation intelligibility through a full-link translation solution. The following explains why the full-link translation solution described in this disclosure needs to possess both "translation" and "full-link" capabilities.

[0113] Because sign language has its own unique vocabulary and grammar, simply segmenting and piecing together natural language text according to its grammar and word structure would make it incomprehensible to hearing-impaired individuals. Therefore, the natural language text needs to be translated into sign language during the sign language video generation process to ensure the generated video conforms to the vocabulary and grammar of sign language, making it truly understandable for the hearing-impaired. In this embodiment, the translation primarily addresses two issues. One is the vocabulary issue. Since the vocabulary of sign language is much smaller than that of natural language, and the meanings of sign language words are usually simple and straightforward, some words in natural language require translation using multiple sign language vocabulary words. For example… Figure 12 The phrase "pleasing to the eye" as shown requires explanation using the sign language words "J" and "color." Similarly, as... Figure 12 The word "feast" in the example needs to be explained using the sign language words "competition" and "meeting." Similarly, "final" in natural language requires the sign language words "last" and "competition." Furthermore, common rhetorical devices and written expressions in natural language also require similar translation using multiple sign language words. Another issue that translation can address is grammatical problems. Because sign language has unique expression patterns—for example, negative words and modifiers are often placed after the verb, while time words and subject words are placed before the verb, and function words and quantifiers are often omitted—there are significant differences between the grammatical word order of natural language and sign language. Translation can convert the grammatical word order of natural language to that of sign language.

[0114] In this embodiment, the end-to-end approach ensures the stability of the translation results. Through the collaboration of multiple modules, it guarantees high availability in business scenarios and is truly acceptable to hearing-impaired individuals. This end-to-end approach can be as follows: Figure 13 The process from Text to Gloss shown involves collaboration between three modules in the entire chain. These three modules are as follows: Figure 13 The module shown includes a product experience enhancement module, a translation workflow collaboration module, and a translation effect improvement module. Because sign language translation is inherently a challenging "minority language" translation, issues such as multimodal attributes, limited relevant data, difficulty in data acquisition, and a degree of uncontrollability in translation models make it difficult to achieve practical application using only a single translation model. Therefore, modules such as... Figure 13 The entire process shown guarantees the stability and controllability of the translation results.

[0115] Specifically, the product experience improvement module can be used to make the speech rate, expressions, habits, etc. more similar to those of real deaf-mute people. The product experience improvement module includes a semantic simplification module, an entity normalization module, and an emotion judgment module. In information broadcasting scenarios, the broadcast content needs to be converted into sign language. However, the speaking speed of the anchor is relatively fast, while the sign language speed of the digital human is relatively slow. If translated word by word, the sign language will lag far behind the spoken language. In addition, if the translation result is too redundant, it is also difficult for deaf people to understand. Therefore, it is necessary to simplify the broadcast content while retaining the original meaning. Thus, in this embodiment, the semantic simplification module can perform semantic simplification on the target natural language text to be translated. For example, the semantic simplification module can perform semantic simplification on the target natural language text through a text summarization model, thereby reducing the lexical complexity in the target natural language text and compressing the sentence length of the target natural language text. Or, the semantic simplification module can retain the core vocabulary in the target natural language text through a lexical complexity calculation method, thereby compressing the sentence length of the target natural language text. The entity normalization module can perform normalization processing on entity information such as names and numbers in information broadcasting scenarios. For example, when translating a name, the first letter of the name in pinyin can be used. For example, if Text is Li Moumou, the corresponding Gloss is Name / L / M / M. For numbers, if they are multi-digit numbers and adjacent numbers are different, for example, taking "256" as an example, when the digital human makes sign language movements, it cannot make the sign language movements corresponding to "2", "5", and "6" in sequence, but make the sign language movements corresponding to "200", "50", and "6" respectively. For multi-digit numbers with adjacent identical numbers, for example, taking "77" as an example, when the digital human makes sign language movements, it can continuously make the sign language movements corresponding to "7" twice. This makes the translation of such entity words more in line with the habits of deaf-mute people and avoids ambiguity. In addition, since expressions are an important element of sign language and also an important factor to help hearing-impaired people understand. Therefore, it is necessary to generate corresponding expressions for the digital human. For example, the emotion judgment module can judge the expression of each Text. Specifically, the emotion judgment module can output emotion labels. In this embodiment, 9 emotion labels are set, namely happy, angry, sad, confused, afraid, disgusted, surprised, painful, and disappointed. That is to say, when the digital human expresses the meaning of this Text through sign language, control the expression of the digital human to be the expression reflected by the emotion label corresponding to this Text. In addition, in some other embodiments, after translating Text into Gloss, it is also possible to judge the emotion label corresponding to each sign language word in the Gloss, so that when the digital human makes the sign language movement corresponding to this sign language word, control the expression of the digital human to match the emotion label corresponding to this sign language word. It can be understood that in this embodiment, the function executed by the semantic simplification module can be executed first. Further, the entity normalization module and the emotion judgment module can execute their respective functions in parallel.

[0116] like Figure 13 As shown, the translation splitting and collaboration module includes the knowledge base matching module, intent analysis module, and translation model, as described above. Their respective functions are as described above and will not be repeated here. Additionally, the translation effect improvement module includes a translation rewriting and correction module and a polysemous word action selection module. The translation model is the core module in the entire translation process. During the training phase of this translation model, the training samples can be the correspondence between the natural language text and the sign language text after translating natural language text into standard sign language text. Alternatively, the training samples can also be the correspondence between the natural language text and the sign language text after obtaining sign language video from natural language text and then obtaining sign language text from the sign language video. Since obtaining sign language video from natural language text and then sign language text from the sign language video is more accurate, the accuracy of the training samples can be improved in this case, thereby improving the training accuracy of the translation model. Furthermore, this embodiment can also perform quality inspection on the sign language text in the training samples. This quality inspection process can be automated, manual, or a combination of automated and manual quality inspection, thereby ensuring high-quality translation data. Furthermore, this translation model can use an encoder-decoder machine translation architecture, such as... Figure 14 As shown. Furthermore, this translation model can employ a pre-training-fine-tuning training mode, experimenting with various decoding schemes including sign language dictionaries, Chinese dictionaries, and a combination of sign language and Chinese dictionaries. Leveraging word alignment knowledge maintained by sign language experts (e.g., "final" needs to be translated as "last / competition"), the translation ability from Chinese vocabulary to sign language vocabulary is enhanced. Additionally, as... Figure 14 As shown, this translation model is trained using multiple loss values, where `gloss loss` represents the first loss value as described above, and `gloss1 loss`, `gloss2 loss`, ..., `glossn loss` represent the second loss values, respectively. Through contrastive learning, i.e., multiple loss values, the translation model's learning effect on sign language grammar and word order can be enhanced. Furthermore, the user interface for hearing-impaired individuals can display the process of translating natural language into sign language animation, as shown in the example... Figure 15 As shown.

[0117] In summary, the embodiments of this disclosure can enhance product experience and acceptability through modules such as semantic simplification and emotion judgment, ultimately reflected in reasonable speech rate and facial expressions. Furthermore, the synergistic effect of the knowledge base matching module, intent analysis module, and translation model can improve translation accuracy and controllability. The translation model ensures the overall fluency of the translation, while word alignment knowledge improves the intelligibility of the final result. The intent analysis module and knowledge base matching module ensure the accuracy of translation for high-frequency phrases and limited scenarios. The polysemous word action selection module ensures the accuracy of actions.

[0118] Figure 16 This is a schematic diagram of the structure of a model training apparatus provided in an embodiment of this disclosure. The model training apparatus provided in this embodiment of the disclosure can execute the processing flow provided in the model training method embodiment, such as... Figure 16 As shown, the model training device 160 includes:

[0119] The acquisition module 161 is used to acquire a first sign language text and at least one second sign language text corresponding to a natural language sample text, wherein the accuracy of the second sign language text is lower than the accuracy of the first sign language text.

[0120] The input module 162 is used to input the natural language sample text into the translation model to be trained, and output the predicted sign language text corresponding to the natural language sample text through the translation model;

[0121] The calculation module 163 is configured to calculate a first loss value of the translation model based on the predicted sign language text and the first sign language text; and to calculate at least one second loss value of the translation model based on the predicted sign language text and the at least one second sign language text.

[0122] Training module 164 is used to train the translation model based on the first loss value and the at least one second loss value.

[0123] Optionally, the first sign language text includes multiple sign language words;

[0124] The second sign language text is obtained by rearranging the order of at least some of the sign language words from the plurality of sign language words; or

[0125] The second sign language text is obtained by replacing at least some of the sign language words among the plurality of sign language words; or

[0126] The second sign language text is the historical sign language text output by the translation model during its historical training process.

[0127] Figure 16The model training apparatus of the illustrated embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0128] Figure 17 This is a schematic diagram of the structure of a natural language translation apparatus provided in an embodiment of this disclosure. The natural language translation apparatus provided in this embodiment can execute the processing flow provided in the natural language translation method embodiment, such as... Figure 17 As shown, the natural language translation device 170 includes:

[0129] Module 171 is used to acquire the target natural language text to be translated;

[0130] The input module 172 is used to input the target natural language text into a pre-trained translation model, and output the target sign language text corresponding to the target natural language text through the translation model. The translation model is obtained according to the model training method described above.

[0131] Optionally, when the input module 172 inputs the target natural language text into a pre-trained translation model and outputs the target sign language text corresponding to the target natural language text through the translation model, it is specifically used to: input the target natural language text into a pre-trained translation model and output the target sign language text corresponding to the target natural language text and the target action identifiers corresponding to polysemous words in the target sign language text through the translation model.

[0132] Optionally, the natural language translation device 170 further includes a correction module 173 and a selection module 174. The correction module 173, after the translation model outputs the target sign language text corresponding to the target natural language text, corrects target words in the target sign language text that are not sign language words to a single sign language word or a combination of at least two sign language words. The selection module 174, based on the context of polysemous words in the target sign language text, selects a target action identifier from multiple action identifiers corresponding to the polysemous words.

[0133] Optionally, the selection module 174 is further configured to: after the correction module 173 corrects the target vocabulary that does not belong to the sign language vocabulary in the target sign language text to a sign language vocabulary or a combination of at least two sign language vocabulary, if the one sign language vocabulary or the combination of at least two sign language vocabulary includes a polysemous word, then select the target action identifier from the multiple action identifiers corresponding to the polysemous word.

[0134] Optionally, the natural language translation device 170 further includes a determining module 175, used to determine whether there exists a preset natural language text matching the target natural language text in the correspondence between the preset natural language text and the preset sign language text. The input module 172 is specifically used to: if there is no preset natural language text matching the target natural language text in the correspondence, then input the target natural language text into the pre-trained translation model.

[0135] Optionally, the determining module 175 is further configured to: if there is no preset natural language text matching the target natural language text in the correspondence, determine whether the intent of the target natural language text is a preset intent. The input module 172 is specifically configured to: if the intent of the target natural language text is not a preset intent, input the target natural language text into a pre-trained translation model.

[0136] Optionally, the determining module 175 is further configured to: if there is a preset natural language text that matches the target natural language text in the correspondence, then use the preset sign language text corresponding to the matched preset natural language text as the target sign language text corresponding to the target natural language text.

[0137] Optionally, the determining module 175 is further configured to: if the intent of the target natural language text is a preset intent, extract keywords from the target natural language text according to the preset slot template corresponding to the preset intent; and generate target sign language text corresponding to the target natural language text according to the keywords.

[0138] Figure 17 The natural language translation device shown in the embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0139] The above describes the internal functions and structure of a model training device or natural language translation device, which can be implemented as an electronic device. Figure 18 A schematic diagram illustrating the structure of an electronic device embodiment provided in this disclosure. For example... Figure 18 As shown, the electronic device includes a memory 181 and a processor 182.

[0140] Memory 181 is used to store programs. In addition to the programs described above, memory 181 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phone book data, messages, pictures, videos, etc.

[0141] The memory 181 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0142] The processor 182 is coupled to the memory 181 and executes the program stored in the memory 181 for:

[0143] Obtain a first sign language text and at least one second sign language text corresponding to a natural language sample text, wherein the accuracy of the second sign language text is lower than the accuracy of the first sign language text;

[0144] The natural language sample text is input into the translation model to be trained, and the translation model outputs the predicted sign language text corresponding to the natural language sample text.

[0145] Calculate the first loss value of the translation model based on the predicted sign language text and the first sign language text;

[0146] Based on the predicted sign language text and the at least one second sign language text, calculate at least one second loss value for the translation model;

[0147] The translation model is trained based on the first loss value and the at least one second loss value.

[0148] Alternatively, processor 182 is also used for:

[0149] Obtain the target natural language text to be translated;

[0150] The target natural language text is input into a pre-trained translation model, and the translation model outputs the target sign language text corresponding to the target natural language text. The translation model is obtained according to the model training method described above.

[0151] Furthermore, such as Figure 18 As shown, the electronic device may also include other components such as a communication component 183, a power supply component 184, an audio component 185, and a display 186. Figure 18 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 18 The components shown.

[0152] Communication component 183 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 183 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 183 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0153] Power supply component 184 provides power to various components of an electronic device. Power supply component 184 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0154] Audio component 185 is configured to output and / or input audio signals. For example, audio component 185 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 181 or transmitted via communication component 183. In some embodiments, audio component 185 also includes a speaker for outputting audio signals.

[0155] Display 186 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0156] In addition, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the model training method or natural language translation method described in the above embodiments.

[0157] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0158] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model training method, wherein, The method includes: Obtain a first sign language text and at least one second sign language text corresponding to a natural language sample text, wherein the accuracy of the second sign language text is lower than the accuracy of the first sign language text; The natural language sample text is input into the translation model to be trained, and the translation model outputs the predicted sign language text corresponding to the natural language sample text. Calculate the first loss value of the translation model based on the predicted sign language text and the first sign language text; Based on the predicted sign language text and the at least one second sign language text, calculate at least one second loss value for the translation model; The translation model parameters are adjusted by controlling the first loss value to decrease while simultaneously controlling each of the second loss values ​​to increase, in order to train the translation model.

2. The method according to claim 1, wherein, The first sign language text includes multiple sign language words; The second sign language text is obtained by rearranging the order of at least some of the sign language words from the plurality of sign language words; or The second sign language text is obtained by replacing at least some of the sign language words among the plurality of sign language words; or The second sign language text is the historical sign language text output by the translation model during its historical training process.

3. A natural language translation method, wherein, The method includes: Obtain the target natural language text to be translated; The target natural language text is input into a pre-trained translation model, and the translation model outputs the target sign language text corresponding to the target natural language text. The translation model is obtained by the model training method according to claim 1 or 2.

4. The method according to claim 3, wherein, The target natural language text is input into a pre-trained translation model, and the translation model outputs the target sign language text corresponding to the target natural language text, including: The target natural language text is input into a pre-trained translation model, which then outputs the target sign language text corresponding to the target natural language text, as well as the target action identifiers corresponding to polysemous words in the target sign language text.

5. The method according to claim 3, wherein, After outputting the target sign language text corresponding to the target natural language text through the translation model, the method further includes: The target vocabulary in the target sign language text that does not belong to the sign language vocabulary is modified into a sign language vocabulary or a combination of at least two sign language vocabulary; Based on the context of the polysemous words in the target sign language text, the target action identifier is selected from multiple action identifiers corresponding to the polysemous words.

6. The method according to claim 5, wherein, After correcting target words in the target sign language text that are not sign language vocabulary to a single sign language word or a combination of at least two sign language words, the method further includes: If a sign language word or a combination of at least two sign language words includes a polysemous word, then the target action sign is selected from the multiple action signs corresponding to the polysemous word.

7. The method according to claim 3, wherein, Inputting the target natural language text into a pre-trained translation model includes: Determine whether there exists a preset natural language text that matches the target natural language text in the correspondence between preset natural language text and preset sign language text; If there is no preset natural language text that matches the target natural language text in the correspondence, then the target natural language text is input into the pre-trained translation model.

8. The method according to claim 7, wherein, If no preset natural language text matching the target natural language text exists in the correspondence, then the target natural language text is input into a pre-trained translation model, including: If there is no preset natural language text matching the target natural language text in the correspondence, then it is determined whether the intent of the target natural language text is the preset intent; If the intent of the target natural language text is not the preset intent, then the target natural language text is input into the pre-trained translation model.

9. The method according to claim 7, wherein, The method further includes: If there exists a preset natural language text in the correspondence that matches the target natural language text, then the preset sign language text corresponding to the matched preset natural language text will be used as the target sign language text corresponding to the target natural language text.

10. The method according to claim 8, wherein, The method further includes: If the intent of the target natural language text is a preset intent, then keywords are extracted from the target natural language text according to the preset slot template corresponding to the preset intent; Based on the keywords, generate the target sign language text corresponding to the target natural language text.

11. A model training device, wherein, include: The acquisition module is used to acquire a first sign language text and at least one second sign language text corresponding to a natural language sample text, wherein the accuracy of the second sign language text is lower than that of the first sign language text. The input module is used to input the natural language sample text into the translation model to be trained, and output the predicted sign language text corresponding to the natural language sample text through the translation model; The calculation module is configured to calculate a first loss value of the translation model based on the predicted sign language text and the first sign language text; and to calculate at least one second loss value of the translation model based on the predicted sign language text and the at least one second sign language text. The training module is used to adjust the model parameters of the translation model by controlling the first loss value to decrease and simultaneously controlling each of the second loss values ​​to increase, so as to train the translation model.

12. A natural language translation device, wherein, include: The acquisition module is used to acquire the target natural language text to be translated; An input module is used to input the target natural language text into a pre-trained translation model, and output the target sign language text corresponding to the target natural language text through the translation model. The translation model is obtained by the model training method according to claim 1 or 2.

13. An electronic device, wherein, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-10.

14. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-10.

Images