Method, device, computer equipment and storage medium for generating sign language video
By extracting summaries and transforming the grammatical structure of texts from hearing individuals, sign language videos synchronized with the audio are generated, solving the problem of asynchronous sign language videos and enabling large-scale application at low cost.
Patent Information
- Application Number
- CN202210114157.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-30
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-01-30
AI Technical Summary
The problem of asynchronous sign language video and audio in existing technologies makes it impossible for hearing-impaired people to understand the audio content normally, and the high cost of generating sign language videos prevents their large-scale application.
By extracting summaries from texts obtained from hearing individuals, sign language text that conforms to the grammatical structure of hearing-impaired individuals is generated and synchronized with the audio to automatically generate sign language videos.
It enables synchronized playback of sign language video and audio, improving intelligibility for hearing-impaired individuals, reducing production costs, and is suitable for various application scenarios such as live video streaming and offline video scenarios.
Smart Images

Figure CN116561294B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of artificial intelligence, and in particular to a method, apparatus, computer equipment and storage medium for generating a sign language video. Background Art
[0002] When watching videos, hearing-impaired people often cannot watch them properly without subtitles, so the audio content of the video needs to be translated into corresponding sign language videos and played synchronously with the video.
[0003] In the related art, it is necessary to obtain the text corresponding to the audio content in advance, and the sign language teacher manually translates the text into sign language, thereby generating a sign language video, which is then added to the video for playback.
[0004] However, due to the limited number of sign language teachers and the high labor cost of implementation, it cannot be applied on a large scale to various application scenarios. Summary of the invention
[0005] The embodiments of the present application provide a method, device, computer equipment and storage medium for generating a sign language video, which can improve the intelligibility of the sign language video, have low implementation cost and can be applied to a large number of application scenarios. The technical solution is as follows:
[0006] On the one hand, an embodiment of the present application provides a method for generating a sign language video, the method comprising:
[0007] Acquire a hearing text, wherein the hearing text is a text that conforms to the grammatical structure of a hearing person;
[0008] Extracting a summary of the listener text to obtain a summary text, wherein the text length of the summary text is shorter than the text length of the listener text;
[0009] Converting the summary text into a sign language text, wherein the sign language text is a text that conforms to the grammatical structure of hearing-impaired persons;
[0010] The sign language video is generated based on the sign language text, and the sign language video is synchronized with the audio corresponding to the listener text.
[0011] On the other hand, an embodiment of the present application provides a device for generating a sign language video, the device comprising:
[0012] An acquisition module, used for acquiring a hearing person text, wherein the hearing person text is a text that conforms to the grammatical structure of a hearing person;
[0013] An extraction module, used for extracting a summary of the listener text to obtain a summary text, wherein the text length of the summary text is shorter than the text length of the listener text;
[0014] A conversion module, used to convert the summary text into a sign language text, wherein the sign language text is a text that conforms to the grammatical structure of the hearing-impaired;
[0015] A generating module is used to generate the sign language video based on the sign language text, wherein the sign language video is synchronized with the audio corresponding to the listener text.
[0016] On the other hand, an embodiment of the present application provides a computer device, which includes a processor and a memory; the memory stores at least one instruction, and the at least one instruction is used to be executed by the processor to implement the method for generating a sign language video as described in the above aspect.
[0017] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one instruction, at least one program, a code set or an instruction set is stored. The at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the method for generating a sign language video as described in the above aspects.
[0018] On the other hand, an embodiment of the present application provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method for generating a sign language video provided in the above aspect.
[0019] The technical solution provided by this application may have the following beneficial effects:
[0020] In the embodiment of the present application, a text summary is extracted from the hearing text to obtain a summary text, thereby shortening the text length of the hearing text so that the finally generated sign language video is synchronized with the audio corresponding to the hearing text. In addition, the summary text is converted into a sign language text that conforms to the grammatical structure of the hearing-impaired, and a sign language video is generated based on the sign language text, thereby improving the intelligibility of the sign language video to the hearing-impaired. The sign language video is automatically generated based on the sign language text without the need for manual signing, which is low in cost and can be applied to a large number of application scenarios, such as live video broadcasting scenarios, offline video playback scenarios, etc. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0022] Figure 1 A schematic diagram showing an implementation environment provided by an exemplary embodiment of the present application is shown;
[0023] Figure 2 A flowchart of a method for generating a sign language video provided by an exemplary embodiment of the present application is shown;
[0024] Figure 3 A schematic diagram showing a principle that a sign language video and its corresponding audio are out of sync provided by an exemplary embodiment of the present application;
[0025] Figure 4 A flowchart of a method for generating a sign language video provided by another exemplary embodiment of the present application is shown;
[0026] Figure 5 A flowchart of a speech recognition process provided by an exemplary embodiment of the present application is shown;
[0027] Figure 6 A schematic diagram of a coder-decoder according to an exemplary embodiment of the present invention is shown;
[0028] Figure 7 A flowchart of a translation model training process provided by an exemplary embodiment of the present application is shown;
[0029] Figure 8 A flowchart of establishing a virtual object provided by an exemplary embodiment of the present application is shown;
[0030] Fig. 9 A flowchart of a method for generating a summary text provided by an exemplary embodiment of the present application is shown;
[0031] Fig.10 A schematic diagram of a dynamic path planning algorithm provided by an exemplary embodiment of the present application is shown;
[0032] Fig.11 A schematic diagram showing a process of a summary text generation method provided by an exemplary embodiment of the present application is shown;
[0033] Fig.12 A flowchart of a method for generating a sign language video provided by an exemplary embodiment of the present application is shown;
[0034] Fig.13 A structural block diagram of a device for generating a sign language video provided by an exemplary embodiment of the present application is shown;
[0035] Fig.14 A structural block diagram of a computer device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION
[0036] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of sensors and methods of making consistent with some aspects of the present application as detailed in the appended claims.
[0037] It should be understood that the "several" mentioned in this article refers to one or more, and "multiple" refers to two or more. "And / or" describes the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.
[0038] The names involved in the embodiments of the present application are introduced below:
[0039] Sign language: A language used by the hearing-impaired, which consists of gestures, body movements, facial expressions and other information. According to the word order, sign language can be divided into natural sign language and gesture sign language. Natural sign language is the language used by the hearing-impaired, while gesture sign language is the language used by the hearing people. Natural sign language and gesture sign language can be distinguished by the word order. For example, "cat / mouse / catch" is natural sign language, and "cat / catch / mouse" is gesture sign language, where " / " is used to separate each phrase.
[0040] Hearing text: text that conforms to the grammatical structure of hearing people, that is, text that conforms to the language habits and grammatical structure of Mandarin Chinese.
[0041] Sign language text: text that conforms to the reading habits and grammatical structure of the hearing-impaired.
[0042] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0043] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0044] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, unmanned driving, automatic driving, drones, robots, smart medical care, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
[0045] In the embodiments of the present application, artificial intelligence is applied to the field of sign language interpretation, which can automatically generate sign language video based on the listener's text and solve the problem of the sign language video being out of sync with the corresponding audio.
[0046] In daily life, when hearing-impaired people watch video programs such as news broadcasts and football game broadcasts, they cannot watch them normally because there are no corresponding subtitles. Or when listening to audio programs, such as radio, hearing-impaired people cannot listen normally because there are no subtitles corresponding to the audio. In related technologies, audio content is usually obtained in advance, and sign language video is pre-recorded according to the audio content, and then synthesized with video or audio before playing, so that hearing-impaired people can understand the corresponding audio content through the sign language video.
[0047] However, since sign language is a language composed of gestures, when the content is the same, the duration of the sign language video is longer than the duration of the audio, which causes the time axis of the generated sign language video to be misaligned with the audio time axis. Especially for videos, it is easy to cause the sign language video to be out of sync with the corresponding audio, affecting the hearing-impaired person's understanding of the audio content. For videos, since the audio content and the video content are consistent, it may also cause differences between the content expressed by the sign language and the video screen. In an embodiment of the present application, by obtaining the listener text and timestamp corresponding to the video, the listener text is extracted to obtain a summary text, thereby shortening the text length of the listener text, so that the time axis of the sign language video generated based on the summary text is aligned with the audio time axis of the audio corresponding to the listener text, thereby solving the problem of the sign language video being out of sync with the corresponding audio.
[0048] The method for generating sign language videos provided in the embodiments of the present application can be applied to a variety of scenarios to provide convenience for the lives of hearing-impaired people.
[0049] In a possible application scenario, the method for generating a sign language video provided by the embodiment of the present application can be applied to a real-time sign language scenario. Optionally, the real-time sign language scenario can be a live broadcast of an event, a live broadcast of news, a live broadcast of a meeting, etc. The method provided by the embodiment of the present application can be used to match the live broadcast content with a sign language video. Taking the live broadcast of news as an example, the audio corresponding to the live broadcast of news is converted into a human-listening text, the human-listening text is compressed to obtain a summary text, and a sign language video is generated based on the summary text, so as to be synthesized with the live broadcast of news video and pushed to the user in real time.
[0050] In another possible application scenario, the method for generating a sign language video provided in the embodiment of the present application can be applied to an offline sign language scenario, where there is an offline text. Optionally, the offline sign language scenario can be a reading scenario of text materials, and the text content can be directly converted into a sign language video for playback.
[0051] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application. The implementation environment may include a terminal 110 and a server 120.
[0052] The terminal 110 installs and runs a client that can watch the sign language video, and the client can be an application or a web client. Taking the client as an application as an example, the application can be a video player, an audio player, etc., which is not limited in the embodiment of the present application.
[0053] Regarding the device type of terminal 110, terminal 110 may include but is not limited to smart phones, tablet computers, e-book readers, Moving Picture Experts Group Audio Layer III (MP3) players, Moving Picture Experts Group Audio Layer IV (MP4) players, laptop computers, desktop computers, intelligent voice interaction devices, smart home appliances, car terminals, etc., and the embodiments of the present application are not limited to this.
[0054] The terminal 110 is connected to the server 120 via a wireless network or a wired network.
[0055] The server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server 120 is used to provide background services for the client. Optionally, the method for generating a sign language video can be executed by the server 120, or by the terminal 110, or by the server 120 and the terminal 110 in collaboration, which is not limited in the embodiments of the present application. For the convenience of description, the following embodiments are described by taking a computer device as an example of an execution subject for generating a sign language video.
[0056] In the embodiment of the present application, the modes in which the server 120 generates sign language videos include an offline mode and a real-time mode.
[0057] In a possible implementation, when the mode in which the server 120 generates the sign language video is offline mode, the server 120 stores the generated sign language video in the cloud. When the user needs to watch the sign language video, the terminal 110 downloads the sign language video from the server by inputting the storage path of the sign language video in the application or web client of the terminal 110.
[0058] In another possible implementation, when the mode in which the server 120 generates the sign language video is real-time mode, the server 120 pushes the sign language video to the terminal 110 in real time, the terminal 110 downloads the sign language video in real time, and the user can watch it by running an application or a web client on the terminal 110.
[0059] The following is an introduction to the method for generating a sign language video in the embodiment of the present application. Figure 2 , which shows a flow chart of a method for generating a sign language video provided by an exemplary embodiment of the present application, the method comprising:
[0060] Step 210, obtaining a hearing person text, where the hearing person text is a text that conforms to the grammatical structure of a hearing person.
[0061] Regarding the type of the listener text, optionally, the listener text can be an offline text or a real-time text.
[0062] Exemplarily, when the listener text is an offline text, it may be a text obtained in a scenario such as video or audio offline download.
[0063] Exemplarily, when the listener text is real-time text, it may be text acquired in scenarios such as live video broadcast and simultaneous interpretation.
[0064] Regarding the source of the listener text, optionally, the listener text can be a text with edited content; it can also be text extracted from a subtitle file, or it can be text extracted from an audio file or a video file, etc., and the embodiments of the present application do not limit this.
[0065] Optionally, in the embodiment of the present application, the language of the listener text is not limited to Chinese, but may be other languages, which is not limited in the embodiment of the present application.
[0066] Step 220 , extract a summary of the listener text to obtain a summary text, wherein the text length of the summary text is shorter than the text length of the listener text.
[0067] like Figure 3 As shown, when expressing the same content, the duration of the sign language video (obtained by sign language translation of the listener text) is longer than the audio duration of the audio corresponding to the listener text, which causes the audio time axis of the audio corresponding to the listener text to be misaligned with the time axis of the finally generated sign language video, thereby causing the sign language video and its corresponding audio to be out of sync. Among them, A1, A2, A3, and A4 are used to represent the timestamp corresponding to the listener text, and V1, V2, V3, and V4 are used to represent the time interval of the sign language video axis. Therefore, in a possible implementation, the computer device can shorten the text length of the listener text so that the finally generated sign language video and its corresponding audio remain synchronized.
[0068] Exemplarily, the computer device obtains the summary text by extracting sentences in the listener text that have an impact on the full text semantics of the listener text.
[0069] Exemplarily, the computer device obtains the summary text by performing text compression processing on the sentences of the listener text.
[0070] In addition, when the listener text is an offline text and when the listener text is a real-time text, the method of extracting the summary of the listener text is different. When the listener text is an offline text, the computer device can obtain the entire content of the listener text, so any one of the above methods or a combination of the two methods can be used to obtain the summary text. When the listener text is a real-time text, since the computer device transmits the listener text in a real-time push manner, it is impossible to obtain the entire content of the listener text, and the summary text can only be obtained by using a method of text compression processing on the sentences of the listener text.
[0071] In another possible implementation, the computer device may synchronize the sign language video with its corresponding audio by adjusting the speed of the sign language gestures in the sign language video. Exemplarily, when the duration of the sign language video is greater than the duration of the audio, the computer device may cause the virtual object performing the sign language gesture to keep shaking naturally between the sign language sentences, waiting for the time axis of the sign language video to align with the audio time axis; when the duration of the sign language video is less than the duration of the audio, the computer device may cause the virtual object performing the sign language gesture to speed up the gesture action between the sign language sentences, aligning the time axis of the sign language video with the audio time axis, thereby synchronizing the sign language video with its corresponding audio.
[0072] Step 230, converting the summary text into a sign language text, where the sign language text is a text that conforms to the grammatical structure of the hearing-impaired.
[0073] In the embodiment of the present application, since the summary text is generated based on the hearing person's text, the summary text is also a text that conforms to the grammatical structure of the hearing person. However, since the grammatical structure of the hearing-impaired person is different from that of the hearing-impaired person, in order to improve the intelligibility of the sign language video to the hearing-impaired person, the computer device converts the summary text into a sign language text that conforms to the grammatical structure of the hearing-impaired person.
[0074] In a possible implementation, the computer device automatically converts the summary text into sign language text based on sign language translation technology.
[0075] Exemplarily, the computer device converts the summary text into sign language text based on natural language processing (NLP) technology.
[0076] Step 240, generating a sign language video based on the sign language text, wherein the sign language video is synchronized with the audio corresponding to the listener's text.
[0077] Based on the different types of listener texts, the mode in which the computer device generates sign language video based on the sign language text is also different.
[0078] In a possible implementation, when the listener text type is offline text, the mode in which the computer device generates a sign language video based on the sign language text is an offline video mode. In the offline video mode, the computer device generates a plurality of sign language video clips from a plurality of sign language text sentences, and synthesizes the plurality of sign language video clips to obtain a complete sign language video, and stores the sign language video to a cloud server for download and use by users.
[0079] In another possible implementation, when the listener text type is real-time text, the mode in which the computer device generates the sign language video based on the sign language text is a real-time push streaming mode. In the real-time push streaming mode, the server generates a sign language video clip from the sign language text sentence, and pushes it sentence by sentence to the client in the form of a video stream, and the user can load and play it in real time through the client.
[0080] In summary, in the embodiment of the present application, a text summary is extracted from the hearing text to obtain a summary text, thereby shortening the text length of the hearing text, so that the finally generated sign language video is synchronized with the audio corresponding to the hearing text. In addition, the summary text is converted into a sign language text that conforms to the grammatical structure of the hearing-impaired, and a sign language video is generated based on the sign language text, which improves the intelligibility of the sign language video to the hearing-impaired, and the sign language video is automatically generated based on the sign language text, without the need for manual signing, and the implementation cost is low, and it can be applied to a large number of application scenarios, such as live video scenarios, offline video scenarios, etc.
[0081] In the embodiment of the present application, in one possible implementation, the computer device may obtain a summary text by performing semantic analysis on the listener text and extracting sentences that affect the semantics of the full text of the listener text; in another possible implementation, the computer device may also obtain a summary text by performing sentence segmentation on the listener text and performing text compression processing on the sentence segmented sentences. The above method is introduced below. Please refer to Figure 4 , which shows a flow chart of a method for generating a sign language video provided by another exemplary embodiment of the present application, the method comprising:
[0082] Step 410, obtaining listener text.
[0083] In the embodiment of the present application, there are multiple ways for a computer device to obtain listener text, and these methods are introduced below.
[0084] In a possible implementation, in an offline scenario, such as a reading scenario, the computer device may directly obtain the input listener text, where the listener text is also the corresponding reading text.
[0085] Optionally, the listener text may be a word file, a pdf file, etc., which is not limited in this embodiment of the present application.
[0086] In another possible implementation, in a video viewing scenario or an audio listening scenario, since the video and audio do not have corresponding subtitles, the computer device may obtain a subtitle file and extract the listener text from the subtitle file.
[0087] The subtitle file refers to the text content containing a timestamp.
[0088] In another possible implementation, in a real-time audio transmission scenario, such as a simultaneous interpretation scenario, a conference live broadcast scenario, etc., a computer device obtains an audio file, further performs speech recognition on the audio file to obtain a speech recognition result, and further generates a listener text based on the speech recognition result.
[0089] Since hearing-impaired people cannot hear sounds, they cannot obtain information from audio files. Computer equipment converts the extracted sounds into text through speech recognition technology, thereby generating text for hearing-impaired people.
[0090] In a possible implementation, the process of speech recognition includes: input - encoding (feature extraction) - decoding - output. Figure 5As shown, it shows the process of speech recognition provided by an exemplary embodiment of the present application. First, the computer device extracts features from the input audio file, that is, converts the audio signal from the time domain to the frequency domain, and provides a suitable feature vector for the sound model. Optionally, the extracted features can be LPCC (Linear Predictive Cepstral Coding, linear prediction cepstral coefficients), MFCC (Mel Frequency Cepstral Coefficients, Mel Frequency Cepstral Coefficients), etc., which are not limited in the embodiment of the present application. Further, the extracted feature vector is input into the acoustic model, and the acoustic model is obtained by training with training data 1. The acoustic model is used to calculate the probability of each feature vector on the acoustic feature according to the acoustic feature. Optionally, the acoustic model can be a word model, a character pronunciation model, a semi-syllable model, a phoneme model, etc., which are not limited in the embodiment of the present application. Further, the probability of the phrase sequence that the feature vector may correspond to is calculated based on the language model. The language model is obtained by training with training data 2. The feature vector is decoded by the acoustic model and the language model to obtain the text recognition result, and then the listener text corresponding to the audio file is obtained.
[0091] In another possible implementation, in a real-time video transmission scenario, such as live sports events, live audio-visual programs, etc., a computer device obtains a video file, and further performs OCR (Optical Character Recognition) on the video frames of the video file to obtain text recognition results, and then obtains the listener text.
[0092] Among them, OCR refers to the technology of analyzing and identifying image files containing text materials to obtain text and layout information.
[0093] In a possible implementation, the process of a computer device obtaining a text recognition result by OCR on a video frame of a video file is as follows: the computer device extracts the video frame of the video file, and each video frame can be regarded as a static picture. Further, the computer device performs image preprocessing on the video frame, and corrects the imaging problems of the image, including geometric transformation, i.e. perspective, distortion, rotation, etc., distortion correction, blur removal, image enhancement and light correction, etc. Further, the computer device performs text detection on the video frame after image preprocessing, and detects the location, range and layout of the text. Further, the computer device performs text recognition on the detected text, converts the text information in the video frame into pure text information, and then obtains the text recognition result. The text recognition result is the listener text.
[0094] Step 420 , semantic analysis is performed on the listener text; key sentences are extracted from the listener text based on the semantic analysis result, where the key sentences are sentences in the listener text that affect the semantics of the entire text; and the key sentences are determined as summary text.
[0095] In a possible implementation, the computer device adopts a sentence-level semantic analysis method for the listener text. Optionally, the sentence-level semantic analysis method may be a shallow semantic analysis or a deep semantic analysis, which is not limited in the embodiment of the present application.
[0096] In a possible implementation, the computer device extracts key sentences from the listener text based on the semantic analysis result, filters out non-key sentences, and determines the key sentences as summary text, wherein the non-key sentences are sentences that have no impact on the semantics of the full text.
[0097] Optionally, the computer device can perform semantic analysis on the listener text based on the TF-IDF (Text Frequency-Inverse Document Frequency) algorithm to obtain key sentences and then generate a summary text. First, the computer device first counts the phrases that appear most frequently in the listener text. Further, weights are assigned to the phrases that appear. The size of the weight is inversely proportional to the commonness of the phrase, that is, phrases that are usually rare but appear many times in the listener text are given higher weights, and phrases that are usually more common are given lower weights. Further, the TF-IDF value is calculated based on the weight value of each phrase. The larger the TF-IDF value, the higher the importance of the phrase to the listener text. Therefore, the phrases with the largest TF-IDF values are selected as keywords, and the text sentences where the phrases are located are the key sentences.
[0098] Exemplarily, the content of the listener text is "The 2022 Winter Games will be held in XX. The mascot of this Winter Games is XXX. The slogan of this Winter Games is 'XXXXX'. I am very proud". The computer device performs semantic analysis on the listener text based on the TF-IDF algorithm and obtains the keyword "Winter Games". Therefore, the sentence containing the keyword "Winter Games" is the key sentence, that is, "The 2022 Winter Games will be held in XX. The mascot of this Winter Games is XXX. The slogan of this Winter Games is 'XXXXX'". And "I am very proud" is a non-key sentence. Non-key sentences that have no effect on the full-text semantics of the listener text are filtered out, so the key sentence "The 2022 Winter Games will be held in XX. The mascot of this Winter Games is XXX. The slogan of this Winter Games is 'XXXXX'" is determined as the summary text.
[0099] Step 430: compress the listener text; and determine the compressed listener text as the summary text.
[0100] In a possible implementation, the computer device performs text compression processing on the listener text according to the compression ratio, and determines the compressed listener text as the summary text.
[0101] Optionally, different types of listener texts have different compression ratios. When the listener text type is offline text, the compression ratios of each sentence in the listener text may be the same or different. When the listener text type is real-time text, in order to reduce delay, the sentences of the listener text are compressed according to a fixed compression ratio to obtain a summary text.
[0102] Optionally, the value of the compression ratio is related to the application scenario. For example, in an interview scenario or a daily communication scenario, since the language is more colloquial, there may be less effective information contained in a sentence, so the value of the compression ratio is larger. In the news broadcast scenario, since the language is concise, there is more effective information contained in a sentence, so the value of the compression ratio is smaller. For example, in an interview scenario, the computer device performs text compression processing on the listener text at a compression ratio of 0.8, while in the news broadcast scenario, the computer device performs text compression processing on the listener text at a compression ratio of 0.3.
[0103] In addition, in the embodiment of the present application, the full-text semantics of the summary text obtained after the text compression processing of the listener text should be consistent with the full-text semantics of the listener text.
[0104] Step 440, input the summary text into a translation model to obtain a sign language text output by the translation model, the translation model is trained based on a sample text pair, and the sample text pair consists of a sample sign language text and a sample listener text.
[0105] Exemplarily, the translation model can be a model built based on the encoder-decoder basic framework. Optionally, the translation model can be an RNN (Recurrent Neural Network) model, a CNN (Convolutional Neural Network) model, an LSTM (Long Short-Time Memory) model, etc., which is not limited in the embodiments of the present application.
[0106] Among them, the basic framework structure of encoder-decoder is as follows Figure 6 As shown, the framework structure is divided into two structural parts: encoder and decoder. In the embodiment of the present application, the summary text is first encoded by the encoder to obtain an intermediate semantic vector, and then the intermediate semantic vector is decoded by the decoder to obtain the sign language text.
[0107] Exemplarily, the process of encoding the summary text through the encoder to obtain the intermediate semantic vector is as follows: first, the word vector (InputEmbedding) of the summary text is input. Further, the word vector and positional encoding are added as the input of the multi-head attention mechanism layer to obtain the output result of the multi-head attention mechanism layer, and the word vector and positional encoding are input into the first Add&Norm (connection & normalization) layer to perform residual connection and normalize the activation value. Further, the output result of the first Add&Norm layer and the output result of the multi-head attention mechanism layer are input into the feedforward layer to obtain the output result corresponding to the feedforward layer, and the output result of the first Add&Norm layer and the output result of the multi-head attention mechanism layer are input into the second Add&Norm layer again to obtain the intermediate semantic vector.
[0108] The process of further decoding the intermediate semantic vector through the decoder to obtain the translation result corresponding to the summary text is as follows: First, the output result of the encoder, that is, the intermediate semantic vector, is used as the input of the decoder (OutputEmbedding). Further, the intermediate semantic vector and the position encoding are added as the input of the first multi-head attention mechanism layer, and the multi-head attention mechanism layer is masked to obtain the output result. At the same time, the intermediate semantic vector and the position encoding are input into the first Add&Norm layer, residual connection is performed, and the activation value is normalized. Further, the output result of the first Add&Norm layer and the output result of the first multi-head attention mechanism layer after masking are input into the second multi-head attention mechanism layer, and the output result of the encoder is also input into the second multi-head attention mechanism layer to obtain the output result of the second multi-head attention mechanism layer. Further, the output result of the first Add&Norm layer and the masked result of the first multi-head attention mechanism layer are input into the second Add&Norm layer to obtain the output result of the second Add&Norm layer. Furthermore, the output of the second multi-head attention mechanism layer and the output of the second Add&Norm layer are input into the feedforward layer to obtain the output of the feedforward layer. At the same time, the output of the second multi-head attention mechanism layer and the output of the second Add&Norm layer are input into the third Add&Norm layer to obtain the output of the third Add&Norm layer. Furthermore, the output of the feedforward layer and the output of the third Add&Norm layer are linearly mapped (Linear) and normalized (Softmax), and finally the output of the decoder is obtained.
[0109] In the embodiment of the present application, the translation model is trained based on sample text pairs, and the training process is as follows: Figure 7 As shown in the figure, the main steps include data processing, model training and inference. Data processing is used to label sample text pairs or expand data.
[0110] In a possible implementation, the sample text pair may be composed of an existing sample listener text and a sample sign language text, as shown in Table 1.
[0111] Table 1
[0112]
[0113] Among them, in the sample sign language text, “ / ” is used to separate each phrase, and “ / / / ” is used to indicate major punctuation marks, such as a period, exclamation mark, question mark, etc., indicating the end of a sentence.
[0114] In another possible implementation, a sample listener text may be obtained by applying a back translation (BT) method to the sample sign language text, thereby obtaining a sample text pair.
[0115] For example, the sample sign language text is shown in Table II.
[0116] Table 2
[0117] Sample Sign Language Text I / want / to / be / a / programmer / hardworking / do / do / do / / a / month / a / little / / / There are many people who may be willing to be programmers and need to work hard to learn.
[0118] Among them, “ / / ” in the sample sign language text is used to indicate small punctuation marks, such as commas, colons, semicolons, etc.
[0119] First, the existing sample listener texts and sample sign language texts are used to train the sign language-Chinese translation model to obtain the trained sign language-Chinese translation model. Secondly, the sample sign language texts in Table 2 are input into the trained sign language-Chinese translation model to obtain the corresponding sample listener texts, and then the sample text pairs are obtained, as shown in Table 3.
[0120] Table 3
[0121]
[0122] The sample text pairs obtained by the above two methods are shown in Table IV.
[0123] Table 4
[0124]
[0125] Further, the computer device trains the translation model based on the sample text pairs shown in Table 4 to obtain a trained translation model. In addition, it should be noted that Table 4 illustrates the contents of the sample text pairs by way of example, and the sample text pairs for training the translation model also include other sample listener texts and corresponding sample sign language texts, which will not be described in detail in the embodiments of the present application.
[0126] Furthermore, the trained translation model is subjected to inference verification, that is, the sample human text is input into the trained translation model to obtain the translation results, as shown in Table 5.
[0127] Table 5
[0128]
[0129] Among them, the space in the translation result means to separate each phrase, and world 1 means it is unique in the world.
[0130] Step 450, obtaining sign language gesture information corresponding to each sign language vocabulary in the sign language text.
[0131] In an embodiment of the present application, after the computer device obtains the sign language text corresponding to the summary text based on the translation model, it further parses the sign language text into individual sign language words, such as eating, going to school, and giving a thumbs up. Sign language gesture information corresponding to each sign language word is established in advance in the computer device. Based on the mapping relationship between sign language words and sign language gesture information, the computer device matches each sign language word in the sign language text to the corresponding sign language gesture information. For example, the sign language gesture information that matches the sign language word "like" is: the thumb is raised upward, and the other four fingers are clenched.
[0132] Step 460: Control the virtual object to execute the sign language gestures in sequence based on the sign language gesture information.
[0133] The virtual object is a digital human image created in advance through 2D or 3D modeling, and each digital human image includes facial features, hairstyle features, body features, etc. Optionally, the digital human can be a simulated human image authorized by a real person, or a cartoon image, etc., which is not limited in the embodiments of the present application.
[0134] Exemplary, combined Figure 8 The process of establishing a virtual object in the embodiment of the present application is briefly described.
[0135] First, input image I (InputimageI), use a pre-trained shape reconstructor (Shapereconstructor) to predict the 3DMM (3D Morphable Model, 3D deformation model) parameters (3DMMcoefficients) and pose parameters p (Posecoefficientsp), and then get the 3DMM mesh (3DMMmesh). Then, use the shape transfer model (shapetransfer) to transform the topology of 3DMMmesh to the game, that is, get the game mesh (Gamemesh). At the same time, image I is decoded (Imageencoder) to further get latent features (Latentfeatures), and the lighting parameters l (Lightingcoefficientsl) are obtained based on the lighting predictor (Lightingpredictor).
[0136] Further, according to Gamemesh, the input image I is UV unwrapped to the UV space to obtain the coarse-grained texture C (CorsetextureC) of the image. Further, the coarse-grained texture C is texture encoded (Textureencoder), and the latent features are extracted, and the latent features of the image and the latent features of the texture are fused (concatenate). Further, texture decoding (Textureencoder) is performed to obtain the refined texture F (RefinedtextureF). The parameters corresponding to Gamemesh, Posecoefficientsp, Lightingcoefficientsl, and RefinedtextureF and other different parameters are input into the differentiable network rendering (DifferentiableRenderer) to obtain the rendered 2D image R (RenderfaceR). During the training process, in order to make the output 2D image R similar to the input image I, the image discriminator (Imagediscriminator) and the texture discriminator (Texturediscriminator) are introduced. The input image I and the 2D image R obtained after each training are judged as real or fake by the image discriminator. The ground truth texture G and the fine texture F obtained after each training are judged as real or fake by the texture discriminator.
[0137] Step 470, generating a sign language video based on the image of the virtual object performing the sign language gesture.
[0138] The computer device renders the virtual object's sign language gestures into individual frames, and splices the static frames into a coherent dynamic video according to the frame rate, thereby forming a video clip. The video clip corresponds to a clause in the sign language text. In order to further improve the color of the video clip, the computer device transcodes each video clip into the YUV format. Among them, YUV refers to a pixel format in which the brightness parameter and the chromaticity parameter are separately represented. Y represents brightness (Luminance), that is, the grayscale value, and U and V represent chromaticity (Chrominance), which is used to describe the color and saturation of the image.
[0139] Furthermore, the computer device splices the video clips to generate a sign language video.
[0140] In a possible implementation, when the listener text is offline text, the sign language video generation mode is offline video mode. After the computer device splices the video clips into a sign language video, the sign language video is stored in a cloud server. When the user needs to watch the sign language video, he needs to enter the storage path of the sign language video in the browser or download software to obtain the complete video.
[0141] In another possible implementation, when the listener text is real-time text, the sign language video generation mode is real-time mode. To avoid delay, the computer device sorts the video clips and pushes them to the user client frame by frame.
[0142] In the embodiments of the present application, text summarization is performed on hearing text in a variety of ways, with the aim of improving the synchronization between the finally generated sign language video and the corresponding audio. In addition, the summary text is converted into a sign language text that conforms to the grammatical structure of the hearing-impaired, and a sign language video is regenerated based on the sign language text, thereby improving the intelligibility of the sign language video to the hearing-impaired. The sign language video is automatically generated, which has a low implementation cost and can be applied to a large number of application scenarios.
[0143] In an embodiment of the present application, when the listener text is an offline text, the computer device can obtain a summary text by performing semantic analysis on the listener text to extract key sentences, or by performing text compression on the listener text to obtain a summary text, or by combining the above two methods to obtain a summary text.
[0144] The previous article has introduced the method that the computer device uses semantic analysis to extract key sentences from the listener's text to obtain the summary text. The following is an introduction to the method that the computer device uses text compression to obtain the summary text. Please refer to Fig. 9 , which shows a flowchart of a method for generating a summary text provided by another exemplary embodiment of the present application, the method comprising:
[0145] Step 901, segment the listener text into sentences to obtain text sentences.
[0146] Since the listener text is an offline text in the embodiment of the present application, the computer device can obtain the entire content of the listener text. In a possible implementation, the computer device divides the listener text into sentences based on punctuation marks to obtain text sentences. The punctuation marks can be punctuation marks such as a period, an exclamation mark, and a question mark that indicate the end of a sentence.
[0147] Exemplarily, the listener text is "The 2022 Winter Games will be held in XX. The mascot of this Winter Games is XXX. The slogan of this Winter Games is 'XXXXX'. I am looking forward to the arrival of the Winter Games". The computer device divides the above listener text into sentences to obtain 3 text sentences. The first text sentence S1 is "The 2022 Winter Games will be held in XX". The second text sentence S2 is "The mascot of this Winter Games is XXX". The third text sentence S3 is "The slogan of this Winter Games is 'XXXXX'". The fourth text sentence S4 is "I am looking forward to the arrival of the Winter Games".
[0148] Step 902: Determine the candidate compression ratio corresponding to each text sentence.
[0149] In a possible implementation, a plurality of candidate compression ratios are preset in the computer device, and the computer device may select a candidate compression ratio corresponding to each text sentence from the preset candidate compression ratios.
[0150] Optionally, the candidate compression ratios corresponding to the various text sentences may be the same or different, which is not limited in this embodiment of the present application.
[0151] Optionally, one text sentence corresponds to multiple candidate compression ratios.
[0152] Exemplarily, as shown in FIG. 6 , the computer device determines three candidate compression ratios for each of the aforementioned four text sentences.
[0153] Table 6
[0154] Text Statement Candidate compression ratio 1 Candidate compression ratio 2 Candidate compression ratio 3 S1 Y11 Y12 Y13 S2 Y21 Y22 Y23 S3 Y31 Y32 Y33 S4 Y41 Y42 Y43
[0155] Among them, Ymn is used for the candidate compression ratio n corresponding to the mth text sentence, for example, Y11 is used to represent the candidate compression ratio 1 corresponding to the first text sentence S1. In addition, in order to reduce the amount of calculation of the computer device, the candidate compression ratios selected for each text sentence are the same. For example, the computer device uses the candidate compression ratio 1 to perform text compression processing on the text sentences S1, S2, S3, and S4. It should be noted that the computer device can also use different candidate compression ratios to perform text compression processing on the text sentences S1, S2, S3, and S4, and the embodiment of the present application is not limited to this.
[0156] Step 903: Perform text compression processing on the text sentence based on the candidate compression ratio to obtain a candidate compressed sentence.
[0157] Exemplarily, the computer device performs text compression processing on text sentences S1, S1, S2, S3, and S4 based on candidate compression ratio 1, candidate compression ratio 2, and candidate compression ratio 3 determined in Table 6, respectively, to obtain candidate compressed sentences corresponding to each text sentence, as shown in Table 7.
[0158] Table 7
[0159]
[0160] Among them, Cmn is used to represent the candidate compressed sentence obtained by performing text compression processing on the mth text sentence with the candidate compression ratio n. For example, C11 is used to represent the candidate compressed sentence obtained by performing text compression processing on the first text sentence S1 with the candidate compression ratio 1.
[0161] Step 904: filter candidate compressed sentences whose semantic similarity with the text sentence is less than a similarity threshold.
[0162] In the embodiment of the present application, in order to ensure the consistency between the finally generated sign language video content and the content of the original hearing text and avoid interfering with the understanding of the hearing-impaired, in the embodiment of the present application, the computer device needs to perform semantic analysis on the candidate compression ratio and compare it with the semantics of the corresponding text sentence to determine the semantic similarity between the candidate compressed sentence and the corresponding text sentence, and filter out the candidate compression ratio that does not match the semantics of the text sentence.
[0163] In a possible implementation, when the semantic similarity is greater than or equal to a similarity threshold, it indicates that the candidate compressed sentence is highly likely similar to the corresponding text sentence, and the computer device retains the candidate compressed sentence.
[0164] In another possible implementation, when the semantic similarity is less than a similarity threshold, it indicates that the candidate compressed sentence is highly likely not similar to the corresponding text compressed sentence, and the computer device filters the candidate compressed sentence.
[0165] Optionally, the similarity threshold is 90%, 95%, 98%, etc., which is not limited in the embodiment of the present application.
[0166] Exemplarily, the computer device filters the candidate compressed sentences in Table 6 based on the similarity threshold to obtain filtered candidate compressed sentences, as shown in Table 8.
[0167] Table 8
[0168]
[0169] The deleted candidate compression statements represent candidate compression statements filtered by the computer device.
[0170] Step 905: Determine the candidate segment duration of the candidate sign language video segment corresponding to the filtered candidate compressed sentence.
[0171] In order to ensure that the time axis of the finally generated sign language video is aligned with the audio time axis of the audio corresponding to the listener text, the computer device first determines the duration of the candidate sign language video segment corresponding to the filtered compressed sentence. Exemplarily, as shown in Table 9, the computer device determines the duration of the candidate sign language video segment corresponding to the filtered candidate compressed sentence.
[0172] Table 9
[0173]
[0174] Among them, Tmn is used to represent the duration of the candidate sign language segment corresponding to the filtered candidate compressed sentence Cmn, and T1, T2, T3, and T4 represent the duration of the audio segments corresponding to the text sentences S1, S2, S3, and S4, respectively.
[0175] Step 906: Determine the duration of the audio segment corresponding to the text sentence based on the timestamp corresponding to the text sentence.
[0176] In the embodiment of the present application, the listener text includes a timestamp. In a possible implementation, the computer device obtains the timestamp corresponding to the listener text while obtaining the listener text, so as to subsequently perform synchronization alignment between the sign language video and the corresponding audio based on the timestamp. The timestamp is used to indicate the time interval of the audio corresponding to the listener text on the audio timeline.
[0177] For example, if the content of the listener text is "Hello, spring", the content of the audio timeline 00:00:00-00:00:70 corresponding to the audio is "Hello", and the content of 00:00:70-00:01:35 is "Spring". Among them, "00:00:00-00:00:70" and "00:00:70-00:01:35" are the timestamps corresponding to the listener text.
[0178] In the embodiment of the present application, since the computer device obtains the listener's text in different ways, the way it obtains the timestamp is also different.
[0179] Exemplarily, when the computer device directly obtains the listener text, it is necessary to convert the listener text into corresponding audio to obtain its corresponding timestamp. Exemplarily, the computer device can also directly extract the timestamp corresponding to the listener text from the subtitle file. Exemplarily, when the computer device obtains the timestamp from the audio file, it is necessary to first perform speech recognition on the audio file, and obtain the timestamp based on the results of speech recognition and the audio timeline. Exemplarily, when the computer device obtains the timestamp from the video file, it is necessary to first perform OCR recognition on the video file, and obtain the timestamp based on the text recognition results and the video timeline.
[0180] Therefore, it can be seen that in the embodiment of the present application, the computer device can obtain the audio segment corresponding to each text sentence based on the timestamp of the listener's text.
[0181] For example, as shown in Table 9, the audio segment length of the audio corresponding to text sentence S1 is T1, the audio segment length of the audio corresponding to text sentence S2 is T2, the audio segment length of the audio corresponding to text sentence S3 is T3, and the audio segment length of the audio corresponding to text sentence S4 is T4.
[0182] Step 907, based on the duration of the candidate sign language segments and the duration of the audio segments, a target compressed sentence is determined from the candidate compressed sentences through a dynamic path planning algorithm, wherein the video timeline of the sign language video corresponding to the text constituted by the target compressed sentence is aligned with the audio timeline of the audio corresponding to the listener text.
[0183] In a possible implementation, the computer device determines the target compressed sentence from the candidate compressed sentences corresponding to each text sentence based on the dynamic path planning algorithm, wherein the path nodes in the dynamic path planning algorithm are the candidate compressed sentences.
[0184] For example, in combination with Table 8 and Fig.10 , the process of the dynamic path planning algorithm is explained. In the dynamic path planning algorithm, each column of path nodes 1001 represents a different candidate compressed sentence of a text sentence. For example, the first column of path nodes 1001 is used to represent different candidate compressed sentences of the text sentence S1. The candidate texts and the video duration of the corresponding sign language videos obtained by the computer device based on the different candidate compressed sentence combinations obtained by the dynamic path planning algorithm are shown in Table 10, wherein the video duration of the sign language video corresponding to the candidate text is obtained by the duration of the candidate sign language video clips corresponding to each candidate compressed sentence.
[0185] Table 10
[0186]
[0187]
[0188] Furthermore, the computer device obtains the timeline of the sign language video corresponding to the candidate text based on the duration of the sign language video corresponding to the candidate text, and matches the audio timeline of the audio corresponding to the listener text, i.e., the text sentences S1, S2, S3 and S4 combination. If the two are aligned, the target candidate text is determined, and the target compressed sentence is determined based on the target candidate text. Then, the computer device determines the target compressed sentence based on the dynamic path planning algorithm. Fig.10 In the example, the target compression statements determined by the computer device based on the dynamic path planning algorithm are C12, C23, C31 and C41.
[0189] Step 908: determine the text composed of the target compressed sentence as a summary text.
[0190] Exemplarily, the computer device determines the text consisting of the target compressed sentence, ie, C12+C23+C31+C41, as the summary text.
[0191] In an embodiment of the present application, a computer device determines a target compressed sentence from candidate compressed sentences based on a similarity threshold and a dynamic path planning algorithm, and then obtains a summary text, thereby shortening the text length of the listener's text, avoiding the problem of the final generated sign language video and its corresponding audio being out of sync, and improving the synchronization of the sign language video and audio.
[0192] In addition, in a possible implementation, when the listener text is an offline text, the computer device can combine a method of extracting key sentences from the listener text with a method of compressing the listener text according to a compression ratio to obtain a summary text. For example, Fig.11 As shown. First, the computer device obtains the listener text and the corresponding timestamp of the video file based on the speech recognition method. Secondly, the computer device performs text summary processing on the listener text. The computer device performs semantic analysis on the listener text, extracts key sentences from the listener text based on the results of the semantic analysis, and obtains the extraction results in Table 1101. The key sentences are text sentences S1 to S2 and text sentences S5 to Sn. At the same time, the computer device performs sentence segmentation processing on the listener text to obtain text sentences S1 to Sn. Further, the computer device performs text compression processing on the text sentences based on the candidate compression ratio to obtain candidate compressed sentences, and obtains compressed results 1 to compressed results m in Table 1101. Among them, Cnm is used to represent candidate compressed sentences.
[0193] Further, the computer device determines the target compressed sentences Cn1, ..., C42, C31, C2m, C11 from the table 1101 based on the dynamic path planning algorithm 1102, wherein the video time axis of the sign language video corresponding to the text constituted by the target compressed sentence is aligned with the audio time axis of the audio corresponding to the listener text. A summary text is generated based on the target compressed sentence. Further, the summary text is translated into sign language to obtain a sign language text, and a sign language video is generated based on the sign language text. Since the listener file is processed for text summarization, the time axis 1104 of the sign language video generated finally is aligned with the audio time axis 1103 of the audio corresponding to the video.
[0194] In the embodiment of the present application, when the listener text is real-time text, the computer device obtains the listener text sentence by sentence, but cannot obtain the entire content of the listener text. Therefore, it is impossible to obtain the summary text by extracting key sentences through semantic analysis of the listener text. In order to reduce the delay, the computer device compresses the listener text according to a fixed compression ratio to obtain the summary text. The method is introduced below:
[0195] 1. Determine the target compression ratio based on the application scenario corresponding to the listener text.
[0196] The target compression ratio is related to the application scenario corresponding to the listener text, and different application scenarios determine different target compression ratios.
[0197] For example, when the application scenario corresponding to the listener text is an interview scenario, since the language used in the listener text in the interview scenario is more colloquial and contains less effective information, the target compression ratio is determined to be a high compression ratio, such as 0.8.
[0198] For example, when the application scenario corresponding to the listener text is a news broadcast scenario or a press conference scenario, the listener text is more concise and contains more effective information, so the target compression ratio is determined to be a low compression ratio, such as 0.4.
[0199] 2. Perform text compression on the listener text based on the target compression ratio to obtain a summary text.
[0200] The computer device compresses the listener text sentence by sentence according to the determined target compression ratio to obtain a summary text.
[0201] In an embodiment of the present application, when the listener text is real-time text, the computer device performs text compression processing on the listener text based on the target compression ratio, shortens the text length of the listener text, and improves the synchronization between the final generated sign language video and its corresponding audio. In addition, different target compression ratios are determined for different application scenarios to improve the accuracy of the final generated sign language video.
[0202] Please refer to Fig.12 , which shows a flow chart of a method for generating a sign language video provided by an exemplary embodiment of the present application. In the implementation of the present application, the method for generating a sign language video includes obtaining a listener text, text summary processing, sign language translation processing, and sign language video generation.
[0203] The first step is to obtain the listener text. The program video source includes audio files, video files, prepared listener text and subtitle files. Taking audio files and video files as examples, for audio files, the computer device extracts audio to obtain the broadcast audio. Further, the computer device processes the broadcast audio through speech recognition technology to obtain the listener text and the corresponding timestamp; for video files, the computer device extracts the listener text and the corresponding timestamp corresponding to the video based on OCR technology.
[0204] The second step is text summary processing. The computer device performs text summary processing on the listener text to obtain a summary text. The processing method includes extracting key sentences based on semantic analysis of the listener text and performing text compression processing on the listener text after dividing the text into sentences. In addition, the type of the listener text is different, and the method of text summary processing by the computer device on the listener text is different. When the type of the listener text is offline text, the computer device can use the method of extracting key sentences based on semantic analysis of the listener text to perform text summary processing on the listener text, or use the method of performing text compression processing on the listener text after dividing the text into sentences to perform text summary processing on the listener text, or it can be a combination of the above two methods. When the type of the listener text is real-time text, the computer device can only use the method of performing text compression processing on the listener text after dividing the text into sentences to perform text summary processing on the listener text.
[0205] The third step is sign language translation processing. The computer device generates a sign language text by using sign language translation based on the summary text generated by the text summary processing.
[0206] Step 4: Generate sign language video. Sign language video is generated in different ways in different modes. In offline mode, the computer device needs to segment the sign language text into sentences and synthesize sentence videos based on text sentences; further, perform 3D rendering on the sentence videos; further, perform video encoding; finally, perform file synthesis on the video encoding files of all sentences to generate the final sign language video. Furthermore, the computer device stores the sign language video in the cloud server, and when the user needs to watch the sign language video, he can download it from the computer device.
[0207] In real-time mode, the computer device does not segment the text sentences of the listener, but requires multiple live broadcasts to be concurrent, thereby reducing latency. The computer device synthesizes sentence videos based on the sign language text; further, 3D rendering is performed on the sentence videos; further, video encoding is performed to generate a video stream. The computer device pushes the video stream to generate a sign language video.
[0208] Please refer to Fig.13 , which shows a structural block diagram of a device for generating a sign language video provided by an exemplary embodiment of the present application. The device may include:
[0209] An acquisition module 1301 is used to acquire a hearing person text, where the hearing person text is a text that conforms to the grammatical structure of a hearing person;
[0210] An extraction module 1302 is used to extract a summary of the listener text to obtain a summary text, wherein the text length of the summary text is shorter than the text length of the listener text;
[0211] A conversion module 1303 is used to convert the summary text into a sign language text, where the sign language text is a text that conforms to the grammatical structure of the hearing-impaired;
[0212] The generating module 1304 is used to generate the sign language video based on the sign language text, and the sign language video is synchronized with the audio corresponding to the listener text.
[0213] Optionally, the extraction module 1302 is used to:
[0214] Performing semantic analysis on the listener text; extracting key sentences from the listener text based on the semantic analysis result, wherein the key sentences are sentences in the listener text that affect the semantics of the entire text; and determining the key sentences as the summary text;
[0215] The listener text is compressed; and the compressed listener text is determined as the summary text.
[0216] Optionally, the extraction module 1302 is used to:
[0217] In the case where the listener text is an offline text, performing semantic analysis on the listener text;
[0218] The step of performing text compression processing on the listener text comprises:
[0219] When the listener text is an offline text or the listener text is a real-time text, text compression processing is performed on the listener text.
[0220] Optionally, the extraction module 1302 is used to:
[0221] In the case where the listener text is an offline text, segmenting the listener text into sentences to obtain text sentences;
[0222] Determining a candidate compression ratio corresponding to each of the text sentences;
[0223] Performing text compression processing on the text sentence based on the candidate compression ratio to obtain a candidate compressed sentence;
[0224] The extraction module 1302 is used to:
[0225] Determining a target compressed sentence from the candidate compressed sentences corresponding to each of the text sentences based on a dynamic path planning algorithm, wherein the path nodes in the dynamic path planning algorithm are the candidate compressed sentences;
[0226] A text composed of the target compressed sentence is determined as the summary text.
[0227] Optionally, the listener text includes a corresponding timestamp, and the timestamp is used to indicate a time interval of the audio corresponding to the listener text on the audio time axis;
[0228] The extraction module 1302 is used to:
[0229] Determining the candidate segment duration of the candidate sign language video segment corresponding to the candidate compressed sentence;
[0230] Determining the duration of an audio segment of the audio corresponding to the text sentence based on the timestamp corresponding to the text sentence;
[0231] Based on the duration of the candidate segments and the duration of the audio segments, the target compressed sentence is determined from the candidate compressed sentences by the dynamic path planning algorithm, wherein the video timeline of the sign language video corresponding to the text constituted by the target compressed sentence is aligned with the audio timeline of the audio corresponding to the listener text.
[0232] Optionally, the device further comprises:
[0233] A filtering module, used for filtering the candidate compressed sentences whose semantic similarity with the text sentence is less than a similarity threshold;
[0234] The extraction module 1302 is used to:
[0235] Determine the candidate segment duration of the candidate sign language video segment corresponding to the candidate compressed sentence after filtering.
[0236] Optionally, the extraction module 1302 is used to:
[0237] In the case that the listener text is a real-time text, text compression processing is performed on the listener text based on a target compression ratio.
[0238] Optionally, the device further comprises:
[0239] The determination module is used to determine the target compression ratio based on the application scenario corresponding to the listener text, wherein different application scenarios correspond to different compression ratios.
[0240] Optionally, the conversion module 1303 is used to:
[0241] The summary text is input into a translation model to obtain the sign language text output by the translation model, wherein the translation model is trained based on a sample text pair, and the sample text pair consists of a sample sign language text and a sample listener text.
[0242] Optionally, the generating module 1304 is used to:
[0243] Obtaining sign language gesture information corresponding to each sign language vocabulary in the sign language text;
[0244] Controlling the virtual object to perform the sign language gestures in sequence based on the sign language gesture information;
[0245] The sign language video is generated based on a picture of the virtual object performing the sign language gesture.
[0246] Optionally, the acquisition module 1301 is used to:
[0247] Obtaining the inputted listener text;
[0248] Acquire a subtitle file; extract the listener text from the subtitle file;
[0249] Acquire an audio file; perform speech recognition on the audio file to obtain a speech recognition result; generate the listener text based on the speech recognition result;
[0250] Acquire a video file; perform OCR recognition on the video frame of the video file to obtain a text recognition result; and generate the listener text based on the text recognition result.
[0251] In summary, in the embodiment of the present application, a text summary is extracted from the hearing text to obtain a summary text, thereby shortening the text length of the hearing text, so that the finally generated sign language video is synchronized with the audio corresponding to the hearing text. In addition, the summary text is converted into a sign language text that conforms to the grammatical structure of the hearing-impaired, and a sign language video is generated based on the sign language text, which improves the intelligibility of the sign language video to the hearing-impaired, and the sign language video is automatically generated based on the sign language text, without the need for manual signing, and the implementation cost is low, and it can be applied to a large number of application scenarios, such as live video scenarios, offline video scenarios, etc.
[0252] It should be noted that: the device provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0253] Fig.14 14 is a schematic diagram of a computer device according to an exemplary embodiment. The computer device 1400 includes a central processing unit (CPU) 1401, a system memory 1404 including a random access memory (RAM) 1402 and a read-only memory (ROM) 1403, and a system bus 1405 connecting the system memory 1404 and the central processing unit 1401. The computer device 1400 also includes a basic input / output system (I / O system) 1406 for helping to transmit information between various devices in the computer device, and a large-capacity storage device 1407 for storing an operating system 1413, an application program 1414 and other program modules 1415.
[0254] The basic input / output system 1406 includes a display 1408 for displaying information and an input device 1409 such as a mouse and a keyboard for user inputting information. The display 1408 and the input device 1409 are connected to the central processing unit 1401 through an input / output controller 1410 connected to the system bus 1405. The basic input / output system 1406 may also include an input / output controller 1410 for receiving and processing inputs from a plurality of other devices such as a keyboard, a mouse, or an electronic stylus. Similarly, the input / output controller 1410 also provides output to a display screen, a printer, or other types of output devices.
[0255] The mass storage device 1407 is connected to the central processing unit 1401 through a mass storage controller (not shown) connected to the system bus 1405. The mass storage device 1407 and its associated computer device readable medium provide non-volatile storage for the computer device 1400. That is, the mass storage device 1407 may include a computer device readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.
[0256] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer device readable instructions, data structures, program modules or other data. Computer device storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM, Digital Video Disc (DVD) or other optical storage, cassette, tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will know that the computer device storage medium is not limited to the above. The above-mentioned system memory 1404 and mass storage device 1407 can be collectively referred to as memory.
[0257] According to various embodiments of the present disclosure, the computer device 1400 can also be connected to a remote computer device on a network through a network such as the Internet. That is, the computer device 1400 can be connected to the network 1411 through the network interface unit 1412 connected to the system bus 1405, or the network interface unit 1415 can be used to connect to other types of networks or remote computer device systems (not shown).
[0258] The memory also includes one or more programs, which are stored in the memory. The central processing unit 1401 implements all or part of the steps of the above-mentioned method for generating sign language video by executing the one or more programs.
[0259] In an embodiment of the present application, a computer-readable storage medium is further provided, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the method for generating a sign language video as described in the above aspects.
[0260] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for generating a sign language video provided in various optional implementations of the above aspects.
[0261] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and embodiments are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0262] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A method for generating a sign language video, characterized in that: The method comprises: Acquire a hearing text, wherein the hearing text is a text that conforms to the grammatical structure of a hearing person; Extracting a summary of the listener text to obtain a summary text, wherein the text length of the summary text is shorter than the text length of the listener text; Converting the summary text into a sign language text, wherein the sign language text is a text that conforms to the grammatical structure of hearing-impaired persons; Generate the sign language video based on the sign language text, wherein the sign language video is synchronized with the audio corresponding to the listener text; The summary text is extracted from the listener text to obtain a summary text, including at least one of the following: Performing semantic analysis on the listener text; extracting key sentences from the listener text based on the semantic analysis result, wherein the key sentences are sentences in the listener text that affect the semantics of the entire text; and determining the key sentences as the summary text; Performing text compression processing on the listener text; determining the compressed listener text as the summary text, wherein the summary text is composed of a target compression sentence, the target compression sentence is determined from candidate compression sentences corresponding to each text sentence based on a dynamic path planning algorithm, the path nodes in the dynamic path planning algorithm are the candidate compression sentences, the text sentence is obtained by sentence segmentation of the listener text, the same text sentence corresponds to multiple candidate compression sentences, and multiple candidate compression sentences are obtained by performing text compression processing based on multiple candidate compression ratios corresponding to the text sentence, and the video timeline of the sign language video corresponding to the text composed of the target compression sentence is aligned with the audio timeline of the audio corresponding to the listener text.
2. The method according to claim 1, characterized in that The semantic analysis of the listener text includes: In the case where the listener text is an offline text, performing semantic analysis on the listener text; The step of compressing the listener text comprises: When the listener text is an offline text or the listener text is a real-time text, text compression processing is performed on the listener text.
3. The method according to claim 2, characterized in that When the listener text is an offline text, performing text compression processing on the listener text includes: In the case where the listener text is an offline text, segmenting the listener text into sentences to obtain the text sentences; Determine the candidate compression ratio corresponding to each of the text sentences; Performing text compression processing on the text sentence based on the candidate compression ratio to obtain the candidate compressed sentence; The step of determining the compressed listener text as the summary text comprises: Determine the target compressed sentence from the candidate compressed sentences corresponding to each of the text sentences based on a dynamic path planning algorithm; A text composed of the target compressed sentence is determined as the summary text.
4. The method according to claim 3, characterized in that The listener text includes a corresponding timestamp, and the timestamp is used to indicate the time interval of the audio corresponding to the listener text on the audio time axis; The step of determining the target compressed sentence from the candidate compressed sentences corresponding to each of the text sentences based on the dynamic path planning algorithm includes: Determining the candidate segment duration of the candidate sign language video segment corresponding to the candidate compressed sentence; Determining the duration of an audio segment of the audio corresponding to the text sentence based on the timestamp corresponding to the text sentence; Based on the duration of the candidate segments and the duration of the audio segments, the target compressed sentence is determined from the candidate compressed sentences by the dynamic path planning algorithm.
5. The method according to claim 4, characterized in that Before determining the candidate segment duration of the candidate sign language video segment corresponding to the candidate compressed sentence, the method further includes: Filtering the candidate compressed sentences whose semantic similarity with the text sentence is less than a similarity threshold; The determining of the candidate segment duration of the candidate sign language video segment corresponding to the candidate compressed sentence includes: Determine the candidate segment duration of the candidate sign language video segment corresponding to the candidate compressed sentence after filtering.
6. The method according to claim 2, characterized in that When the listener text is a real-time text, performing text compression processing on the listener text includes: In the case that the listener text is a real-time text, text compression processing is performed on the listener text based on a target compression ratio.
7. The method according to claim 6, characterized in that Before performing text compression processing on the listener text based on the target compression ratio, the method further includes: The target compression ratio is determined based on an application scenario corresponding to the listener text, wherein different application scenarios correspond to different compression ratios.
8. The method according to any one of claims 1 to 7, characterized in that: The step of converting the summary text into a sign language text comprises: The summary text is input into a translation model to obtain the sign language text output by the translation model, wherein the translation model is trained based on a sample text pair, and the sample text pair consists of a sample sign language text and a sample listener text.
9. The method according to any one of claims 1 to 7, characterized in that: The generating the sign language video based on the sign language text comprises: Obtaining sign language gesture information corresponding to each sign language vocabulary in the sign language text; Controlling the virtual object to perform the sign language gestures in sequence based on the sign language gesture information; The sign language video is generated based on a picture of the virtual object performing the sign language gesture.
10. The method according to any one of claims 1 to 7, characterized in that: The method of obtaining the listener text includes at least one of the following methods: Obtaining the inputted listener text; Acquire a subtitle file; extract the listener text from the subtitle file; Acquire an audio file; perform speech recognition on the audio file to obtain a speech recognition result; generate the listener text based on the speech recognition result; Get the video file; Perform OCR recognition on the video frame of the video file to obtain a text recognition result; and generate the listener text based on the text recognition result.
11. A device for generating a sign language video, characterized in that: The device comprises: An acquisition module, used for acquiring a hearing person text, wherein the hearing person text is a text that conforms to the grammatical structure of a hearing person; An extraction module, used for extracting a summary of the listener text to obtain a summary text, wherein the text length of the summary text is shorter than the text length of the listener text; A conversion module, used to convert the summary text into a sign language text, wherein the sign language text is a text that conforms to the grammatical structure of the hearing-impaired; A generating module, configured to generate the sign language video based on the sign language text, wherein the sign language video is synchronized with the audio corresponding to the listener text; Wherein, the extraction module is used to: Performing semantic analysis on the listener text; extracting key sentences from the listener text based on the semantic analysis result, wherein the key sentences are sentences in the listener text that affect the semantics of the entire text; and determining the key sentences as the summary text; Performing text compression processing on the listener text; determining the compressed listener text as the summary text, wherein the summary text is composed of a target compression sentence, the target compression sentence is determined from candidate compression sentences corresponding to each text sentence based on a dynamic path planning algorithm, the path nodes in the dynamic path planning algorithm are the candidate compression sentences, the text sentence is obtained by sentence segmentation of the listener text, the same text sentence corresponds to multiple candidate compression sentences, and multiple candidate compression sentences are obtained by performing text compression processing based on multiple candidate compression ratios corresponding to the text sentence, and the video timeline of the sign language video corresponding to the text composed of the target compression sentence is aligned with the audio timeline of the audio corresponding to the listener text.
12. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method for generating a sign language video as described in any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that: The readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method for generating a sign language video as described in any one of claims 1 to 10.
14. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; a processor reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for generating a sign language video as described in any one of claims 1 to 10.
