Sign language generation method and device, electronic equipment, storage medium and product

Abstract

The invention provides a sign language generation method and device, electronic equipment, a storage medium and a product, and the method comprises the steps: carrying out the emotion classification of each audio frame in the voice audio of a speaker, and determining an emotion feature sequence corresponding to the voice audio; and based on the voice audio and the emotion feature sequence, adjusting the facial action and the hand action of the speaker in the speaker action picture frame, and generating a speaker sign language action picture sequence corresponding to the voice audio. By adopting the technical scheme of the invention, the voice audio can be combined with the emotional characteristics of the voice audio to adjust the facial actions and hand actions of the speaker in the action picture frame of the speaker, so that the sign language actions and facial expressions of the speaker have the emotional characteristics, and the emotion degree of sign language generation is improved. Besides, the styles of the facial actions and the hand actions of the speaker in the speaker sign language action picture sequence are the same as the sign language expression style of the hearing-impaired person, so that the sign language generation accuracy is improved, and the hearing-impaired person can understand the sign language more conveniently.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a sign language generation method, device, electronic device, storage medium and product. Background Art

[0002] With the development of current technology, more and more applications have emerged in translating the speaker's voice or text into sign language movements, generating corresponding fluent and accurate sign language videos for hearing-impaired people to communicate. The existing sign language generation method is to obtain the sign language videos recorded by standard sign language teachers or sign language announcers to build a sign language movement database, and generate the sign language corresponding to the audio by learning the sign language movements in the sign language movement database. However, the generated sign language may be different from the hearing-impaired people who are currently communicating, resulting in low accuracy of sign language generation. In addition, the rhythm of the generated sign language is monotonous and dull, and cannot reflect the emotions and rhythm in the audio, resulting in low emotionality of the generated sign language. Summary of the invention

[0003] Based on the above needs, the present application proposes a sign language generation method, device, electronic device, storage medium and product, which can improve the accuracy and emotionality of sign language generation.

[0004] To achieve the above objectives, this application proposes the following technical solutions:

[0005] According to a first aspect of an embodiment of the present application, a sign language generation method is provided, comprising:

[0006] Performing emotion classification on each audio frame in the speaker's speech audio to determine an emotion feature sequence corresponding to the speech audio;

[0007] Based on the speech audio and the emotion feature sequence, adjusting the speaker's facial movements and hand movements in the speaker action picture frame to generate a speaker sign language action picture sequence corresponding to the speech audio;

[0008] The style of the speaker's facial movements and hand movements in the speaker's sign language movement picture sequence is the same as the sign language expression style of the hearing-impaired person.

[0009] Optionally, the method further includes:

[0010] Performing face and hand masking processing on the speaker action picture frame to obtain a masked picture of the speaker, and performing facial recognition on the speaker action picture frame to obtain facial identity features of the speaker;

[0011] Based on the speech audio and the emotion feature sequence, adjusting the speaker's facial movements and hand movements in the speaker action picture frame to generate a speaker sign language action picture sequence corresponding to the speech audio, including:

[0012] Based on the speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity feature of the speaker, a predetermined sign language action generation algorithm is used to generate the speaker's facial actions and hand actions in the mask image of the speaker, and obtain a speaker sign language action image sequence corresponding to the speech audio;

[0013] The sign language movement generation algorithm is determined by learning and generating sign language movements using pre-recorded sign language movement videos of hearing-impaired persons.

[0014] Optionally, based on the speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity feature of the speaker, a predetermined sign language action generation algorithm is used to generate the speaker's facial actions and hand actions in the mask image of the speaker, and obtain a speaker sign language action image sequence corresponding to the speech audio, including:

[0015] Based on the speech audio and the emotion feature sequence, determining a rhythm amplitude feature corresponding to the speech audio;

[0016] Based on the mask image of the speaker, the facial identity features of the speaker and the rhythm amplitude features corresponding to the speech audio, the speaker's facial movements and hand movements in the mask image of the speaker are generated to obtain a speaker sign language movement image sequence corresponding to the speech audio.

[0017] Optionally, based on the mask image of the speaker, the facial identity features of the speaker, and the rhythm amplitude features corresponding to the speech audio, the facial movements and hand movements of the speaker in the mask image of the speaker are generated to obtain a sequence of speaker sign language movement images corresponding to the speech audio, including:

[0018] Determining the action optical flow feature corresponding to the speech audio based on the rhythm amplitude feature corresponding to the speech audio;

[0019] Based on the mask image of the speaker, the facial identity features of the speaker and the action optical flow features corresponding to the speech audio, the speaker's facial movements and hand movements in the mask image of the speaker are generated to obtain a speaker sign language action image sequence corresponding to the speech audio.

[0020] Optionally, based on the speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity feature of the speaker, a predetermined sign language action generation algorithm is used to generate the speaker's facial actions and hand actions in the mask image of the speaker, and obtain a speaker sign language action image sequence corresponding to the speech audio, including:

[0021] Inputting the speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity feature of the speaker into a pre-trained sign language action generation model to obtain a speaker sign language action image sequence corresponding to the speech audio;

[0022] The sign language action generation model is obtained by training the sign language action generation based on pre-recorded sign language action videos of hearing-impaired persons.

[0023] Optionally, the sign language action generation model includes: a rhythm amplitude feature encoder, an optical flow feature encoder and a sign language action generator;

[0024] The speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity feature of the speaker are input into a pre-trained sign language action generation model to obtain a speaker sign language action image sequence corresponding to the speech audio, including:

[0025] Inputting the speech audio and the emotion feature sequence into the rhythm amplitude feature encoder to obtain the rhythm amplitude feature corresponding to the speech audio;

[0026] Inputting the rhythm amplitude feature corresponding to the speech audio into the optical flow feature encoder to obtain the action optical flow feature corresponding to the speech audio;

[0027] The mask image of the speaker, the facial identity features of the speaker and the action optical flow features corresponding to the speech audio are input into the sign language action generator to obtain a speaker sign language action image sequence corresponding to the speech audio.

[0028] Optionally, the training process of the sign language action generation model includes:

[0029] Obtain sign language action videos corresponding to sample audios of hearing-impaired persons in different emotional states and sample emotional features corresponding to the sample audios;

[0030] Masking the face and hands of the video frames in the sign language action video to obtain a sample masked image of the hearing-impaired person, and performing facial recognition on the video frames in the sign language action video to obtain sample facial identity features of the hearing-impaired person;

[0031] Inputting the sample audio, the sample emotion feature, the sample mask image of the hearing-impaired person, and the sample facial identity feature of the hearing-impaired person into a pre-built sign language action generation model to obtain a predicted sign language action image sequence corresponding to the sample audio;

[0032] Based on the difference between the predicted sign language action picture sequence corresponding to the sample audio and the sample sign language action picture sequence, the model parameters of the sign language action generation model are adjusted; wherein the sample sign language action picture sequence is obtained by extracting frames from the sign language action video corresponding to the sample audio.

[0033] Optionally, the sample audio, the sample emotion feature, the sample mask image of the hearing-impaired person, and the sample facial identity feature of the hearing-impaired person are input into a pre-built sign language action generation model to obtain a predicted sign language action image sequence corresponding to the sample audio, including:

[0034] Inputting the sample audio and the sample emotion feature into a rhythm amplitude feature encoder to obtain a sample rhythm amplitude feature corresponding to the sample audio, and training the rhythm amplitude feature encoder based on the difference between the annotated rhythm amplitude feature and the sample rhythm amplitude feature; wherein the annotated rhythm amplitude feature is a rhythm amplitude feature manually annotated for the sample audio;

[0035] Inputting the sample rhythm amplitude feature corresponding to the sample audio into an optical flow feature encoder to obtain the sample action optical flow feature corresponding to the sample audio;

[0036] Inputting the sample action optical flow features, the sample mask image of the hearing-impaired person, and the sample facial identity features of the hearing-impaired person into a sign language action generator to obtain a predicted sign language action image sequence corresponding to the sample audio;

[0037] Based on the difference between the sample action optical flow features and the annotated optical flow features, and the difference between the predicted sign language action picture sequence and the sample sign language action picture sequence, the optical flow feature encoder and the sign language action generator are jointly trained; wherein the annotated optical flow features are obtained by extracting optical flow features from the sign language action video corresponding to the sample audio.

[0038] According to a second aspect of an embodiment of the present application, a sign language generating device is provided, comprising:

[0039] The emotion classification module is used to perform emotion classification on each audio frame in the speaker's speech audio and determine the emotion feature sequence corresponding to the speech audio;

[0040] A sign language generation module, configured to adjust the speaker's facial movements and hand movements in the speaker's action picture frame based on the speech audio and the emotion feature sequence, and generate a speaker's sign language action picture sequence corresponding to the speech audio;

[0041] The style of the speaker's facial movements and hand movements in the speaker's sign language movement picture sequence is the same as the sign language expression style of the hearing-impaired person.

[0042] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including: a memory and a processor;

[0043] The memory is connected to the processor and is used to store programs;

[0044] The processor is used to implement the above-mentioned sign language generation method by running the program in the memory.

[0045] According to a fourth aspect of an embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned sign language generation method is implemented.

[0046] According to a fifth aspect of an embodiment of the present application, there is provided a computer program product, comprising computer program instructions, which, when executed by a processor, enable the processor to implement the above-mentioned sign language generation method.

[0047] The sign language generation method proposed in the present application performs emotion classification on each audio frame in the speaker's speech audio to determine the emotion feature sequence corresponding to the speech audio; based on the speech audio and the emotion feature sequence, the speaker's facial movements and hand movements in the speaker's action picture frame are adjusted to generate a speaker's sign language action picture sequence corresponding to the speech audio; wherein the style of the speaker's facial movements and hand movements in the speaker's sign language action picture sequence is the same as the sign language expression style of the hearing-impaired person. By adopting the technical solution of the present application, the speaker's facial movements and hand movements in the speaker's action picture frame can be adjusted by combining the speech audio with the emotion features of the speech audio, so that the speaker's sign language movements and facial expressions have emotion features, thereby improving the emotion level of sign language generation. In addition, the style of the speaker's facial movements and hand movements in the speaker's sign language action picture sequence is the same as the sign language expression style of the hearing-impaired person, thereby improving the accuracy of sign language generation and making it easier for the hearing-impaired person to understand. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0049] Figure 1 A flowchart of a sign language generation method provided in an embodiment of the present application;

[0050] Figure 2 A flowchart of another sign language generation method provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of a processing flow for determining a speaker's sign language action picture sequence provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of a processing flow for training a sign language action generation model provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of a processing flow for determining a predicted sign language action picture sequence provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of the structure of a sign language generating device provided in an embodiment of the present application;

[0055] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The technical solution of the embodiment of the present application is applicable to the application scenario of generating actions corresponding to audio, and is specifically used in the application scenario of generating sign language actions. By adopting the technical solution of the embodiment of the present application, the facial actions and hand actions of the speaker in the speaker action picture frame can be adjusted by combining the speech audio with the emotional characteristics of the speech audio, so that the speaker's sign language actions and facial expressions have emotional characteristics, thereby improving the emotionality of the sign language generation. In addition, the style of the speaker's facial actions and hand actions in the speaker's sign language action picture sequence is the same as the sign language expression style of the hearing-impaired person, which improves the accuracy of the sign language generation and makes it easier for the hearing-impaired person to understand.

[0057] With the development of current technology, more and more applications have emerged that translate the speaker's voice or text into sign language movements, generating corresponding fluent and accurate sign language videos for the hearing-impaired to communicate.

[0058] The current mainstream solutions have the following characteristics:

[0059] First, in the 3D solution, neural networks are often used to generate 3D human skeleton points or various feature points, and finally 3D animation is rendered through the animation engine as the final form of expression. Low-cost animation rendering models are difficult to achieve accurate rhythm, amplitude, and facial expression driving. The corresponding labor costs, equipment costs, and time costs of highly realistic 3D animation are extremely high.

[0060] Second, in the two-dimensional solution, the sign language movement generation of two-dimensional animation mostly obtains sign language videos recorded by standard sign language teachers or sign language announcers to build a sign language movement database, and generates sign language corresponding to the audio by learning the sign language movements in the sign language movement database. However, the generated sign language may be different from the hearing-impaired person who is currently communicating, resulting in low accuracy of sign language generation. In addition, the rhythm of the generated sign language is monotonous and dull, and cannot reflect the emotion and rhythm in the audio, resulting in low emotionality of the sign language generation.

[0061] Based on this, the present application proposes a sign language generation method. This technical solution can adjust the speaker's facial movements and hand movements in the speaker's action picture frame by combining the speech audio with the emotional characteristics of the speech audio, so that the speaker's sign language movements and facial expressions have emotional characteristics, and the style of the speaker's facial movements and hand movements in the speaker's sign language action picture sequence is the same as the sign language expression style of the hearing-impaired, thereby solving the problem of low accuracy and emotionality of sign language generation in the prior art.

[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] Exemplary Methods

[0064] See also Figure 1 As shown, the embodiment of the present application proposes a method for generating sign language. The method includes:

[0065] S101, performing emotion classification on each audio frame in the speaker's speech audio to determine an emotion feature sequence corresponding to the speech audio.

[0066] In order to achieve normal communication between the speaker and the hearing-impaired, it is necessary to generate sign language movements with the same meaning as the speech audio spoken by the speaker, so that the hearing-impaired can understand what the speaker wants to express based on the sign language movements with the same meaning as the speech audio when they cannot hear the speech audio spoken by the speaker. However, when the speaker speaks, he or she expresses emotions. Even if the content spoken is the same, different emotions may express different meanings. For example, when the speaker says "there is a cat", if he or she says it with a happy emotion, the content expressed includes love for cats. If he or she says it with a fearful emotion, the content expressed includes fear of cats. Therefore, in order to improve the expression accuracy of the corresponding sign language movements generated according to the speaker's speech audio, it is also necessary to integrate the emotions in the speaker's speech audio into the sign language movements so that the hearing-impaired can understand the content that the speaker wants to express more accurately.

[0067] Therefore, this embodiment needs to perform emotion classification on each audio frame in the speaker's speech audio, determine the emotion to be expressed by each audio frame in the speech audio, thereby obtaining the emotion features corresponding to each audio frame in the speech audio, and then collect the emotion features corresponding to all audio frames according to the time sequence of the audio frames in the speech audio to obtain the emotion feature sequence corresponding to the speaker's speech audio.

[0068] In this embodiment, the audio frames are subjected to emotion classification. First, the audio frames need to be preprocessed, such as noise reduction processing, standardization processing, etc., to improve the accuracy of the subsequent extraction of sound features corresponding to the audio frames. Then, the preprocessed audio frames are subjected to sound feature extraction to extract emotion-related sound features in the audio frames, such as pitch, speech rate, spectrum, Mel-spectral coefficient (MFCC), linear prediction coding (LPC), etc. These features can reflect the emotional information in the audio frames. Finally, the sound features corresponding to the audio frames are subjected to emotion classification to determine the emotional features of the audio frames, wherein the emotional features of the audio frames can reflect the emotional category of the audio frames, etc. In this embodiment, a pre-trained emotion classification model can be used to perform emotion classification on the sound features corresponding to the audio frames. The emotion classification model can be trained using pre-collected sample audio, and each sample audio frame in the sample audio is manually annotated with emotion features in advance as the emotion label of each sample audio frame. Then, the sample audio frame is input into the emotion classification model to predict the predicted emotion features of the sample audio frame. With the goal of minimizing the difference between the predicted emotion features of the sample audio frame and the emotion label, the model parameters of the emotion classification model are adjusted, thereby completing the training of the emotion classification model. In this embodiment, the emotion classification model preferably uses an existing classification network such as a BiLSTM network, an AlexNet network, or a wav2vec2 network.

[0069] S102: Based on the speech audio and the emotion feature sequence, the speaker's facial movements and hand movements in the speaker action picture frame are adjusted to generate a speaker sign language action picture sequence corresponding to the speech audio.

[0070] This embodiment pre-acquires a front picture frame of the speaker as a speaker action picture frame. The speaker action picture frame may be acquired from a speaking video of the speaker by extracting frames, or may be acquired by taking a frontal shot of the speaker using an image acquisition device (such as a camera, etc.).

[0071] After acquiring the emotion feature sequence corresponding to the speech audio through the above steps, this embodiment adjusts the speaker's hand movements in the speaker action picture frame according to the content expressed by the speech audio and the emotion feature sequence, so that the speaker's hand movements can express the content expressed by the speech audio and the emotion expressed by the emotion feature sequence, and adjusts the speaker's facial movements in the speaker action picture frame according to the content expressed by the speech audio and the emotion feature sequence, so that the speaker's facial movements can express the emotion in the speech audio. Finally, after adjusting the speaker's facial movements and hand movements in the speaker action picture frame, a speaker sign language action picture corresponding to each audio frame in the speech audio is obtained, and all speaker sign language action pictures are collected according to the time sequence of each audio frame in the speech audio to obtain a speaker sign language action picture sequence corresponding to the speech audio, so that the speaker in the speaker sign language action picture sequence corresponding to the speech audio can not only show the sign language movements with the same meaning as the speech audio, but also show the same emotion as the speech audio in the facial expressions corresponding to the speaker's sign language movements and facial movements.

[0072] In addition, in this embodiment, the adjustment method of adjusting the speaker's facial movements and hand movements in the speaker action picture frame according to the speech audio and emotion feature sequence is determined by learning the sign language expression style and facial expression expression style of the hearing-impaired person, so that when the speaker's facial movements and hand movements in the speaker action picture frame are adjusted, they conform to the sign language expression style of the hearing-impaired person, that is, the style of the speaker's facial movements and hand movements in the speaker sign language action picture sequence is the same as the sign language expression style of the hearing-impaired person. When the hearing-impaired person watches the speaker sign language action picture sequence with the same sign language expression style as his or her own, it is easier for the hearing-impaired person to understand the content expressed by the speaker and improve the accuracy of the hearing-impaired person's understanding, that is, the generated speaker sign language action picture sequence has a higher accuracy.

[0073] Furthermore, after generating a sequence of speaker sign language action pictures corresponding to the speech audio, this embodiment needs to merge all pictures in the speaker sign language action picture sequence into a video in order, so as to obtain the speaker's sign language video corresponding to the speech audio, so that the hearing-impaired person can communicate with the speaker.

[0074] From the above introduction, it can be seen that the sign language generation method proposed in the embodiment of the present application performs emotion classification on each audio frame in the speaker's speech audio, and determines the emotion feature sequence corresponding to the speech audio; based on the speech audio and the emotion feature sequence, the speaker's facial movements and hand movements in the speaker's action picture frame are adjusted to generate a speaker's sign language action picture sequence corresponding to the speech audio; wherein, the style of the speaker's facial movements and hand movements in the speaker's sign language action picture sequence is the same as the sign language expression style of the hearing-impaired person. By adopting the technical solution of this embodiment, the speaker's facial movements and hand movements in the speaker's action picture frame can be adjusted by combining the speech audio with the emotion features of the speech audio, so that the speaker's sign language movements and facial expressions have emotion features, and the emotion degree of the sign language generation is improved. In addition, the style of the speaker's facial movements and hand movements in the speaker's sign language action picture sequence is the same as the sign language expression style of the hearing-impaired person, which improves the accuracy of sign language generation and is more convenient for the hearing-impaired person to understand.

[0075] As an optional implementation, the embodiment of the present application also proposes a sign language generation method.

[0076] See also Figure 2 As shown, the method includes:

[0077] S201, performing face and hand masking processing on the speaker action picture frame to obtain a masked picture of the speaker, and performing facial recognition on the speaker action picture frame to obtain facial identity features of the speaker.

[0078] This embodiment needs to obtain a speaker action picture frame. The method of obtaining the speaker action picture frame has been specifically described in the above embodiment and will not be repeated in this embodiment. Before adjusting the speaker's facial movements and hand movements in the speaker action picture frame, this embodiment needs to first mask the speaker's face and hands in the speaker action picture frame to obtain a masked picture of the speaker. Among them, to mask the face and hands of the speaker action picture frame, it is necessary to first perform image analysis and detection on the speaker action picture frame to detect the speaker's facial contour and hand contour in the speaker action picture frame, and then mask the speaker's face and hands in the speaker action picture frame according to the speaker's facial contour and hand contour.

[0079] Specifically, the present embodiment can detect the facial contour of the speaker in the speaker action picture frame by using an existing face detection algorithm, locate the face area in the speaker action picture frame, and then use an existing edge detection algorithm (such as Canny edge detection, Sobel operator, Prewitt operator, etc.) to extract the facial contour of the face area. The present embodiment can detect the hand contour of the speaker in the speaker action picture frame by using an existing hand detection algorithm (such as shape-based hand detection, deep learning-based hand detection, etc.) to locate the hand area in the speaker action picture frame, and then use an existing edge detection algorithm to extract the hand contour of the hand area. According to the facial contour and hand contour of the speaker, the face and hands of the speaker in the speaker action picture frame are masked, which can be achieved by using an existing Mask mask processing method.

[0080] This embodiment also requires facial recognition of the speaker action picture frame, extracting the speaker's facial features in the speaker action picture frame, and using the facial features as the speaker's facial identity features. This embodiment preferably uses a facial recognizer to perform facial recognition on the speaker action picture frame, detects and tracks faces in the speaker action picture frame based on image processing technology and machine learning algorithms, locates and preprocesses the detected faces, such as de-graying, denoising, and normalizing, and then extracts features from the located and preprocessed faces, thereby obtaining the speaker's facial identity features. Among them, the facial recognizer can use the existing SphereFace network, ArcFace network, etc. This embodiment identifies the facial identity features of the speaker so that when the facial movements of the speaker in the speaker action picture frame are adjusted, the face remains the speaker's face. If the face in the speaker action picture frame is to be replaced with the face of another person, the picture frame of the other person can be obtained, and the facial identity features of the other person can be identified from the picture frame of the other person. Then, the facial movements of the speaker in the speaker action picture frame can be adjusted according to the facial identity features of the other person. The face of the speaker can be replaced with the face of another person, thereby realizing the function of flexibly replacing the speaker's face.

[0081] S202: Perform emotion classification on each audio frame in the speaker's speech audio to determine an emotion feature sequence corresponding to the speech audio.

[0082] S203, based on the speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity features of the speaker, using a predetermined sign language action generation algorithm, generate the speaker's facial actions and hand actions in the mask image of the speaker, and obtain a speaker sign language action image sequence corresponding to the speech audio.

[0083] This embodiment predetermines a sign language action generation algorithm, which is determined by learning to generate sign language actions through a pre-recorded sign language action video of a hearing-impaired person. The hearing-impaired person is a hearing-impaired person who is currently communicating. Through the pre-recorded sign language action video of the hearing-impaired person, the sign language action generation algorithm learns the generation method of the sign language expression style of the hearing-impaired person, so that the sign language action picture sequence generated by the sign language action generation algorithm conforms to the sign language expression style of the hearing-impaired person, making it easier for the hearing-impaired person to understand and improving the accuracy of understanding.

[0084] Specifically, this embodiment needs to generate the speaker's facial movements and hand movements in the speaker's mask picture according to the speech audio, the emotional feature sequence, the speaker's mask picture and the speaker's facial identity features, using a predetermined sign language movement generation algorithm, so as to obtain a speaker's sign language movement picture sequence corresponding to the speech audio. The sign language movement generation algorithm can analyze the rhythm amplitude of each audio frame in the speech audio according to the speech audio and the emotional feature sequence, and then determine the face and hand movement information of each audio frame according to the rhythm amplitude of each audio frame, and finally supplement the hand position in the speaker's mask picture with the moving hand movement and the face position with the moving facial movement on the speaker's facial identity features according to the movement information, so as to obtain a speaker's sign language movement picture corresponding to each audio frame. According to the above method, after determining the speaker's sign language movement picture corresponding to each audio frame in the speech audio, the speaker's sign language movement picture sequence is assembled according to the time sequence of each audio frame in the speech audio.

[0085] Furthermore, this step specifically includes the following:

[0086] First, based on the speech audio and emotion feature sequence, the rhythm amplitude characteristics corresponding to the speech audio are determined.

[0087] This embodiment analyzes the rhythm features and amplitude features that match the emotions in the speech audio based on the speech audio and the emotion feature sequence, wherein the rhythm features include the rhythm features corresponding to each audio frame in the speech audio, that is, the rhythm features are a sequence of rhythm features corresponding to each audio frame, and the amplitude features include the amplitude features corresponding to each audio frame in the speech audio, that is, the amplitude features are a sequence of amplitude features corresponding to each audio frame. In this embodiment, the rhythm and amplitude features corresponding to the speech audio include the rhythm features corresponding to the speech audio and the amplitude features corresponding to the speech audio, and the rhythm features and the amplitude features can be separated or spliced ​​together.

[0088] In this embodiment, the predetermined sign language action generation algorithm includes a rhythm amplitude feature determination algorithm, and the rhythm amplitude feature determination algorithm is used to perform speech and emotion analysis on the speech audio and the corresponding emotion feature sequence to determine the rhythm amplitude feature corresponding to the speech audio. The rhythm amplitude feature determination algorithm in this embodiment is obtained by performing rhythm amplitude feature determination training using sample audio, emotion features manually annotated on the sample audio, and rhythm amplitude features manually annotated on the sample audio, that is, based on the sample audio and the emotion features manually annotated on the sample audio, the rhythm amplitude feature determination algorithm is used to determine the predicted rhythm amplitude feature corresponding to the sample audio, and then the rhythm amplitude feature determination algorithm is adjusted and improved according to the difference between the predicted rhythm amplitude feature and the rhythm amplitude feature manually annotated on the sample audio, with the goal of minimizing the difference.

[0089] Second, based on the speaker's mask image, the speaker's facial identity features and the rhythm amplitude features corresponding to the speech audio, the speaker's facial movements and hand movements in the speaker's mask image are generated to obtain a speaker sign language movement image sequence corresponding to the speech audio.

[0090] This embodiment is based on a pre-acquired mask image of the speaker, the facial identity features of the speaker, and the rhythm and amplitude features corresponding to the speech audio, and determines the facial movement information and hand movement information of the speaker according to the rhythm and amplitude features and the facial identity features of the speaker, and then determines the facial movements and hand movements of the face position and hand position in the mask image of the speaker according to the facial movement information and hand movement information of the speaker, and fills the facial movements and hand movements into the mask image of the speaker, thereby obtaining a sequence of speaker sign language movement images corresponding to the speech audio.

[0091] Specifically, this embodiment first determines the action optical flow features corresponding to the speech audio based on the rhythm amplitude features corresponding to the speech audio.

[0092] In this embodiment, according to the rhythm amplitude feature corresponding to the speech audio, it is necessary to determine the movement information of the face moving according to the rhythm amplitude feature as the facial action optical flow feature, and according to the rhythm amplitude feature corresponding to the speech audio, it is necessary to determine the movement information of the hand moving according to the rhythm amplitude feature as the hand action optical flow feature, and the set of the facial action optical flow feature and the hand action optical flow feature is used as the action optical flow feature corresponding to the speech audio. Among them, the facial action optical flow feature can be used to determine the facial action trajectory, and the hand action optical flow feature can be used to determine the hand action trajectory.

[0093] In this embodiment, the predetermined sign language action generation algorithm also includes an optical flow feature determination algorithm. Based on the rhythm amplitude characteristics corresponding to the speech audio, the optical flow feature determination algorithm is used to determine the action optical flow characteristics corresponding to the speech audio.

[0094] Then, based on the speaker's mask image, the speaker's facial identity features and the action optical flow features corresponding to the speech audio, the speaker's facial movements and hand movements in the speaker's mask image are generated to obtain a speaker sign language action image sequence corresponding to the speech audio.

[0095] After determining the action optical flow features corresponding to the speech audio, this embodiment can determine the hand motion trajectory of the speaker according to the hand action optical flow features in the action optical flow features, thereby determining the speaker's hand motion, determine the facial motion trajectory according to the facial action optical flow features in the action optical flow features, and determine the speaker's facial motion according to the facial motion trajectory based on the speaker's facial identity features. Then, the speaker's facial motion and hand motion are filled into the speaker's mask image, that is, the speaker's facial motion is filled into the face area of ​​the speaker's mask image, and the speaker's hand motion is filled into the hand area of ​​the speaker's mask image, and finally a speaker's sign language action picture sequence corresponding to the speech audio is obtained.

[0096] In this embodiment, the predetermined sign language action generation algorithm also includes an action picture determination algorithm. Based on the speaker's mask picture, the speaker's facial identity features, and the action optical flow features corresponding to the voice audio, the action picture determination algorithm can be used to generate the speaker's facial movements and hand movements in the speaker's mask picture, and obtain a speaker's sign language action picture sequence corresponding to the voice audio.

[0097] Step S202 in this embodiment is the same as step S101 in the above embodiment, and this embodiment will not specifically describe the execution method of step S202. In addition, this embodiment does not limit the execution order between step S201 and step S202, and step S201 may be executed first, and then step S202, or step S202 may be executed first, and then step S201, or step S201 and step S202 may be executed simultaneously.

[0098] As an optional implementation, see Figure 3 As shown, in another embodiment of the present application, it is disclosed that in step S203 in the above embodiment, based on the speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity feature of the speaker, a predetermined sign language action generation algorithm is used to generate the speaker's facial action and hand action in the mask image of the speaker, and obtain the speaker's sign language action image sequence corresponding to the speech audio, which specifically includes the following steps:

[0099] S301, inputting the speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity feature of the speaker into a pre-trained sign language action generation model to obtain a speaker sign language action image sequence corresponding to the speech audio.

[0100] After acquiring the emotion feature sequence, the mask image of the speaker and the facial identity features of the speaker corresponding to the speech audio, this embodiment inputs the speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity features of the speaker into the pre-trained sign language action generation model to obtain the speaker sign language action picture sequence corresponding to the speech audio. The sign language action generation model in this embodiment is obtained by training the sign language action generation based on the pre-recorded sign language action video of the hearing-impaired person. The hearing-impaired person is the hearing-impaired person who is currently communicating. Through the pre-recorded sign language action video of the hearing-impaired person, the sign language action generation model learns the generation method of the sign language expression style of the hearing-impaired person, so that the sign language action picture sequence generated by the sign language action generation model conforms to the sign language expression style of the hearing-impaired person, which is more convenient for the hearing-impaired person to understand and improves the accuracy of understanding.

[0101] In this embodiment, the sign language action generation model is a model that uses the sign language action generation algorithm in the above embodiment to generate sign language actions.

[0102] As an optional implementation, another embodiment of the present application discloses that the sign language action generation model includes: a rhythm amplitude feature encoder, an optical flow feature encoder and a sign language action generator. Step S301 in the above embodiment specifically includes the following steps:

[0103] First, the speech audio and emotion feature sequence are input into the rhythm amplitude feature encoder to obtain the rhythm amplitude feature corresponding to the speech audio.

[0104] In this embodiment, the speech audio and the emotion feature sequence corresponding to the speech audio are input into the rhythm amplitude feature encoder in the sign language action generation model, and the rhythm amplitude feature encoder can execute the rhythm amplitude feature determination algorithm in the above embodiment to obtain the rhythm amplitude feature corresponding to the speech audio. Among them, the rhythm amplitude feature encoder is composed of a convolutional neural network (CNN) and a long short-term memory network (LSTM).

[0105] Second, the rhythm amplitude features corresponding to the speech audio are input into the optical flow feature encoder to obtain the action optical flow features corresponding to the speech audio.

[0106] In this embodiment, the rhythm amplitude feature corresponding to the speech audio is input into the optical flow feature encoder, which can execute the optical flow feature determination algorithm in the above embodiment to obtain the action optical flow feature corresponding to the speech audio. The optical flow feature encoder is composed of a convolutional neural network (CNN) and a recurrent neural network (RNN).

[0107] Third, the speaker's mask image, the speaker's facial identity features, and the action optical flow features corresponding to the speech audio are input into the sign language action generator to obtain a sequence of the speaker's sign language action images corresponding to the speech audio.

[0108] In this embodiment, the mask image of the speaker, the facial identity features of the speaker, and the action optical flow features corresponding to the speech audio are input into the sign language action generator, and the sign language action generator can execute the action image determination algorithm in the above embodiment to obtain the speaker's sign language action image sequence corresponding to the speech audio. Among them, the sign language action generator is composed of a generative adversarial network (GAN).

[0109] As an optional implementation, see Figure 4 As shown, in another embodiment of the present application, it is disclosed that the training process of the sign language action generation model includes the following steps:

[0110] S401, obtaining sign language action videos corresponding to sample audios of hearing-impaired persons in different emotional states and sample emotion features corresponding to the sample audios.

[0111] The hearing-impaired person needs to use a sign language generation method to convert the speaker's voice audio into sign language movements so that the hearing-impaired person can understand what the speaker is saying. In order to make the sign language movements converted from the speaker's voice audio more consistent with the expression style of the hearing-impaired person communicating with him, this embodiment needs to use the sign language movement video of the hearing-impaired person for training when training the sign language movement generation model, so that the sign language movement generation model can learn the sign language expression style of the hearing-impaired person. Therefore, in order to train the sign language movement generation model, this embodiment first needs to obtain the sign language movement video corresponding to the sample audio of the hearing-impaired person in different emotional states, and determine the sample emotional features corresponding to the sample audio according to the emotional state corresponding to the sample audio.

[0112] For example, for emotional states such as happiness, anger, excitement, and the texts "I ran two kilometers in the park today", "The cat is sleeping quietly", "The store is filled with various goods", "A few yellow leaves fell from the tree", and "He is doing homework", this embodiment can record the sign language action videos of hearing-impaired people according to different emotional states for each text. Five different texts and three different states can record 15 different videos. In order to improve the accuracy of the sign language action generation model, it can be achieved by expanding the sign language action videos recorded by hearing-impaired people. Among them, the sample audio is the audio corresponding to the above text combined with the emotional state corresponding to the text.

[0113] In this embodiment, the sample emotion feature corresponding to the sample audio can be determined by manually marking the emotion feature through the emotion state corresponding to the sample audio, and the manually marked emotion feature is used as the emotion feature corresponding to each frame of the sample audio frame in the sample audio, and the emotion features corresponding to each frame of the sample audio frame are combined into the sample emotion feature corresponding to the sample audio; it is also possible to determine the emotion feature of each sample audio frame by using an emotion classification network to perform emotion classification on each sample audio frame in the sample audio, and combine the emotion features corresponding to each frame of the sample audio frame into the sample emotion feature corresponding to the sample audio.

[0114] S402, masking the face and hands of the video frames in the sign language action video to obtain sample masked images of the hearing-impaired person, and performing facial recognition on the video frames in the sign language action video to obtain sample facial identity features of the hearing-impaired person.

[0115] This embodiment needs to perform a frame extraction operation on the sign language action video of the hearing-impaired person to obtain a video frame in the sign language action video, and then mask the face and hands of the hearing-impaired person in the video frame to obtain a sample masked image of the hearing-impaired person. In this embodiment, the specific method used for masking the face and hands of the video frame in the sign language action video is the same as the specific method used for masking the face and hands of the speaker action picture frame in the above embodiment, and will not be described in detail in this embodiment.

[0116] This embodiment needs to perform facial recognition on the video frames in the sign language action video obtained by the frame extraction operation to obtain the sample facial identity features of the hearing-impaired person. The specific method used for performing facial recognition on the video frames in the sign language action video is the same as the specific method used for performing facial recognition on the speaker action picture frames in the above embodiment, and will not be described in detail in this embodiment.

[0117] S403, inputting the sample audio, sample emotion features, sample mask images of hearing-impaired persons, and sample facial identity features of hearing-impaired persons into a pre-built sign language action generation model to obtain a predicted sign language action image sequence corresponding to the sample audio.

[0118] This embodiment inputs sample audio, sample emotion features, sample mask images of hearing-impaired persons, and sample facial identity features of hearing-impaired persons into a pre-built sign language motion generation model, and uses the sign language motion generation model to generate facial movements and hand movements, thereby obtaining a predicted sign language motion image sequence corresponding to the sample audio.

[0119] S404: Adjust model parameters of the sign language action generation model based on the difference between the predicted sign language action picture sequence corresponding to the sample audio and the sample sign language action picture sequence.

[0120] After acquiring the sign language action video corresponding to the sample audio, this embodiment performs a frame extraction operation on the sign language action video to obtain all video frames contained in the sign language action video, and organizes all video frames contained in the sign language action video into a sample sign language action picture sequence according to their time sequence in the sign language action video.

[0121] This embodiment adjusts the model parameters of the sign language action generation model based on the difference between the predicted sign language action picture sequence corresponding to the sample audio and the sample sign language action picture sequence, with the goal of minimizing the difference.

[0122] As an optional implementation, see Figure 5 As shown, in another embodiment of the present application, it is disclosed that the sign language action generation model includes: a rhythm amplitude feature encoder, an optical flow feature encoder and a sign language action generator. Step S403 in the above embodiment specifically includes the following steps:

[0123] S501. Input the sample audio and the sample emotion feature into the rhythm amplitude feature encoder to obtain the sample rhythm amplitude feature corresponding to the sample audio, and train the rhythm amplitude feature encoder based on the difference between the labeled rhythm amplitude feature and the sample rhythm amplitude feature.

[0124] In this embodiment, the sample audio and the corresponding sample emotion feature are input into the rhythm amplitude feature encoder in the sign language action generation model to obtain the sample rhythm amplitude feature corresponding to the sample audio. In this embodiment, the rhythm amplitude feature of the sample audio also needs to be manually annotated in advance to determine the annotated rhythm amplitude feature corresponding to the sample audio. Among them, the manual annotation of the rhythm amplitude feature of the sample audio can be to annotate the rhythm feature and the amplitude feature of the sample audio separately. If the sample rhythm amplitude feature corresponding to the sample audio generated by the rhythm amplitude feature encoder is a feature spliced ​​by the rhythm feature and the amplitude feature, then the manually annotated rhythm feature and amplitude feature need to be spliced ​​according to the format of the sample rhythm amplitude feature.

[0125] This embodiment needs to train the rhythm amplitude feature encoder based on the difference between the marked rhythm amplitude feature corresponding to the sample audio and the sample rhythm amplitude feature corresponding to the sample audio, with the goal of minimizing the difference. Specifically, a first loss function between the marked rhythm amplitude feature corresponding to the sample audio and the sample rhythm amplitude feature corresponding to the sample audio can be calculated, and then the parameters of the rhythm amplitude feature encoder are adjusted according to the first loss function to make the first loss function smaller and smaller, until the first loss function reaches a preset loss function range, and the training of the rhythm amplitude feature encoder is completed.

[0126] S502: Input the sample rhythm amplitude feature corresponding to the sample audio into the optical flow feature encoder to obtain the sample action optical flow feature corresponding to the sample audio.

[0127] In this embodiment, the sample rhythm amplitude feature output by the rhythm amplitude feature encoder in the sign language motion generation model is input into the optical flow feature encoder in the sign language motion generation model to obtain the sample motion optical flow feature corresponding to the sample audio.

[0128] S503, inputting the sample action optical flow features, the sample mask images of the hearing-impaired person, and the sample facial identity features of the hearing-impaired person into a sign language action generator to obtain a predicted sign language action image sequence corresponding to the sample audio.

[0129] In this embodiment, the sample mask images of the hearing-impaired persons, the sample facial identity features of the hearing-impaired persons, and the sample action optical flow features output by the optical flow feature encoder are input into the sign language action generator in the sign language action generation model to obtain a predicted sign language action image sequence corresponding to the sample audio.

[0130] S504: Jointly train the optical flow feature encoder and the sign language action generator based on the difference between the sample action optical flow features and the annotated optical flow features, and the difference between the predicted sign language action picture sequence and the sample sign language action picture sequence.

[0131] This embodiment needs to use an existing mature optical flow extractor to extract optical flow features from each video frame in the sign language action video corresponding to the acquired sample audio, obtain the optical flow features corresponding to each video frame in the sign language action video, and combine the optical flow features corresponding to all video frames in the sign language action video into a sequence according to the time sequence of the video frames in the sign language action video to obtain the labeled optical flow features. Among them, the existing mature optical flow extractor is composed of a Transformer network, such as a FLowNet network, a RAFT network, etc.

[0132] This embodiment also needs to perform a frame extraction operation on the sign language action video corresponding to the acquired sample audio to obtain all video frames contained in the sign language action video, and organize all video frames contained in the sign language action video into a sample sign language action picture sequence according to their time sequence in the sign language action video.

[0133] This embodiment is based on the difference between the sample action optical flow features and the annotated optical flow features, and the difference between the predicted sign language action picture sequence and the sample sign language action picture sequence, and aims to minimize both differences, and jointly trains the optical flow feature encoder and the sign language action generator. Specifically, the second loss function between the sample action optical flow features and the annotated optical flow features, and the third loss function between the predicted sign language action picture sequence and the sample sign language action picture sequence are calculated, and then the optical flow feature encoder and the sign language action generator are parameter adjusted according to the second loss function and the third loss function, so that the second loss function and the third loss function become smaller and smaller, until the second loss function and the third loss function both reach the preset loss function range, and the joint training of the optical flow feature encoder and the sign language action generator is completed.

[0134] This embodiment uses the recorded sign language movement videos of the hearing-impaired to train the sign language movement generation model. Compared with the existing two-dimensional solutions that require standard sign language teachers or sign language announcers to record and shoot sign language movement videos, the labor cost is reduced. Compared with the existing three-dimensional solutions that require special equipment for motion capture and collection, require art designers to model, and require three-dimensional software for rendering, this embodiment only needs to use image acquisition equipment to shoot the front two-dimensional video of the hearing-impaired, which can also reduce labor costs.

[0135] Exemplary Devices

[0136] Accordingly, the present application also provides a sign language generating device, see Figure 6 As shown, the device comprises:

[0137] The emotion classification module 100 is used to perform emotion classification on each audio frame in the speaker's speech audio and determine the emotion feature sequence corresponding to the speech audio;

[0138] The sign language generation module 110 is used to adjust the speaker's facial movements and hand movements in the speaker action picture frame based on the speech audio and the emotion feature sequence, and generate a speaker sign language action picture sequence corresponding to the speech audio;

[0139] Among them, the styles of the speaker's facial movements and hand movements in the speaker's sign language action picture sequence are the same as the sign language expression style of the hearing-impaired person.

[0140] From the above introduction, it can be seen that the sign language generation device proposed in the embodiment of the present application can adjust the speaker's facial movements and hand movements in the speaker action picture frame by combining the speech audio with the emotional characteristics of the speech audio, so that the speaker's sign language movements and facial expressions have emotional characteristics, thereby improving the emotionality of the sign language generation. In addition, the style of the speaker's facial movements and hand movements in the speaker's sign language action picture sequence is the same as the sign language expression style of the hearing-impaired person, thereby improving the accuracy of the sign language generation and making it easier for the hearing-impaired person to understand.

[0141] As an optional implementation, another embodiment of the present application discloses that the sign language generation device also includes: a picture frame processing module.

[0142] The picture frame processing module is used to perform face and hand masking processing on the speaker action picture frame to obtain the speaker's masked picture, and perform facial recognition on the speaker action picture frame to obtain the speaker's facial identity features;

[0143] The sign language generation module 110 is specifically used to generate the speaker's facial movements and hand movements in the speaker's mask image based on the speech audio, the emotion feature sequence, the speaker's mask image and the speaker's facial identity features, and obtain a speaker's sign language movement image sequence corresponding to the speech audio using a predetermined sign language movement generation algorithm;

[0144] Among them, the sign language movement generation algorithm is determined by learning and generating sign language movements using pre-recorded sign language movement videos of hearing-impaired people.

[0145] As an optional implementation, another embodiment of the present application discloses that the sign language generation module 110 includes: a rhythm amplitude determination unit and an action picture determination unit.

[0146] A rhythm amplitude determination unit, used to determine the rhythm amplitude feature corresponding to the speech audio based on the speech audio and the emotion feature sequence;

[0147] The action picture determination unit is used to generate the speaker's facial movements and hand movements in the speaker's mask picture based on the speaker's mask picture, the speaker's facial identity features and the rhythm amplitude features corresponding to the speech audio, and obtain a speaker sign language action picture sequence corresponding to the speech audio.

[0148] As an optional implementation manner, another embodiment of the present application discloses an action picture determination unit, which is specifically used to:

[0149] Based on the rhythm amplitude feature corresponding to the speech audio, determine the action optical flow feature corresponding to the speech audio;

[0150] Based on the speaker's mask image, the speaker's facial identity features and the action optical flow features corresponding to the speech audio, the speaker's facial movements and hand movements in the speaker's mask image are generated to obtain a speaker sign language action image sequence corresponding to the speech audio.

[0151] As an optional implementation, disclosed in another embodiment of the present application, the sign language generation module 110 includes: a model processing unit.

[0152] A model processing unit, used to input the speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity feature of the speaker into a pre-trained sign language action generation model to obtain a speaker sign language action image sequence corresponding to the speech audio;

[0153] The sign language movement generation model is trained based on pre-recorded sign language movement videos of hearing-impaired people.

[0154] As an optional implementation, in another embodiment of the present application, it is disclosed that the sign language action generation model includes: a rhythm amplitude feature encoder, an optical flow feature encoder and a sign language action generator;

[0155] The model processing unit is specifically used for:

[0156] Input the speech audio and emotion feature sequence into the rhythm amplitude feature encoder to obtain the rhythm amplitude feature corresponding to the speech audio;

[0157] Input the rhythm amplitude feature corresponding to the speech audio into the optical flow feature encoder to obtain the action optical flow feature corresponding to the speech audio;

[0158] The speaker's mask image, the speaker's facial identity features and the action optical flow features corresponding to the speech audio are input into the sign language action generator to obtain a sequence of the speaker's sign language action images corresponding to the speech audio.

[0159] As an optional implementation, another embodiment of the present application discloses that the sign language generation device also includes: a video acquisition module, a sample picture frame processing module, a sample sign language generation module and a model parameter adjustment module.

[0160] A video acquisition module, used to acquire the sign language action videos corresponding to the sample audios of the hearing-impaired persons in different emotional states and the sample emotional features corresponding to the sample audios;

[0161] The sample picture frame processing module is used to perform face and hand masking on the video frames in the sign language action video to obtain sample masked pictures of hearing-impaired persons, and perform facial recognition on the video frames in the sign language action video to obtain sample facial identity features of hearing-impaired persons;

[0162] A sample sign language generation module is used to input sample audio, sample emotion features, sample mask images of hearing-impaired persons, and sample facial identity features of hearing-impaired persons into a pre-built sign language action generation model to obtain a predicted sign language action image sequence corresponding to the sample audio;

[0163] The model parameter adjustment module is used to adjust the model parameters of the sign language action generation model based on the difference between the predicted sign language action picture sequence corresponding to the sample audio and the sample sign language action picture sequence; wherein the sample sign language action picture sequence is obtained by extracting frames from the sign language action video corresponding to the sample audio.

[0164] As an optional implementation, another embodiment of the present application discloses a sample sign language generation module, which is specifically used for:

[0165] Input the sample audio and the sample emotion feature into the rhythm amplitude feature encoder to obtain the sample rhythm amplitude feature corresponding to the sample audio, and train the rhythm amplitude feature encoder based on the difference between the annotated rhythm amplitude feature and the sample rhythm amplitude feature; wherein the annotated rhythm amplitude feature is the rhythm amplitude feature manually annotated for the sample audio;

[0166] Input the sample rhythm amplitude feature corresponding to the sample audio into the optical flow feature encoder to obtain the sample action optical flow feature corresponding to the sample audio;

[0167] Inputting the sample action optical flow features, the sample mask images of the hearing-impaired person, and the sample facial identity features of the hearing-impaired person into the sign language action generator to obtain a predicted sign language action image sequence corresponding to the sample audio;

[0168] Based on the difference between the sample action optical flow features and the annotated optical flow features, as well as the difference between the predicted sign language action picture sequence and the sample sign language action picture sequence, the optical flow feature encoder and the sign language action generator are jointly trained; wherein the annotated optical flow features are obtained by extracting optical flow features from the sign language action video corresponding to the sample audio.

[0169] The sign language generation device provided in this embodiment belongs to the same application concept as the sign language generation method provided in the above embodiments of this application, can execute the sign language generation method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of executing the sign language generation method. For technical details not fully described in this embodiment, please refer to the specific processing content of the sign language generation method provided in the above embodiments of this application, and will not be repeated here.

[0170] Exemplary Electronic Devices

[0171] Another embodiment of the present application also provides an electronic device, see Figure 7As shown, the device includes:

[0172] Memory 200 and processor 210;

[0173] The memory 200 is connected to the processor 210 and is used to store programs;

[0174] The processor 210 is used to implement the sign language generation method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0175] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0176] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus.

[0177] A bus may include a pathway that transfers information between components of a computer system.

[0178] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the scheme of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0179] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0180] The memory 200 stores a program for executing the technical solution of the present invention, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0181] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0182] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0183] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0184] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement each step of any sign language generation method provided in the above embodiments of the present application.

[0185] Exemplary computer program products and storage media

[0186] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the sign language generation method according to various embodiments of the present application described in the above-mentioned "Exemplary Method" section of this specification.

[0187] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0188] In addition, an embodiment of the present application may also be a storage medium on which a computer program is stored, and the computer program is executed by a processor to execute the steps of the sign language generation method according to various embodiments of the present application described in the above "Exemplary Method" section of this specification.

[0189] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0190] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0191] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0192] The modules and sub-modules in the devices and terminals of the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0193] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0194] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0195] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0196] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0197] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0198] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0199] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A sign language generation method, characterized in that: include: Performing emotion classification on each audio frame in the speaker's speech audio to determine an emotion feature sequence corresponding to the speech audio; Based on the speech audio and the emotion feature sequence, adjusting the speaker's facial movements and hand movements in the speaker action picture frame to generate a speaker sign language action picture sequence corresponding to the speech audio; The style of the speaker's facial movements and hand movements in the speaker's sign language movement picture sequence is the same as the sign language expression style of the hearing-impaired person.

2. The method according to claim 1, characterized in that: The method further comprises: Performing face and hand masking processing on the speaker action picture frame to obtain a masked picture of the speaker, and performing facial recognition on the speaker action picture frame to obtain facial identity features of the speaker; Based on the speech audio and the emotion feature sequence, adjusting the speaker's facial movements and hand movements in the speaker action picture frame to generate a speaker sign language action picture sequence corresponding to the speech audio, including: Based on the speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity feature of the speaker, a predetermined sign language action generation algorithm is used to generate the speaker's facial actions and hand actions in the mask image of the speaker, and obtain a speaker sign language action image sequence corresponding to the speech audio; The sign language movement generation algorithm is determined by learning and generating sign language movements using pre-recorded sign language movement videos of hearing-impaired persons.

3. The method according to claim 2, characterized in that Based on the speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity feature of the speaker, a predetermined sign language action generation algorithm is used to generate the speaker's facial action and hand action in the mask image of the speaker, and obtain a speaker sign language action image sequence corresponding to the speech audio, including: Based on the speech audio and the emotion feature sequence, determining a rhythm amplitude feature corresponding to the speech audio; Based on the mask image of the speaker, the facial identity features of the speaker and the rhythm amplitude features corresponding to the speech audio, the speaker's facial movements and hand movements in the mask image of the speaker are generated to obtain a speaker sign language movement image sequence corresponding to the speech audio.

4. The method according to claim 3, characterized in that Based on the mask image of the speaker, the facial identity feature of the speaker, and the rhythm amplitude feature corresponding to the speech audio, the facial movements and hand movements of the speaker in the mask image of the speaker are generated to obtain a speaker sign language movement image sequence corresponding to the speech audio, including: Determining the action optical flow feature corresponding to the speech audio based on the rhythm amplitude feature corresponding to the speech audio; Based on the mask image of the speaker, the facial identity features of the speaker and the action optical flow features corresponding to the speech audio, the speaker's facial movements and hand movements in the mask image of the speaker are generated to obtain a speaker sign language action image sequence corresponding to the speech audio.

5. The method according to claim 2, characterized in that: Based on the speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity feature of the speaker, a predetermined sign language action generation algorithm is used to generate the speaker's facial action and hand action in the mask image of the speaker, and obtain a speaker sign language action image sequence corresponding to the speech audio, including: Inputting the speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity feature of the speaker into a pre-trained sign language action generation model to obtain a speaker sign language action image sequence corresponding to the speech audio; The sign language action generation model is obtained by training the sign language action generation based on pre-recorded sign language action videos of hearing-impaired persons.

6. The method according to claim 5, characterized in that The sign language action generation model includes: a rhythm amplitude feature encoder, an optical flow feature encoder and a sign language action generator; The speech audio, the emotion feature sequence, the mask image of the speaker and the facial identity feature of the speaker are input into a pre-trained sign language action generation model to obtain a speaker sign language action image sequence corresponding to the speech audio, including: Inputting the speech audio and the emotion feature sequence into the rhythm amplitude feature encoder to obtain the rhythm amplitude feature corresponding to the speech audio; Inputting the rhythm amplitude feature corresponding to the speech audio into the optical flow feature encoder to obtain the action optical flow feature corresponding to the speech audio; The mask image of the speaker, the facial identity features of the speaker and the action optical flow features corresponding to the speech audio are input into the sign language action generator to obtain a speaker sign language action image sequence corresponding to the speech audio.

7. The method according to claim 5, characterized in that The training process of the sign language action generation model includes: Obtain sign language action videos corresponding to sample audios of hearing-impaired persons in different emotional states and sample emotional features corresponding to the sample audios; Masking the face and hands of the video frames in the sign language action video to obtain a sample masked image of the hearing-impaired person, and performing facial recognition on the video frames in the sign language action video to obtain sample facial identity features of the hearing-impaired person; Inputting the sample audio, the sample emotion feature, the sample mask image of the hearing-impaired person, and the sample facial identity feature of the hearing-impaired person into a pre-built sign language action generation model to obtain a predicted sign language action image sequence corresponding to the sample audio; Based on the difference between the predicted sign language action picture sequence corresponding to the sample audio and the sample sign language action picture sequence, the model parameters of the sign language action generation model are adjusted; wherein the sample sign language action picture sequence is obtained by extracting frames from the sign language action video corresponding to the sample audio.

8. The method according to claim 7, characterized in that Inputting the sample audio, the sample emotion feature, the sample mask image of the hearing-impaired person, and the sample facial identity feature of the hearing-impaired person into a pre-built sign language action generation model to obtain a predicted sign language action image sequence corresponding to the sample audio, including: Inputting the sample audio and the sample emotion feature into a rhythm amplitude feature encoder to obtain a sample rhythm amplitude feature corresponding to the sample audio, and training the rhythm amplitude feature encoder based on the difference between the annotated rhythm amplitude feature and the sample rhythm amplitude feature; wherein the annotated rhythm amplitude feature is a rhythm amplitude feature manually annotated for the sample audio; Inputting the sample rhythm amplitude feature corresponding to the sample audio into an optical flow feature encoder to obtain the sample action optical flow feature corresponding to the sample audio; Inputting the sample action optical flow features, the sample mask image of the hearing-impaired person, and the sample facial identity features of the hearing-impaired person into a sign language action generator to obtain a predicted sign language action image sequence corresponding to the sample audio; Based on the difference between the sample action optical flow features and the annotated optical flow features, and the difference between the predicted sign language action picture sequence and the sample sign language action picture sequence, the optical flow feature encoder and the sign language action generator are jointly trained; wherein the annotated optical flow features are obtained by extracting optical flow features from the sign language action video corresponding to the sample audio.

9. A sign language generating device, characterized in that: include: The emotion classification module is used to perform emotion classification on each audio frame in the speaker's speech audio and determine the emotion feature sequence corresponding to the speech audio; A sign language generation module, configured to adjust the speaker's facial movements and hand movements in the speaker's action picture frame based on the speech audio and the emotion feature sequence, and generate a speaker's sign language action picture sequence corresponding to the speech audio; The style of the speaker's facial movements and hand movements in the speaker's sign language movement picture sequence is the same as the sign language expression style of the hearing-impaired person.

10. An electronic device, characterized in that: include: Memory and processor; The memory is connected to the processor and is used to store programs; The processor is used to implement the sign language generation method according to any one of claims 1 to 8 by running the program in the memory.

11. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the sign language generation method according to any one of claims 1 to 8 is implemented.

12. A computer program product, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to implement the sign language generation method according to any one of claims 1 to 8.

Images