A sign language animation generation method, device and medium

Abstract

The application provides a sign language animation generation method, device and medium. The method comprises: obtaining an action posture sequence of a target sign language word from a preset sign language library, the target sign language word being a sign language word required for a sign language to be generated, each frame of the action posture sequence at least indicating an upper limb posture of a virtual human body model; splicing action posture sequences corresponding to all target sign language words in a sequence according to a semantic of the sign language to be generated to obtain a target action posture sequence, wherein a trained neural network is used to perform smooth transition processing on the connection of the action posture sequences corresponding to two target sign language words to be spliced in the predetermined splicing mode; and driving the posture of the virtual human body model to change according to the target action posture sequence to generate a target sign language animation. The transition between sign language word animations in the generated sign language animation is more coherent and natural, and the generation efficiency of the sign language animation is improved.

Description

Technical Field

[0001] This invention relates to the fields of sign language and machine learning, specifically to the field of sign language animation technology, and more specifically, to a method, device, and medium for generating sign language animation. Background Technology

[0002] Based on how humans perceive language, language can be divided into auditory language and visual language.

[0003] Typically, most human communication uses auditory language. The speaker, based on the pronunciation rules of natural language (such as Chinese, English, and Japanese), uses their lips to pronounce words, conveying their intended thoughts. The receiver listens to the speaker's pronunciation, and the captured sound waves are transmitted to the brain to extract the expressed thought. Thus, the speaker transmits their intended thought to the receiver. In other words, the receiver perceives the thought conveyed by the speaker through hearing, and the medium of communication is the speech corresponding to natural language text.

[0004] However, communication between individuals with hearing and / or language impairments (i.e., deaf-mute individuals) is generally achieved through visual language, such as sign language. In sign language, the signer performs various gestures according to sign language standards (e.g., the National Common Sign Language Dictionary implemented in China in October 2019). These gestures primarily mimic images or syllables to convey their intended thoughts. The receiver observes these gestures visually and transmits the captured images to the brain to extract the intended message. In other words, the receiver perceives the message visually, and the medium of communication is the physical gestures corresponding to natural language text.

[0005] With the advancement of computer science and technology, 3D computer animation has emerged. 3D computer animation (also known as model animation) is a technique that uses computers to simulate the changes of scenes and shapes in three-dimensional space over time. Specifically, it involves constructing three-dimensional models using computers and automatically generating a series of realistic, continuous dynamic images by controlling the movement of the models, virtual cameras, and virtual light sources. As model animation is applied in various scenarios, the industrial and commercial sectors have begun to use this technology to generate sign language animation, that is, to create sign language animation using virtual human models to provide better services for the deaf and mute. For example, virtual human model animation can be used to create sign language animations for movies so that deaf and mute viewers can watch them.

[0006] In natural language, words combine to form sentences, sentences combine to form paragraphs, and paragraphs combine to form chapters. Sign language is similar; its smallest unit is the sign word, which combines to form sign sentences, sign sentences combine to form sign paragraphs, and sign paragraphs combine to form sign chapters. When creating sign language animations, sign word animations are typically created beforehand based on sign language standards (such as the National Common Sign Language Dictionary implemented in China in October 2019). Then, according to the semantics of the desired sign language animation, the required sign word animations are connected in the order of their meanings. Transition frames are generated between two consecutive sign word animations that need to be connected sequentially. These transition frames act as a bridge between the two animations, and their quality directly affects the evaluation of the seamless transition between them.

[0007] There are two main traditional methods for generating transition frames: optical flow-based methods and distance loss determination methods.

[0008] Optical flow-based methods utilize the temporal changes of pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous and current frames, thereby calculating the motion information of objects between adjacent frames. Therefore, optical flow-based methods first select the insertion position of the transition frame, and then use interpolation to insert the transition frame sequence at the determined insertion position. The optical flow-based method for generating transition frames has two main drawbacks: First, because it calculates transition frames based on image sequences, the amount of computation is large, resulting in slow transition frame generation and high time costs. This is because selecting the insertion position of the transition frame requires comparing each frame of the preceding and following sign language animation segments one by one, making the transition frame generation slow, and the number of comparisons is proportional to the product of the number of frames in the preceding and following sign language animation segments. Second, the quality of the generated transition frames is poor, mainly manifested in defects such as ghosting, abrupt switching, inconsistent speed before and after the action, and severe distortion of the gesture. The reason is that the interpolation calculation does not distinguish whether a pixel belongs to the virtual human body model, but treats each pixel equally. Furthermore, the speed of the virtual human body model's limb movements and the correlation between the limb movements before and after are not considered during the stitching process. Therefore, the quality of the generated transition frames is poor, which makes it impossible to correctly represent sign language actions.

[0009] The distance loss determination method generally uses the skeletal data of a virtual human model to find the insertion position of transition frames. Specifically, it typically assesses the differences between animation frames by calculating the distance loss between bones, determines the insertion position based on the differences, and then inserts the transition frame sequence at the insertion position using linear interpolation or spherical interpolation. Transition frames generated based on the distance loss determination method generally have the two drawbacks mentioned above: First, the generation speed of transition frames is slow, resulting in high time costs. This is because the selection of the insertion position of the transition frame is relatively fixed and rigid, requiring the determination of the skeletal coordinates of the virtual human model in each frame of animation, and then determining the distance loss between bones based on the skeletal coordinates in each frame of animation, resulting in a long selection time for the insertion position of the transition frame. Second, the quality of the generated transition frames is poor. This is because the method of determining the transition frame position only considers distance, and the defects of the interpolation method are added during interpolation (see above), resulting in poor flexibility of the generated transition frames, a relatively direct smoothing effect, and an unnatural connection between the generated transition frames and the preceding and following sign language animations, losing continuity. Summary of the Invention

[0010] Therefore, the purpose of this invention is to provide an animation generation method, apparatus, electronic device, storage medium, chip, and computer program product that can improve the generation efficiency of sign language animation and the quality of transition frames between sign language words in the generated sign language animation to a certain extent.

[0011] According to a first aspect of the present invention, a method for generating sign language animation is provided. The method includes the following steps: obtaining a sequence of action postures of target sign language words from a preset sign language lexicon, wherein the target sign language words are sign language words required to generate sign language, and each frame of the action posture sequence indicates at least the upper limb posture of a virtual human body model; concatenating the action posture sequences corresponding to all target sign language words according to a predetermined concatenation method in the semantic order of the sign language to be generated, to obtain a target action posture sequence, wherein the predetermined concatenation method utilizes a trained neural network to perform smooth transition processing on the connection between the action posture sequences corresponding to the two target sign language words to be concatenated; and driving the posture of the virtual human body model to change according to the target action posture sequence to generate a target sign language animation.

[0012] According to a second aspect of the present invention, a sign language animation generation apparatus is provided. The apparatus includes: a first acquisition module, configured to acquire a sequence of action postures of target sign language words from a preset sign language lexicon, wherein the target sign language words are sign language words required to generate sign language, and each frame of the action posture sequence indicates at least the upper limb posture of a virtual human body model; a first splicing module, configured to splice the action posture sequences corresponding to all target sign language words according to a predetermined splicing method in the semantic order of the sign language to be generated, to obtain a target action posture sequence, wherein the predetermined splicing method utilizes a trained neural network to perform smooth transition processing on the connection between the action posture sequences corresponding to two target sign language words to be spliced; and a first driving module, configured to drive the posture of the virtual human body model to change according to the target action posture sequence, thereby generating a target sign language animation.

[0013] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored, which can be executed by a processor to implement the steps of the sign language animation generation method as described in the first aspect.

[0014] A fourth aspect of this application provides an electronic device including one or more processors and a storage device storing one or more computer programs executable on the processors, the one or more computer programs being executed by the one or more processors to enable the electronic device to perform the steps of the sign language animation generation method as described in the first aspect.

[0015] In this embodiment, a trained neural network is used to concatenate the action posture sequences corresponding to two sign language words that need to be sequentially concatenated from all the target sign language words required to generate the sign language animation. This results in a target action posture sequence, which is then used to drive the posture transformation of a virtual human model, thereby generating the target sign language animation. This not only improves the quality of transition frames between adjacent sign language word animations—making the transitions between adjacent sign language word animations more coherent, natural, and smooth—but also increases the efficiency of sign language animation generation, meeting real-time requirements. Attached Figure Description

[0016] The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

[0017] Figure 1 A flowchart of the sign language animation generation method provided in the embodiments of this application;

[0018] Figure 2 This is a schematic diagram of the neural network structure for smooth transition processing at splicing points provided in an embodiment of this application;

[0019] Figure 3 A schematic diagram of the structure of the first neural network provided in an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of the structure of a recurrent gating unit in a first neural network provided in an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of the structure of the first attention mechanism layer in the first neural network provided in an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of the structure of the second neural network provided in an embodiment of this application;

[0023] Figure 7 A flowchart of some steps in the animation generation method provided in the embodiments of this application;

[0024] Figure 8 This is an illustrative representation of the effects of several different splicing methods;

[0025] Figure 9 The embodiments provided in this application provide for the Figure 2 The neural network shown is trained by selecting different selection parameters as the selection parameters for the sequence selection layer in the second neural network. Figure 2 The overall accuracy of the neural network shown;

[0026] Figure 10 This is a schematic diagram of the structure of the animation generation device provided in the embodiments of this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. It should be understood that the specific embodiments described herein are only used to explain the invention and are not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments in this application are within the scope of protection of this application.

[0028] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that the terms "first," "second," etc., can be interchanged where appropriate, achieving the same effect of distinguishing similar objects. Furthermore, the objects distinguished by "first," "second," etc., are usually of the same class, and the number of objects is not limited; for example, the first object can be one or more. In addition, "and / or" in the specification and claims indicates at least one of the connected objects (i.e., a single object or a combination of connected objects), and the character " / " generally indicates that the preceding and following objects are in an "or" relationship. In this application specification, unless otherwise specified, vectors are represented in bold, such as vector...

[0029] As mentioned in the background section, existing sign language word splicing techniques suffer from drawbacks such as slow generation speed and poor quality of the generated transition frames. Considering these shortcomings, the inventors of this application have conducted in-depth research on the application of machine learning and deep learning technologies in sign language animation production. The inventors have discovered that semantically continuous sign language animations (and / or sign language word animations) can be used as raw data, processed, and then input into a neural network for training. Using this trained neural network to assist in sign language animation production not only improves the quality of transition frames between sign language word animations but also achieves higher efficiency than the distance loss determination method.

[0030] Therefore, this application provides a method for generating sign language animation. In this method, a trained neural network is used to concatenate all the action posture sequences corresponding to the two sign language words that need to be sequentially concatenated in the target sign language words required for the animation. This results in a target action posture sequence, which is then used to drive the posture transformation of a virtual human model, thereby generating the target sign language animation. This not only improves the quality of transition frames between adjacent sign language word animations—that is, makes the transitions between adjacent sign language word animations more coherent, natural, and smooth—but also improves the generation efficiency of the sign language animation, meeting real-time requirements.

[0031] For ease of understanding, some keywords involved in the embodiments of this application will be explained below. Unless otherwise specified, when the same keywords appear in the following text, their meanings are the same as those explained here.

[0032] Sign language is a form of language that uses hand gestures to express information. Based on the length of the information conveyed, sign language can be categorized into sign words, sign sentences, sign paragraphs, and sign texts. A sign sentence is composed of multiple sign words arranged chronologically according to their semantics, while a sign paragraph or sign text is composed of multiple sign sentences arranged chronologically according to their semantics.

[0033] A sign language word is a series of gestures arranged sequentially to represent the meaning of a word. These gestures are primarily upper limb movements (see various sign language standards, such as the National Common Sign Language Dictionary implemented in China in October 2019). A sign language word is the smallest unit of sign language.

[0034] Sign language sentences are sentences composed of multiple sign language words. In other words, sign language sentences are sentences formed by combining multiple sign language words in sequence according to the desired semantic meaning. That is, sign language sentences are combinations of body movements that arrange a series of sign language words in chronological order to represent the semantic meaning of a sentence. This combination of body movements is formed by connecting the body movements corresponding to each sign language word according to the semantic order of the sentence.

[0035] A sign language word gesture sequence is a mathematically represented sequence. Each element (or gesture frame or gesture unit) in this sequence indicates a gesture in the limb movements that constitute the sign language word. The gesture frames in the sequence are arranged according to the temporal order of the limb movements in the sign language word. The gestures in this sequence can be gesture control data for a gesture (corresponding to the sign language word) performed by a virtual human model (which can be a lifelike figure, cartoon character, or any other object that can express sign language through limbs). The gesture control data can be the skeletal rotation coordinates of the virtual human model (also referred to as joint angles in some literature), or the spatial coordinates of the joints of the virtual human model. Of course, other representations are also possible, such as using skinned vertex coordinates or a combination of skeletal rotation coordinates and skinned vertex coordinates. It should be noted that the aforementioned gesture representation methods are only illustrative and do not preclude other representation methods.

[0036] Sign language word animation is an animation created by driving a virtual human model to change postures and make corresponding limb movements according to the sequence of movement postures of sign language words.

[0037] Hand gesture animation is an animation created by driving a virtual human body model to change postures and perform corresponding limb movements according to the sequence of hand gestures.

[0038] Sign language animation can be an animation created by driving a virtual human model to change postures and perform the corresponding limb movements according to the sequence of actions and postures that correspond to the semantics of sign language sentences, sign language paragraphs, or sign language chapters (or sign language articles).

[0039] The following description, in conjunction with the accompanying drawings, provides a detailed account of a sign language animation generation method, sign language animation generation device, electronic device, storage medium, chip, and computer program product provided in this application, through exemplary embodiments and exemplary application scenarios.

[0040] like Figure 1 The diagram shows a flowchart of a sign language animation generation method provided in an embodiment of this application. The sign language animation generation method can be applied to electronic devices, such as computers, smartphones, tablets, wearable devices (e.g., smartwatches), or servers in distributed systems, cloud servers, intelligent cloud servers with artificial intelligence technology, or intelligent cloud hosts. Of course, it is also possible to apply it to multiple electronic devices that interact with each other based on communication principles (see the description of step S101 below; for example, in some optional embodiments, step S101 can be implemented when multiple electronic devices interact based on communication principles). Please refer to... Figure 1 The sign language animation generation method includes the following steps S101 to S103:

[0041] Step S101: Obtain the action posture sequence of the target sign language word from the preset sign language word library.

[0042] According to one embodiment of the present invention, the sign language lexicon is a database that can be deployed in a predetermined storage medium or device. The storage medium or device for deploying the sign language lexicon can be an electronic device executing the sign language animation generation method; when the action posture sequence of the target sign language word is needed, it can be retrieved directly from the local storage. Alternatively, the storage medium or device for deploying the sign language lexicon can be a separate storage medium or device deployed independently of the electronic device executing the sign language animation generation method. When the action posture sequence of the target sign language word is needed, the electronic device executing the sign language animation generation method accesses the storage medium or device deploying the sign language lexicon based on communication principles and retrieves the action posture sequence of the target sign language word from the storage medium or device deploying the sign language lexicon.

[0043] The sign language lexicon stores computer-readable sign language word files (e.g., sign language word animations, images or videos of sign language words taken by humans in real-time) and / or the corresponding action posture sequences of the sign language word animations. The sign language word files are electronic data files pre-created based on sign language standards for each sign language word. In this embodiment, the sign language (word) animation is skeletal animation. However, it is not excluded that the sign language (word) animation can be other forms of computer 3D model animation; for example, it is not excluded that the sign language (word) animation can also be vertex animation or skinned animation.

[0044] Please refer to the previous explanation for the action posture sequence of sign language words. Each frame of the action posture sequence indicates at least the upper limb posture of the virtual human body model. A frame in the action posture sequence refers to a posture unit (corresponding to element, posture frame) in that action posture sequence.

[0045] Taking sign language word animation as an example, the action posture sequence corresponding to the sign language word animation refers to the posture sequence composed of the virtual human body model in each frame of the sign language word animation arranged according to the time sequence of the animation frames. The posture sequence corresponding to a sign language word animation is a mathematical representation of a series of limb movements that the virtual human body model needs to make in time sequence for the sign language word.

[0046] An animated sign language word and its corresponding motion sequence can be mutually converted. On one hand, the motion sequence of the virtual human model corresponding to the sign language word can be extracted from the animated sign language word (for example, the spatial coordinates of the joints of the virtual human model can be used to represent the posture of each frame in the motion sequence). On the other hand, the posture of the virtual human model can be transformed based on the motion sequence corresponding to the sign language word, thereby generating the animated sign language word.

[0047] For example, there exists a sign language word corresponding to a sign language word (denoted as P) as P = [P1, P2...P... N The sign language word animation includes N frames of animation and the corresponding action posture sequence (denoted as P) as P = [P1, P2...P...]. N ], schematically, where animation frame P N This is the Nth frame of the sign language word animation P, the gesture P. N Let N be the pose of the Nth frame in the action pose sequence P of the sign language word. Where N ∈ N * N * This represents the set of positive integers. The symbol "[...]" represents a sequence in which the elements are arranged in a specific order. Sign language word animation P and action / gesture sequence P can be "converted" to each other using the aforementioned conversion method.

[0048] Like sign language animation, sign language videos are composed of individual video frames arranged sequentially. Therefore, when sign language words are conveyed through video, similar to sign language animation, the sequence of gestures and actions for the sign language words can be extracted from the video. Each frame in sign language animation and video is an image. Extracting the sequence of gestures and actions from these images essentially involves performing human (or human model) recognition and gesture recognition on each frame of the sign language animation or video. Similarly, when sign language words are conveyed through images, the sequence of gestures and actions can be extracted from the sign language images.

[0049] Specifically, "sign language to be generated" refers to the sign language animation to be generated. For the sign language to be generated, it is necessary to obtain the action posture sequence of sign language words that can completely represent the semantics of the sign language to be generated through semantic combinations. This is the action posture sequence of the target sign language words, meaning the target sign language words are the sign language words required for the sign language to be generated. According to the semantics of the sign language to be generated, the action posture sequence of the target sign language words is sequentially concatenated to obtain the action posture sequence of the sign language to be generated. In this embodiment, the virtual human model is driven to change postures according to the action posture sequence to generate the sign language animation. Therefore, it is necessary to obtain the action posture sequence of the target sign language words from a pre-set sign language lexicon so that it can be used later to concatenate the action posture sequence corresponding to the sign language to be generated (i.e., the target action posture sequence).

[0050] For example, if the semantic meaning of the sign language animation to be generated is "The weather is nice today", the text "The weather is nice today" can be segmented into three words: "today", "weather" and "nice". Then, the action posture sequence corresponding to each of the sign language words that can represent the meaning of "today", "weather" and "nice" needs to be obtained from the pre-set sign language vocabulary.

[0051] In some embodiments, if the action posture sequence of a sign language word is stored in a preset sign language dictionary, the action posture sequence of the target sign language word can be obtained directly from the sign language dictionary.

[0052] Step S102: Concatenate the action posture sequences corresponding to all target sign language words according to a predetermined concatenation method, following the semantic order of the sign language to be generated, to obtain the target action posture sequence. The predetermined concatenation method utilizes a trained neural network to perform smooth transition processing on the connection between the action posture sequences corresponding to the two target sign language words to be concatenated.

[0053] The sign language to be generated can be a sign phrase, a sign paragraph, or a sign text. Similar to natural language text, sign phrases are composed of individual sign words connected in sequence, sign phrases then form sign paragraphs, and sign paragraphs then form sign texts. When the sign language to be generated is a sign text animation or a sign paragraph animation, after obtaining the action posture sequence of the target sign word from the sign language lexicon, it is necessary to "assemble" the obtained target sign word action posture sequence into the action posture sequence corresponding to the sign phrase (animation) according to the semantics of the sign language to be generated. Then, the action posture sequence corresponding to the sign phrase (animation) is arranged in sequence into the action posture sequence corresponding to the sign text (animation) or sign paragraph (animation). That is, according to the semantics of the sign phrases in the sign language animation to be generated, the action posture sequences of the target sign words are first connected (or spliced) in sequence to form the action posture sequence of the sign phrase (animation), and then the action posture sequences of the individual sign phrases (animations) are assembled into the action posture sequence of the sign paragraph (animation) or sign text (animation).

[0054] It should be noted that the following description will mainly take the generation of sign language animation with the semantic meaning of "the weather is nice today" as an example, especially the connection of two sign language words with the meanings of "today" and "weather". Based on the description of the embodiments of this application, those skilled in the art can directly and without doubt understand the generation method of sign language chapter animation or sign language segment animation. For example, regarding the sign language meaning "The weather is nice today, let's go play," according to the sign language animation generation method provided in the embodiments of this application, those skilled in the art, based on the description of the embodiments of this application, can directly and without doubt know that it is necessary to obtain the action posture sequences corresponding to the six sign language words (animations) with meanings of "today," "weather," "nice," "we," "go," and "play," respectively. Then, the action posture sequences corresponding to the three sign language words (animations) with meanings of "today," "weather," and "nice" are concatenated into one action posture sequence, and the action posture sequences corresponding to the three sign language words (animations) with meanings of "we," "go," and "play" are concatenated into one action posture sequence. Then, based on these two action posture sequences, two sign language animations with meanings of "The weather is nice today" and "Let's go play," respectively, are generated. Finally, these two sign language animations are arranged according to their meanings into one sign language animation to represent the meaning of "The weather is nice today, let's go play." Alternatively, two action posture sequences with the meanings "The weather is nice today" and "Let's go play" can be concatenated into one action posture sequence, and then the posture of the virtual human body model can be directly driven based on the concatenated action posture to generate a sign language animation with the meaning "The weather is nice today, let's go play".

[0055] It should also be noted that the acquired action sequence of the target sign language word is reusable. For example, if the semantic representation of the sign language to be generated is a dialogue between two people, A and B: "A says 'The weather is nice today,' and B replies: 'Yes, the weather is quite nice today.'", when acquiring the action sequence of the target sign language word for this target sign language, it is not necessary to acquire the action sequence of the sign language words with the meanings of "today" and "weather" twice. After obtaining the action sequence of the target sign language word from the sign language dictionary, it can be stored in a buffer. When the action sequence of the sign language word needs to be reused, it can be retrieved directly from the buffer without having to retrieve it from the sign language dictionary again. This relatively reduces the time spent acquiring the action sequence of the target sign language word and improves the efficiency of sign language animation generation.

[0056] For example, after obtaining the action posture sequences corresponding to the sign language words representing the three meanings of "today," "weather," and "good" (referred to as sign language word I, sign language word II, and sign language word III respectively), that is, obtaining the action posture sequence of sign language word I (whose semantic meaning is "today"): P = [P1, P2...P... N ], The sequence of gestures for sign language word II (meaning "weather"): Q = [Q1, Q2...Q M ], The sequence of gestures for sign language word III (meaning "good"): S = [S1, S2...S... Z [The sequence consists of N frames for the action posture sequence of sign language word I, M frames for the action posture sequence of sign language word II, and Z frames for the action posture sequence of sign language word III, where N, M, and Z are positive integers for illustrative purposes. P] N Q represents the Nth element in the sequence of gestures for the sign language word I. M S represents the M-th element in the action sequence of sign language word II. Z This represents the Z-th element in the action sequence of sign language word III. The meanings of the remaining elements are similar and will not be elaborated here. The action sequence needs to be arranged according to the semantic order of "The weather is nice today," that is, according to the order of "I II III," or P, Q, S: [P1, P2...P...]. N [Q1, Q2...Q] and [Q1, Q2...Q] M [Q1, Q2...Q] M [S1, S2...S] and [S1, S2...S] Z The pieces are then assembled sequentially.

[0057] When splicing the action posture sequences of target sign language words to assemble the action posture sequences corresponding to sign language words, it is necessary to smooth the splicing points of the action posture sequences corresponding to the two sequentially spliced ​​sign language words.

[0058] For example, when concatenating motion pose sequences P and Q, it is necessary to modify animation frame P. N The connection between P and Q1 is smoothed, including but not limited to: removing motion pose sequences P and Q from frames P close to the animation frame P. N The motion pose sequence of Q1 and / or a transition motion pose sequence are generated and inserted into the connection between motion pose sequences P and Q. For example, for animation frame P... N Smoothing the connection between Q1 and Q1 involves deleting the animation frames after frame I in the motion pose sequence P and before frame J in the motion pose sequence Q, and inserting a transitional motion pose sequence F = [F1, F2...F1] frames at the connection. K ], where I, J, and K are all positive integers, and I < N, J < M. Thus, the action posture sequence Y is obtained. P+Q = [P1, P2...P I F1, F2...F K Q J+1 Q J+2 ...Q M In the same manner, the action pose sequences Q and S are concatenated to obtain the action pose sequence Y. Q+S = [Q1, Q2...Q A E1, E2...E B S D+1 S D+2 ...S Z ], where the posture sequence E = [E1, E2...E B [ is] the transitional motion posture sequence between motion posture sequences Q and S, where A, B, and D are all positive integers, and J < A < M, D < Z. Then, the motion posture sequence Y... P+Q and Y Q+S By concatenating the sequences, we obtain the target action posture sequence Y, representing the sign language to be generated, which is "The weather is nice today". P+Q+S = [P1, P2...P I F1, F2...F K Q J+1 Q J+2 ...Q A E1, E2...E B S D+1 S D+2 ...S Z ].

[0059] In this embodiment, a trained neural network is used to perform a smooth transition by connecting the action posture sequences corresponding to the two target sign language words to be concatenated. This trained neural network can be a neural network capable of handling sequence-to-sequence tasks, such as a combination of one or more neural networks including Lastm, GRU, ConS2S, or Transformer. Alternatively, to adapt to the sequence data used in this embodiment, certain improvements or selections can be made to the aforementioned sequence-to-sequence neural networks before combining the improved or selected networks for smooth transition processing. Exemplarily, in some optional implementations, the following methods can be used... Figures 2 to 7 The neural network described in the illustrated embodiment undergoes smooth transition processing.

[0060] Based on in-depth research into the application of machine learning and deep learning technologies in sign language animation production, the inventors of this application have conducted extensive research and experiments on the application of sequence-to-sequence neural networks (such as LSTM neural networks, GRU neural networks, ConS2S neural networks, and Transformer neural networks) in sign language animation generation, and have proposed, for example... Figure 2 The neural network shown is used for smooth transition processing at the splice joint.

[0061] like Figure 2 The diagram illustrates a schematic representation of the neural network structure provided in an embodiment of this application for smooth transition processing at splicing points. Please refer to [link to relevant documentation]. Figure 2 The neural network used for smooth transition processing at the splicing point includes: a first neural network 21 (also known as a target position selection module, splicing position search network, SPSN), a second neural network 22 (also known as a transition action pose sequence generation module, transition frame generation network, TFGN), and a filter splicing layer 23 (also known as a filter splicing module, Filter Layer). Both the first neural network 21 and the second neural network 22 can be sequence-to-sequence neural network models.

[0062] To facilitate understanding, before elaborating on the neural network proposed in the embodiments of this application, we will still use the aforementioned example of "the weather is nice today" to illustrate an exemplary application scenario of the neural network proposed in the embodiments of this application for smooth transition processing at the splicing point. The following will utilize... Figure 2The following is an overview of the neural network used for smooth transition processing at the splicing point to splice the action posture sequence of two sign language words (animation) with the meanings of "today" and "weather".

[0063] After obtaining the movement sequence P = [P1, P2...P] corresponding to sign language words I and II with the meanings "today" and "weather" respectively, N ] and Q = [Q1, Q2...Q M After that, it is necessary to follow the action sequence P = [P1, P2...P1] corresponding to the sign language word I which means "today". N The sequence of actions and postures corresponding to the sign language word II, which means "weather," is Q = [Q1, Q2...Q]. M The following are the sequences to be assembled.

[0064] During the splicing process, the action posture sequence P and the action posture sequence Q need to be combined into a single action posture sequence X = [P1, P2, ..., PQ]. N Q1, Q2...Q M ], using the action pose sequence X as Figure 2 The image shows the input to the neural network used for smooth transition processing at the splicing point. This is for ease of subsequent processing... Figure 2 The neural network used for smooth transition processing at the splicing points is explained (especially its internal processing), which represents the action pose sequence X = [P1, P2...P...]. N Q1, Q2...Q M The equivalent representation is the first action pose sequence X = [X1, X2, ..., X]. t ...X N X N+1 X N+2 ...X N+M ], where t represents the time step, t is a positive integer, and the maximum value of t is N+M. The attitude sequence [X1, X2...X... t ...X N This is equivalent to the attitude sequence P = [P1, P2...P...]. N ], attitude sequence [X N+1 X N+2 ...X N+M The pose sequence is equivalent to [Q1, Q2...Q...]. M ].

[0065] Input the action posture sequence X Figure 2 The general process of the neural network used for smooth transition processing at the splicing point is as follows:

[0066] The first neural network 21 receives and processes the first action pose sequence X, and outputs the target position (preIndex, postIndex). The target position is the position of the frame to be replaced in the first action pose sequence predicted by the first neural network 21. Here, preIndex represents the target start position, and postIndex represents the target end position.

[0067] The second neural network 22 receives the first action pose sequence X and the target position (preIndex, postIndex), processes them, and outputs a transition action pose sequence F = [F1, F2...F...]. K ].

[0068] The filtering and splicing layer 23 receives the target position (preIndex, postIndex), the transitional action posture sequence F, and the first action posture sequence X. It replaces the action posture sequence located at the target position (preIndex, postIndex) in the first action posture sequence X with the transitional action posture sequence F, thus completing the splicing of the action posture sequences of the two target sign language words I and II, which correspond to the meanings of "today" and "weather" respectively, to obtain the action posture sequence Y output by the filtering and splicing layer 23. P+Q = [P1, P2...P I F1, F2...F K Q J+1 Q J+2 ...Q M ].

[0069] The assembly method for the action sequence of sign language words II and III, respectively, meaning "weather" and "good," is similar and will not be repeated here. When assembling the sequence [Q1, Q2...Q...]... M S1, S2...S Z ]enter Figure 2 The neural network shown is used for smoothing the transition at the stitching point, resulting in the action pose sequence Y output by the filtering stitching layer 23. Q+S = [Q1, Q2...Q A E1, E2...E B S D+1 S D+2 ...S Z ]

[0070] Based on this, the first neural network 21, the second neural network 22, and the filtering and splicing layer 23 will be described in detail below with some illustrative embodiments.

[0071] like Figure 3The diagram shows a schematic representation of the structure of the first neural network in the neural network provided in this embodiment for smooth transition processing at the splicing point. Please refer to... Figure 3 The first neural network 21 includes: a first encoder layer 211, a first decoder layer 212, a first attention layer 213, a first prediction layer 214, and a first output layer 215. In some optional embodiments, both the first encoder 211 and the first decoder 212 include at least one LSTM unit, where each LSTM unit is a recurrent gating unit. The following description uses an LSTM network with an attention mechanism as an illustrative example of the first neural network 211:

[0072] The first encoder 211 is used to encode each frame of the first action posture sequence in time steps to obtain the hidden state output by the first encoder at the corresponding time step.

[0073] The first encoder 211 encodes the pose frame corresponding to the time step in the first action pose sequence at one time step to obtain the hidden state and cell state output at that time step.

[0074] For example, the first encoder 211 can encode the pose of each frame in the first action pose sequence X in time steps. Specifically, at encoding time step t, the pose of the t-th frame in the first action pose sequence X is encoded. t Encode the hidden state at time step t to obtain the hidden state. and cell state Until the last time step t of the encoding N+M For the pose X of the N+Mth frame in the first action pose sequence X N+M Encode the hidden state. and cell state

[0075] like Figure 4 The diagram illustrates the structure of a recurrent gating unit in a neural network used for smooth transition processing at splicing points, as provided in an embodiment of this application. Please refer to... Figure 4 In the first encoder 211, the input to the cyclic gating unit includes: the cell state (or memory state) C obtained from the previous time step. t-1 (exist Figure 3 The transfer of the memory state between the two loop gating units in the diagram is shown by a relatively thin, parabolic arrow (H) obtained at the previous time step. t-1 (exist Figure 3The diagram shows the transfer of hidden states between the two loop gating units and between the loop gating unit and the first attention mechanism layer 213 (indicated by relatively thick arrows), as well as a frame of pose X from the action pose sequence input at the current time step. t .

[0076] The first encoder 211 contains three cyclic gating units: a forget gate, an input gate, and an output gate. The cyclic gating unit checks the attitude X of frame t at time step t. t The mathematical expressions for encoding processing are shown in equations (1)-(6), where I t It is the output of the input gate, F t It is the output of the forget gate, O t It is the output of the output gate, W xi W xf W xo and W hi W hf W ho It is the weight parameter, b i b f b o σ is the bias parameter, and σ is the sigmoid activation function.

[0077] I t =σ(X) t W xi +H t-1 W hi +b i (1)

[0078] F t =σ(X) t W xf +H t-1 W hf +b f (2)

[0079] O t =σ(X) t W xo +H t-1 W ho +b o (3)

[0080] The output of each cyclic gating unit includes the cell state C corresponding to the current time step t. t and hidden state H t Its mathematical expression is shown in formulas (4)-(6), where is Candidate state, C t By F t C t-1 I t and Together, tanh refers to the hyperbolic tangent function, and ⊙ represents the dot product operation.

[0081]

[0082]

[0083]

[0084] The first decoder 212 is used to decode each frame of the first action pose sequence based on the hidden state output by the last time step of the first encoder in a time step, so as to obtain the hidden state output by the first decoder at the corresponding time step.

[0085] During the decoding process, the first decoder 212, based on the hidden state output by the last time step of the first encoder, performs decoding processing on the pose frame corresponding to the time step in the first action pose sequence within a time step. (Note: This does not mean inputting the pose frames in the first action pose sequence into the decoder for decoding. "Decoding the pose frame corresponding to the time step in the first action pose sequence" means that the decoding at the current time step is equivalent to decoding at the time step in the first action pose sequence that is "aligned" with the current decoding time step. For example, the decoding of the first decoder 212 at time step t is equivalent to decoding the pose X of the first action pose sequence.) t (Decoding at time step t) yields the hidden state and cell state output at that time step.

[0086] For example, the first encoder 211 at the last time step t of the encoding... N+M Hidden state of output and cell state It will be passed to the first decoder 212 for decoding. The first decoder 212 receives the hidden state. and cell state Decoding then begins. The input for the first time step t1 of decoding includes: the hidden state. Cell state and the starting flag vector (e.g.) Figure 3 (As shown in the start diagram), decode to obtain the hidden state. Cell state The data input to the first decoder 212 at decoding time step t (other than the first time step t1) includes: Y output from the first attention mechanism layer. t-1 The memory state output by the first decoder 212 at the previous time step The hidden state output by the first decoder 212 at the previous time step The first decoder decodes until time step t. N+M End (e.g.) Figure 3 (as shown in the Stop section).

[0087] The first decoder 212 and the first encoder 211 have similar structures, the difference being that the first decoder 212 (at all time steps except the first time step t1) will use the output of the first attention mechanism layer 213 at time step t-1 as the input at time step t. The calculation formulas in the cyclic gating unit of the first decoder 212 are shown in formulas (7)-(12):

[0088] I t =σ(Y) t W xi +H t-1 W hi +b i (7)

[0089] F t =σ(Y) t W xf +H t-1 W hf +b f (8)

[0090] O t =σ(Y) t W xo +H t-1 W ho +b o (9)

[0091]

[0092]

[0093]

[0094] If a sequence-to-sequence neural network only has an encoder and a decoder, the encoder encodes the input sequence into a content unit and directly passes it to the decoder. However, this content unit is usually represented by a vector, and vectors have a limited dimension, inevitably losing some information from the input sequence. This results in the decoder relying on flawed underlying data, leading to low prediction accuracy. To mitigate this issue and reduce prediction accuracy, this embodiment introduces an attention mechanism into the sequence-to-sequence neural network. Figure 2 or Figure 3 The first attention mechanism layer 213 is shown in the figure. This attention mechanism is based on the hidden state generated at time step t during the decoding process of the first decoder 212. The hidden state H generated during the encoding process of the first encoder 211 (e)Determine the first attention feature value. During prediction, the hidden state of the first decoder 212 output at time step t is determined based on the first attention feature value. By concatenating the hidden states, we obtain the hidden states "selected" through the attention mechanism. And based on the obtained hidden state To improve the prediction accuracy of neural networks, predictions are made. The first attention mechanism layer 213 is explained below:

[0095] like Figure 5 The diagram illustrates the structure of the first attention mechanism layer in the neural network used for smooth transition processing at the splicing point, as provided in an embodiment of this application. Please refer to [reference needed]. Figure 3 and Figure 5 The first attention mechanism layer 213 is used to determine the first attention feature value. In some optional implementations, the first attention feature value G... t The first attention mechanism layer 213 utilizes the hidden state H output by the first encoder 211 at each time step. (e) The hidden state output by the first decoder 212 at the current time step Attention value a t The hidden state H output at each time step of the first encoder 211 (e) The result is obtained by weighted summation.

[0096] For example, still taking the first action pose sequence X as an example, the first attention feature value G is determined. t The mathematical operations are shown in the following formulas (13)-(16).

[0097]

[0098]

[0099]

[0100] G t =a t H (e) (16)

[0101] Where score is the scoring function, and exp is the exponential function with the natural constant e as the base. W is the transpose operator. a The weight matrix is ​​used. First, the hidden state generated by the first decoder 212 at time step t is calculated in formulas (13) and (14). For the hidden state generated by the first encoder 211 at time step t (“alignment”) Attention value a t(That is, the first attention mechanism layer 213 first determines the hidden state H output by the first encoder 211 at each time step.) (e) The hidden state output by the first decoder 212 at the current time step t Attention value a t Then, in formulas (15) and (16), the attention value a will be used. t The hidden state H output by the first encoder 211 at each time step (e) We perform a weighted summation to obtain the first attention feature value G. t (i.e., the first attention mechanism layer 213 then applies the attention value a) t The hidden state H output by the first encoder 211 at each time step (e) (Perform weighted summation).

[0102] The first prediction layer 214 uses the hidden state output by the first decoder and the first attention feature value determined by the first attention mechanism layer to make predictions at each time step, thereby obtaining a probability distribution matrix indicating the target position at each time step, and determining the candidate value for each time step based on the maximum value of each column in the probability distribution matrix of the target position at each time step.

[0103] Please refer to the above. Figure 3 and Figure 5 For example, still taking the first action pose sequence X as the input, when the first prediction layer 214 makes a prediction at time step t, it first sets the first attention feature value G. t The hidden state generated by the first decoder 212 at time step t splicing (i.e.) Figure 5 The `concat` operation described above (where `concat` is a function typically used to join two or more arrays) creates a hidden state. The calculation process is shown in formula (17). Then, for the hidden state... Perform a linear transformation and use softMax to obtain the probability distribution matrix Y. t The calculation process is shown in formula (18), and the probability distribution matrix Y is... t The position of each frame in the predicted first action pose sequence X at time step t is the confidence level of the target position. Next, the probability distribution matrix Y is determined. t Find the maximum value (max) in each column of Y', and start from Y'. t Extract the maximum value (max) from each column in Y as an element, and then sort it according to its position in Y. t Arrange the columns in the matrix Y in a row to form a new matrix Y. t That is, to obtain the candidate value Y predicted at time step t. tThe calculation process is shown in formula (19).

[0104]

[0105]

[0106] Y t =argMax(Y t ′) (19)

[0107] Please continue reading Figure 3 The first output layer 215 is used to determine the target position based on the candidate values ​​at each time step and the preset target position flag vector.

[0108] In some optional implementations, the target location marker vector includes a preset start marker vector and a preset end marker vector. When the first output layer 215 determines the target location based on the candidate value of each time step and the preset target location marker vector, it does so by the following steps: determining a first distance between the candidate value of each time step and the preset start marker vector and a second distance between the candidate value of each time step and the preset end marker vector, and determining the first candidate value with the smallest first distance and the second candidate value with the smallest second distance as the coordinate value of the target location.

[0109] For example, continuing with the first action pose sequence X as the input, as described above, the first prediction layer obtains the results at time step t1 to time step t2. N+M All candidate values ​​output {Y1, Y2...Y} t ...Y N+M After that, determine all candidate values ​​{Y1, Y2...Y}. t ...Y N+M The first distance between each candidate value Yt in} and the preset start flag vector preIndexFlag and the second distance between each candidate value Yt and the preset end flag vector postIndexFlag are used to determine the first candidate value with the smallest first distance and the second candidate value with the smallest second distance as the coordinates of the target position. That is, the first candidate value with the smallest first distance is determined as the target start position, and the second candidate value with the smallest second distance is determined as the target end position. The calculation process is shown in formulas (20)-(21).

[0110] preIndex = min t ||Y t -preIndexFlag|| 2 (20)

[0111] postIndex = min t ||Y t-postIndexFlag|| 2 (twenty one)

[0112] In some alternative implementations, in the first output layer 215, after determining the target position represented by candidate values, it is converted into its frame position in the first action pose sequence X. For example, after determining the target position at time step t... m Predicted candidate values and at time step t n Predicted candidate values Let preIndex be the target start position and postIndex be the target end position, then time step t in the first action pose sequence X is... m corresponding attitude frame and time step t n corresponding attitude frame The coordinates of the current location in frames m and n are determined as the starting and ending target positions, respectively. Here, m and n are both positive integers, and 1 < m < n < N + M.

[0113] Please see Figure 1 The first output layer 215 will input the coordinates of the determined target location into the second neural network 22 and the filtering and splicing layer 23.

[0114] While the above illustrative embodiments provide one way to implement a first neural network using an LSTM network with an attention mechanism, it should be understood that those skilled in the art can employ other possible implementation methods. For example, the LSTM network in the above illustrative embodiments can be replaced with a GRU network, with each GRU unit replacing an LSTM unit. Furthermore, although the GRU network does not have cell states, it does have hidden states. Therefore, a first attention mechanism layer can be added to the GRU network based on the principles of the above illustrative embodiments, thereby forming a GRU network with an attention mechanism. Alternatively, the first neural network can also be implemented using traditional sequence-to-sequence neural networks such as LSTM networks and GRU networks.

[0115] like Figure 6 The diagram illustrates the structure of the second neural network in the neural network provided in this application embodiment for smooth transition processing at the splicing point. Please refer to... Figure 6 The second neural network 22 includes: a sequence selection layer 221 (Culling Layer), a Transformer subnetwork, and a second output layer 227. The following section discusses... Figure 6 The second neural network 22 shown is explained in detail below:

[0116] The sequence selection layer 221 is used to perform position selection processing on the first action posture sequence to obtain the selected first action posture sequence.

[0117] The processing efficiency of the Transformer subnetwork decreases as the input sequence increases. In order to improve the processing efficiency of the Transformer subnetwork, a sequence selection layer 221 is added before the Transformer subnetwork in this embodiment. The sequence selection layer 221 selects the first action posture sequence input to the second neural network 22 based on the target position received from the first neural network 21, so as to reduce the amount of data of the sequence input to the Transformer subnetwork and thus improve the processing efficiency of the Transformer subnetwork.

[0118] According to one embodiment of the present invention, the selection process performed by the sequence selection layer 221 on the first action pose sequence includes: deleting pose sequences located at the target position in the first pose sequence according to the target position, thereby obtaining the selected first action pose sequence. For example, the first action pose sequence X described above is still taken as input. Upon receiving the target position (preIndex, postIndex) and the first action pose sequence X transmitted by the first neural network 21, the sequence selection layer 221 deletes pose frames located between the target position (preIndex, postIndex) in the first action pose sequence X, thereby obtaining the selected first action pose sequence X. c = [X1, X2...X t ...X preIudex X postIndex ...X N+M ].

[0119] According to one embodiment of the present invention, the sequence selection process of the second neural network further includes the following sub-step: acquiring a preset number of frames before the target position and a preset number of frames after the target position to form a selected first action pose sequence. The preset number of frames is the number of frames selected from multiple sets of candidate frames that optimizes the accuracy of the trained neural network. In other words, the selection process performed by the sequence selection layer 221 on the first action pose sequence includes: extracting the poses of each frame fr before and after the target position from the first action pose sequence based on the target position and the preset number of frames fr (which can be considered as a hyperparameter of the network) to obtain the selected first action pose sequence. fr, i.e., the preset number of frames, is the number of frames selected from multiple sets of candidate frames that optimizes the overall accuracy of the first and second neural networks after training. For example, the aforementioned first action pose sequence X is used as the input. Upon receiving the target position (preIndex, postIndex) and the first action pose sequence X transmitted by the first neural network 21, the sequence selection layer 221 extracts pose frames located before and after the target position (preIndex, postIndex) from the first action pose sequence X to obtain the selected first action pose sequence X. c Sequence X c It will be input into the Transformer subnetwork, sequence X c Represented as X c =[X preIndex-fr+1 , ..., X preIudex X preIndex , ..., X preIndex+fr-1 The technical solution of this embodiment can achieve at least the following beneficial technical effects: This embodiment selects the first action posture sequence by choosing the number of frames that optimizes the overall accuracy of the trained neural network from multiple sets of candidate frames. This not only shortens the length of the input sequence, reduces the amount of data processing, and improves the splicing efficiency, but also improves the overall accuracy of the trained neural network in smoothly transitioning the connection between the action posture sequences corresponding to the two target sign language words to be spliced, thus improving the processing effect and making the sign language animation smoother and more fluid, thereby enhancing the user experience.

[0120] The Transformer sub-network includes: a sequence embedding layer 222, a position embedding layer 223, a second encoder 224, a second decoder 225, and a linear regression layer 226 (Linear & SoftMax Layer). Overall, the Transformer sub-network performs sequence embedding, position encoding, encoding, decoding, and linear regression processing on the selected first action pose sequence to obtain candidate action pose sequences. The Transformer sub-network is further explained below:

[0121] Sequence embedding layer 222 will input data X c The dimension is increased by d, and the calculation process is shown in formula (22). E is the output tensor after the dimension is increased.

[0122] E = Embedding(X) c ) (twenty two)

[0123] The position encoding layer 223 first generates a position embedding tensor, using a fixed position encoding method, and the calculation formula is shown in (23):

[0124]

[0125] Here, pos represents the position of the first dimension of data X. In the position embedding matrix, even positions are calculated using sin, and odd positions are calculated using cos. Finally, the generated position embedding tensor is added to the tensor output by the sequence embedding layer to obtain the tensor P embedded with position information.

[0126] The transition frame sequence generation model consists of two parts: an encoder and a decoder. The encoder encodes the input data and outputs the encoded data sequence and a sequence mask to the decoder. The encoder and decoder have similar network structures; in this embodiment, P... (e) As input to the encoding part, P (d) As input to the decoding part.

[0127] The second encoder 224 includes a multi-head attention layer and a feedforward layer.

[0128] A multi-head attention layer is composed of multiple self-attention layers. Each self-attention layer first processes the input P... (e) Perform a linear transformation to generate three tensors Q iK i and V i Next, calculate Q. i and To prevent the product of these terms from becoming too large, a scaling factor is introduced. Where d k For matrix K i The second dimension is then normalized using softmax, and multiplied by V. i The summation representation of the weights is obtained from the head. i Finally, all heads i The results are aggregated and transformed through a learnable linear projection to produce the final output Z. The calculation formulas are shown in (24)-(26), where... W is the weight matrix of the i-th self-attention layer. o It is a linear projection transformation.

[0129]

[0130]

[0131] Z=MHA(Q,K,V)=[head1,…head h ]·W o (26)

[0132] The feedforward layer provides nonlinear transformation and mainly consists of three layers. The first and third layers are linear transformation layers, and the second layer is a nonlinear activation layer with ReLU as the activation function. Finally, after residual connection, the output result O is calculated as shown in (27), where W1, W3, b1 and b3 are the weight parameters and bias parameters of the first and third layers, respectively.

[0133] O (e) =max(0,ZW1+b1)W3+b3+Z (27)

[0134] The second decoder 225 includes a masked multi-head attention layer, an encoder-decoder multi-head attention layer, and a feedforward layer.

[0135] Masked Multi-Head Attention introduces a mask on top of the multi-head attention layer in the encoding part. This mask is used to eliminate interference from input data after a certain time step when predicting the result at a certain time step. It requires the summation representation of the weights in the head. i When a mask matrix M is added, the calculation formula is shown in (28).

[0136]

[0137] The difference between the encoder-decoder multi-head attention layer and the multi-head attention layer in the second encoder 224 is that the input data comes not only from the information of the decoding part, but also from the output information of the encoding part. The calculation formulas are shown in (29)-(31).

[0138]

[0139]

[0140]

[0141] The feed-forward layer is the same as the corresponding layer in the coding part, and its output is O. (d) .

[0142] Linear and regression layers 226 (Linear & Softmax) will O (d) The input is a fully connected layer (not shown in the figure) and softmax normalized. The calculation formulas are shown in (32)-(33), where W L F is the weight matrix of the fully connected layer, and F′ is the probability distribution matrix (probability distribution matrix of candidate action pose sequences) after processing by the softmax function. t This represents the predicted value of the transitional motion posture of the posture frame corresponding to time step t.

[0143] F t = softmax(W) L O (d) (32)

[0144] F t =argMax(F t ′) (33)

[0145] The second output layer 227 is used to determine the transition action posture sequence based on the candidate action posture sequence and the preset flag frame indicating the transition action posture sequence.

[0146] In some optional implementations, the second output layer 227 is used to determine a transitional action posture sequence based on the candidate action posture sequence and a preset flag frame indicating a transitional action posture sequence, including: determining a third distance between each candidate frame in the candidate action posture sequence and a preset start flag frame and a fourth distance between each candidate frame and a preset end flag frame, and determining the candidate frame between the first candidate frame with the smallest third distance and the second candidate frame with the smallest fourth distance as the transitional action posture sequence.

[0147] For example, still using the aforementioned first action pose sequence X as input. After obtaining the predicted transition action poses {F1, F2...F...} at all time steps... t ...F N+M After that, calculate the predicted transition poses for all time steps and the preset start flag frame Y. preIndex The third distance between them and the preset end flag frame Y postIndex The fourth distance between them determines the candidate frames between the first candidate frame with the smallest third distance and the second candidate frame with the smallest fourth distance as the transition action pose sequence O = (F a F a+1 F b The calculation process is shown in formulas (34)-(35).

[0148] a = min t ||F t -Y preIndex || 2 (34)

[0149] b = min t ||F t -Y postIndex || 2 (35)

[0150] The filter splicing layer 23 is used to replace the action posture sequence located at the target position in the first action posture sequence with the transition action posture sequence, thereby completing the splicing of the action posture sequences of the two target sign language words before and after.

[0151] For example, still taking the aforementioned first motion pose sequence X as input. The filtering and stitching layer 23 receives the first motion pose sequence X, the target position (preIndex, postIndex), and the transitional motion pose sequence O. The filtering and stitching layer 23 replaces the motion poses in the first motion pose sequence X located between the preIndex and postIndex frames with the transitional motion pose sequence O, and finally outputs the stitched motion pose sequence Y. Sequence Y and Y P+Q Same, that is, Y = Y P+Q That is, Y = [X1, X2, ..., X...] preIndex F a F2, ..., F b X postIndex X postIndex+1 , ..., X N+M ].

[0152] While the above illustrative embodiments illustrate one way of implementing a second neural network using a Transformer network with a sequence selection layer, it should be understood that those skilled in the art can employ other possible implementations. For example, the Transformer subnetwork in the above illustrative embodiments can be replaced with a ConvS2S network or an LSTM. Therefore, a sequence selection layer can also be added to a ConvS2S network or LSTM based on the principles of the above illustrative embodiments, thereby forming a ConvS2S network or LSTM with a sequence selection layer. Alternatively, the second neural network can also employ a traditional Transformer network, ConvS2S network, or LSTM network.

[0153] Neural networks need to be trained before they can be used for actual predictions. Figure 2 Based on the explanation of the structure of the neural network shown, the following will discuss... Figure 2 The training of the neural network shown is explained.

[0154] According to an embodiment of the present invention, the first neural network 21 and the second neural network 22 are obtained through the following steps:

[0155] Obtain a training set, which includes multiple samples and a label for each sample. Each sample includes the action posture sequences corresponding to the preceding and following sign language words to be concatenated. The labels include a position label indicating the frame to be replaced in the corresponding sample and a transition action posture sequence label indicating the ground truth value of the transition action posture sequence for the corresponding sample. The position label is used to indicate the position of the frame to be replaced in the corresponding sample.

[0156] The first neural network and the second neural network are trained iteratively through multiple rounds using the training set to obtain the trained first neural network and the trained second neural network. Specifically, during the training of the first neural network, gradients are calculated based on the probability distribution matrix of the target position at all time steps and the loss determined by the position labels, and the parameters of the first encoder, first decoder, first attention mechanism layer, and first prediction layer are updated via backpropagation. Since the first output layer of the first neural network has no trainable parameters, it does not need to be updated. During the training of the second neural network, gradients are calculated based on the probability distribution matrix of the candidate action pose sequence and the loss determined by the transition action pose sequence labels, and the parameters of the Transformer sub-network are updated via backpropagation. Since the sequence selection layer and the second output layer of the second neural network have no trainable parameters, they do not need to be updated.

[0157] Below is an example of a training process. Assume the original dataset consists of two parts: each sign language word A (before the virtual human model begins to sign the word, the upper limbs are naturally vertically positioned at the sides of the body with the hands close to the waist; when the sign language begins, the upper limbs, especially the hands to the elbows, are raised from the waist to make the gesture; after the gesture for a sign language word is completed, the upper limbs, especially the hands to the elbows, return to their natural position at the sides of the body with the hands close to the waist. Therefore, the initial and final segments of the sign language word animation generally contain these useless animation frames of the arms raising and / or falling, which are not necessary for conveying the "thought") and sentences B composed of these words. Each sign language word A and each sentence B corresponds to a complete motion capture data file (skeleton rotation data). Each sign language sentence B is divided into several consecutive segments, each segment may be a word segment C or a transition segment D, with C and D alternating. An animation segment of a word C is the middle part of an animation segment of one word from each word A, with start and end markers. During training, sentence B is segmented according to word intervals to form the structure C1DC2. Thus, A1 and A2, corresponding to C1 and C2, become the input to the dataset. The start marker of C2, the end marker of C1, and D are the labels. Position labels can be represented in the following way:

[0158]

[0159] Where Y is the location label corresponding to the sample input to the model. It is the starting position label of the frame being replaced. It is the end position label of the frame being replaced.

[0160] The first neural network 21 is trained based on cross-entropy loss, and the calculation formula is shown in formula (36):

[0161]

[0162] In each round, the gradient is calculated based on the output probability distribution matrix φ' of the first prediction layer and the label Y, and the first neural network is updated based on backpropagation of the gradient. It should be noted that during backpropagation update, the parameters of the first output layer 215 in the first neural network 21 are not updated, but the parameters of the first prediction layer 214, the first attention mechanism layer 213, the first decoder 212, and the first encoder 211 are updated.

[0163] Transitional action posture sequence labels can be represented as:

[0164]

[0165] in, and The sequence of actions and poses between them is the pose sequence corresponding to the actual transition frames.

[0166] The second neural network 22 is trained based on cross-entropy loss, and the calculation formula is shown in formula (37):

[0167]

[0168] In each round, the probability distribution matrix F of the candidate action pose sequence obtained by the Transformer subnetwork is... t The gradient between label Y and the gradient is calculated, and the second neural network 22 is updated based on backpropagation of the gradient. It should be noted that during the backpropagation update, the parameters of the second output layer 227 in the second neural network 21 are not updated, but the parameters of the Transformer subnetwork and the sequence selection layer 221 are updated.

[0169] In some optional implementations, during the pre-training of the first neural network 21 and the second neural network 22, candidate parameters fr are selected from a preset set of selection parameters {fr} as selection parameters for the sequence selection layer 221 in the second neural network 22 to perform multiple rounds of training on the first neural network 21 and the second neural network 22. In each round of training, a candidate frame number fr that has not yet participated in training is selected from the set of candidate frame numbers {fr} to participate in the training.

[0170] The overall accuracy of the first neural network 21 and the second neural network 22 obtained in each of the multiple training rounds is evaluated. The overall accuracy is used to describe the positional error between the target position predicted by the first neural network 21 and the position label of the sample, and the length error between the transition action posture sequence predicted by the second neural network 22 and the transition action posture sequence label of the sample.

[0171] In some alternative implementations, the overall accuracy of the model describes the overall error between the predicted sign language segment Y and the actual sign language segment T. Illustratively, the overall accuracy is calculated using the following formula:

[0172]

[0173] Where t is the pose frame number, L is the actual sign language segment length, and T t It is the t-th pose frame, Y t×K / L It is the label in the transitional action pose sequence label that is closest to the value of the t-th pose frame, where K is the predicted sign language segment length, ||T t -Y t×K / L || is the accuracy of the predicted attitude frame, ||LK|| is the length error, and λ is a coefficient used to balance the accuracy and length error of the predicted attitude frame.

[0174] Step S103: Based on the target action posture sequence, drive the posture of the virtual human body model to change, and generate the target sign language animation.

[0175] In some optional implementations, after obtaining the target action posture sequence corresponding to the sign language to be generated, the virtual human body model is driven to change its posture using this target action posture sequence. That is, the virtual human body model is driven to sequentially and continuously perform each posture in the target action posture sequence according to the time sequence, thereby forming the target sign language animation. This target sign language animation can be recorded as a video, inserted into a video that requires sign language translation, or displayed in real time in a live video that requires sign language translation (e.g., inserting a display window of a virtual human body model into the interface of a live video).

[0176] For example, when obtaining the target action posture sequence Y P+Q+S Then, the virtual human body model can be driven to press [P1, P2...P... I F1, F2...F K Q J+1 Q J+2 ...Q A E1, E2...E B S D+1 S D+2 ...S Z In the order of [], poses P1, P2, ..., P3 are made sequentially. I Posture F1, Posture F2, ..., Posture F K Posture Q J+1 Posture Q J+2 ..., posture Q A ..., Posture E1, ..., Posture E2, ..., Posture E B ..., posture S D+1 ..., posture S D+2 ..., posture S Z And make the process into an animation X P+Q+S Obtain the target sign language animation X P+Q+S = [P1, P2...P I F1, F2...F K Q J+1 Q J+2 ...Q A E1, E2...E B S D+1 S D+2 ...S Z The sign language animation means "The weather is nice today." Specifically, P1 to S... Z They represent the target sign language animation X respectively.P+Q+S In the animation frames, for example, P1 represents the target sign language animation X. P+Q+S The first animation frame in the video.

[0177] In this embodiment, a trained neural network is used to concatenate the action posture sequences corresponding to two sign language words that need to be sequentially concatenated from all the target sign language words required to generate the sign language animation. This results in a target action posture sequence, which is then used to drive the posture transformation of a virtual human model, thereby generating the target sign language animation. This not only improves the quality of transition frames between adjacent sign language word animations—making the transitions between adjacent sign language word animations more coherent, natural, and smooth—but also increases the efficiency of sign language animation generation, meeting real-time requirements.

[0178] In the Figure 2 The illustrated process of training the neural network is as follows: Figure 7 The diagram shows a flowchart of some steps in the animation generation method provided in this application embodiment. Please refer to... Figure 7 In the aforementioned step S102, the predetermined splicing method includes processing the action posture sequence corresponding to any two consecutive target sign language words that need to be spliced ​​sequentially among all the target sign language words in the following manner:

[0179] S701, For two target sign language words that need to be spliced ​​in sequence, merge the action posture sequences corresponding to the two target sign language words into the first action posture sequence.

[0180] S702, the first action posture sequence is input into the trained first neural network for processing to obtain the target position, wherein the target position is the position of the frame to be replaced in the first action posture sequence predicted by the first neural network.

[0181] S703, the target position and the first action posture sequence are input into a trained second neural network for processing to obtain a transition action posture sequence;

[0182] S704, replace the action posture sequence located at the target position in the first action posture sequence with the transition action posture sequence to complete the splicing of the action posture sequences of the two target sign language words.

[0183] For the detailed process of steps S701 to S704, please refer to the previous text. Figure 2 The structure and working principle of the neural network shown are explained, and the processing procedures are the same or similar, so they will not be repeated here.

[0184] It should be noted that although the steps are described in a specific order above, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently, or even in a different order, as long as the required function can be achieved.

[0185] The sign language animation generation method provided in this application can be executed by a sign language animation generation method. This application uses a sign language animation generation method as an example to illustrate the sign language animation generation apparatus provided in this application.

[0186] like Figure 10 The diagram shows a schematic representation of the sign language animation generation device provided in an embodiment of this application. Please refer to [link / reference]. Figure 10 The sign language animation generation device 90 includes:

[0187] The first acquisition module 901 is used to acquire the action posture sequence of the target sign language word from a preset sign language lexicon. The target sign language word is the sign language word required to generate the sign language. Each frame of the action posture sequence indicates at least the upper limb posture of the virtual human body model.

[0188] The first splicing module 902 is used to splice the action posture sequences corresponding to all target sign language words according to a predetermined splicing method in the semantic order of the sign language to be generated, so as to obtain the target action posture sequence. In the predetermined splicing method, a trained neural network is used to perform smooth transition processing on the connection between the action posture sequences corresponding to the two target sign language words to be spliced.

[0189] The first driving module 903 is used to drive the virtual human body model to change its posture according to the target action posture sequence, thereby generating the target sign language animation.

[0190] In some optional implementations, the first splicing module 902 includes:

[0191] The first splicing submodule is used to merge the action posture sequences corresponding to the two target sign words that need to be spliced ​​in sequence into a first action posture sequence.

[0192] A first neural network is used to process the first action posture sequence to obtain a target position, wherein the target position is the position of the frame to be replaced in the first action posture sequence predicted by the first neural network.

[0193] The second neural network is used to input the target position and the first action posture sequence into the trained second neural network for processing to obtain a transition action posture sequence.

[0194] A filtering and splicing layer is used to replace the action posture sequence located at the target position in the first action posture sequence with the transition action posture sequence, thereby completing the splicing of the action posture sequences of the two target sign language words before and after.

[0195] In some optional implementations, the first neural network is a sequence-to-sequence model, including a first encoder, a first decoder, a first attention mechanism layer, and a first prediction layer. The first prediction layer makes a prediction at each time step using the hidden state output by the first decoder and a first attention feature value determined by the first attention mechanism layer. The first attention feature value is obtained by weighted summing of the hidden state output by the first encoder at each time step relative to the attention value of the hidden state output by the first decoder at the current time step.

[0196] In some alternative implementations, in the first neural network,

[0197] The first encoder is used to encode each frame of the first action pose sequence in time steps to obtain the hidden state output by the first encoder at the corresponding time step.

[0198] The first decoder is used to perform time-step decoding on each frame of the first action pose sequence based on the hidden state output by the last time step of the first encoder, so as to obtain the hidden state output by the first decoder at the corresponding time step.

[0199] The first attention mechanism layer is used to determine the first attention feature value;

[0200] The first prediction layer is used to make predictions at each time step using the hidden state output by the first decoder and the first attention feature value determined by the first attention mechanism layer, to obtain a probability distribution matrix indicating the target position at each time step, and to determine the candidate value at each time step based on the maximum value of each column in the probability distribution matrix of the target position at each time step.

[0201] The first output layer is used to determine the target position based on the candidate values ​​at each time step and the preset target position flag vector.

[0202] In some optional implementations, the target location marker vector includes a preset start marker vector and a preset end marker vector. The first output layer is used to determine the target location based on the candidate value at each time step and the preset target location marker vector, including: determining a first distance between the candidate value at each time step and the preset start marker vector and a second distance between the candidate value at each time step and the preset end marker vector, and determining the first candidate value with the smallest first distance and the second candidate value with the smallest second distance as the coordinate value of the target location.

[0203] In some alternative implementations, the second neural network includes:

[0204] A sequence selection layer is used to perform position selection processing on the first action posture sequence to obtain a selected first action posture sequence.

[0205] The Transformer subnetwork is used to perform sequence embedding, position encoding, encoding, decoding and linear regression processing on the selected first action pose sequence to obtain candidate action pose sequences;

[0206] The second output layer is used to determine the transition action posture sequence based on the candidate action posture sequence and the preset flag frame indicating the transition action posture sequence.

[0207] In some alternative implementations, the second output layer includes:

[0208] The transition action posture sequence determination layer is used to determine the third distance between each candidate frame in the candidate action posture sequence and the preset start flag frame and the fourth distance between each candidate frame and the preset end flag frame. The candidate frames between the first candidate frame with the smallest third distance and the second candidate frame with the smallest fourth distance are determined as the transition action posture sequence.

[0209] In some alternative implementations, the first neural network and the second neural network are trained through the following steps:

[0210] Obtain a training set, which includes multiple samples and labels corresponding to each sample. Each sample includes the action posture sequences corresponding to the preceding and following sign words that need to be concatenated. The labels include a position label indicating the frame to be replaced in the corresponding sample and a transition action posture sequence label indicating the ground truth value of the transition action posture sequence of the corresponding sample.

[0211] The first neural network and the second neural network are trained in multiple rounds of alternating iterations using the training set to obtain the trained first neural network and the trained second neural network.

[0212] In this embodiment, a trained neural network is used to concatenate the action posture sequences corresponding to two sign language words that need to be sequentially concatenated from all the target sign language words required to generate the sign language animation. This results in a target action posture sequence, which is then used to drive the posture transformation of a virtual human model, thereby generating the target sign language animation. This not only improves the quality of transition frames between adjacent sign language word animations—making the transitions between adjacent sign language word animations more coherent, natural, and smooth—but also increases the efficiency of sign language animation generation, meeting real-time requirements.

[0213] The sign language animation generation device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0214] To verify the effectiveness of the model, the inventors of this application conducted a comparative experiment based on the National (China) General Sign Language 3D Database. This experiment mainly demonstrates the prediction results of different neural network frameworks and conducts a comparative analysis. First, the dataset is introduced; second, the evaluation criteria are given; then, the interpolation transition results of the dataset on four different models are presented; finally, the accuracy and effect are evaluated.

[0215] (1) Dataset

[0216] The national standard sign language 3D database used includes 6707 sign language words. Each sign language word is the rotation data of 53 levels of human skeleton, represented in the form of skeletal animation, starting and ending with the arms falling naturally. The experiment of this invention uses 10000 concatenated words composed of 100 of these sign language words.

[0217] First, the 3D rotation data of each frame of the sign language animation is converted into position coordinates (x, y, z). Then, the coordinate data is normalized to the range [-0.5, 0.5]. Next, the coordinates of the 53 bones are unfolded to form a 159-dimensional vector. The predefined flag vectors preIndexFlag and postIndexFlag are 159-dimensional vectors composed of the numbers 0.6004 and 0.6005, respectively.

[0218] (2) Evaluation indicators

[0219] To evaluate the overall performance, the overall accuracy of the first neural network 21 and the second neural network 22 obtained in each of the multiple training rounds can be assessed. The overall accuracy is used to describe the positional error between the target position predicted by the first neural network 21 and the position label of the sample, as well as the accuracy between the transitional action posture sequence predicted by the second neural network 22 and the transitional action posture sequence label of the sample.

[0220] Overall accuracy: The experiment was defined using the aforementioned formula (38).

[0221] Bilingual Evaluation Understudy (BLEU): BLEU is used to measure the similarity between candidate sentences and reference sentences in natural language processing tasks. This invention uses it to measure the degree of overlap between predicted transition frames and actual transition frames. When evaluating BLEU scores, the experiment divides the predicted frame sequence into subsequences of different lengths n and calculates the proportion of the predicted sequence appearing in the actual sequence. For example, given the predicted sequence [T1, T2, T3, ..., T...]... m The true sequence [L1, L2, L3, ..., L] is given. m When calculating the BELU-2 score, the predicted sequence can be divided into m-n+1 subsequences, namely m-1 subsequences [T1, T2], [T2, T3], ..., [T...]. m-1 T m ], when [L] exists i L j When a value equals one of the predicted subsequences, the model considers that the predicted subsequence to appear in the true subsequence. In this invention, the experimentally measured BLEU range is n∈[1,…,4], and the corresponding result is represented as BLEU-n.

[0222] The proportion of successfully predicted splicing position data to all data (PTA): In this work, the model's ability to predict splicing positions is crucial, so this invention proposes a metric:

[0223]

[0224] Where N P N is the number of test set samples that conform to Formula 13. T It represents the total number of data points in the test set.

[0225] The ratio of the predicted splicing position to the actual position (PAA): Even if some models can predict the splicing position, the predicted splicing position may not be the same as the actual splicing position. Therefore, this invention proposes a metric:

[0226]

[0227] Where N A This indicates the number of test set samples where the predicted splicing position is basically the same as the actual splicing position (with a position error between plus or minus one).

[0228] (3) Experimental Results

[0229] This experiment was conducted using an NVIDIA GeForce RTX 3090 on a Windows 10 platform, with PyCharm as the experimental environment.

[0230] The comparison models include ConvS2S, Lstm, and Transformer. ConvS2S has an encoder and decoder each consisting of 20 corresponding layers, an embedding and hidden layer size of 512, a CNN kernel size of 3, and a batch size of 100. Lstm has an embedding and hidden layer size of 512, and a batch size of 32. Transformer contains one encoder layer and one decoder layer, an attention module with 8 heads, and an embedding size of 512.

[0231] All four models were trained using SGD with an initial learning rate of 10.-3 The results are as follows.

[0232] The experiment first attempted to find the optimal parameters for the Culling layer by using different fr values. The overall accuracy varies with increasing fr values ​​as follows: Figure 9 As shown in the figure, the overall accuracy was optimal when fr = 8 in this experiment.

[0233] Table 1 shows the index results of each model in the comparative experiment, including BELU-1 to BELU-4, PTA, PAA, and Accuary index.

[0234] Table 1 compares the index results of each model in the experiment.

[0235]

[0236] Table 2 Average Forecast Time

[0237]

[0238] (4) Results Analysis and Evaluation

[0239] As shown in Table 1, the ConvS2S, LSTM, and Transformer models achieved high BLEU-1 scores, indicating that they perform well in predicting single data points. However, their BLEU-2 / 3 / 4 scores dropped sharply, while the present invention remained stable, suggesting that the present invention outperforms the other three models in predicting long sequences.

[0240] As shown in Table 1, the PTA of ConvS2S is 0.251, meaning that this model almost fails to predict the splice positions. The PTAs of LSTM and Transformer are close to 1, indicating that they predict almost all splice positions. However, their PAA is as low as 0.784 and 0.601, respectively, proving that most of the predicted splice positions are incorrect. The PTA and PAA metrics of this invention are superior to the other three models, demonstrating the effectiveness of the model in predicting transition positions.

[0241] (5) Visualization of sign language animation splicing

[0242] This experiment only visualized data in the x and y directions. To highlight the sign language gestures, only the 47 bones that represent the upper body posture were displayed. The visualization results are as follows: Figure 8 As shown, that is Figure 8 The embodiments provided in this application are illustrated schematically. Figure 2 The neural network shown is the experimental group, with ConvS2S networks, LSTM networks, and Transformer networks serving as control groups. (From...) Figure 8It is understood that the quality of the transition (pose) frame (or transition animation frame) generated between the pose frame (or animation frame) of the preceding sign word and the pose frame (or animation frame) of the following sign word in the embodiments of this application is better than the transition (pose) frames generated by ConvS2S network, LSTM network and Transformer network.

[0243] Experimental results show that the model of this invention is almost identical to the actual model in terms of transition frame position and gesture prediction, and there is no problem of a sharp drop in accuracy leading to severe gesture distortion as the number of transition frames increases.

[0244] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0245] Computer-readable storage media can be tangible devices that hold and store instructions for use by an instruction execution device. Computer-readable storage media can be, for example, including but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0246] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating sign language animation, characterized in that, The method includes the following steps: Obtain the action posture sequence of the target sign language word from a pre-set sign language lexicon. The target sign language word is the sign language word required to generate the sign language. Each frame of the action posture sequence indicates at least the upper limb posture of the virtual human body model. According to a predetermined splicing method, all action posture sequences corresponding to the target sign language words are spliced ​​together in the semantic order of the sign language to be generated to obtain the target action posture sequence. The predetermined splicing method utilizes a trained neural network to smoothly transition the connection between the action posture sequences corresponding to two target sign language words to be spliced. The trained neural network includes a trained first neural network and a trained second neural network. The predetermined splicing method includes processing the action posture sequences corresponding to any two target sign language words that need to be spliced ​​sequentially in the following manner: for any two target sign language words that need to be spliced ​​sequentially... For each target sign language word, the corresponding action posture sequences of the two target sign language words are sequentially merged into a first action posture sequence. The first action posture sequence is then input into a trained first neural network for processing to obtain a target position, which is the position of the frame to be replaced in the first action posture sequence predicted by the first neural network. The target position and the first action posture sequence are then input into a trained second neural network for processing to obtain a transition action posture sequence. The transition action posture sequence is then used to replace the action posture sequence located at the target position in the first action posture sequence, thus completing the splicing of the action posture sequences of the two target sign language words. Based on the target action posture sequence, the posture of the virtual human body model is driven to change, generating the target sign language animation.

2. The method according to claim 1, characterized in that, The first neural network is a sequence-to-sequence model, including a first encoder, a first decoder, a first attention mechanism layer, and a first prediction layer. The first prediction layer makes a prediction at each time step using the hidden state output by the first decoder and the first attention feature value determined by the first attention mechanism layer. The first attention feature value is obtained by weighted summing of the hidden state output by the first encoder at each time step relative to the attention value of the hidden state output by the first decoder at the current time step.

3. The method according to claim 2, characterized in that, In the first neural network, The first encoder is used to encode each frame of the first action pose sequence in time steps to obtain the hidden state output by the first encoder at the corresponding time step. The first decoder is used to perform time-step decoding on each frame of the first action pose sequence based on the hidden state output by the first encoder at the last time step, so as to obtain the hidden state output by the first decoder at the corresponding time step. The first attention mechanism layer is used to determine the first attention feature value; The first prediction layer is used to make predictions at each time step using the hidden state output by the first decoder and the first attention feature value determined by the first attention mechanism layer, to obtain a probability distribution matrix indicating the target position at each time step, and to determine the candidate value at each time step based on the maximum value of each column or the maximum value of each row in the probability distribution matrix indicating the target position at each time step. The first output layer is used to determine the target position based on the candidate value of each time step and the preset target position flag vector.

4. The method according to claim 3, characterized in that, The target location flag vector includes a preset start flag vector and a preset end flag vector. The first output layer is used to determine the target location based on the candidate value at each time step and the preset target location flag vector in the following manner: Determine the first distance between the candidate value of each time step and the starting flag vector and the second distance between the candidate value and the ending flag vector, and determine the first candidate value with the smallest first distance and the second candidate value with the smallest second distance as the coordinate value of the target position.

5. The method according to claim 1, characterized in that, The second neural network includes: A sequence selection layer is used to perform position selection processing on the first action posture sequence to obtain a selected first action posture sequence. The Transformer subnetwork is used to perform sequence embedding, position encoding, encoding, decoding and linear regression processing on the selected first action pose sequence to obtain candidate action pose sequences; The second output layer is used to determine the transition action posture sequence based on the candidate action posture sequence and the preset flag frame indicating the transition action posture sequence.

6. The method according to claim 5, characterized in that, The flag frames include a start flag frame indicating the beginning of the transition action posture and an end flag frame indicating the end of the transition action posture. The second output layer is used to determine the transition action posture sequence based on the candidate action posture sequence and the preset flag frames indicating the transition action posture sequence in the following manner: determine the third distance between each candidate frame in the candidate action posture sequence and the start flag frame and the fourth distance between each candidate frame and the end flag frame, and determine the candidate frame between the first candidate frame with the smallest third distance and the second candidate frame with the smallest fourth distance as the transition action posture sequence.

7. The method according to any one of claims 1-6, characterized in that, The first neural network and the second neural network were obtained through the following steps: Obtain a training set, which includes multiple samples and labels corresponding to each sample. Each sample includes the action posture sequences corresponding to the preceding and following sign words that need to be concatenated. The labels include a position label indicating the frame to be replaced in the corresponding sample and a transition action posture sequence label indicating the ground truth value of the transition action posture sequence of the corresponding sample. The first neural network and the second neural network are trained in multiple rounds of alternating iterations using the training set to obtain the trained first neural network and the trained second neural network.

8. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method according to any one of claims 1 to 7.

9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the electronic device to perform the steps of the method as described in any one of claims 1 to 7.

Images