Sign language animation automatic generation method and device based on semantic driving, and computer storage medium

Abstract

The invention discloses a sign language animation automatic generation method and device based on semantic driving and a computer storage medium, and the method comprises the steps: obtaining a text, carrying out the semantic analysis of the text, and obtaining a target sign language word; the target sign language word is a sign language word required by a sign language to be generated; according to a semantic analysis result, obtaining a sign language skeleton animation of the target sign language word from a sign language vocabulary library; each frame of the sign language skeleton animation at least comprises point-line architecture postures of the upper limbs and the head; and splicing all the obtained sign language skeleton animations according to a semantic sequence of the text, and generating a transition skeleton animation between two adjacent sign language skeleton animations to obtain a target skeleton animation. In the prior art, a mode of directly calculating and processing opponent word animations often needs to process a large number of details and overall effects, and according to the sign language animation generation method, only the postures of the point-line architecture need to be recognized and calculated in the generation stage, so that the calculation amount is greatly reduced, and the sign language animation generation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of sign language animation, and in particular to a semantically driven sign language animation automatic generation method and device, and a computer storage medium. Background Art

[0002] In natural language, words are combined into sentences, sentences are combined into paragraphs, and paragraphs are combined into chapters. Sign language is similar, with the smallest granularity being a sign language word, which is combined into a sign language sentence, which is combined into a sign language paragraph, and which is combined into a sign language chapter.

[0003] When making a sign language animation, usually the sign language word animation corresponding to each sign language word is made in advance based on the sign language standard (such as the "National General Sign Language Dictionary" implemented in China in October 2019), and then the sign language word animation required for the sign language animation to be made is connected in the order of the semantics of the sign language animation to be made. A transition frame needs to be generated between the two sign language word animations that need to be connected in sequence, and the transition frame plays a role in connecting the two sign language word animations.

[0004] However, conventional sign language animation generation methods directly calculate and process sign language word animations, which requires large amounts of data and calculations. In addition, the transition amplitudes between sign language word animations are different, so the transition time between each gesture is also different. Transition animations are made by relying on the interpolation method, which fails to take into account the speed and correlation of the previous and next movements. The quality of the generated transition frames is poor, resulting in the inability to correctly express sign language movements.

[0005] Therefore, it is necessary to provide a semantically driven automatic generation method and device for sign language animation, and a computer storage medium to solve the above technical problems. Summary of the invention

[0006] The present invention overcomes the deficiencies of the prior art and provides a method and device for automatically generating a sign language animation based on semantics drive, and a computer storage medium.

[0007] To achieve the above object, the technical solution adopted by the present invention is: a method for automatically generating sign language animation based on semantics drive, comprising the following steps: Acquire a text, perform semantic analysis on the text, and obtain a target sign language word and an emotional metaphor type; the target sign language word is a sign language word required for the sign language to be generated; According to the semantic analysis result, a sign language skeleton animation of the target sign language word is obtained from a sign language vocabulary library; each frame of the sign language skeleton animation at least includes: a point-line structure posture of an upper limb and a head; splicing all acquired sign language skeleton animations according to the semantic order of the text, generating a transition skeleton animation between two adjacent sign language skeleton animations, and adjusting the rhythm of the corresponding sign language skeleton animation and the transition skeleton animation in combination with the emotional metaphor type to obtain a target skeleton animation; The target skeleton animation is rendered to generate a target sign language animation.

[0008] In a preferred embodiment of the present invention, the semantic analysis includes word segmentation, part-of-speech tagging, syntactic analysis, semantic role tagging, and emotional metaphor recognition; Performing emotional metaphor recognition on the text, including: Using a sentiment classification model, sentiment analysis is performed on the text to identify sentiment-containing words and sentiment-expressing parts of the text, and the rest are declarative parts; The emotional metaphor type of the emotional expression part is further identified.

[0009] In a preferred embodiment of the present invention, the generation of the transition skeleton animation includes: Obtaining an end frame and a start frame of two of the sign language skeleton animations before and after the transition skeleton animation; Determine the longest motion trajectory according to the point-line frame postures in the end frame and the start frame; the motion trajectory is the movement path of the same point in the process of changing the point-line frame posture of the end frame to the point-line frame posture of the start frame; Based on the longest motion trajectory, determining the number of frames of the transition skeleton animation; According to the motion trajectory from the end frame point-line frame posture to the start frame point-line frame posture, and the number of frames, a point-line frame posture image of each frame of the transition skeleton animation is produced, and the transition skeleton animation is obtained by splicing.

[0010] In a preferred embodiment of the present invention, the method for constructing the sign language vocabulary library comprises the following steps: A sign language vocabulary set and a sign language skeleton animation set are constructed respectively; the sign language vocabulary set is a set of all target sign language words; A mapping relationship between each of the target sign language words and the sign language skeleton animation is established to construct a sign language vocabulary library.

[0011] In a preferred embodiment of the present invention, the generation of the sign language skeleton animation comprises the following steps: Based on the sign language standard, obtain the sign language video corresponding to each sign language word; The movements of upper limbs and head in sign language animation are disassembled frame by frame and converted into point-line structure postures: joints and limb ends are represented by points, and bones are represented by line segments; The point-line structure posture of each frame is combined according to the order of the original sign language animation to obtain the sign language skeleton animation.

[0012] In a preferred embodiment of the present invention, in the sign language vocabulary library, the point-line structure posture of each frame of the sign language skeleton animation is digitized and converted into a character string that can be processed by a computer.

[0013] In a preferred embodiment of the present invention, the calculation of the motion trajectory of the transition skeleton animation includes: Splice the end frame point-line frame pose data + virtual data + start frame point-line frame pose data to create a target string; Matching is performed in the sign language vocabulary library to find the character string with the highest similarity to the target character string, and multi-frame point-line frame posture data between the end frame and the start frame in the sign language skeleton animation corresponding to the character string is extracted as a preset transition skeleton animation; Based on the preset transition skeleton animation, the point-line frame posture data between adjacent frames are compared to draw the motion trajectory of each part in the preset transition skeleton animation.

[0014] In a preferred embodiment of the present invention, the calculation of the number of frames of the transition skeleton animation is specifically as follows: the number of frames of the transition skeleton animation is calculated according to the average limb speed V in the sign language video, in units of mm / frame. , where L is the value of the longest motion trajectory, in mm.

[0015] In a preferred embodiment of the present invention, the step of adjusting the rhythm of the corresponding sign language skeleton animation and the transition skeleton animation in combination with the emotion metaphor type includes: A matching strategy between the emotion metaphor type and the emotion intensity is preset, wherein the emotion metaphor types include: happiness, good, anger, sadness, fear, evil, and shock; and the emotion intensity corresponding to the emotion metaphor is a ratio relative to the standard sign language rhythm; According to the emotional metaphor type of the emotional expression part, matching the corresponding emotional intensity; The emotional intensity is mapped to a plurality of sign language skeleton animations of the emotional expression part and the transitional skeleton animations therebetween, and the duration of the emotional expression part and the number of frames of the transitional skeleton animations during this period are extended or shortened in equal proportion.

[0016] The automatic generation device of sign language animation based on semantics drive includes: A semantic analysis unit, used to obtain text, perform semantic analysis on the text, and obtain a target sign language word and an emotional metaphor type; the target sign language word is a sign language word required for the sign language to be generated; The sign language skeleton animation acquisition unit acquires the sign language skeleton animation of the target sign language word from the sign language vocabulary library according to the semantic analysis result; each frame of the sign language skeleton animation at least includes: the point-line structure posture of the upper limbs and the head; A target skeleton animation generating unit is used to splice all acquired sign language skeleton animations according to the semantic order of the text, generate a transition skeleton animation between two adjacent sign language skeleton animations, and adjust the rhythm of the corresponding sign language skeleton animation and the transition skeleton animation in combination with the emotional metaphor type to obtain a target skeleton animation; A rendering unit renders the target skeleton animation to generate a target sign language animation.

[0017] A computer storage medium includes computer program instructions, which, when executed by a processor, enable the processor to execute any of the above-mentioned semantic-driven automatic generation methods for sign language animation.

[0018] The present invention solves the defects existing in the background technology and has the following beneficial effects: (1) The method and device for automatically generating sign language animation based on semantics drive and computer storage medium of the present invention optimize and compress the sign language word animation into skeleton animation, which at least includes the point-line structure posture of the upper limbs and the head, and reduces the computing power occupied when splicing and generating transition skeleton animation according to the semantic order. In the prior art, the method of directly calculating and processing the sign language word animation often requires processing a large number of details and overall effects. In the generation stage, the present invention only needs to identify and calculate the point-line structure posture, which greatly reduces the amount of calculation and improves the generation efficiency of the sign language animation.

[0019] (2) The present invention uses the skeleton animation data in the sign language vocabulary library, first presets the motion trajectory of each part in the transition skeleton animation, and accurately calculates the number of transition frames based on the longest motion trajectory and the average speed of the limbs in the sign language video, then draws and splices the point-line frame posture image frame by frame to generate a smooth and coherent transition animation. This process ensures the difference and naturalness of the motion amplitude of each part in the animation, making the connection between the previous and next skeleton animations appear particularly realistic and coherent, and successfully achieving a smooth transition between two sign language actions.

[0020] (3) The present invention uses semantic analysis technology to extract emotional metaphor components in the text. These emotional colors are not directly expressed, but implicit in the text. Subsequently, based on the influence of such emotions on the rhythm and movements of human expression, the present invention designs a set of emotional intensity matching strategies. According to this strategy, the duration of the emotional expression part will be extended or shortened proportionally. At the same time, the number of transition skeleton animation frames in the process is also adjusted accordingly. The sign language animation generated in this way can show a variety of rhythmic characteristics, thereby realizing the implicit and practical expression of emotional information, which is closer to the real sign language communication scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. Figure 1 is a flow chart of a method for automatically generating sign language animation according to a preferred embodiment of the present invention; Figure 2 It is a schematic diagram of a sign language animation generating device according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0024] Application Overview There are two main traditional transition frame generation methods: optical flow-based method and distance loss determination method.

[0025] The optical flow-based method uses the pixel changes in the image sequence and the correlation between adjacent frames to calculate the motion information of the object between adjacent frames, thereby selecting the insertion position of the transition frame and inserting the transition frame sequence at the determined insertion position by interpolation. However, this method has two main disadvantages: first, due to the need to compare the animation frame by frame, the amount of data calculated is large, resulting in a slow speed and high time cost for generating transition frames; second, the quality of the generated transition frames is poor, and there may be problems such as ghosting, abrupt action switching, inconsistent speed, and gesture deformation, because the interpolation calculation does not distinguish whether the pixel is part of the virtual human model, and does not consider the speed and correlation of limb movements.

[0026] The distance loss determination rule evaluates the difference between animation frames by calculating the distance loss between the skeletal data of the virtual human model, determines the insertion position based on the difference, and inserts the transition frame sequence at the insertion position by linear interpolation or spherical interpolation. However, this method also has the problems of slow generation speed and poor transition frame quality. The reason for the slow generation speed is that it is necessary to determine the skeletal coordinates in each frame of animation, and then calculate the distance loss based on the skeletal coordinates, resulting in a long time for selecting the insertion position. The reason for the poor quality of the transition frame is that the method of determining the transition frame position only considers the distance, and the defects of the superimposed interpolation method during interpolation result in poor flexibility of the generated transition frame, direct smoothing effect, and unnatural connection with the previous and subsequent sign language animations, resulting in a loss of coherence.

[0027] The present invention obtains text and performs semantic analysis to determine the target sign language word, and obtains the corresponding sign language skeleton animation from the sign language vocabulary library. Subsequently, a transition skeleton animation production method based on the longest motion trajectory and the average speed of the limbs is adopted. The method determines the longest motion trajectory by analyzing the point-line frame postures in the end frame and the start frame of two adjacent sign language skeleton animations, and calculates the number of frames of the transition animation accordingly. Then, a corresponding number of point-line frame posture images are produced and spliced ​​according to the motion trajectory to form a smooth transition skeleton animation. This improvement not only ensures the continuity of the animation, but also significantly improves the naturalness and realism of the transition animation.

[0028] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0029] Exemplary methods: Figure 1 A diagram showing a method for automatically generating sign language animation according to an embodiment of the present invention is shown.

[0030] like Figure 1 As shown, the present invention provides a method for automatically generating sign language animation based on semantics driving, and controls the following steps: S1, obtain the text, perform semantic analysis on the text, and obtain the target sign language word; S2. According to the semantic analysis result, a sign language skeleton animation of the target sign language word is obtained from the sign language vocabulary library; each frame of the sign language skeleton animation at least includes: a point-line structure posture of the upper limbs and the head; S3, splicing all acquired sign language skeleton animations according to the semantic order of the text, and generating a transition skeleton animation between two adjacent sign language skeleton animations to obtain a target skeleton animation; S4. Render the target skeleton animation to generate a target sign language animation.

[0031] The present invention optimizes and compresses the sign language word animation into a skeleton animation, which includes at least the point-line structure posture of the upper limbs and the head, and reduces the computing power occupied when splicing and generating transition skeleton animation according to the semantic order. In the prior art, the method of directly calculating and processing the sign language word animation often requires processing a large number of details and overall effects. In the generation stage, the present invention only needs to identify and calculate the point-line structure posture, which greatly reduces the amount of calculation and improves the generation efficiency of the sign language animation.

[0032] Below, each step will be described in detail.

[0033] In step S1, a text is obtained and semantic analysis is performed on the text to obtain a target sign language word.

[0034] That is to say, first, a text input is required, which can be a spoken sentence or a written text. If it is a spoken sentence, the received spoken sentence needs to be recognized and converted into a written text. Before performing semantic analysis on it, it is preferred to use natural language processing pyltp technology to perform word segmentation, part-of-speech tagging, syntactic analysis and semantic role tagging on the written text. Among them, word segmentation is to decompose the acquired written text string into independent and meaningful words. Part-of-speech tagging is to assign a part-of-speech category (noun, verb, adjective, etc.) to each word in the word segmentation result. Syntactic analysis involves understanding the structure and grammatical relationship of the sentence, which is achieved through dependency syntactic analysis, in which each word in the written text is marked with its relationship with other words in the sentence. Semantic role tagging is to identify the arguments of the predicate in the sentence (such as agent, patient, time, place, etc.). Then, based on the results of the semantic analysis, find the synonyms and near-synonyms corresponding to each word, and then expand them into target sign language words, and each target sign language word corresponds to a sign language word animation.

[0035] In step S2, according to the result of semantic analysis, a sign language skeleton animation of the target sign language word is obtained from the sign language vocabulary library; each frame of the sign language skeleton animation at least includes: a point-line structure posture of the upper limbs and the head.

[0036] That is to say, according to the result of the semantic analysis in step S1, the sign language skeleton animation corresponding to the target sign language word is obtained from the sign language vocabulary library, and the sign language vocabulary library includes several sign language skeleton animations. Each frame of each sign language skeleton animation includes at least: the point-line structure posture of the upper limbs and the head, and facial expressions, etc.

[0037] Furthermore, the generation of the sign language skeleton animation includes the following steps: S21. Based on the sign language standard, obtaining the sign language video corresponding to each sign language word; S22. Decompose the movements of upper limbs and head in the sign language animation frame by frame and convert them into point-line structure postures: joints and limb ends are represented by points, and bones are represented by line segments; S23, combining the point-line structure posture of each frame according to the original sign language animation sequence to obtain the sign language skeleton animation.

[0038] Through the above steps, starting from the standard of sign language, complex sign language movements are converted into a series of simplified point-line structure gestures that are easy for computers to process.

[0039] It is worth noting that in step S22, a point-line frame posture is first constructed based on the human body structure in the sign language video, and a mapping relationship between the human body structure and the point-line frame posture is established; wherein, limb joints and ends (such as fingertips) are represented by points, bones are represented by line segments, and line segments connect the points. The head obtains the human head contour line based on edge detection, and the corresponding human body structure is connected to form a point-line frame posture. Then, computer vision technology is used to extract the changes in upper limbs, head and facial expressions in the sign language video, and map them to the point-line frame posture in real time, that is, the point-line frame posture changes consistent with the human body structure movements in the sign language video are obtained, and then the sign language skeleton animation is obtained.

[0040] Furthermore, the method for constructing a sign language vocabulary library includes the following steps: Construct a sign language vocabulary set and a sign language skeleton animation set respectively; the sign language vocabulary set is a collection of all target sign language words, including letters, titles, occupations, surnames, clothing, food, daily necessities, time, space, life, social interaction, philosophy, psychology, ethnicity, religion, transportation, post and telecommunications, agriculture, animal husbandry, sideline production and fishery, culture, sports, health, science, mathematics, physics, chemistry, biology, names of European and American countries, animals and plants, etc.

[0041] A mapping relationship between each target sign language word and the sign language skeleton animation is established to construct a sign language vocabulary library.

[0042] It is worth mentioning that the point-line structure posture of each frame of the sign language skeleton animation in the sign language vocabulary library is digitized and converted into a character string that can be processed by a computer.

[0043] In the sign language vocabulary, each sign language skeleton animation is assigned a corresponding serial number. When obtaining the corresponding sign language skeleton animation through the target sign language word, it is only necessary to extract the corresponding sign language skeleton animation. For example, for a specific target sign language word, there is a corresponding sign language skeleton animation, which is recorded as P. This sign language skeleton animation P is composed of a series of animation frames arranged in order, forming a sequence P = [P1, P2, ..., PN]. Here, N represents the total number of frames of the sign language skeleton animation, which belongs to the positive integer set Each frame of animation, such as PN, is a component of the sign language skeleton animation P, which records in detail the movement and posture of the target sign language word at a specific moment.

[0044] At the same time, the target sign language word also corresponds to a sequence of action postures, also recorded as P (although the symbols are the same, they can be clearly distinguished according to the context). This sequence P consists of a series of postures arranged in order, that is, P = [P1', P2', ..., PN']. Among them, PN' represents the specific action posture of the target sign language word at the Nth frame of the animation.

[0045] Regarding the relationship between the animation P and the action posture sequence P, there is a specific conversion mechanism between the two. Specifically, each frame PN in the animation P can be regarded as a visual presentation of the corresponding posture PN' in the action posture sequence P. Conversely, each posture PN' in the action posture sequence P can also be understood as an abstract expression of the specific action and posture captured by the corresponding frame PN in the animation P.

[0046] In step S3, all acquired sign language skeleton animations are spliced ​​according to the semantic order of the text, and a transition skeleton animation is generated between two adjacent sign language skeleton animations to obtain a target skeleton animation.

[0047] That is, according to the semantic analysis result obtained in step S1, the order of each target sign language word in the sentence is determined, and these sign language skeleton animations are spliced ​​together in sequence to form a preliminary animation sequence composed of multiple sign language skeleton animations. Then, a transition skeleton animation is generated between two adjacent sign language skeleton animations, and all the spliced ​​sign language skeleton animations and the generated transition skeleton animations are combined together to form a complete and coherent target skeleton animation, thereby obtaining the target skeleton animation.

[0048] In the preliminarily spliced ​​animation sequence, there may be discontinuities in posture between two adjacent sign language skeleton animations. In order to smooth out such discontinuities, a transition skeleton animation needs to be generated.

[0049] Furthermore, the generation of transition skeleton animation includes: S31. Obtain the end frame and the start frame of the two sign language skeleton animations before and after the transition skeleton animation; that is, obtain the last frame image of the sign language skeleton animation before the transition skeleton animation, and the first frame image of the sign language skeleton animation after the transition skeleton animation. The transition skeleton animation needs to connect the end frame and the start frame to ensure the visual continuity and smoothness of the two adjacent sign language skeleton animations and improve the overall quality.

[0050] S32, determining the longest motion trajectory according to the point-line frame postures in the end frame and the start frame; the motion trajectory is the movement path of the same point in the process of changing the point-line frame posture of the end frame to the point-line frame posture of the start frame; Among them, the calculation of transition skeleton animation motion trajectory includes: Splice the end frame point-line frame pose data + virtual data + start frame point-line frame pose data to create a target string; Matching is performed in the sign language vocabulary library to find the character string with the highest similarity to the target character string, and multiple frames of point-line frame gesture data between the end frame point-line frame gesture data and the start frame point-line frame gesture data in the sign language skeleton animation corresponding to the character string are extracted as the preset transition skeleton animation; Based on the preset transition skeleton animation, the point-line frame posture data between adjacent frames are compared to draw the motion trajectory of each part in the preset transition skeleton animation. Further, the motion trajectory of all parts and the length of each motion trajectory are obtained.

[0051] S33. Determine the number of frames of the transition skeleton animation based on the longest motion trajectory; calculate the number of frames of the transition skeleton animation according to the average limb speed V in the sign language video, in units of mm / frame, where L is the value of the longest motion trajectory, in units of mm.

[0052] S34, according to the motion trajectory from the end frame point-line frame posture to the start frame point-line frame posture, produce point-line frame posture images of the number of frames determined in step S33, and splice them to obtain the transition skeleton animation.

[0053] It is worth mentioning that the changes in facial expressions in sign language animation are all small changes, so no transition animation is required.

[0054] Through the above steps, the present invention uses the skeleton animation data in the sign language vocabulary library, first presets the motion trajectory of each part in the transition skeleton animation, and accurately calculates the number of transition frames based on the longest motion trajectory and the average speed of the limbs in the sign language video, and then draws and splices the point-line frame posture image frame by frame to generate a smooth and coherent transition animation. This process ensures the difference and naturalness of the motion amplitude of each part in the animation, makes the connection between the front and back skeleton animations appear particularly real and coherent, and successfully achieves a smooth transition between two sign language actions.

[0055] In step S4, the target skeleton animation is rendered to generate a target sign language animation.

[0056] That is to say, what is obtained through steps S1-S3 is the target skeleton animation, that is, a coherent animation composed of point-line frame gestures. It is also necessary to further render the target skeleton animation, preferably using real-time rendering technology, that is, using a deep learning model to process animation frames in real time, and finally generate a realistic sign language animation.

[0057] The semantic analysis in step S1 also includes: using a sentiment classification model to perform sentiment analysis on the hand sentence, identifying the emotional words and emotional expression parts of the expression, and the rest are the declarative parts, and further, performing metaphor recognition on the emotional expression part.

[0058] It is worth mentioning that the sentiment classification model is a model based on machine learning, which uses known data to predict the sentiment of unknown data. Support vector machines are preferably used. Specifically: first, the text data annotated by feature engineering is trained using a machine learning algorithm to obtain a sentiment classification model; then, new text data with the same structure is input into the sentiment classification model to predict the sentiment type of the new text; finally, the true label of the new text data is compared with the predicted value label to obtain the accuracy of the label in the data set.

[0059] The matching strategy of preset emotional metaphor type and emotional intensity is to match the corresponding emotional intensity according to the emotional metaphor type of the emotional expression part. The emotional metaphor types include: happiness, good, anger, sadness, fear, hate, and surprise. The emotional intensity corresponding to the emotional metaphor is expressed in terms of the ratio relative to the standard sign language rhythm, as shown in Table 1.

[0060] Table 1 Matching table between emotion metaphor type and emotion intensity

[0061] In step S3, the intensity of the emotion is mapped to several sign language skeleton animations of the emotion expression part and the transition skeleton animations between them, and the duration of the emotion expression part and the number of frames of the transition skeleton animations during this period are proportionally extended or shortened to adjust the sign language rhythm and the intensity of the movement to achieve implicit expression of emotion, which is closer to the actual sign language expression.

[0062] Specifically, speeding up the rhythm means shortening the duration of the corresponding emotional expression part and reducing the number of frames; slowing down the rhythm means the opposite.

[0063] Exemplary devices: like Figure 2 As shown, the present invention also provides a semantically driven automatic generation device for sign language animation, comprising: a semantic analysis unit, used to obtain text, perform semantic analysis on the text, and obtain a target sign language word and an emotional metaphor type; the target sign language word is the sign language word required for the sign language to be generated; a sign language skeleton animation acquisition unit, which obtains the sign language skeleton animation of the target sign language word from the sign language vocabulary library according to the semantic analysis result; each frame of the sign language skeleton animation at least includes: a point-line structure posture of the upper limbs and the head; a target skeleton animation generation unit, which is used to splice all the acquired sign language skeleton animations according to the semantic order of the text, generate a transition skeleton animation between two adjacent sign language skeleton animations, and adjust the rhythm of the corresponding sign language skeleton animation and the transition skeleton animation in combination with the emotional metaphor type to obtain the target skeleton animation; a rendering unit, which renders the target skeleton animation to generate the target sign language animation.

[0064] In one embodiment, the text acquired by the semantic analysis unit may be a spoken sentence or a written text. If it is a spoken sentence, the received spoken sentence needs to be recognized and converted into a written text.

[0065] In one embodiment, when performing semantic analysis on the text, it is preferred to use natural language processing pyltp technology to perform word segmentation, part-of-speech tagging, syntactic analysis and semantic role tagging on the written text.

[0066] In one embodiment, the construction of the sign language vocabulary library includes: constructing a sign language vocabulary set and a sign language skeleton animation set respectively; establishing a mapping relationship between each target sign language word and the sign language skeleton animation to construct a sign language vocabulary library. Furthermore, the point-line frame posture of each frame of the sign language skeleton animation in the sign language vocabulary library is digitized and converted into a character string that can be processed by a computer.

[0067] In one embodiment, the generation of the sign language skeleton animation includes: based on the sign language standard, obtaining the sign language video corresponding to each sign language word; disassembling the movement of the upper limbs and the head in the sign language animation frame by frame, and converting it into point-line frame postures: the joints and limb ends are represented by points, and the skeleton is represented by line segments; combining the point-line frame postures of each frame according to the original sign language animation sequence to obtain the sign language skeleton animation. Starting from the sign language standard, the complex sign language movements are converted into a series of simplified point-line frame postures that are easy for computers to process.

[0068] In one embodiment, the target skeleton animation generation unit includes: a transition skeleton animation generation component and a splicing component. The transition skeleton animation generation component is used to generate a transition skeleton animation, including: obtaining the end frame and the start frame of the two sign language skeleton animations before and after the transition skeleton animation; determining the longest motion trajectory according to the point-line frame posture in the end frame and the start frame; determining the number of frames of the transition skeleton animation based on the longest motion trajectory; making a point-line frame posture image of each frame of the transition skeleton animation according to the motion trajectory from the end frame point-line frame posture to the start frame point-line frame posture, and the number of frames, and splicing to obtain the transition skeleton animation. The splicing component is used to splice all the sign language skeleton animations and the transition skeleton animation in sequence to obtain the target skeleton animation.

[0069] Exemplary Computer Storage Media: The embodiment of the present invention may also be a computer storage medium, including computer program instructions, which, when executed by a processor, enable the processor to execute the semantic-driven automatic generation method of sign language animation as described in the above embodiment.

[0070] Computer readable storage medium can adopt any combination of one or more readable media.Readable medium can be readable signal medium or readable storage medium.Readable storage medium can for example include but not limited to electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any above combination.More specific example (non-exhaustive list) of readable storage medium comprises: have one or more wires electrical connection, portable disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device or above-mentioned any suitable combination.

[0071] The above is based on the ideal embodiment of the present invention. Through the above description, relevant personnel can make various changes and modifications without departing from the technical concept of the present invention. The technical scope of the present invention is not limited to the content in the specification, and the technical scope must be determined according to the scope of the claims.

Claims

1. A semantically driven sign language animation automatic generation method, characterized in that: The following steps are involved: Acquire a text, perform semantic analysis on the text, and obtain a target sign language word and an emotional metaphor type; The target sign language word is the sign language word required for the sign language to be generated; According to the semantic analysis result, obtaining the sign language skeleton animation of the target sign language word from the sign language vocabulary library; Each frame of the sign language skeleton animation at least includes: point-line structure postures of the upper limbs and the head; splicing all acquired sign language skeleton animations according to the semantic order of the text, generating a transition skeleton animation between two adjacent sign language skeleton animations, and adjusting the rhythm of the corresponding sign language skeleton animation and the transition skeleton animation in combination with the emotional metaphor type to obtain a target skeleton animation; The target skeleton animation is rendered to generate a target sign language animation.

2. The method for automatically generating sign language animation based on semantics drive according to claim 1, characterized in that: The semantic analysis includes word segmentation, part-of-speech tagging, syntactic analysis, semantic role tagging and emotional metaphor recognition; Performing emotional metaphor recognition on the text, including: Using a sentiment classification model, sentiment analysis is performed on the text to identify sentiment-containing words and sentiment-expressing parts of the text, and the rest are declarative parts; The emotional metaphor type of the emotional expression part is further identified.

3. The method for automatically generating sign language animation based on semantics drive according to claim 1, characterized in that: The generation of the transition skeleton animation includes: Obtaining an end frame and a start frame of two of the sign language skeleton animations before and after the transition skeleton animation; Determine the longest motion trajectory according to the point-line frame postures in the end frame and the start frame; the motion trajectory is the movement path of the same point in the process of changing the point-line frame posture of the end frame to the point-line frame posture of the start frame; Based on the longest motion trajectory, determining the number of frames of the transition skeleton animation; According to the motion trajectory from the end frame point-line frame posture to the start frame point-line frame posture, and the number of frames, a point-line frame posture image of each frame of the transition skeleton animation is produced, and the transition skeleton animation is obtained by splicing.

4. The method for automatically generating sign language animation based on semantics drive according to claim 1, characterized in that: The method for constructing the sign language vocabulary library comprises the following steps: A sign language vocabulary set and a sign language skeleton animation set are constructed respectively; the sign language vocabulary set is a set of all target sign language words; A mapping relationship between each of the target sign language words and the sign language skeleton animation is established to construct a sign language vocabulary library.

5. The method for automatically generating sign language animation based on semantics drive according to claim 1, characterized in that: The generation of the sign language skeleton animation comprises the following steps: Based on the sign language standard, obtain the sign language video corresponding to each sign language word; The movements of upper limbs and head in sign language animation are disassembled frame by frame and converted into point-line structure postures: joints and limb ends are represented by points, and bones are represented by line segments; The point-line structure posture of each frame is combined according to the order of the original sign language animation to obtain the sign language skeleton animation.

6. The method for automatically generating sign language animation based on semantics drive according to claim 3, characterized in that: The calculation of the transition skeleton animation motion trajectory includes: Splice the end frame point-line frame pose data + virtual data + start frame point-line frame pose data to create a target string; Matching is performed in the sign language vocabulary library to find the character string with the highest similarity to the target character string, and multi-frame point-line frame posture data between the end frame and the start frame in the sign language skeleton animation corresponding to the character string is extracted as a preset transition skeleton animation; Based on the preset transition skeleton animation, the point-line frame posture data between adjacent frames are compared to draw the motion trajectory of each part in the preset transition skeleton animation.

7. The method for automatically generating sign language animation based on semantics drive according to claim 3 is characterized in that: The calculation of the transition skeleton animation frame number is specifically as follows: the frame number of the transition skeleton animation is calculated according to the average limb speed V in the sign language video. , where L is the value of the longest motion trajectory.

8. The method for automatically generating sign language animation based on semantics drive according to claim 2, characterized in that: The step of adjusting the rhythm of the corresponding sign language skeleton animation and the transition skeleton animation in combination with the emotion metaphor type includes: A matching strategy between the emotion metaphor type and the emotion intensity is preset, wherein the emotion metaphor types include: happiness, good, anger, sadness, fear, evil, and shock; and the emotion intensity corresponding to the emotion metaphor is a ratio relative to the standard sign language rhythm; According to the emotional metaphor type of the emotional expression part, matching the corresponding emotional intensity; The emotional intensity is mapped to a plurality of sign language skeleton animations of the emotional expression part and the transitional skeleton animations therebetween, and the duration of the emotional expression part and the number of frames of the transitional skeleton animations during this period are extended or shortened in equal proportion.

9. The automatic generation device of sign language animation based on semantics drive is characterized by: include: A semantic analysis unit, used for acquiring text, performing semantic analysis on the text, and obtaining target sign language words and emotional metaphor types; The target sign language word is the sign language word required for the sign language to be generated; A sign language skeleton animation acquisition unit, which acquires the sign language skeleton animation of the target sign language word from a sign language vocabulary library according to the semantic analysis result; Each frame of the sign language skeleton animation at least includes: point-line structure postures of the upper limbs and the head; A target skeleton animation generating unit is used to splice all acquired sign language skeleton animations according to the semantic order of the text, generate a transition skeleton animation between two adjacent sign language skeleton animations, and adjust the rhythm of the corresponding sign language skeleton animation and the transition skeleton animation in combination with the emotional metaphor type to obtain a target skeleton animation; A rendering unit renders the target skeleton animation to generate a target sign language animation.

10. A computer storage medium, characterized in that The method comprises computer program instructions, which, when executed by a processor, enable the processor to execute the method for automatically generating sign language animation based on semantic driving according to any one of claims 1 to 8.

Images