Real-time scene gesture generation method and device and digital human interaction system

Abstract

The invention relates to a real-time scene gesture generation method and device and a digital human interaction system, and belongs to the technical field of artificial intelligence. The method comprises the steps of obtaining an input text; generating a basic gesture action of the digital human based on the input text by adopting an action generation model; adopting a similarity matching method to retrieve an index semantic action corresponding to the input text from a pre-established semantic enhancement data set; and superposing the index semantic action into the sequence of the basic gesture actions to generate the gesture action of the digital human. According to the method, a large model technology is introduced, accurate semantic text descriptions of semantic gestures are generated in batches, retrieval matching is carried out through word vector similarity calculation, and index semantic actions are superposed in the generation process of a speaking digital person, so that the semantic effect expressed by the gestures of the digital person is enhanced, and the accuracy of the speech digital person is improved. Unification of language text semantic content and gesture content expression is achieved on the premise that the real-time effect is guaranteed.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a real-time scene gesture generation method, device and digital human interaction system. Background Art

[0002] When digital humans express their speech, they usually use corresponding gestures. The text content of speech and gestures belong to two different modalities. In order to match gestures with speech content or generate corresponding actions, it is first necessary to establish a mapping relationship between text and gestures.

[0003] In the prior art, there are several methods for establishing a mapping relationship:

[0004] (1) Rule matching method based on manual annotation. For small batches of text and gestures, the mapping between the two modal data can be directly established through manual annotation. In downstream retrieval tasks, given the spoken text content, it can be matched according to the size of the similarity by keyword matching based on named entity recognition or calculating the text word vector similarity based on pre-trained models such as BERT and BGE. The advantage of this method is that it provides an accurate hit rate basis for downstream retrieval tasks and is fast and can be real-time, but the disadvantage is that manual annotation itself takes time, and the accuracy of matching between texts is easily affected by the domain content of the annotated text.

[0005] (2) Mapping method based on deep learning. Existing methods implement text2motion or text2gesture based on the text and action dataset HumanML3D. Deep learning methods use structures such as diffusion or AutoEncoder. This end-to-end method can directly establish a mapping from text to generated gestures, but it is limited by the size of the existing data and the mismatch of the data space. For example, the existing dataset mainly reflects actions such as walking, running, jumping, and standing rather than actions in a fixed narrative state. Therefore, it is difficult to directly apply it to the gesture expression of digital human real-time dialogue interaction scenarios.

[0006] Therefore, how to solve the technical problem that it is difficult to achieve clear semantic expression of gestures generated by 3D digital humans in real time is a topic worth studying for those skilled in the art. Summary of the invention

[0007] In view of the above analysis, the embodiments of the present invention aim to provide a real-time scene gesture generation method, device and digital human interaction system, so as to achieve the unification of the expression of language text semantic content and gesture content while ensuring the real-time effect.

[0008] In a first aspect of the present application, a real-time scene gesture generation method is provided, comprising:

[0009] Get input text;

[0010] Using an action generation model to generate basic gesture actions of the digital human based on the input text;

[0011] Retrieving the indexed semantic actions corresponding to the input text from a pre-established semantic enhancement dataset using a similarity matching method;

[0012] Superimposing the indexed semantic action onto the sequence of basic gesture actions to generate the gesture action of the digital human;

[0013] The construction process of the semantic enhancement dataset includes: inputting the annotated semantic gesture corpus into a large language model, generating enhanced text samples based on the semantic gestures, and the enhanced text samples are used to reflect the semantic information expressed by the gestures; after screening the enhanced text samples, inputting them into the large language model to iteratively generate enhanced semantic descriptions; and storing the mapping relationship between the gestures and the enhanced semantic descriptions in the semantic enhancement dataset for retrieval.

[0014] Optionally, the semantic gesture corpus includes intention information, name information, action description, simple text instance, and action occurrence time of each gesture.

[0015] Optionally, after screening the enhanced text samples, inputting them into a large language model to iteratively generate enhanced semantic descriptions includes:

[0016] Manually screening the enhanced text samples to remove text samples that are not semantically relevant;

[0017] The filtered text samples are input into the large language model again for iteration to generate enhanced semantic descriptions;

[0018] The enhanced semantic description is checked for semantic consistency to ensure that the enhanced semantic description is consistent with the initial semantic description. Figure 1 To.

[0019] Optionally, the using a similarity matching method to retrieve an indexed semantic action corresponding to the input text from a pre-established semantic enhancement data set includes:

[0020] Using a pre-trained word vector model to calculate the word vector for the entire sentence of the input text;

[0021] In a pre-established semantic enhancement dataset, a pre-trained word vector model is used to calculate word vectors for the semantic descriptions in the semantic enhancement database;

[0022] According to the cosine similarity matching method, the word vector with the highest overall semantic matching with the input text is retrieved from the semantic enhancement dataset, and the action corresponding to the word vector is used as the indexed semantic action;

[0023] Sentences in the input text are segmented, and word vectors are calculated for each word obtained through segmentation to determine the key words that best match the overall semantics of the input text.

[0024] Optionally, the step of superimposing the indexed semantic action onto the sequence of basic gesture actions to generate a digital human gesture action includes:

[0025] The index semantic action is superimposed on the sequence of the basic gesture action by using the Bezier curve interpolation method to generate the gesture action of the digital human.

[0026] Optionally, before the step of using the Bezier curve interpolation method to superimpose the index semantic action onto the sequence of the basic gesture actions, the step further includes:

[0027] Determine the time range of the basic gesture action P according to the position of the key words in the sentence [t start ,t end ];

[0028] Determine the time length T of the occurrence of the index semantic action ΔP e ;

[0029] T e ≥1s, constraint t end -t start ≥1s, if not exceeded, fill in;

[0030] Judgement end -t start ≥T e , calculate t start ,t end The midpoint t mid , with t mid To achieve the splicing of two fragments for reference, update the alignment range:

[0031] Judgement end -t start <T e , then according to t end To achieve the splicing of two fragments, update the alignment range t start =t end -T e , t end Still t end ;

[0032] T eWhen <1s, the action start and end times are covered from ΔP to P according to the median value without smoothing.

[0033] Optionally, the using of a Bezier curve interpolation method to superimpose the index semantic action onto the sequence of basic gesture actions comprises:

[0034] Take a certain buffer range at the end of the basic gesture action and the beginning of the index semantic action: p left is the buffer range of the tail of the basic gesture action; p right The buffer range at the beginning of the index semantic action;

[0035] If the buffer range padding>10, then consider padding=10; if padding<10, use Bezier curve interpolation to smoothly transition the splicing area of ​​P and ΔP:

[0036] Defining reference fragments

[0037] Construct the sampling point and target matrices: x combined =[x A ,x B ],y combined =[A ref ,B ref ];

[0038] Construct interpolation function S(x): S(x combined )=y combined ;

[0039] Sample the middle segment and get the smoothed result: x smooth =linspace(1,3,p left ),

[0040] y smooth =S(x smooth );

[0041] Update completed stitching: P′=vstack(P,ΔP).

[0042] In a second aspect of the present application, a real-time scene gesture generation device is provided, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, a real-time scene gesture generation method according to any one of the above is implemented.

[0043] A third aspect of the present application provides a digital human interaction system, comprising the real-time scene gesture generation device described above.

[0044] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the real-time scene gesture generation method according to any one of the above-mentioned methods is implemented.

[0045] The real-time scene gesture generation method provided by the present application obtains input text; adopts an action generation model to generate basic gestures of a digital human based on the input text; adopts a similarity matching method to retrieve indexed semantic actions corresponding to the input text from a pre-established semantic enhancement data set; and superimposes the indexed semantic actions on the sequence of basic gesture actions to generate gestures of the digital human. The present application introduces large model technology to batch generate accurate semantic text descriptions of semantic gestures, performs retrieval and matching through word vector similarity calculation, and superimposes indexed semantic actions on the generation process of a speaking digital human to enhance the semantic effect of digital human gesture expression, thereby achieving the unification of the expression of language text semantic content and gesture content while ensuring real-time effects. In addition, the present application also provides a real-time scene gesture generation device, a digital human interaction system, and a computer-readable storage medium having the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this specification. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0047] Figure 1 A flowchart of a specific implementation of the real-time scene gesture generation method provided by this application;

[0048] Figure 2 This is a schematic diagram of the overall process of the real-time scene gesture generation method provided by this application;

[0049] Figure 3 This is a schematic diagram of semantic enhancement;

[0050] Figure 4 A schematic diagram of the process of inputting the large language model for iteration provided by this application;

[0051] Figure 5 This is a schematic diagram of the process of the step-by-step semantic similarity matching method of this application;

[0052] Figure 6 It is a schematic diagram of the step-by-step semantic similarity matching method;

[0053] Figure 7This is a structural block diagram of the real-time scene gesture generation device provided in this application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. It should be noted that, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be combined, separated, interchanged and / or rearranged with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0055] The terms used here are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used here, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used here, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0056] A flowchart of a specific implementation of the real-time scene gesture generation method provided in this application is as follows Figure 1 As shown, the method specifically includes:

[0057] S101: Obtain input text.

[0058] It is understandable that the input text may be directly input by the user, or may be obtained by voice conversion as a basis for subsequent processing.

[0059] S102: using an action generation model to generate basic gesture actions of the digital human based on the input text.

[0060] Specifically, a semantic analysis is performed on the input text. Based on the result of the semantic analysis, the most appropriate gesture is selected from a predefined gesture library. These gestures are usually a collection of hand movements, body postures, facial expressions, etc., which can express different emotions, intentions or contexts. It can be understood that the gesture library defines many common mapping relationships between gestures and sentences.

[0061] S103: using a similarity matching method, retrieving an indexed semantic action corresponding to the input text from a pre-established semantic enhancement data set.

[0062] The construction process of the semantic enhancement dataset includes: inputting the annotated semantic gesture corpus into a large language model, generating enhanced text samples based on the semantic gestures, and the enhanced text samples are used to reflect the semantic information expressed by the gestures; after screening the enhanced text samples, inputting them into the large language model to iteratively generate enhanced semantic descriptions; and storing the mapping relationship between the gestures and the enhanced semantic descriptions in the semantic enhancement dataset for retrieval.

[0063] The semantic gesture corpus includes the following information: intention, name, description, simple text instance, and time of action. The annotated semantic gesture corpus is input into the large language model, which outputs enhanced text samples that should be able to reflect the semantic information expressed by the gesture. These text samples generate richer semantic examples based on the input initial corpus. The generated semantic examples reflect the semantic connotation of the gesture rather than the description of the gesture itself. The content should be consistent with the initial simple text instance and cover more context.

[0064] Furthermore, for a large number of generated text samples, some samples may not be completely relevant to the semantics and need to be manually screened and removed. For the screened text samples, context can be added to make them clearer, or the generation strategy can be adjusted to improve the semantic matching.

[0065] The filtered samples are input into the large language model again, and further iterations are used to generate richer and more accurate semantic descriptions. Semantic consistency checks can also be performed to ensure that the enhanced semantics are consistent with the initial intent. All gestures and their enhanced semantic information are stored as a retrievable semantic enhancement dataset to facilitate subsequent model training and calling.

[0066] Semantically enhanced datasets can be retrieved using word vector technology. Specifically, the text can be converted into a mathematically computable vector form, and the most suitable gesture action can be matched by vector similarity calculation as the index semantic action. This step ensures that the selected gesture action is consistent with the semantics of the speech content.

[0067] A large-scale language model is used to generate semantic text descriptions of gestures, and text describing each gesture is automatically created based on the speech content, ensuring that each action has a clear semantic meaning.

[0068] S104: superimposing the indexed semantic action into the sequence of the basic gesture action to generate the gesture action of the digital human.

[0069] In this way, when the digital human is speaking, the selected gestures will be superimposed on the 3D model and performed synchronously with the speech. This not only enhances the semantic expression of the gestures, but also ensures the real-time nature of the interaction. The indexed semantic actions are superimposed on the sequence of the basic gestures to generate the gestures of the digital human, making the semantics of the language text and the expression of the gestures more unified, and improving the naturalness and accuracy of the digital human interaction experience.

[0070] The overall flow chart of the real-time scene gesture generation method provided in this application is as follows Figure 2 As shown, it can be understood that the real-time scene gesture generation method provided by the present application can be used in various 3D real-time digital human interaction systems.

[0071] The following takes a method for using Bi-LSTM to realize the generation of text to digital human gestures as an example to demonstrate the specific implementation process.

[0072] Suppose the input text is T, and the text audio obtained through text-to-speech technology (TTS) is A(T), and the audio sampling rate is 16000. The basic gesture action corresponding to the audio is P, and the frame rate is 30fps. The basic digital human gesture generation method is:

[0073] P = f(A(T))

[0074] Among them, f represents the Bi-LSTM method for implementing audio to action sequence mapping. To perform semantic enhancement based on this model, it is necessary to introduce an external data set D(p,i), in which p represents a semantic gesture and i represents an action description and a simple example for the gesture. The embodiment of the present application first uses a large language model to enhance i in D(p,i) to generate D′(p,i), then retrieves the indexed semantic action ΔP based on the text T, and finally enhances the retrieved gesture to a sequence of basic gesture actions. The gesture action after semantic enhancement is represented as:

[0075] Δp=p i～T|D′

[0076] P′=P+ΔP

[0077] Among them, ΔP is an indexed semantic action for semantic enhancement, and obtaining ΔP and integrating ΔP into the original sequence of speech gesture output P is the goal of this application.

[0078] In an embodiment of the present application, the semantic description of action gestures that are not labeled in detail is enhanced based on the semantic understanding and generation capabilities of a large language model. In order to ensure the accuracy of the semantic description of the gesture action annotations, the semantic enhancement part needs to include key information annotation (including intention, name, simple description, simple text instance annotation, action occurrence time and other information), speech text sample generation based on a large model, speech text sample screening and addition (manually eliminating unreasonable speech text instances), and finally screening out multiple speech text samples corresponding to each gesture, and inputting them into the large model together for enhanced semantic description generation. The enhanced semantic description obtained can effectively reflect the semantic information after sufficient iteration. All semantic gestures and enhanced semantic descriptions constitute a searchable semantic enhancement data set, such as Figure 3 The semantic enhancement diagram is shown in the figure.

[0079] like Figure 4 As shown in the schematic diagram of the process of inputting into the large language model for iteration provided in the present application, after screening the enhanced text sample, inputting into the large language model for iteration to generate the enhanced semantic description includes:

[0080] S401: Manually screen the enhanced text samples to remove text samples that are not semantically relevant.

[0081] S402: Input the filtered text samples into the large language model again to iteratively generate enhanced semantic descriptions.

[0082] S403: Perform semantic consistency check on the enhanced semantic description to ensure that the enhanced semantic description is consistent with the initial semantic description. Figure 1 To.

[0083] The generated gesture description text is verified and optimized through manual feedback to ensure semantic accuracy and adaptability. This process helps improve the quality and relevance of gesture descriptions. The combination of large models and manual feedback is used to generate abstract semantic action descriptions that integrate multiple fields, avoiding the time-consuming manual annotation and the interference of information in the field of the annotated text, thereby increasing the accuracy of the gesture semantic text description while ensuring efficiency.

[0084] like Figure 5 As shown in the process diagram of the step-by-step semantic similarity matching method of the present application, using the similarity matching method to retrieve the indexed semantic actions corresponding to the input text from the pre-established semantic enhancement data set includes:

[0085] S501: Use a pre-trained word vector model to calculate the word vector for the entire sentence of the input text.

[0086] S502: In a pre-established semantic enhancement data set, a pre-trained word vector model is used to calculate word vectors for the semantic descriptions in the semantic enhancement database.

[0087] S503: According to the cosine similarity matching method, the word vector with the highest overall semantic matching with the input text is retrieved from the semantic enhancement data set, and the action corresponding to the word vector is used as the indexed semantic action.

[0088] S504: Segment the sentences of the input text, calculate the word vector for each word obtained by the segmentation, and determine the key words that have the highest overall semantic match with the input text.

[0089] Specifically, refer to Figure 6 Schematic diagram of the step-by-step semantic similarity matching method. This application specifically uses the pre-trained word vector model BGE-m3. First, the word vectors of all enhanced generated description texts in the semantic database are calculated to obtain E(D′). Next, for text input, the BGE-m3 pre-trained model is used to obtain the vector corresponding to the sentence. The highest-ranked word vector is obtained according to the cosine similarity calculation method and the corresponding action is obtained according to D′. Jieba is then used to segment the sentence, and then BGE is used to calculate the word vector for each word and obtain the similarity. Rank 1 is also taken to obtain the key words that are closest to the overall semantics.

[0090] It is understandable that in the step-by-step semantic similarity matching based on the word vector model, ChatGLM's embedding or BERT can be used to replace BGE to obtain the word vectors corresponding to all text contents. Similarly, when the accuracy and latency are feasible, ChatGLM with smaller parameters such as 3B or 6B can be fine-tuned to implement the input text and output the target retrieval action and key words.

[0091] This application uses a pre-trained BGE word vector model and combines it with a sentence-to-word ladder similarity matching method to calculate the similarity between the spoken text and the semantic action description, thereby retrieving matching gestures and achieving time alignment. This method fully considers the multi-layer semantic information of the spoken text while effectively avoiding the response delay that may be caused by matching with a large language model.

[0092] In any of the above embodiments, superimposing the index semantic action onto the sequence of the basic gesture actions to generate the gesture action of the digital human includes: using the Bezier curve interpolation method to superimpose the index semantic action onto the sequence of the basic gesture actions to generate the gesture action of the digital human, so as to smoothly merge the two actions (the basic action PP and the enhanced action ΔPΔP) in the time sequence to ensure the natural transition of the action.

[0093] Based on the retrieved corresponding action ΔP and key words, as well as the initial action P, the time range of the basic gesture action P is determined according to the position of the key words in the sentence [t start ,t end ].

[0094] Determine the time length T of the occurrence of the index semantic action ΔP e .

[0095] Set T e ≥1s, if not exceeded, complete.

[0096] Constraint end -t start ≥1s, if not exceeded, complete.

[0097] Judgement end -t start ≥T e , calculate t start , t end The midpoint t mid , with t mid To achieve the splicing of two fragments for reference, update the alignment range:

[0098] Judgement end -t start <T e , then according to t end To achieve the splicing of two fragments, update the alignment range t start =t end -T e , t end Still t end .

[0099] T e When <1s, the action start and end times are covered from ΔP to P according to the median value without smoothing.

[0100] After determining the specific start and end time of the splicing, we need to consider the smoothing method of motion fusion. This application uses the Bezier curve interpolation method, which is limited to obtaining the padding on the left and right sides. left 、p right If padding is greater than 10, then consider padding = 10. Assuming padding < 10, the interpolation function is S(x), with t start Node (corresponding to p left ) are spliced ​​as an illustration.

[0101] Through Bezier curve interpolation, the joint area of ​​P and ΔP is smoothly transitioned:

[0102] Defining reference fragments

[0103] Construct the sampling point and target matrices:

[0104] x combined =[x A ,x B ],y combined =[A ref ,B ref ];

[0105] Construct interpolation function S(x): S(x combined )=y combined ;

[0106] Sample the middle segment and get the smoothed result: x smooth =linspace(1,3,p left ),

[0107] y smooth =S(x smooth );

[0108] Update completed stitching: P′=vstack(P,ΔP).

[0109] It is understandable that, in the stage of body motion fusion, in addition to the Bezier curve interpolation method, the present application may also use other methods such as local weighted smoothing or Gaussian smoothing, which are not limited here.

[0110] The real-time scene gesture generation method provided by the present application obtains input text; uses an action generation model to generate basic gestures of a digital human based on the input text; uses a similarity matching method to retrieve indexed semantic actions corresponding to the input text from a pre-established semantic enhancement data set; and superimposes the indexed semantic actions onto the sequence of basic gesture actions to generate gestures of the digital human. The present application introduces large model technology to batch generate accurate semantic text descriptions of semantic gestures, performs retrieval and matching through word vector similarity calculations, and superimposes indexed semantic actions onto the generation process of a speaking digital human, to enhance the semantic effect of digital human gesture expression, and to achieve the unification of the expression of language text semantic content and gesture content while ensuring real-time effects.

[0111] In addition, the present application also provides a real-time scene gesture generation device, such as Figure 7As shown in the structural block diagram of the real-time scene gesture generation device provided in the present application, the device specifically includes a memory 71 and a processor 72, and the memory 71 stores a computer program. When the computer program is executed by the processor 72, it implements the real-time scene gesture generation method according to any one of the above-mentioned methods.

[0112] In addition, the present application also provides a digital human interaction system, including any of the real-time scene gesture generation devices described above.

[0113] In addition, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the real-time scene gesture generation method according to any one of the above-mentioned methods is implemented.

[0114] Computer readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0115] The professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0116] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0117] The specific implementation methods described above further illustrate the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A real-time scene gesture generation method, characterized in that: include: Get input text; Using an action generation model to generate basic gesture actions of the digital human based on the input text; Retrieving the indexed semantic actions corresponding to the input text from a pre-established semantic enhancement dataset using a similarity matching method; Superimposing the indexed semantic action onto the sequence of basic gesture actions to generate the gesture action of the digital human; The construction process of the semantic enhancement dataset includes: inputting the annotated semantic gesture corpus into a large language model, generating enhanced text samples based on the semantic gestures, and the enhanced text samples are used to reflect the semantic information expressed by the gestures; after screening the enhanced text samples, inputting them into the large language model to iteratively generate enhanced semantic descriptions; and storing the mapping relationship between the gestures and the enhanced semantic descriptions in the semantic enhancement dataset for retrieval.

2. The real-time scene gesture generation method according to claim 1, characterized in that: The semantic gesture corpus includes the intention information, name information, action description, simple text instance, and action occurrence time of each gesture.

3. The real-time scene gesture generation method according to claim 1, characterized in that: After screening the enhanced text samples, inputting them into a large language model to iteratively generate enhanced semantic descriptions includes: Manually screening the enhanced text samples to remove text samples that are not semantically relevant; The filtered text samples are input into the large language model again for iteration to generate enhanced semantic descriptions; A semantic consistency check is performed on the enhanced semantic description to ensure that the enhanced semantic description is consistent with the initial intent.

4. The real-time scene gesture generation method according to any one of claims 1 to 3, characterized in that: The method of using a similarity matching method to retrieve an indexed semantic action corresponding to the input text from a pre-established semantic enhancement data set includes: Using a pre-trained word vector model to calculate the word vector for the entire sentence of the input text; In a pre-established semantic enhancement dataset, a pre-trained word vector model is used to calculate word vectors for the semantic descriptions in the semantic enhancement database; According to the cosine similarity matching method, the word vector with the highest overall semantic matching with the input text is retrieved from the semantic enhancement dataset, and the action corresponding to the word vector is used as the indexed semantic action; Sentences in the input text are segmented, and word vectors are calculated for each word obtained through segmentation to determine the key words that best match the overall semantics of the input text.

5. The real-time scene gesture generation method according to claim 4, characterized in that: The step of superimposing the indexed semantic action onto the sequence of basic gesture actions to generate the gesture action of the digital human comprises: The index semantic action is superimposed on the sequence of the basic gesture action by using the Bezier curve interpolation method to generate the gesture action of the digital human.

6. The real-time scene gesture generation method according to claim 5, characterized in that: Before superimposing the index semantic action onto the sequence of the basic gesture action using the Bezier curve interpolation method, the method further includes: Determine the time range of the basic gesture action P according to the position of the key words in the sentence [t start ,t end ]; Determine the time length T of the occurrence of the index semantic action ΔP e ; T e ≥1s, constraint t end -t start ≥1s, if not exceeded, fill in; Judgement end -t start ≥T e , calculate t start ,t end The midpoint t mid , with t mid To achieve the splicing of two fragments for reference, update the alignment range: Judgement end -t start <T e , then according to t end To achieve the splicing of two fragments, update the alignment range t start =t end -T e , t end Still t end ; T e When <1s, the action start and end times are covered from ΔP to P according to the median value without smoothing.

7. The real-time scene gesture generation method according to claim 6, characterized in that: The step of using the Bezier curve interpolation method to superimpose the index semantic action onto the sequence of the basic gesture action comprises: Take a certain buffer range at the end of the basic gesture action and the beginning of the index semantic action: p left is the buffer range of the tail of the basic gesture action; p right The buffer range at the beginning of the index semantic action; If the buffer range padding>10, then consider padding=10; if padding<10, use Bezier curve interpolation to smoothly transition the splicing area of ​​P and ΔP: Defining reference fragments Construct the sampling point and target matrices: x combined =[x A ,x B ],y combined =[A ref ,B ref ]; Construct interpolation function S(x): S(x combined )=y combined ; Sample the middle segment and get the smoothed result: x smooth =linspace(1,3,p left ),y smooth =S(x smooth ); Update completed stitching:

8. A real-time scene gesture generation device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the real-time scene gesture generation method according to any one of claims 1 to 7 is implemented.

9. A digital human interaction system, characterized in that: Comprising a real-time scene gesture generating device according to claim 8.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the real-time claim according to any one of claims 1 to 7 is implemented. ZSP240648CN Scene gesture generation method.

Images