Method, device and medium for generating three-dimensional companion gesture in real time based on random walk

Abstract

The application discloses a three-dimensional accompanying gesture real-time generation method and device based on random walk and a medium. First, a certain number of human body key point three-dimensional coordinate samples of human short-time speech are collected. According to sample sources and maximum standard deviation, the samples are divided into several accompanying gesture groups. Then, in each group, a one-to-many directed connection graph between samples is generated by calculating the vector distance between the first and last frames of different samples. Then, the continuous low-amplitude region (speech pause) of the input audio is searched, and the length of each voice is calculated. According to the length of each voice, a random number is used to select nodes (samples) and adjacent nodes in the directed connection graph, and through the insertion of transition frames, a human body key point three-dimensional coordinate sequence with the same length as the input audio is finally spliced. The advantage of the application is that the implementation cost is lower than that of the prior art. Compared with repeatedly playing recorded videos, the accompanying gestures generated by the application are almost free of repetition, and the effect is better.

Description

Technical Field

[0001] This invention relates to a method, device, and medium for real-time generation of three-dimensional accompanying gestures based on random walks. Background Technology

[0002] In recent years, information technology has developed rapidly, especially with the emergence of numerous new technologies and methods in artificial intelligence. Virtual Humans, also known as Embodied Conversational Agents, interact with natural people by simulating their appearance and voice, replacing inefficient human-to-human interactions and rigid traditional human-computer interfaces. This provides a more realistic experience while reducing human resource costs across industries. If this technology meets the requirements of industry and commerce, it can be widely applied to customer service across various sectors, and can also serve as a digital assistant for individuals or organizations, used for information gathering, providing information, and caring for patients, making people's work and lives more convenient.

[0003] Animated conversational agents are essentially a form of graphical (image-based) human-computer interaction, capable of interacting with humans or other similar entities in the environment through language, facial expressions, and body language. Since Justine Cassell et al. formally proposed the concept of animated conversational agents in 2000, there has been a growing demand for multimodal (text, speech, image) machine intelligence. One significant challenge for the fields of artificial intelligence and image synthesis is making the appearance of the conversational agent closely resemble that of a natural person, both statically and dynamically. Static anthropomorphism requires that the machine-generated appearance of the conversational agent be perceptible to a natural person and closely resemble their appearance. Dynamic anthropomorphism requires that the appearance of the conversational agent during human-computer interaction (including lip movements, facial expressions, and body language) be perceptible to a natural person and closely resemble their corresponding aspects.

[0004] In the past two years, a large number of technical achievements have emerged in the field of academic research on concrete conversational agents, especially in the field of speech-to-animation, but there are also some shortcomings.

[0005] In terms of real-time performance, most publicly available solutions generate video from speech, rather than outputting it in real time. From an implementation perspective, they often employ highly redundant software architectures and inefficient programming languages ​​(such as Python). For example, VOCA's algorithm steps are highly repetitive, and converting a few seconds of audio into video often takes more than ten minutes. From a software deployment perspective, these results often use adversarial networks for mapping, requiring significant floating-point operations and storage space, placing high demands on hardware. Based on observations of human communication, the vividness of speech-to-animation depends not only on the synchronization and accuracy of speech, lip movements, and graphic / image output, but also on the speaker's facial expressions and body language, and the interaction between these and lip movements (speech changes). Accompanying gestures for non-verbal communication can enhance the human-computer interaction experience, thereby improving the anthropomorphism of the concrete conversational agent.

[0006] Most importantly, with the exception of a few papers, most speech-to-animation results in both academic research and industrial applications do not incorporate accompanying gestures. This is one of the reasons why concrete conversational agents struggle to produce a satisfactory interactive experience in terms of human-like anthropomorphism. All of these factors significantly limit the applicability and application scope of existing academic solutions in the industrial field.

[0007] Currently, some solutions combine movement and speech. For example, an end-to-end neural network model has been proposed for physical robots, which can convert text into robot arm movements. However, due to the limitations of robot capabilities, it only considers arm movements; accompanying gestures usually involve joints in the neck, spine, fingers, etc., thus limiting its applicability. Other methods use recurrent neural networks to establish a bidirectional mapping between whole-body movements and language descriptions, and dynamic Bayesian networks to attempt to establish a connection between speech content and human actions. However, most methods use mostly two-dimensional gesture data and almost none consider finger movements, thus limiting the vividness and expressiveness of human actions. Summary of the Invention

[0008] To address the current technical problem of insufficient expressiveness in the combination of actions and speech, this invention provides a method, device, and medium for real-time generation of three-dimensional accompanying gestures based on random walks.

[0009] To achieve the above-mentioned technical objectives, the technical solution of the present invention is as follows:

[0010] A method for real-time generation of 3D speech gestures based on random walks includes the following steps:

[0011] Step 1: Collect the three-dimensional coordinate sequences of key points on the human body of different people while they are speaking, as different samples;

[0012] Step two: Classify all samples based on their source, and then group them according to the magnitude of the change in the three-dimensional coordinates of each sample in each category;

[0013] Step 3: Generate a connection graph based on the samples in each group, where each node in the connection graph corresponds to a sample. Then, calculate the vector distance between each node and connect each node with the first d nodes with the smallest vector distance to it with directed edges, thus forming a one-to-many directed connection graph.

[0014] Step 4: Collect audio data and divide the audio frames into multiple subsequences based on pauses in the speech, either consisting entirely of pause frames or entirely of non-pause frames; then select a group and a corresponding directed connection graph based on the audio content, usage scenario, and requirements.

[0015] Step 5: For each subsequence consisting entirely of paused frames, generate a corresponding sequence of three-dimensional coordinates of human key points for the same number of frames based on the three-dimensional coordinates of human key points in the default pose.

[0016] Step six: For each subsequence consisting entirely of non-pause frames, in the connection graph, based on the directed connections between samples, multiple nodes are randomly selected sequentially, and transition frames are added before the first node, after the last node, and between adjacent nodes; then the samples and transition frames are spliced ​​together to obtain a three-dimensional coordinate sequence of human key points with the same number of frames as the subsequence consisting entirely of non-pause frames; thus, together with the result of step five, a three-dimensional speech-accompanied gesture matching the speech audio is formed.

[0017] In the method described above, step one, the key points of the human body include the various joints of the human body and the extremities.

[0018] In the method described above, step two involves classifying all samples based on their source, specifically by classifying them according to the gender, age, and occupation of the people from whom the samples originated.

[0019] In the method described above, step two involves grouping based on the following steps:

[0020] For each sample after classification, i.e., a three-dimensional coordinate sequence Calculate the maximum standard deviation :

[0021]

[0022] Where L is the length of the three-dimensional coordinate sequence S; k , represents the frame number in S; i , represents the index in S; The number of key points on the human body;

[0023] For all samples in the nth class, divide them into several groups according to the numerical distribution of the maximum standard deviation.

[0024] The method described herein, wherein step three includes the following steps:

[0025] For containing The nth group of samples; construct a connection graph. ,in , representing the set of nodes and the set of edges. That is, each node in G corresponds to a sample.

[0026] Take the j-th sample For all and ,calculate End frame With sample First frame Vector distance between ;

[0027] For all and Take the d samples with the smallest vector distance and arrange them in ascending order ( Arranged in G, with corresponding nodes To the node Construct a directed edge with weight . , d represents the degree of a one-to-many directed graph.

[0028] The method described herein, wherein step four includes:

[0029] Divide speech and audio into Frame, in which The duration of human speech audio. The output frame rate for accompanying gestures is determined by sequentially checking audio frames whose amplitude is below a threshold A. If at least x consecutive audio frames meet this condition, they are considered pauses in speech and are called pause frames. Audio frames other than pause frames are non-pause frames. At this point, the entire audio is represented as an audio frame sequence. ,in The subsequence is represented by either all pause frames or all non-pause frames; then, a group and a connection graph G generated from this group are selected based on the speech audio content, usage scenario, and requirements.

[0030] The method described herein, wherein step five includes:

[0031] First, the three-dimensional coordinate sequence of key points of the human body is obtained. Set it to an empty set, and then for the i-th subsequence If all n audio frames are pause frames, then in the sequence Add n human body key point 3D coordinates ,Right now ,in For a number of n The three-dimensional coordinate sequence of key points on the human body; The three-dimensional coordinates of key human body points in the preset default pose are shown, and ⨁ indicates sequence splicing.

[0032] The method described herein, wherein step six includes:

[0033] For the i-th subsequence If all n audio frames are non-pause frames, then divide them into three subsequences. , and ,and , The number of frames is m, where m <n / 2;

[0034] right Get random numbers Take the sample corresponding to the j-th node in G. Then take its first frame. Interpolation is used to obtain the transition frame sequence. ,in ;Tr is a Sigmoid-like function;

[0035]

[0036] in ; a and b are vectors; then let ;

[0037] right ,if If the length L > n-2m, then let , where the function Indicates the truncation sequence The first n-2m frames; otherwise let And begin the random walk process: generate random numbers. Search by node Starting with edge p, find the endpoint of the edge whose weight is closest to p, and take the sample corresponding to the endpoint. ,Pick First frame and End frame Use the function Tr as a transition frame ,in Then let Repeat this process until... The number of frames added is equal to n-2m;

[0038] right ,Pick End frame Calculate the transition frame sequence ,in Then let .

[0039] An electronic device, comprising:

[0040] One or more processors;

[0041] Storage device for storing one or more programs.

[0042] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0043] A computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned method.

[0044] The technical advantages of this invention are as follows: compared to the widely adopted deep learning methods, it has lower computational costs and does not require independent floating-point computing units (such as GPUs), resulting in lower deployment costs. Furthermore, this invention has lower requirements for the number of samples collected, especially the sample duration. The connection graph generation step allows this invention to theoretically generate accompanying gestures of arbitrary duration with smooth gesture changes, thus reducing sampling costs. Unlike the methods widely used in industry that repeatedly play recorded accompanying gesture videos, this invention uses a random walk approach, making the probability of repeated accompanying gestures within a short period low, making them less noticeable and increasing their anthropomorphism. Testing has shown that this invention meets the requirements for real-time performance and high anthropomorphism. Attached Figure Description

[0045] Figure 1 This is a numerical distribution chart of the maximum standard deviation of 2240 samples collected in this embodiment of the invention;

[0046] Figure 2 This is a one-to-many directed connection graph with a degree of 3 generated from 2240 sample nodes in this embodiment of the invention;

[0047] Figure 3 This is a 3D coordinate sequence of human key points with a frame interval of 5, i.e., a skeleton node view, in an embodiment of the present invention.

[0048] Figure 4 This is a bar chart showing the average score of the anthropomorphism of accompanying gestures in embodiments of the present invention. Detailed Implementation

[0049] This embodiment provides a method for real-time generation of 3D speech gestures based on random walks for highly realistic figurative intelligent agents. Specifically, it consists of two parts: connection graph generation and graph random walks.

[0050] Connection graph generation

[0051] Connection graph generation consists of three processes: sample collection, clustering, and connection graph generation. A sample is collected from a human while they are speaking. Three-dimensional coordinate sequence of key points of an individual ,in L is the sequence length (number of frames). , Human key points include joints and extremities, with a sampling rate of [missing information - likely a percentage]. Hertz, then the duration of a sample is Seconds. The first dimension of the sample is the number of human keypoints, and the dimension is flattened for easier processing. Assume the output frame rate of the graph random walk is... The generation of the connection graph involves the following steps.

[0052] (1) Collected from several individuals The above samples are divided into several categories based on their source. The source of the samples can be determined according to specific needs and can generally include gender, age, occupation, etc.

[0053] (2) For each sample of the nth class Calculate its maximum standard deviation , This represents the magnitude of the movement:

[0054]

[0055] Where L is the length of the three-dimensional coordinate sequence S; k , represents the frame number in S; i , represents the index in S, here This is because three-dimensional coordinates include three axes: X, Y, and Z axes.

[0056] (3) For all samples in the nth class, divide them into several groups according to the numerical distribution of the maximum standard deviation calculated above. Specifically, in this embodiment, two maximum standard deviation thresholds, one large and one small, are set based on the number of samples in a class. Samples with a larger threshold are grouped into one group, samples with a smaller threshold are grouped into another group, and the rest are grouped into a single group, thus obtaining three groups. In this embodiment, the thresholds are set to ensure that the number of samples in the three groups is as equal as possible, so that each group has a relatively balanced number of samples in subsequent use. Of course, the grouping criteria can be adjusted as needed in specific use.

[0057] (4) Repeat steps (2) and (3) above for all classes. The final result is a set of all classes and all groups that are labeled with different data sources and other characteristics of accompanying gesture groups.

[0058] (5) For the nth group, assume it contains One sample; plotting Node set edge set That is, each node in graph G corresponds to a sample. If Then, a common interpolation algorithm is used to increase the number of frames in the sample, so that the sampling rate of all samples is equal. .

[0059] (6) Take the j-th sample For all and ,calculate End frame With sample First frame Vector distance between .

[0060] (7) For all and Take the d samples with the smallest vector distance and sort them in the order of smallest, second smallest, and second smallest. Arranged in G, with corresponding nodes To the node Construct a directed edge with weight . , d is called the degree of a one-to-many directed graph.

[0061] (8) Repeat steps (6) and (7) for all samples in the nth group. At this point, the one-to-many directed connection graph G is generated.

[0062] (9) Repeat steps (5), (6), (7), and (8) for all groups. At this point, each group has a one-to-many directed connection graph.

[0063] Random walk

[0064] Graph random walks are used to generate a sequence of 3D coordinates of human keypoints for a duration equal to the input audio. This process requires the following input: a duration of... Human voice audio, key points of the human body in the default posture ( ( ) three-dimensional coordinates (i.e., the 3D coordinates of key human points in the default pose), the directed connected graph G corresponding to the group selected according to the application scenario, the set of random numbers R satisfying the average distribution, and the output frame rate. This process outputs a sequence of three-dimensional coordinates for key human body points. The process includes the following steps.

[0065] (1) Human speech audio is divided into The audio is processed sequentially, examining each frame for an amplitude below a threshold A. If at least x consecutive audio frames meet this condition, they are considered pauses in speech and are called pause frames; the other audio frames are non-pause frames. The entire audio can then be represented as a sequence of audio frames. , The sequence is divided into subsequences, and each subsequence consists entirely of either paused or non-paused frames. Then, a 3D coordinate sequence of human keypoints is initialized. .

[0066] (2) For the i-th subsequence If all n audio frames are pause frames, then in the sequence Add n default poses for the three-dimensional coordinates of human key points , that is to say . That is, containing n The sequence of three-dimensional coordinates of key human body points. This is because, under normal circumstances, when a person pauses during continuous speech, their movements also pause, so the same default pose is used in all pause frames here.

[0067] (3) For the i-th subsequence If all n audio frames are non-pause frames, then divide them into three subsequences. , and .in , The number of frames is m (m <n / 2)。

[0068] (4) Get random numbers Take the sample corresponding to the j-th node in the directed connected graph G. Then take its first frame. Interpolation is used to obtain the transition frame sequence. ,in Tr is a sigmoid-like function.

[0069]

[0070] in Let a and b be vectors. Then let .

[0071] (5) ,if The length L>n-2m, add The function Indicates the truncation sequence The first n-2m elements. Otherwise, append. And begin the random walk process: generate random numbers. Search by node Starting with edge p, find the endpoint of the edge whose weight is closest to p, and take the sample corresponding to the endpoint. ,Pick First frame and The last frame uses the function Tr as a transition frame and appends the transition frame to the end. , that is to say Repeat this process until this step. The number of additional frames is equal to n-2m.

[0072] (6) ,Pick End frame Calculate the transition frame sequence ,in .make .

[0073] (7) For all subsequences Repeat steps (2) or (3)(4)(5)(6).

[0074] Ultimately, a sequence of three-dimensional coordinates of human key points is obtained, with the same number of frames as human speech audio. It can then synchronize with video and audio rendering to output a concrete intelligent agent's accompanying gestures.

[0075] In this embodiment, the number of key points of the human body is collected. =55. Single sample sequence length L=64. Human keypoint sampling rate =15 Hz. The output frame rate of the graph random walk is 15 Hz. =60 frames per second. Number of samples collected. >2000. Cubic spline interpolation can be used as the interpolation algorithm. In this embodiment, gender, age, occupation, and movement amplitude from the data source are used as features of the accompanying gesture group, and the degree d=3 of the one-to-many directed connection graph is taken. The Euclidean distance function dist is used for the vector distance function. Additionally, the audio threshold A = 0.02 × 32768, x = 30. Subsequence v in v out The number of frames is m=16. Figure 1 The numerical distribution of the maximum standard deviation of the 2240 collected samples is shown. Figure 2 This shows a one-to-many directed connection graph with a degree of 3 generated from 2240 sample nodes. Figure 3 It displays a sequence of 3D coordinates of human key points with a frame interval of 5, i.e., a view of skeletal nodes.

[0076] Based on this embodiment, tests were conducted by inputting audio of different durations and detecting the time required to generate a coordinate sequence (graph random walk). The experimental environment consisted of an Intel 64 Family 6 Model 165 Stepping 5 Genuine Intel ~3792 MHz CPU and 32GB of RAM. Some experimental results are shown in the table below.

[0077] Table 1. Experimental results on the time taken to generate the coordinate sequence (graph random walk).

[0078]

[0079] In this embodiment, the time required to generate the three-dimensional coordinate sequence of human body key points is approximately 0.1% to 0.4% of the input audio duration, demonstrating strong real-time performance.

[0080] The 16 accompanying gesture videos generated in this embodiment were mixed with 16 real-life accompanying gesture videos, and 30 volunteers were randomly invited to watch 4 of them and rate them. Both types of videos are skeletal node views. A score of 5 indicates that the volunteers think it is very close to a real person, a score of 4 indicates that it is relatively close to a real person, and a score from 0 indicates that it is not like a real person at all. Figure 4 The average score of 32 videos is presented. Light-colored bars represent the average score of the accompanying gesture videos generated in this embodiment, while dark-colored bars represent the average score of the accompanying gesture videos featuring real people. The difference between the two is not significant, indicating that the accompanying gestures generated in this embodiment are relatively close to those of real people.

[0081] According to embodiments of the present invention, the present invention also provides an electronic device and a computer-readable medium.

[0082] Electronic devices include:

[0083] One or more processors;

[0084] Storage device for storing one or more programs.

[0085] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0086] In practical use, users can interact with servers, which are also electronic devices, via a network to receive or send messages. Terminal devices are generally various electronic devices equipped with a display and used through a human-computer interface, including but not limited to smartphones, tablets, laptops, and desktop computers. Various specific application software can be installed on these terminal devices as needed, including but not limited to web browsers, instant messaging software, social media platforms, and shopping apps.

[0087] A server is a network service provider that offers various services, such as a backend server that performs calculations on 3D coordinate sequences of key human body points and audio data transmitted from terminal devices. This enables the generation of 3D accompanying gestures that match the speech, and the return of the final 3D accompanying gestures to the terminal device.

[0088] The method provided in this embodiment is generally executed by the server. In practical applications, the terminal device can also directly execute this method under necessary conditions.

[0089] Similarly, the computer-readable medium of the present invention stores a computer program thereon, which, when executed by a processor, implements the three-dimensional accompanying gesture generation method of the embodiments of the present invention.

Claims

1. A method for real-time generation of three-dimensional accompanying gestures based on random walk, characterized in that, Includes the following steps: Step 1: Collect the three-dimensional coordinate sequences of key points on the human body of different people while they are speaking, as different samples; Step two: Classify all samples based on their source, and then group them according to the magnitude of the change in the three-dimensional coordinates of each sample in each category; Step 3: Generate a connection graph based on the samples in each group, where each node in the connection graph corresponds to a sample. Then, calculate the vector distance between each node and connect each node with the first d nodes with the smallest vector distance to it with directed edges, thus forming a one-to-many directed connection graph. Step 4: Collect audio data and divide the audio frames into multiple subsequences based on pauses in the speech, either consisting entirely of pause frames or entirely of non-pause frames; then select a group and a corresponding directed connection graph based on the audio content, usage scenario, and requirements. Step 5: For each subsequence consisting entirely of paused frames, generate a corresponding sequence of three-dimensional coordinates of human key points for the same number of frames based on the three-dimensional coordinates of human key points in the default pose. Step six: For each subsequence consisting entirely of non-pause frames, in the connection graph, according to the directed connections between samples, multiple nodes are randomly selected sequentially, and transition frames are added before the first node, after the last node, and between adjacent nodes; then the samples and transition frames are spliced ​​together to obtain a three-dimensional coordinate sequence of human key points with the same number of frames as the subsequence consisting entirely of non-pause frames; thus, together with the result of step five, a three-dimensional speech-accompanied gesture matching the speech audio is formed.

2. The method according to claim 1, characterized in that, In step one, the key points of the human body include the various joints and extremities of the body.

3. The method according to claim 1, characterized in that, In step two, all samples are classified based on their source, which is done according to the gender, age, and occupation of the people from whom the samples originated.

4. The method according to claim 1, characterized in that, In step two, the grouping is achieved based on the following steps: For each sample after classification, i.e., a three-dimensional coordinate sequence Calculate the maximum standard deviation : ； Where L is the length of the three-dimensional coordinate sequence S; k , represents the frame number in S; i , represents the index in S; The number of key points on the human body; For all samples in the nth class, divide them into several groups according to the numerical distribution of the maximum standard deviation.

5. The method according to claim 3, characterized in that, Step three includes the following steps: For containing The nth group of samples; construct a connection graph. ,in , representing the set of nodes and the set of edges. That is, each node in G corresponds to a sample. Take the j-th sample For all and ,calculate End frame With sample First frame Vector distance between ; For all and Take the d samples with the smallest vector distance and arrange them in ascending order ( Arranged in G, with corresponding nodes To the node Construct a directed edge with weight . , d represents the degree of a one-to-many directed graph.

6. The method according to claim 1, characterized in that, Step four includes: Divide speech and audio into Frame, in which The duration of human speech audio. The output frame rate for accompanying gestures is determined by sequentially checking audio frames whose amplitude is below a threshold A. If at least x consecutive audio frames meet this condition, they are considered pauses in speech and are called pause frames. Audio frames other than pause frames are non-pause frames. At this point, the entire audio is represented as an audio frame sequence. ,in The subsequence is represented by either all pause frames or all non-pause frames; then, a group and a connection graph G generated from this group are selected based on the speech audio content, usage scenario, and requirements.

7. The method according to claim 6, characterized in that, Step five includes: First, the three-dimensional coordinate sequence of key points of the human body is... Set it to an empty set, and then for the i-th subsequence If all n audio frames are pause frames, then in the sequence Add n human body key point 3D coordinates ,Right now ,in For a number of n The three-dimensional coordinate sequence of key points on the human body; The three-dimensional coordinates of key human body points in the preset default pose are shown, and ⨁ indicates sequence splicing.

8. The method according to claim 7, characterized in that, Step six includes: For the i-th subsequence If all n audio frames are non-pause frames, then divide them into three subsequences. , and ,and , The number of frames is m, where m <n / 2; right Get random numbers Take the sample corresponding to the j-th node in G. Then take its first frame. Interpolation is used to obtain the transition frame sequence. ,in ;Tr is a Sigmoid-like function; ； in ; a and b are vectors; then let ; right ,if If the length L > n-2m, then let , where the function Indicates the truncation sequence The first n-2m frames; otherwise let And begin the random walk process: generate random numbers. Search by node Starting with edge p, find the endpoint of the edge whose weight is closest to p, and take the sample corresponding to the endpoint. ,Pick First frame and End frame Use the function Tr as a transition frame ,in Then let Repeat this process until... The number of frames added is equal to n-2m; right ,Pick End frame Calculate the transition frame sequence ,in Then let .

9. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

Images