Method and system for matching hand actions of digital doll shape through human voice

Abstract

The invention discloses a method and system for matching digital doll shape hand actions through human voice, and the method comprises the steps: S1, collecting voice data containing different voice instructions and corresponding digital doll hand action data, and carrying out the pairing and marking processing; s2, performing feature extraction on the voice data and the hand action data, and converting the voice data and the hand action data into feature vectors; s3, dividing the data set into a training set, a verification set and a test set for model training to obtain a voice-action matching model; and S4, receiving voice input of a user, performing feature extraction on the input voice, predicting a hand action matched with the voice input by using the trained voice-action matching model, applying the predicted hand action to the digital doll image, and displaying the predicted hand action on the digital doll image in real time. According to the invention, through the deep learning model and the multi-modal feature extraction technology, high-precision matching of voice and hand actions is realized, and the smoothness and naturalness of action generation are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and computer graphics, and more specifically to a method and system for matching the hand movements of a digital puppet image through human voice. The method and system are particularly suitable for interactive scenarios such as virtual assistants, game characters, virtual reality, and augmented reality, which require real-time generation of voice-driven movements. Background Art

[0002] A digital puppet is a virtual character created through technologies such as computer graphics, motion capture, image rendering, and artificial intelligence. A digital puppet possesses a human-like appearance, behavior, and interactive capabilities, and exists in a virtual environment, such as a computer screen, virtual reality (VR), or augmented reality (AR) device. Its image, movements, and voice can be customized according to different needs and scenarios. The hand movements of existing digital puppets are mainly controlled and generated by voice-action matching methods that are rule-driven or statically mapped. These methods typically associate specific voice commands with fixed hand movements through a predefined rule base or a simple mapping table. For example, in a virtual assistant or game character, the system triggers predefined gestures through keyword recognition. However, this technology has significant shortcomings: First, it relies on a fixed rule base or mapping relationship, making it difficult to handle complex and changing voice input and refined action requirements, resulting in stiff, discontinuous, and unnatural action generation; second, existing technologies are relatively simple in feature extraction, usually relying only on a single voice feature (such as pitch or volume) or simple hand movement data, and cannot fully utilize the multimodal information of voice and movement, resulting in low matching accuracy and poor adaptability; in addition, existing technologies generally lack efficient real-time processing capabilities, and the delay from voice input to action generation is high, making it difficult to meet the needs of real-time interactive scenarios; finally, existing technologies generally lack user feedback mechanisms and cannot optimize the system based on the user's actual usage, resulting in a lack of personalized and intelligent interactive experience. These problems limit the widespread application of existing technologies in fields such as virtual assistants, game characters, and virtual reality. Therefore, it is necessary to improve and optimize the algorithm processing method for digital puppet hand movements. Summary of the Invention

[0003] In view of the above problems, the present invention provides a method for matching hand movements of a digital puppet image through human voice, comprising the following steps:

[0004] S1, data collection and pairing annotation, collecting audio sample voice data containing different voice commands and digital puppet hand movement data corresponding to the voice commands, and pairing and annotating the voice data with the digital puppet hand movement data;

[0005] S2, feature extraction, performing voice feature extraction on the audio sample voice data and converting the voice signal into a feature vector;

[0006] Extracting hand motion features from the hand motion data of the digital puppet, and converting the hand motion data of the digital puppet into feature vectors;

[0007] S3, model training, dividing the dataset into a training set, a validation set, and a test set, selecting a model type, and using the training set to train the selected model type to obtain a speech-action matching model;

[0008] S4, voice and action matching, receives the user's voice input and extracts features of the input voice, uses the trained voice-action matching model to predict the hand movements that match the voice input, and applies the predicted hand movements to the digital puppet image, and displays the predicted hand movements on the digital puppet image in real time.

[0009] As a further illustration of the present invention, in step S2, the audio sample speech data and the digital puppet hand motion data are pre-processed before feature extraction.

[0010] Furthermore, in step S2, MFCC (Mel-Frequency Cepstral Coefficient), Chroma or Spectrogram feature extraction method is used to extract speech features from the audio sample speech data, and the speech signal is converted into a feature vector.

[0011] Furthermore, in step S2, PCA (Principal Component Analysis) or DTW (Dynamic Time Warping) method is used to extract hand motion features from the digital puppet hand motion data, and the digital puppet hand motion data is also converted into feature vectors.

[0012] Furthermore, in step S3, the model type is one of LSTM (Long Short-Term Memory Network), GRU (Gated Recurrent Unit), CNN (Convolutional Neural Network) or a combination thereof.

[0013] Furthermore, in step S4, when the hand movements of the predicted digital puppet image are displayed in real time based on the user's voice input, user feedback on the movement matching effect is collected, and the voice-movement matching model is optimized according to the user feedback.

[0014] On the other hand, the present invention also provides a system for matching the hand movements of a digital puppet image through human voice, which specifically includes a voice acquisition module, a voice feature extraction module, a hand movement feature extraction module, a matching model training module, a movement generation module and a real-time feedback module;

[0015] The voice acquisition module is used to collect the user's voice signal and pre-process the voice signal;

[0016] The speech feature extraction module is used to extract features from the collected speech signal and convert the speech signal into a high-dimensional feature vector;

[0017] The hand motion feature extraction module is used to obtain the hand motion data of the digital puppet and perform feature extraction and dimensionality reduction processing on the hand motion data;

[0018] The matching model training module is used to perform matching training on the speech feature vector and the hand movement feature vector, and generate a high-precision speech-movement matching model through iterative optimization of the training set, the validation set and the test set;

[0019] The action generation module is used to receive the voice signal input by the user, call the matching model in real time to predict the corresponding hand movement, and transmit the action data to the control system of the digital puppet image through the driving interface to realize the real-time generation and display of the hand movement;

[0020] The real-time feedback module is used to collect user feedback on the action matching effect and use the feedback data for continuous optimization of the speech-action matching model.

[0021] Furthermore, the system adopts a distributed architecture, the voice acquisition module, voice feature extraction module, hand movement feature extraction module and matching model training module are connected through a high-speed data bus, and the movement generation module and the control system of the digital puppet image realize real-time data transmission through a low-latency communication protocol.

[0022] Beneficial effects of the present invention:

[0023] (1) By introducing multimodal speech feature extraction methods such as MFCC and Chroma, and combining PCA and DTW to perform efficient feature extraction and dimensionality reduction on hand motion data, the accuracy and robustness of speech and hand motion matching are significantly improved. It can adapt to complex and changeable speech input and refined motion requirements, solve the problem of stiff and incoherent motion generation in existing technologies, and achieve high-precision and high-efficiency speech and motion feature matching.

[0024] (2) A deep learning model combining LSTM and CNN is used to fully utilize the temporal and spatial features of speech and motion data to achieve high-precision speech-motion matching, making the hand movements of the digital puppet more natural and smooth, greatly improving the user experience.

[0025] (3) Through distributed architecture and low-latency communication protocols, the system can efficiently process voice input and generate corresponding hand movements in real time, significantly reducing response delays and meeting the needs of real-time interactive scenarios.

[0026] (4) The present invention introduces a real-time feedback optimization mechanism, which collects user feedback on the action matching effect through a real-time feedback module, and uses the feedback data to continuously optimize the voice-action matching model, so that the system can adapt to the voice characteristics and personalized needs of different users, further enhancing the system's intelligence level and interactive experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of the method for matching the hand movements of a digital puppet image with human voice according to the present invention;

[0028] Figure 2 This is a module structure diagram of the system for matching the hand movements of a digital puppet image with human voice in the present invention. DETAILED DESCRIPTION

[0029] The following is a detailed description of the embodiments of the present invention with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0030] In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", "first", "second", etc., indicating directions or positions or sequential relationships, are based on the directions or positions or sequential relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention.

[0031] A method for matching hand movements of a digital puppet image with human voice is invented, which specifically includes the following steps:

[0032] S1, data collection and pairing annotation, collects audio sample voice data containing different voice commands and digital puppet hand movement data corresponding to the voice commands, and pairs and annotates the voice data with the digital puppet hand movement data.

[0033] S2, feature extraction, performing voice feature extraction on the audio sample voice data and converting the voice signal into a feature vector;

[0034] Extracting hand motion features from the hand motion data of the digital puppet, and converting the hand motion data of the digital puppet into feature vectors;

[0035] S3, model training, divides the data set into a training set, a validation set, and a test set, selects a model type, and uses the training set to train the selected model type to obtain a speech-to-motion matching model; during the model training process, the validation set and the test set can be used to verify and test the performance of the speech-to-motion matching model. If the performance of the speech-to-motion matching model does not meet the design standards, the hyperparameters can be adjusted and training can be continued to optimize the model performance so that it meets the designed application standards.

[0036] S4, voice and action matching, receives the user's voice input and extracts features of the input voice, uses the trained voice-action matching model to predict the hand movements that match the voice input, and applies the predicted hand movements to the digital puppet image, and displays the predicted hand movements on the digital puppet image in real time.

[0037] The main principle of the present invention is to collect a large number of audio samples containing different voice commands and the corresponding hand movement data (such as gestures, finger movements, etc.) as basic data. After pre-processing such as noise reduction and normalization, corresponding feature extraction is performed to form a data set that can be used for training. A voice-action matching model that can predict the hand movements of a digital puppet based on human voice is trained. The algorithm and model are then integrated into practical applications to predict and generate the hand movements of a digital puppet. It can be applied to fields such as virtual assistants, game characters and virtual reality, solving the problems of low accuracy and poor real-time performance of voice-driven hand movement matching in the existing technology.

[0038] Specifically, preferably, in step S2, the audio sample speech data and the digital puppet hand motion data are preprocessed before feature extraction. As mentioned above, this preprocessing includes noise reduction, normalization, etc., to improve the accuracy of subsequent processing.

[0039] Specifically, in step S2, MFCC (Mel-Frequency Cepstral Coefficients), Chroma, or Spectrogram feature extraction methods are used to extract speech features from the audio sample speech data, converting the speech signal into a feature vector. PCA (Principal Component Analysis) or DTW (Dynamic Time Warping) methods are used to extract hand motion features from the digital puppet hand motion data, also converting the digital puppet hand motion data into a feature vector. By combining MFCC, Chroma, and other speech feature extraction methods with PCA and DTW hand motion feature extraction methods, high-precision and high-efficiency matching of speech and motion features is achieved.

[0040] Specifically, in step S3, the model type is one of LSTM (long short-term memory network), GRU (gated recurrent unit), CNN (convolutional neural network) or a combination thereof. For example, a model combining LSTM and CNN can be used to make full use of the temporal and spatial features of speech and motion data to improve the accuracy and naturalness of the speech-motion matching model.

[0041] Specifically, as a preference, in step S4, when the hand movements of the predicted digital puppet image are displayed in real time based on the user's voice input, user feedback on the movement matching effect is collected, and the voice-movement matching model is optimized according to the user feedback, which can adapt to the voice characteristics and personalized needs of different users, and further enhance the intelligence level and interactive experience.

[0042] To implement the method of matching the hand movements of a digital puppet image with human voice in the present invention, a system for matching the hand movements of a digital puppet image with human voice can be implemented in practical applications. The system specifically includes a voice acquisition module, a voice feature extraction module, a hand movement feature extraction module, a matching model training module, a movement generation module, and a real-time feedback module.

[0043] The voice acquisition module is used to collect the user's voice signal and pre-process the voice signal;

[0044] The speech feature extraction module is used to extract features from the collected speech signal and convert the speech signal into a high-dimensional feature vector;

[0045] The hand motion feature extraction module is used to obtain the hand motion data of the digital puppet and perform feature extraction and dimensionality reduction processing on the hand motion data;

[0046] The matching model training module is used to perform matching training on the speech feature vector and the hand movement feature vector, and generate a high-precision speech-movement matching model through iterative optimization of the training set, the validation set and the test set;

[0047] The action generation module is used to receive the voice signal input by the user, call the matching model in real time to predict the corresponding hand movement, and transmit the action data to the control system of the digital puppet image through the driving interface to realize the real-time generation and display of the hand movement;

[0048] The real-time feedback module is used to collect user feedback on the action matching effect and use the feedback data for continuous optimization of the speech-action matching model.

[0049] As a preferred embodiment, the system adopts a distributed architecture, the voice acquisition module, voice feature extraction module, hand movement feature extraction module and matching model training module are connected through a high-speed data bus, and the movement generation module and the control system of the digital puppet image realize real-time data transmission through a low-latency communication protocol.

[0050] The above description is merely an explanation of the preferred embodiments of the present invention and should not be construed as limiting the claims. The present invention is not limited to the above embodiments, and variations in the specific structure are permitted. In short, all variations made within the scope of the independent claims of the present invention are also within the scope of protection of the present invention.

Claims

1. A method for matching hand movements of a digital puppet image through human voice, characterized in that: The following steps are involved: S1, data collection and pairing annotation, collecting audio sample voice data containing different voice commands and digital puppet hand movement data corresponding to the voice commands, and pairing and annotating the voice data with the digital puppet hand movement data; S2, feature extraction, performing voice feature extraction on the audio sample voice data and converting the voice signal into a feature vector; Extracting hand motion features from the hand motion data of the digital puppet, and converting the hand motion data of the digital puppet into feature vectors; S3, model training, dividing the dataset into a training set, a validation set, and a test set, selecting a model type, and using the training set to train the selected model type to obtain a speech-action matching model; S4, voice and action matching, receives the user's voice input and extracts features of the input voice, uses the trained voice-action matching model to predict the hand movements that match the voice input, and applies the predicted hand movements to the digital puppet image, and displays the predicted hand movements on the digital puppet image in real time.

2. The method of matching hand movements of a digital puppet image through human voice according to claim 1, characterized in that: In the step S2, the audio sample speech data and the digital puppet hand movement data are pre-processed before feature extraction.

3. The method of matching hand movements of a digital puppet image through human voice according to claim 1, characterized in that: In the step S2, the MFCC, Chroma or Spectrogram feature extraction method is used to extract speech features from the audio sample speech data, and the speech signal is converted into a feature vector.

4. The method of matching hand movements of a digital puppet image through human voice according to claim 1, characterized in that: In the step S2, the PCA or DTW method is used to extract hand motion features from the hand motion data of the digital puppet, and the hand motion data of the digital puppet is also converted into a feature vector.

5. The method of matching hand movements of a digital puppet image through human voice according to claim 1, characterized in that: In step S3, the model type is one of LSTM, GRU, and CNN, or a combination thereof.

6. The method of matching hand movements of a digital puppet image through human voice according to claim 1, characterized in that: In step S4, when the hand movements of the predicted digital puppet image are displayed in real time based on the user's voice input, user feedback on the movement matching effect is collected, and the voice-movement matching model is optimized according to the user feedback.

7. A system for matching hand movements of a digital puppet image with human voice, characterized in that: It includes voice acquisition module, voice feature extraction module, hand movement feature extraction module, matching model training module, movement generation module and real-time feedback module; The voice acquisition module is used to collect the user's voice signal and pre-process the voice signal; The speech feature extraction module is used to extract features from the collected speech signal and convert the speech signal into a high-dimensional feature vector; The hand motion feature extraction module is used to obtain the hand motion data of the digital puppet and perform feature extraction and dimensionality reduction processing on the hand motion data; The matching model training module is used to perform matching training on the speech feature vector and the hand movement feature vector, and generate a high-precision speech-movement matching model through iterative optimization of the training set, the validation set and the test set; The action generation module is used to receive the voice signal input by the user, call the matching model in real time to predict the corresponding hand movement, and transmit the action data to the control system of the digital puppet image through the driving interface to realize the real-time generation and display of the hand movement; The real-time feedback module is used to collect user feedback on the action matching effect and use the feedback data for continuous optimization of the speech-action matching model.

8. The system for matching hand movements of a digital puppet image through human voice according to claim 7, characterized in that: The system adopts a distributed architecture. The voice acquisition module, voice feature extraction module, hand movement feature extraction module and matching model training module are connected through a high-speed data bus. The movement generation module and the control system of the digital puppet image realize real-time data transmission through a low-latency communication protocol.

Images