Somatosensory dance game method for generating virtual character based on user voice information

Abstract

The invention discloses a somatosensory dance game method and device for generating a virtual character based on user voice information and a computer readable storage medium, and the method comprises the steps: activating a preset voice assistant to carry out interactive communication with a user when the virtual character of a somatosensory dance game is created; collecting first voice data of communication between the user and the voice assistant, and sending a reading request for reading a specified text to the user when the first voice data meets a preset condition; collecting second voice data when the user reads the specified text; based on the second voice data, utilizing a pre-trained machine learning model to analyze biological characteristics and character characteristics of the user, and generating a timbre model matched with the timbre of the user; generating an image of the virtual character based on the biological characteristics and the character characteristics; and generating the voice of the virtual character based on the voice model. The somatosensory dance game method for generating the virtual character based on the user voice information has the advantage of intelligently and individually creating the virtual character.

Description

Technical Field

[0001] The present invention relates to the field of somatosensory game technology, and in particular to a somatosensory dance game method, device and computer-readable storage medium for generating a virtual character based on user voice information. Background Art

[0002] With the rapid development of virtual reality (VR) and somatosensory gaming technologies, users are increasingly demanding the creation of personalized virtual characters in virtual environments. Existing somatosensory gaming and VR systems usually rely on users to manually input or select preset parameters to create virtual characters. Although this method can provide a certain degree of personalization, it has significant shortcomings in practical applications:

[0003] First, the method of manually inputting or selecting preset parameters lacks accurate recognition of the user's real biometric characteristics. Users usually need to select or input basic information such as height, body shape, gender, etc. This method is not only time-consuming and laborious, but also easily affected by the user's subjective judgment, resulting in the generated virtual character not matching the user's actual characteristics, making it difficult to fully reflect individual differences and personalized needs.

[0004] Secondly, the process of manually inputting or presetting parameters itself can easily make users feel cumbersome and complicated, especially for users who are not familiar with technical details. This not only reduces the user's enthusiasm for participation, but may also cause some users to give up using these functions, thus affecting the market acceptance of the product.

[0005] In view of the above problems, the prior art urgently needs a more intelligent and personalized virtual character creation method. Summary of the invention

[0006] The embodiment of the present application aims to realize intelligent and personalized creation of virtual characters by providing a somatosensory dance game method for generating virtual characters based on user voice information.

[0007] To achieve the above-mentioned purpose, the embodiment of the present application provides a method for generating a somatosensory dance game of a virtual character based on user voice information, comprising:

[0008] When creating a virtual character for a somatosensory dance game, activate the preset voice assistant to interact with the user;

[0009] Collecting first voice data of the user communicating with the voice assistant, and sending a reading request to the user to read aloud the specified text when the first voice data meets a preset condition;

[0010] Collecting second voice data when the user reads the specified text;

[0011] Based on the second voice data, using a pre-trained machine learning model to analyze the user's biological characteristics and personality characteristics, and generate a timbre model that matches the user's timbre;

[0012] generating an image of the virtual character based on the biological characteristics and personality characteristics;

[0013] The voice of the virtual character is generated based on the voice model.

[0014] To achieve the above-mentioned objectives, an embodiment of the present application also proposes a somatosensory dance game device for generating a virtual character based on user voice information, comprising a memory, a processor, and a somatosensory dance game program for generating a virtual character based on user voice information stored in the memory and executable on the processor, wherein when the processor executes the somatosensory dance game program for generating a virtual character based on user voice information, a somatosensory dance game method for generating a virtual character based on user voice information as described in any one of the above items is implemented.

[0015] To achieve the above-mentioned objectives, an embodiment of the present application also proposes a computer-readable storage medium, on which is stored a somatosensory dance game program for generating a virtual character based on user voice information. When the somatosensory dance game program for generating a virtual character based on user voice information is executed by a processor, a somatosensory dance game method for generating a virtual character based on user voice information as described in any of the above items is implemented.

[0016] The somatosensory dance game method of the technical solution of the present application, through the interactive communication between the voice assistant and the user, eliminates the user's vigilance and establishes initial trust with the user, and then by collecting the voice data of the user when reading the specified text, combined with the pre-trained machine learning model, it is possible to analyze the user's biological characteristics, personality characteristics and timbre model based on the voice data, and then automatically generate a virtual character that is highly matched with the user based on these biological characteristics, personality characteristics and timbre models. This method not only eliminates the cumbersome process of manually inputting or selecting preset parameters in traditional somatosensory games and virtual reality technologies, but also significantly improves the personalization and realism of virtual characters. In addition, because the technical solution can automatically capture and analyze the user's voice characteristics, the character creation process becomes more natural and seamless, which enhances the user's interactive experience and immersion. Therefore, the somatosensory dance game method of the technical solution of the present application has the advantage of creating virtual characters intelligently and individually. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0018] Figure 1 This is a module structure diagram of an embodiment of a somatosensory dance game device for generating a virtual character based on user voice information according to the present invention;

[0019] Figure 2 The figure is a flow chart of an embodiment of a method for generating a somatosensory dance game of a virtual character based on user voice information according to the present invention.

[0020] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0021] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0022] In order to better understand the above technical solution, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0023] It should be noted that in the claims, any reference symbols placed between brackets shall not be constructed as limitations to the claims. The presence of "comprising" in the text does not exclude the presence of components or steps not listed in the claims. The quantifier "one" or "an" preceding a component does not exclude the presence of a plurality of such components. The present invention can be implemented by means of hardware comprising several different components and by means of appropriately programmed computers. In a unit claim that lists several means, several of these means may be embodied by the same hardware item. The use of "first", "second", and "third" etc. does not indicate any order, and these words may be interpreted as names.

[0024] like Figure 1 As shown, Figure 1 It is a schematic diagram of the structure of a server 1 (also called a somatosensory dance game device for generating a virtual character based on user voice information) of a hardware operating environment involved in an embodiment of the present invention.

[0025] The server of the embodiment of the present invention is a device with display function, such as "Internet of Things devices", smart air conditioners, smart lights, smart power supplies with networking functions, AR / VR devices with networking functions, smart speakers, self-driving cars, PCs, smart phones, tablet computers, e-book readers, portable computers, etc.

[0026] like Figure 1 As shown, the server 1 includes: a memory 11, a processor 12 and a network interface 13.

[0027] The memory 11 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the server 1, such as a hard disk of the server 1. In other embodiments, the memory 11 may also be an external storage device of the server 1, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc., equipped on the server 1.

[0028] Furthermore, the memory 11 may include an internal storage unit of the server 1 and an external storage device. The memory 11 may be used not only to store application software and various data installed on the server 1, such as the code of the somatosensory dance game program 10 for generating a virtual character based on user voice information, but also to temporarily store data that has been output or is to be output.

[0029] In some embodiments, the processor 12 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, which is used to run the program code or process data stored in the memory 11, such as executing a somatosensory dance game program 10 that generates a virtual character based on user voice information.

[0030] The network interface 13 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and is generally used to establish a communication connection between the server 1 and other electronic devices.

[0031] The network may be the Internet, a cloud network, a wireless fidelity (Wi-Fi) network, a personal area network (PAN), a local area network (LAN), and / or a metropolitan area network (MAN). Various devices in the network environment may be configured to connect to the communication network according to various wired and wireless communication protocols. Examples of such wired and wireless communication protocols may include, but are not limited to, at least one of the following: Transmission Control Protocol and Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), ZigBee, EDGE, IEEE 802.11, Light Fidelity (Li-Fi), 802.16, IEEE 802.11s, IEEE 802.11g, multi-hop communication, wireless access point (AP), device-to-device communication, cellular communication protocol, and / or Bluetooth (Blue Tooth) communication protocol or a combination thereof.

[0032] Optionally, the server may further include a user interface, which may include a display, an input unit such as a keyboard, and an optional user interface may further include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be referred to as a display screen or a display unit, which is used to display information processed in the server 1 and to display a visual user interface.

[0033] Figure 1 Only the server 1 having components 11-13 and a somatosensory dance game program 10 for generating a virtual character based on user voice information is shown. It can be understood by those skilled in the art that Figure 1 The structure shown does not constitute a limitation on the server 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0034] In this embodiment, the processor 12 can be used to call the somatosensory dance game program for generating a virtual character based on user voice information stored in the memory 11, and perform the following operations:

[0035] When creating a virtual character for a somatosensory dance game, activate the preset voice assistant to interact with the user;

[0036] Collecting first voice data of the user communicating with the voice assistant, and sending a reading request to the user to read aloud the specified text when the first voice data meets a preset condition;

[0037] Collecting second voice data when the user reads the specified text;

[0038] Based on the second voice data, using a pre-trained machine learning model to analyze the user's biological characteristics and personality characteristics, and generate a timbre model that matches the user's timbre;

[0039] generating an image of the virtual character based on the biological characteristics and personality characteristics;

[0040] The voice of the virtual character is generated based on the voice model.

[0041] Based on the hardware architecture of the somatosensory dance game device for generating a virtual character based on user voice information, an embodiment of the somatosensory dance game method for generating a virtual character based on user voice information of the present invention is proposed. The somatosensory dance game method for generating a virtual character based on user voice information of the present invention aims to realize the intelligent and personalized creation of virtual characters.

[0042] Reference Figure 2 , Figure 2 This is an embodiment of a method for generating a virtual character's somatosensory dance game based on user voice information of the present invention. The method for generating a virtual character's somatosensory dance game based on user voice information comprises the following steps:

[0043] S10. When creating a virtual character for a somatosensory dance game, activate a preset voice assistant to interactively communicate with the user.

[0044] A virtual character is a digital image or role in a motion-sensing dance game. This image will represent the user's presence and performance in the game and interact with the game environment.

[0045] A voice assistant is an intelligent voice interaction system built into the game that can interact with the user in natural language. The assistant has the ability to understand and respond to user voice input and is designed to guide the user through the character creation process through voice.

[0046] Specifically, when the user selects the option to create a character, the system automatically starts the voice assistant. The voice assistant greets the user in a soft and friendly tone, such as "Hello, welcome to the character creation session. I will help you customize your unique character." Then, the voice assistant communicates with the user through guiding questions, such as "What kind of music do you like?", "What type of dance do you prefer?", etc. These questions can not only help users better understand the character creation process, but also provide the necessary interactive basis for voice data collection in subsequent steps, and help build user trust and eliminate users' vigilance.

[0047] S20: Collect first voice data of the user communicating with the voice assistant, and send a reading request for reading a specified text to the user when the first voice data meets a preset condition.

[0048] Among them, the first voice data refers to the voice signal generated by the user during the interaction with the voice assistant. These data include characteristics such as the user's speaking speed, pitch, timbre, intonation and emotional expression.

[0049] The preset condition refers to the system's judgment on whether the user's voice data in communication with the voice assistant meets specific requirements, such as whether the user's voice shows sufficient relaxation or positive emotions, and the tone feature analysis results meet the predetermined standards.

[0050] The designated text is a specific text content that the game system requires the user to read aloud. The content of the designated text usually includes various types of sentences, short articles or phrases, which are designed to be representative so as to capture the user's natural voice characteristics.

[0051] Optionally, specify the following features for the text: Phoneme diversity: The text content contains a variety of phonemes to ensure that the user's voice pronunciation characteristics can be fully captured. Emotional expression: The text can contain different emotional expressions, such as questions, statements, commands, etc., to capture the user's voice characteristics in different contexts. Sentence structure: The sentences in the text can be long or short, and contain different grammatical structures, which helps capture the user's speaking speed and intonation changes in different language expressions.

[0052] Specifically, when the user answers questions asked by the voice assistant through voice, the system automatically collects these voice inputs. The system can record the user's voice through the microphone device and process it in real time in the background. The processing includes filtering background noise, removing irrelevant sound interference, and extracting key voice features. When the first voice data meets the preset conditions, the system sends a request to the user to read the specified text aloud.

[0053] In some embodiments, the preset condition refers to judging that the user is in a relaxed state based on the voice data of the user in the preliminary communication, and the judgment of the relaxed state includes at least one of the following characteristics:

[0054] 1. The user's speaking speed is maintained within the preset speaking speed range.

[0055] The speech rate refers to the number of words or syllables pronounced by the user per minute during the communication process. The predetermined speech rate range is usually set to a numerical range that reflects the natural conversation of the user, such as 100 to 150 words per minute.

[0056] Specifically, when the user has an initial communication with the voice assistant, the system will calculate the user's speaking speed in real time. If the user's speaking speed is always maintained within a predetermined range (for example, 100 to 150 words per minute), the system will determine that the user may be in a relaxed state. The stability of this speaking speed range usually indicates that the user is communicating with the voice assistant in a natural and relaxed manner, rather than speeding up or slowing down due to tension or anxiety.

[0057] 2. The user's pitch is maintained within the preset pitch range.

[0058] Among them, pitch refers to the height of the fundamental frequency in the user's voice, which is usually related to the user's emotions and physiological state. The preset pitch range is a reasonable range set according to the user's normal speaking pitch, such as 85 to 180 Hz.

[0059] Specifically, during the initial communication, the system analyzes the fundamental frequency of the user's voice to determine whether the pitch is maintained within a preset range. If the pitch remains stable and there is no obvious drastic fluctuation, the system will assume that the user is likely to be in a relaxed state. This is because when the user is relaxed, the pitch tends to be stable, and the pitch will not rise or fall as when nervous.

[0060] 3. The user's tone change frequency is lower than the preset tone threshold.

[0061] The intonation change frequency refers to the frequency of changes in the user's intonation during communication, that is, the number of ups and downs of the voice. The preset intonation threshold is a standard that limits the number of intonation changes, for example, no more than 3 times per minute.

[0062] Specifically, when analyzing the user's voice data, the system calculates the frequency of voice changes. If the user's voice change frequency is lower than a preset threshold (such as no more than 3 times per minute), the system will determine that the user's voice is stable, suggesting that the user is in a relaxed state. A lower frequency of voice changes usually indicates that the user is emotionally stable and does not frequently adjust the voice due to anxiety or excitement.

[0063] 4. The user's voice rhythm is maintained within the preset voice range;

[0064] Among them, speech rhythm refers to the speed, pauses and continuity of the user's speech during communication. The preset speech range usually reflects the speech rhythm characteristics in natural conversation, including the uniformity and coherence of speech.

[0065] Specifically, the system determines whether the speech rhythm is within the preset range by analyzing indicators such as pause intervals and pronunciation continuity in the user's speech. If the user's speech rhythm is even and there is no obvious rush or interruption, the system will think that the user may be in a relaxed state. This kind of rhythm balance is usually a manifestation of relaxation and natural communication.

[0066] 5. Through the emotion recognition algorithm, the emotional characteristics in the user's voice meet the preset standards of calm or happy state.

[0067] Among them, the emotion recognition algorithm refers to an algorithm model that judges the user's emotional state by analyzing voice features (such as speaking speed, pitch, intonation, etc.). The preset standard is based on the score of calm or pleasant emotions given by the emotion recognition model, which is usually set to a score higher than 0.7.

[0068] Specifically, the system uses an emotion recognition algorithm to process and analyze the user's voice data. If the emotion recognition model scores the user's current emotional state higher than 0.7 and shows a calm or happy state, the system will consider the user to be in a relaxed state. This score usually reflects the user's overall emotional state, ensuring the naturalness and comfort of communication.

[0069] It can be understood that by establishing user trust and eliminating guarded psychology through initial communication with the voice assistant, the system can collect the user's voice data and analyze its tone characteristics in a natural and relaxed environment, which helps guide the user to read the specified text so that the user can express more real emotions and obtain more real and accurate voice characteristics. In addition, this interactive communication method makes the character creation process more natural and smooth, significantly improving the user's interactive experience and immersion.

[0070] S30: collecting second voice data when the user reads the specified text.

[0071] Specifically, the system can guide the user to read the specified text through a graphical interface or voice prompts. When the user reads the specified text, the system collects the voice data of the user reading the specified text through a configured microphone array or a single microphone device. Among them, if a microphone array is used, more accurate sound source positioning and higher noise resistance can be achieved. This is because the microphone array collects the user's voice signal through multiple channels at the same time, which helps to reduce the interference of background noise.

[0072] S40. Based on the second voice data, use a pre-trained machine learning model to analyze the user's biological characteristics and personality characteristics, and generate a timbre model that matches the user's timbre.

[0073] Among them, pre-trained machine learning models refer to models that have been trained on a large amount of labeled data and have certain predictive capabilities. These models learn patterns and features in a large amount of data during the training process, and can output prediction results for user biometrics (such as age, gender, voiceprint features, etc.) and personality traits (such as extroversion, introversion, etc.) after given input data (such as voice features).

[0074] Optionally, the machine learning model can be at least one of a deep neural network, a convolutional neural network, a recurrent neural network, a long short-term memory network, a support vector machine, and a random forest.

[0075] Specifically, the system inputs the collected second voice data into the pre-trained machine learning model, which automatically extracts and analyzes the user's biological characteristics, personality characteristics, and timbre model. The analysis process includes steps such as feature extraction, feature selection, and classification, and finally outputs the user's relevant feature information.

[0076] S50: Generate the appearance of the virtual character based on the biological characteristics and personality characteristics.

[0077] Specifically, the system selects the virtual character template that best matches the user's biological and personality characteristics analyzed by the machine learning model, and then adjusts the appearance attributes of the virtual character template according to the user's characteristics to make it closer to the user's actual characteristics. For example, the virtual character's gender, height, skin condition and other attributes are adjusted according to the user's biological characteristics (such as gender, age, height, etc.), and the virtual character's skin color, hairstyle, hair color, makeup and other attributes are adjusted according to the user's personality characteristics. For example, an extroverted user may get a lively and dynamic character, while an introverted user may get a calm and low-key character.

[0078] S60: Generate the voice of the virtual character based on the voice model.

[0079] Among them, the voice of the virtual character refers to the vocal characteristics of the virtual character generated based on the user's timbre model. This voice will be used for the virtual character's dialogue, performance and interaction in the game.

[0080] Specifically, the system first uses the text-to-speech (TTS) technology to generate the voice of the virtual character using the timbre model. For example, when the user enters a text or the character needs to speak according to the game plot, the system will call the timbre model to convert the text into a voice similar to the user's timbre.

[0081] It can be understood that the somatosensory dance game method of the technical solution of the present application, through the interactive communication between the voice assistant and the user, eliminates the user's vigilance and establishes initial trust with the user, and then collects the voice data of the user when reading the specified text, and combines it with the pre-trained machine learning model, it can analyze the user's biological characteristics, personality characteristics and timbre model based on the voice data, and then automatically generate a virtual character that is highly matched with the user based on these biological characteristics, personality characteristics and timbre models. This method not only eliminates the cumbersome process of manually inputting or selecting preset parameters in traditional somatosensory games and virtual reality technologies, but also significantly improves the personalization and realism of virtual characters. In addition, because the technical solution can automatically capture and analyze the user's voice characteristics, the character creation process becomes more natural and seamless, which enhances the user's interactive experience and immersion. Therefore, the somatosensory dance game method of the technical solution of the present application has the advantage of creating virtual characters intelligently and individually.

[0082] In some embodiments, the voice assistant communicates with the user in a preset tone and preset text, and the preset tone and the preset text are designed for the purpose of building user trust and / or eliminating the user's vigilance.

[0083] Among them, the preset tone refers to the tone characteristics used by the voice assistant when communicating with users, which is usually soft, friendly, and kind, aiming to make users feel relaxed and trusting.

[0084] Preset text refers to the text content used by the voice assistant during initial communication. It is designed to be simple, easy and stress-free, with the aim of establishing initial trust with the user and eliminating the user's vigilance.

[0085] Specifically, when the user starts the virtual character creation process, the voice assistant first greets the user in a friendly and relaxed tone, such as "Hello! Welcome to our game world. We will create a unique virtual character together." Then, the voice assistant asks simple, stress-free open-ended questions through designed preset texts, such as "Have you played any interesting games recently?" or "What type of music do you like?" The purpose of these questions is to make the user feel relaxed and naturally enter a communication state, thereby laying the foundation for subsequent voice data collection and analysis.

[0086] It can be understood that through the preset tone and language text, user trust can be quickly established and vigilance can be eliminated. The system can collect the user's voice data and analyze its tone characteristics in a natural and relaxed environment, thereby achieving accurate judgment of the user's emotional state.

[0087] In some embodiments, analyzing the user's biometrics based on the second voice data using a pre-trained machine learning model includes:

[0088] S110: extracting pitch features, formant features, and spectrum features from the second speech data.

[0089] Among them, the pitch features reflect the basic frequency of the user's voice, which is usually related to the vibration of the vocal cords. The system can use the autocorrelation method or the pitch tracking algorithm to extract the fundamental frequency (F0) from the speech signal to obtain the pitch features.

[0090] Furthermore, there are the following differences in pitch between men and women:

[0091] Physiological differences: Pitch is determined by the vibration frequency of the vocal cords (fundamental frequency, F0). Generally, men's vocal cords are longer and thicker, with a lower vibration frequency, so the fundamental frequency is usually lower than that of women. Women's vocal cords are shorter and thinner, with a higher vibration frequency, so the fundamental frequency is usually higher.

[0092] Gender differences: Statistically, the average pitch of a male is usually between 85Hz and 180Hz, while the average pitch of a female is usually between 165Hz and 255Hz.

[0093] Formant Features are energy-concentrated areas in the speech spectrum that reflect the shape and size of the user's vocal tract. The system can extract multiple formant frequencies (such as F1, F2, F3) as features through linear predictive coding (LPC) or formant tracking algorithms.

[0094] Furthermore, there are the following differences in formant peaks between men and women:

[0095] Physiological differences: Formants are determined by the shape and size of the vocal tract, reflecting the physical resonant properties of the vocal tract. Men typically have longer vocal tracts (including the pharynx and oral cavity), which results in lower-frequency formants. Women typically have shorter vocal tracts, resulting in higher-frequency formants.

[0096] Gender differences: Generally, the first formant (F1) and the second formant (F2) are lower in frequency in male speech and higher in frequency in female speech. For example, a male's F1 may be between 100Hz and 900Hz, and F2 between 900Hz and 2400Hz; whereas a female's F1 and F2 are higher.

[0097] Spectral features reflect the energy distribution of speech signals at different frequencies and can be extracted through short-time Fourier transform (STFT) or Mel-frequency cepstral coefficients (MFCC). Spectral features can capture the sound quality and timbre characteristics of user voices.

[0098] Furthermore, there are the following differences between men and women on the spectrum:

[0099] Physiological differences: Spectral characteristics reflect the energy distribution of speech signals in different frequency ranges. Due to the differences in the vocal tract structure between men and women, the energy of speech signals is distributed differently in the spectrum. Male speech usually has higher energy in the low-frequency part, while female speech has higher energy in the mid- and high-frequency parts.

[0100] Gender differences: By analyzing the spectral envelope or MFCC (Mel Frequency Cepstral Coefficients) of the speech signal, the difference in energy distribution between different genders can be observed. Generally, the spectrum of female speech tends to show higher energy in the higher frequency range, while the spectrum characteristics of male speech are more concentrated in the low frequency band.

[0101] S120, normalizing and reducing the dimension of the pitch features, formant features, and spectrum features.

[0102] The purpose of normalization is to scale the numerical ranges of different features to the same scale (such as between 0 and 1) to eliminate the scale differences between different features. This can prevent features with a larger range of eigenvalues ​​from having too much impact on the model.

[0103] Dimensionality reduction refers to using dimensionality reduction techniques to reduce feature dimensions. The purpose of this step is to remove redundant information while retaining the main components that best reflect the user's gender characteristics. The reduced feature vector will contain fewer dimensions, but the information will be more concentrated.

[0104] Specifically, after extracting various types of speech features, the system normalizes and reduces the dimensionality of these features to reduce the differences and redundancy between the features, thereby improving the prediction accuracy and efficiency of the model.

[0105] Optionally, features such as pitch features may be normalized using methods such as minimum-maximum normalization, Z-Score normalization, and maximum absolute value normalization.

[0106] Optionally, principal component analysis, linear discriminant analysis, independent component analysis, singular value decomposition and other methods may be used to reduce the dimension of features such as pitch features.

[0107] S130, merging the pitch features, formant features, and spectrum features after normalization and dimensionality reduction processing into a first feature vector.

[0108] Specifically, pitch features, formant features, and spectrum features can be fused into a feature vector using methods such as feature concatenation, weighted fusion, and feature averaging.

[0109] S140: Input the first feature vector into a pre-trained gender recognition model to predict the gender feature of the user.

[0110] Among them, the pre-trained gender recognition model can be a support vector machine (SVM), a deep neural network (DNN) or a convolutional neural network (CNN), etc. The model has been trained on a large-scale labeled dataset and can accurately map the input feature vector to the gender classification result.

[0111] Specifically, the system inputs the first feature vector into the pre-trained gender recognition model, and the model analyzes the input feature vector and outputs the user's gender prediction result. The model outputs the user's gender characteristics, usually as a binary classification (such as male / female), or it can output a probability distribution (such as the probability of male is 0.8 and the probability of female is 0.2).

[0112] It can be understood that through the feature extraction, processing and fusion process performed in the above embodiment, the system can accurately extract and identify gender features from the user's voice data. This process utilizes the multi-dimensional information of pitch, formant and spectral features. By inputting these features into the pre-trained machine learning model, the model can use the correlation and difference between these features to perform a comprehensive analysis, thereby accurately predicting the user's gender, thereby achieving high accuracy and robustness in gender recognition.

[0113] In some embodiments, analyzing the user's biometrics using a pre-trained machine learning model based on the second voice data further includes:

[0114] S210: Calculate the total number of syllables and the duration of voice activity in the voice data.

[0115] The total number of syllables refers to the number of syllables contained in the speech signal extracted from the speech data. A syllable is the basic unit of speech, usually consisting of a vowel or a combination of a vowel and a consonant. The active speech duration refers to the total duration of actual speech content in the speech signal when the user reads a specified text. The active speech duration does not include silence or background noise, and only calculates the actual speaking time.

[0116] Specifically, the system first uses the voice activity detection (VAD) algorithm to distinguish between voice activity and silence in the voice signal and extract the duration of voice activity. Next, the system calculates the total number of syllables in the voice data by detecting peaks or syllable changes in the voice. Finally, the system uses this information as a basis to further analyze the user's speaking speed characteristics. By analyzing the voice data, the system calculates the total number of syllables produced by the user when reading the specified text, as well as the duration of voice activity (that is, the actual length of time the user speaks when reading aloud).

[0117] S220: Calculate speech rate features according to the total number of syllables and the duration of speech activity.

[0118] Among them, the speech rate feature refers to the number of syllables uttered by the user per unit time, which is used to indicate the user's pronunciation speed. The speech rate feature is an important indicator reflecting the speed of speech, usually expressed in the form of "syllables / second" or "syllables / minute". The speech rate feature can reflect the user's speaking habits, which is usually related to age. For example, young people usually speak faster, while older people speak slower.

[0119] Specifically, the system obtains the speech rate feature by dividing the total number of syllables and the speech activity duration calculated in step S210. The calculation formula is: speech rate = total number of syllables / speech activity duration.

[0120] S230, extracting fundamental frequency features from the speech data, and calculating intonation range, intonation rising pattern, intonation falling pattern, intonation mean and intonation standard deviation according to the fundamental frequency features.

[0121] Among them, the fundamental frequency feature refers to the fundamental frequency (F0) in the user's voice signal, which reflects the frequency of vocal cord vibration and is usually used to describe the change in pitch. Intonation range refers to the range of changes in the fundamental frequency in the user's voice, representing the ups and downs of the user's voice. Intonation rise mode and intonation fall mode respectively represent the pattern or trend of the rise and fall of the fundamental frequency in the user's voice. Intonation mean refers to the average value of the fundamental frequency in the user's voice, reflecting the pitch level of the overall voice. Intonation standard deviation indicates the fluctuation range of the fundamental frequency, reflecting the stability and variation of the voice.

[0122] Specifically, the system first extracts fundamental frequency features from speech data through short-time Fourier transform (STFT) or other frequency analysis methods, and then calculates the intonation range, intonation rise and fall patterns, intonation mean and intonation standard deviation based on the fundamental frequency features.

[0123] S240, integrating the intonation range, intonation rising pattern, intonation falling pattern, intonation mean and intonation standard deviation to obtain intonation features.

[0124] Specifically, the system performs a fusion operation such as feature concatenation or weighted averaging on the intonation range, intonation rising pattern, intonation falling pattern, intonation mean, and intonation standard deviation calculated in step S230 to form a unified intonation feature vector, i.e., a second feature vector. The second feature vector can fully reflect the intonation characteristics of the user and serve as one of the inputs of the subsequent age recognition model.

[0125] S250: Merge the speech rate feature, intonation feature, pitch feature, formant feature and spectrum feature into a second feature vector.

[0126] Specifically, the system sequentially concatenates the speech rate features, intonation features, pitch features, formant features, and spectrum features generated in the previous steps into a unified feature vector, the second feature vector. This vector incorporates a variety of speech feature information and can provide comprehensive feature input for the machine learning model, thereby improving the model's prediction accuracy.

[0127] S260: Input the second feature vector into a pre-trained age recognition model to predict the user's age feature.

[0128] Specifically, the system inputs the second feature vector generated in step S250 into the pre-trained age recognition model, and the model outputs the user's age prediction result by analyzing and processing these features. The model can be a machine learning algorithm such as a deep neural network (DNN), a support vector machine (SVM), a random forest, etc., which can accurately predict the user's age characteristics after being trained with a large amount of data.

[0129] It is worth noting that the age recognition model can be a classification model or a regression model. If the age recognition model is a classification model, the age output result is the age group to which the user belongs (such as 10-20 years old, 20-30 years old, etc.). If the age recognition model is a regression model, the output result is the specific age value of the user.

[0130] It can be understood that by extracting and integrating multiple speech features such as speech rate, intonation, pitch, formant and spectrum, a comprehensive second feature vector is constructed and input into the pre-trained age recognition model. The system can more accurately predict the user's age characteristics, thereby improving the personalization and realism of the virtual character. In addition, through the fusion processing of multiple features, the system can fully capture the physiological and behavioral characteristics of the user's voice, thereby improving the robustness and adaptability of the age prediction model.

[0131] In some embodiments, after inputting the second feature vector into a pre-trained age recognition model, analyzing the user's biological characteristics based on the voice data using a pre-trained machine learning model also includes: selecting an age recognition model of the corresponding gender from a preset model library based on the recognition result of the user's gender characteristics.

[0132] Among them, the preset model library refers to a collection of multiple age recognition models pre-stored in the system. These models are specially trained and optimized according to different gender characteristics to adapt to the differences in voice characteristics between men and women.

[0133] Specifically, the system first selects the age recognition model corresponding to the user's gender (such as male or female) from the preset model library based on the user's gender characteristics (such as male or female). The purpose of this is to use models trained specifically for different genders to improve the accuracy of age prediction. Since men and women have significant differences in voice characteristics (such as pitch, intonation, speaking speed, etc.), using models of different genders can better capture these differences and provide more accurate age predictions.

[0134] In some embodiments, when collecting the second voice data of the user reading the specified text, a microphone array consisting of at least two microphones distributed at different heights and positions is used to collect the second voice data of the user reading the specified text.

[0135] Specifically, based on the configuration of the microphone array, the system is able to infer the user's physical height by analyzing the time difference between the sound reaching different microphones, and merge it with the preliminary predicted height to obtain a more accurate final height of the user.

[0136] Analyzing the user's biometric features using a pre-trained machine learning model based on the second voice data, further comprising:

[0137] S310: Input the second feature vector into a pre-trained region recognition model to predict the region where the user is located.

[0138] Among them, the pre-trained region recognition model refers to a machine learning model that has been trained on large-scale labeled data and can predict the user's region based on the input feature vector.

[0139] It is worth noting that due to the diversity of languages ​​and regional cultures, people in different regions will show different patterns in speech features. For example, people in some regions speak faster, and some dialects have specific resonance peak frequencies, which can be reflected by feature vectors. The second feature vector is a comprehensive feature vector generated by integrating multiple speech features (such as speaking speed, intonation, pitch, resonance peaks, spectrum, etc.). These features can reflect the user's pronunciation habits, intonation patterns, speaking speed, pitch changes and other speech features, which often vary significantly in different geographical regions.

[0140] Then, during the model training process, the model can be targeted to learn the mapping relationship between the second eigenvector and the region where the user is located. For example, certain eigenvector patterns may correspond to specific regions. In this way, by inputting new eigenvectors, the model can predict the region where the user may come from based on the learned patterns and rules. Back to the technical solution of the present application, by inputting the second eigenvector into the pre-trained region recognition model, the model predicts the user's geographic location or cultural area by analyzing these voice features, and then accurately infers the region where the user is located.

[0141] S320: Obtain corresponding population statistical heights from a preset height database according to the gender characteristics, age characteristics and the region where the user is located.

[0142] The preset height database refers to a database stored in the system that contains statistical data on the height of people of different genders, age groups and regions. The database is built based on extensive demographic data and can reflect the height distribution of different groups.

[0143] Specifically, the system extracts statistical height data corresponding to the user's characteristics from a preset height database based on the user's gender, age and location. These data usually include statistical information such as average height, median height and standard deviation.

[0144] S330: Predict the user's preliminary height based on the statistical height of the population and a pre-built statistical model.

[0145] Among them, population statistical height refers to the statistical data extracted from the preset height database, which reflects the height distribution of the population matching the user's characteristics.

[0146] Pre-built statistical models refer to mathematical models built based on large-scale data analysis and regression techniques, which are used to predict individual characteristics (such as height) based on statistical data.

[0147] Optionally, the statistical model can be any one of a linear regression model, a logistic regression model, a polynomial regression model, a Bayesian statistical model, a principal component analysis, a random forest, a K-means clustering, a naive Bayes classifier, a survival analysis model, and a hidden Markov model.

[0148] Specifically, the system uses a pre-built statistical model to calculate based on the demographic height data extracted from the database and the user's gender, age and regional characteristics, and outputs a preliminary height prediction value that conforms to statistical laws.

[0149] It can be understood that since the user's region is predicted based on voice features, and the population statistical height is obtained from a preset database in combination with gender, age and region information, the system uses a pre-built statistical model to make a preliminary height prediction, thereby effectively improving the accuracy and personalization of height prediction. In addition, by integrating multi-dimensional data, the system can provide height prediction results that are more in line with the actual situation based on the user's specific characteristics. At the same time, this solution uses a statistical model to predict height, which also reduces the error caused by individual differences, making height prediction more stable and reliable.

[0150] S340, calculating the user's head position according to the time difference between the user's voice reaching different microphones.

[0151] The time difference between the user's voice reaching different microphones refers to the time difference between the voice signal emitted by the user and the different microphones in the microphone array. Since the propagation speed of sound in the air is limited, the time it takes for the voice signal to reach different microphones from the user's head position will be different.

[0152] The user's head position refers to the position coordinates of the user's head in three-dimensional space inferred by calculating the time difference between the sound reaching different microphones.

[0153] Specifically, the system first measures the time it takes for the user's voice to reach each microphone, and then uses time difference method (TDOA) or triangulation to calculate the user's head position. By analyzing these time differences, the system is able to determine the relative position of the user's head in the microphone array (including height, horizontal position, etc.).

[0154] S350: Calculate the user's physical height according to the user's head position.

[0155] Among them, physical height refers to the actual height of the user from the top of the head to the ground, reflecting the actual height of the user.

[0156] Specifically, the system calculates the user's physical height based on the user's head position combined with the known positions of the microphones in the microphone array.

[0157] For example, if the height of the user's head relative to the ground is known, the system can use this as an estimate of physical height. Taking into account that the user may be in different standing or sitting positions, the system can apply some adjustments or calibrations to ensure the accuracy of physical height.

[0158] For example, assuming that the user's head position is represented as (xhead, yhead, zhead), and the lowest microphone position in the microphone array is (xmic, ymic, zmic), then the user's head height h head It is the vertical coordinate value of the user's head position, that is, zhead.

[0159] Then, considering the position of the user's head relative to the microphone array (i.e., the horizontal offset may affect the measurement accuracy), the projection of the user's head relative to the microphone array in the vertical direction can be calculated, which is expressed as:

[0160] h head =z head -z mic , where zmic is the height coordinate of the microphone and zhead is the height coordinate of the user's head.

[0161] Furthermore, the user's physical height Huser can be calculated by the following formula:

[0162] H user =h head +h mic , where h mic is the distance from the lowest microphone to the ground, h head is the distance from the user's head to the lowest microphone.

[0163] S360: Merge the physical height with the preliminary height to obtain the final height of the user.

[0164] Among them, the final height is the user's comprehensive height obtained by integrating the physical height and the preliminary height, which combines the results of physical measurement and statistical prediction.

[0165] Specifically, the system combines the physical height and the preliminary height, using methods such as weighted averaging and Bayesian inference. Through this fusion method, the system can comprehensively consider the advantages and disadvantages of the two methods to obtain a more accurate estimate of the user's height. For example, the physical height and the preliminary height can be added with a certain weight to balance the results based on physical measurement and statistical prediction, thereby obtaining the user's final height.

[0166] In some embodiments, fusing the physical height with the preliminary height to obtain a final height of the user includes:

[0167] S361. Calculate the posterior distribution of the user's height based on the predefined prior distribution and likelihood function, combined with the physical height and the preliminary height. The posterior distribution calculation formula is as follows:

[0168] P(H|H physical , H pred )∝P(H physical |H)·P(H pred |H)·P(H), where

[0169] Among them, P(H) is the prior distribution, which conforms to the normal distribution, μ prior is the average height based on population statistics, σ prior 2 is the variance of height,

[0170] Among them, P(H physical |H) is the physical height likelihood function, H physical is the physical height predicted by the microphone array, σ physical 2 is the variance of the physical prediction error,

[0171] Among them, P(H pred |H) is the initial height likelihood function, H pred is the initial height, σ pred 2 is the error variance of the statistical model;

[0172] S362. Use the mean of the posterior distribution as the final height. The calculation formula of the mean of the posterior distribution is as follows:

[0173]

[0174] S370: Use the final height as the height feature of the user.

[0175] Specifically, the system uses the final fused height as the user's height feature. This final height feature will be used to generate personalized virtual characters, making the generated virtual characters closer to the user's actual height, thereby improving the authenticity of the virtual experience.

[0176] It can be understood that since a microphone array is used when collecting voice data and the user's head position is calculated based on the time difference between the sound reaching different microphones, the system can infer the user's physical height based on the actual measurement results, thereby improving the accuracy of height prediction. By integrating the physical height with the preliminary predicted height, the system further eliminates the limitations of a single prediction method and ultimately obtains a more accurate user height feature. In addition, this method of combining physical measurement and statistical prediction not only improves the accuracy of the system, but also enhances the personalization and realism of the virtual character, thereby providing users with a more immersive interactive experience.

[0177] In some embodiments, the designated text includes designated lines in a preset plot content, and the preset plot content is displayed to the user when creating a virtual character of a somatosensory dance game, wherein displaying the preset plot content to the user includes: dividing the preset plot content into a plurality of plot segments, and displaying the plurality of plot segments in sequence according to a preset arrangement strategy, wherein the arrangement strategy is designed based on a preset personality test method to stimulate the user's expression in multiple personality dimensions through multi-dimensional situational changes.

[0178] Among these, preset plot content refers to a scene or story content pre-set according to the plot of the game, which is used to guide users into a specific situation to guide users to express specific emotions or attitudes.

[0179] The division of the preset plot content refers to splitting the complete plot content into multiple logically or situationally relatively independent plot segments, each of which has a specific situational or emotional expression goal.

[0180] Designated lines refer to specific text content that users need to read aloud in the preset plot content. These lines can guide the user's voice expression to facilitate subsequent character analysis.

[0181] The arrangement strategy refers to the pre-set order of clip display. The arrangement strategy is usually designed according to the requirements of the personality test method to ensure that the order of clips can effectively stimulate the user's expression in different personality dimensions. For example, the arrangement strategy can arrange the clips according to increasing emotional intensity, increasing decision complexity, emotional contrast arrangement, increasing situation complexity, increasing pressure, increasing cultural and social situation, language style changes, personal value conflicts, etc.

[0182] Personality testing methods refer to standardized methods for evaluating user personality traits, such as MBTI, the Big Five personality trait model, etc. The ranking strategy is designed based on these methods to fully stimulate the user's expression in multiple personality dimensions through multi-dimensional situational changes. For example, the expression of the extroversion / introversion dimension can be stimulated by different social situations, and the expression of the thinking / feeling dimension can be triggered by different decision-making situations.

[0183] Multi-dimensional situational changes refer to stimulating users' expressions in multiple personality dimensions by showing different types of situations (such as social, decision-making, and emotional). The situational changes designed in each clip can prompt users to express different aspects of their personality traits in different situations, thereby providing rich data support for personality analysis.

[0184] Personality dimensions are different aspects or dimensions used in psychology and personality testing to describe and classify individual personality traits. These dimensions usually reflect a person's tendencies in behavior, thinking, and emotions, and are an important basis for analyzing and understanding individual personality. The definition and classification of personality dimensions may vary in different personality testing methods, but they are usually described by a set of relative characteristics.

[0185] Here are some common personality test methods and explanations of their personality dimensions:

[0186] 1.MBTI (Myers-Briggs Type Indicator), MBTI is a personality type assessment tool developed based on Carl Jung's psychological type theory. It uses four binary personality dimensions to describe a person's personality: Extraversion (E) vs. Introversion (I); Sensing (S) vs. Intuition (N); Thinking (T) vs. Feeling (F); Judging (J) vs. Perceiving (P). 2. The Big Five Personality Traits, the Big Five Personality Traits is a personality assessment model widely used in psychology. It uses five independent dimensions to describe an individual's personality. 3. Jung's psychological type theory, Carl Jung's psychological type theory is the basis of MBTI, but it focuses more on the classification of internal psychological processes.

[0187] Specifically, when requesting the user to read a specified text, the system can show the user a preset plot content related to the game story, and present the clip to the user through visual or audio means. When showing the preset plot content to the user, the system divides the plot content into multiple plot clips, and displays these plot clips in sequence according to the arrangement strategy designed by the personality test method. In this way, through multi-dimensional situational changes, the system can guide users to make diverse expressions in different personality dimensions, and extract corresponding voice features from them, providing more comprehensive and accurate data support for personality analysis. This method can capture various aspects of the user's personality in a more detailed manner, thereby improving the personalized matching of virtual characters.

[0188] It is worth pointing out that when creating a virtual character, this plot content can be displayed to the user as an independent unit to collect voice data related to the personality in a targeted manner, or only the plot content can be displayed to the user, because the voice data generated when the user reads the lines is also sufficient to analyze the user's biological characteristics and timbre model.

[0189] It can be understood that through carefully designed plot content, the system can trigger specific emotional expressions of users when reading designated lines, such as excitement, calmness, tension, etc. This emotional expression can significantly affect the voice features, allowing the system to more accurately capture and analyze the user's personality type. At the same time, by showing plot content in different situations, the system can collect voice data of users in multiple situations. This helps to obtain richer voice features, thereby improving the accuracy and comprehensiveness of personality analysis.

[0190] In some embodiments, the multidimensional situational changes include at least one of the following situational changes: changes in emotional intensity, changes in situational complexity, changes in role type, changes in decision-making difficulty, changes in sociocultural background, changes in language style, changes in stressful situations, conflicts in personal values, and time sequence.

[0191] It can be understood that through these multi-dimensional situational changes, the system can stimulate the user's personality characteristics in many aspects, thereby more comprehensively analyzing the user's personality type. These changing dimensions not only enrich the user's voice feature data, but also provide a multi-level and multi-angle basis for personality analysis, making the generation of virtual characters more accurate and personalized.

[0192] In some embodiments, displaying preset plot content to the user further includes: selecting matching plot content from a preset plot library based on the gender characteristics of the user.

[0193] It is understandable that this selection of plot content based on gender characteristics can ensure that users are more natural and authentic when expressing themselves, thus providing more reliable data support for personality analysis. In addition, gender-matched plot content can also enhance the user's sense of immersion, making it easier for them to enter the role and situation, thereby more accurately showing their personality characteristics.

[0194] In some embodiments, based on the second voice data, analyzing the personality characteristics of the user using a pre-trained machine learning model includes:

[0195] S410: extracting corresponding speech features from the speech data of each plot segment according to a preset character dimension analysis requirement, wherein the character dimension analysis requirement of each plot segment is determined according to a preset arrangement strategy.

[0196] The voice data of the plot segment refers to the sound signal collected by the voice input device when the user reads the specified lines in the plot segment. Since the plot content is divided into multiple plot segments, the system can save the voice data collected when the user reads the specified lines in the plot segment in segments.

[0197] The need for personality dimension analysis is determined based on specific personality testing methods (such as MBTI, Big Five personality model, DISC). For example, if the MBTI personality type test method is used, the personality dimensions include extraversion (E), introversion (I), sensing (S), intuitive (N), etc. The system extracts voice features related to these dimensions, such as speaking speed, tone, volume, etc., based on the specific personality dimension analysis needs of each plot segment.

[0198] Specifically, the system first displays the designed plot segments in sequence according to the preset arrangement strategy. The display content of each segment (for example, if the target is the extroversion / introversion dimension, the segment content may involve social situations; if the target is the judgment / perception dimension, the segment content may involve decision-making scenarios) has been customized according to the personality dimension analysis needs. Therefore, the user's performance in each segment will directly reflect his or her characteristics in the corresponding personality dimension. After collecting the voice data, the system extracts the corresponding features from the voice data through the preset personality dimension analysis requirements to ensure the pertinence and effectiveness of the analysis. For example, when analyzing the extroversion / introversion dimension, the system may focus on speech speed, volume, and changes in intonation; when analyzing the thinking / emotional dimension, the system may focus on the stability of intonation and the richness of emotional expression.

[0199] S420. According to different personality dimensions, select an applicable machine learning model to analyze the extracted voice features to generate the user's personality characteristics under each personality dimension.

[0200] Among them, different personality testing methods (such as MBTI, Big Five personality model, DISC) require different machine learning models for analysis. For example, the MBTI personality type test requires the use of a specific model to analyze the user's extroversion and introversion, while the Big Five personality model requires the use of other models to evaluate the user's openness, conscientiousness and other dimensions.

[0201] Specifically, after extracting the corresponding voice features, the system selects the machine learning model that best fits the personality dimension for analysis. For example, the system can use a support vector machine (SVM) model to analyze speech rate and volume data to generate the user's personality features in the extroversion / introversion dimension. At the same time, for the thinking / emotional dimension, the system may select a neural network model to process intonation changes and emotion recognition features. Through the analysis of these models, the system generates the user's specific performance characteristics in each personality dimension.

[0202] S430: Based on the personality characteristics of each personality dimension, generate the overall personality type of the user as the personality characteristics of the user.

[0203] Among them, the personality characteristics of each personality dimension refer to the performance results of the user in different personality dimensions obtained through the machine learning model analysis in step S420.

[0204] Personality type refers to the personality classification result of a user under a specific personality test method (MBTI personality type, Big Five personality model, DISC personality type). The overall personality type is a comprehensive personality classification result that integrates the user's performance in various personality dimensions, such as a certain type in MBTI (such as INTJ, ESFP, etc.).

[0205] Specifically, after obtaining the user's personality characteristics under each personality dimension, the system generates the user's overall personality type by comprehensively analyzing these characteristics. For example, the system may integrate the personality characteristics of each dimension through methods such as splicing, weighted averaging, and rule-based decision-making to obtain the user's MBTI type or personality type results under other personality testing methods.

[0206] It can be understood that by extracting corresponding voice features from the voice data of each plot segment and combining it with the preset segment arrangement strategy, the system can accurately collect and analyze data for different personality dimensions, thereby improving the accuracy and comprehensiveness of personality analysis; by selecting an applicable machine learning model to analyze the extracted voice features, the system can generate the user's personality characteristics under each personality dimension, ensuring the scientificity and reliability of the personality analysis results; by integrating the personality characteristics of each personality dimension, the system generates the user's overall personality type, thereby providing users with a personalized virtual character experience that fits their personality.

[0207] In some embodiments, according to different personality dimensions, an applicable machine learning model is selected to analyze the extracted voice features to generate the user's personality characteristics under each personality dimension, including:

[0208] S510: When analyzing the personality dimension of the current plot segment, extract voice features related to the current personality dimension from voice data of other plot segments.

[0209] Among them, the current plot segment refers to the specific plot segment currently being analyzed, which is designed to stimulate the user's performance on a specific personality dimension.

[0210] Other plot segments refer to other plot segments that are different from the current plot segment and are preset in the system. These segments may involve the user's performance in other situations.

[0211] The voice features related to personality dimensions refer to the voice features that can reflect the user's performance in a specific personality dimension, such as speaking speed, volume, and intonation changes.

[0212] Specifically, when the system analyzes the personality dimension in the current plot segment, it not only considers the voice features extracted from the current segment, but also checks the voice data in other plot segments to find those features related to the current personality dimension analysis needs. For example, if the current plot segment mainly involves the user's extroversion / introversion dimension, the system may extract relevant features such as speech rate and volume from other segments to supplement the analysis data in the current segment. This cross-segment feature extraction helps to build a more complete and accurate user personality trait portrait.

[0213] S520: weighted integration of the speech features extracted from the current plot segment and the related speech features extracted from other plot segments.

[0214] Among them, weighted integration means assigning different weights to speech features according to different feature sources. Optionally, a higher weight can be assigned to the speech features of the current segment, while lower weights can be assigned to the speech features of other segments, and these weighted features can be integrated into a comprehensive feature vector.

[0215] Specifically, after obtaining the voice features of the current plot segment, the system regards these features as the main data source and assigns them a higher weight, because the current segment is designed for this personality dimension and the features have the highest relevance and accuracy. At the same time, the system will also extract auxiliary voice features related to the current personality dimension from other plot segments, but these features have lower weights to ensure that they do not excessively affect the analysis results during the integration process. Finally, the system integrates the weighted features to generate a comprehensive feature vector for more accurate personality analysis.

[0216] It can be understood that by giving higher weight to the speech features of the current plot segment, the system can ensure that the personality analysis mainly relies on the feature data with clear design goals, thereby improving the accuracy of the analysis; by extracting and weighting less relevant features from other segments, the system can supplement and improve the personality analysis data, thereby improving the comprehensiveness and consistency of the analysis. Ultimately, this weighted integration strategy makes personality analysis more accurate and the generated personality features more reliable.

[0217] S530: Input the weighted integrated voice features into the corresponding machine learning model to generate the user's personality characteristics under each personality dimension.

[0218] Specifically, the system inputs the weighted integrated speech features into a pre-trained machine learning model. This model is specially optimized for the personality temperature analysis needs of the current plot segment (such as extroversion / introversion or thinking / emotional dimensions). By analyzing the input comprehensive feature vector, the user's personality characteristics under this personality dimension are generated. This result can help further understand and define the user's overall personality type and be used for subsequent virtual character generation and personalization.

[0219] It can be understood that when analyzing the personality dimension of the current plot segment, the system not only extracts the voice features of the current segment, but also extracts voice features related to the current personality dimension from other plot segments. Therefore, the system can more comprehensively capture the user's personality performance in different situations, thereby providing richer data support for the analysis of the current personality dimension. In addition, by giving a higher weight to the voice features extracted from the current plot segment and a lower weight to the relevant voice features extracted from other segments, the system can effectively integrate the information in different segments, ensure that personality analysis mainly relies on targeted feature data, and thus improve the accuracy and consistency of personality analysis.

[0220] In some embodiments, based on the second voice data, using a pre-trained machine learning model to analyze and generate a timbre model that matches the user's timbre includes:

[0221] S610: Preprocess the second voice data, wherein the preprocessing includes cleaning background noise and removing non-voice components, and extracting key timbre features of the user's voice.

[0222] Among them, background noise cleaning refers to the use of audio processing technology to remove environmental noise and interference in voice data that may affect the analysis, ensuring the purity of the voice signal.

[0223] Non-speech component removal refers to removing irrelevant parts of the speech data, such as silent segments, breathing sounds, coughing sounds, etc., to ensure that only the user's valid speech information is included in the analysis.

[0224] Key timbre feature extraction refers to extracting features that are crucial to timbre modeling from preprocessed speech data, such as pitch, timbre, prosody, formants, etc.

[0225] Specifically, the system can use filters (such as low-pass filters, high-pass filters) or other noise suppression algorithms to remove background sounds in the environment. Then, the system identifies and removes non-speech components in the speech signal, such as silence, breathing sounds, etc. Next, the system uses audio analysis tools to extract key timbre features. For example, the system can extract pitch through autocorrelation methods or Fourier transforms, and extract timbre features through linear predictive coding (LPC) or Mel-frequency cepstral coefficients (MFCCs).

[0226] S620: Input the preprocessed speech features into a pre-trained speech machine learning model.

[0227] Among them, the pre-trained speech machine learning model can recognize and process speech features and generate corresponding timbre representations. The model can be a deep neural network (such as a convolutional neural network CNN, a long short-term memory network LSTM) or other machine learning algorithms suitable for processing speech data.

[0228] Specifically, the system uses the timbre features extracted from S610 as input data to the pre-trained machine learning model. The model further processes and analyzes these features through its hierarchical structure to identify the user's unique timbre characteristics. For example, the model may analyze the pitch change pattern, the brightness of the timbre, the rhythm of the intonation, etc., to generate a high-dimensional feature vector or parameter set that matches the user's voice characteristics.

[0229] S630: Generate a timbre model that matches the user's timbre based on the user's voice features and the output of the pre-trained model.

[0230] Specifically, the system matches and optimizes the output of the pre-trained model (such as high-dimensional feature vectors) with the user's original voice features to generate a timbre model. The timbre model may contain multiple parameters, such as pitch range, timbre adjustment coefficient, resonance peak frequency, etc., which are used to describe the key characteristics of the user's voice. The generated timbre model can not only reflect the user's natural voice characteristics, but can also be used in the subsequent text-to-speech (TTS) process to ensure that the voice of the virtual character is highly consistent with the user's real timbre. After the model is generated, the system can verify and fine-tune it by comparing the similarity between the original voice data and the generated voice to improve the accuracy of the model and the user's matching degree.

[0231] It can be understood that by preprocessing the collected user voice data, extracting key timbre features, and inputting them into the pre-trained machine learning model for analysis and generation, the system can generate a timbre model that highly matches the user's timbre. This timbre model can accurately reflect the user's voice characteristics, thereby providing a basis for the generation of the virtual character's voice, ensuring that the virtual character's voice performance is natural and realistic, and significantly enhancing the user's immersion and personalized experience. In addition, the generated timbre model further improves the similarity with the user's voice through matching and optimization, enhancing the emotional connection between the virtual character and the user.

[0232] In some embodiments, generating an image of a virtual character based on the biological characteristics and personality characteristics includes:

[0233] S710: According to the biological characteristics and personality characteristics of the user, select a virtual character template corresponding to the biological characteristics and personality type from a preset template library.

[0234] Among them, the preset template library contains a variety of virtual character templates, each of which corresponds to specific biological characteristics and personality types. For example, the template library may have character templates for different heights, body shapes, and genders, as well as appearance designs suitable for different personality types (such as lively, stable, mysterious, etc.). The system analyzes the user's biological characteristics (such as height, gender, skin quality, etc.) and personality characteristics (such as MBTI type, Big Five personality score, etc.), and selects the virtual character template that best matches these characteristics from the template library.

[0235] Specifically, the system first selects templates that match the user's biological characteristics (such as age, height, and gender), and then further narrows the template selection range based on the user's personality characteristics (such as extroversion and emotional stability), and finally determines the most suitable virtual character template. This process ensures that the basic image of the virtual character matches the user's real characteristics.

[0236] S720: Map the user's biological characteristics to the physical attributes of the virtual character, and map the personality characteristics to the appearance attributes of the virtual character.

[0237] The mapping process involves concretizing the user’s biological and personality traits into the physical and appearance attributes of the avatar. For example, the user’s height can be directly mapped to the avatar’s height attribute, while the user’s personality traits such as extroversion may be mapped to a brighter and bolder hairstyle or hair color.

[0238] Specifically, the system adjusts the physical attributes of the virtual character based on the user's height, gender, face shape and other biological characteristics, so that the character's height, body shape, facial contours, etc. match the user's actual characteristics. At the same time, the system adjusts the appearance details of the character based on the user's personality characteristics, such as choosing bright hair colors and lively hairstyles for extroverted users, or designing softer, low-key makeup and styling for introverted users. This mapping process ensures that the virtual character is not only similar to the user in physical appearance, but also reflects the user's personality characteristics.

[0239] S730: Generate a virtual character image that matches the user according to the mapping result.

[0240] Specifically, the system integrates all mapped attributes to generate the final virtual character image. This character image will have the user's biological characteristics such as height, body shape, skin texture, etc., while appearance details such as hairstyle, hair color, clothing style, etc. will also reflect the user's personality characteristics. The generated virtual character can be used in somatosensory dance games to interact and operate games on behalf of the user, allowing the user to feel a high degree of personalization and realism in the virtual environment.

[0241] It can be understood that by generating a virtual character through dual mapping of biological characteristics and personality characteristics, the character can highly match the user's real characteristics in both physical appearance and personality performance. This method not only improves the personalization and realism of the virtual character, but also enhances the user's immersive experience in the game.

[0242] In some embodiments, generating the voice of the virtual character based on the voice model includes:

[0243] S810. Combining the text or game script input by the user with the timbre model through text-to-speech conversion technology to generate a virtual character voice that matches the user's timbre characteristics.

[0244] Among them, the text input by the user or the game script refers to the text content directly input by the user or preset in the game scenario, and the system needs to convert these contents into the voice of the virtual character.

[0245] Specifically, when a user enters text in the game (such as dialogue content) or when the game script requires a character to speak, the system will call TTS technology. The TTS system will use the previously generated timbre model to convert the text content into voice output consistent with the user's voice characteristics. The parameters in the timbre model (such as pitch, intonation, and timbre) will directly affect the characteristics of the generated voice, thereby ensuring that the voice of the virtual character is highly similar to the user's real voice and providing a personalized voice experience.

[0246] S820: Dynamically adjust and generate the emotional expression of the virtual character's voice according to the current situation and game scene of the virtual character.

[0247] Among these, the virtual character's context and game scene refer to the character's current game context (such as dancing, stage performances, interaction with other characters, etc.) and specific game scenes (such as stage, concert, rehearsal room, etc.). These factors determine the character's emotional expression method.

[0248] Dynamically adjusting the emotional expression of generated speech means that the system adjusts the intonation, pitch, speaking speed and other parameters of the speech in real time according to situational requirements to reflect the emotional state of the character in different dance scenes (such as excitement, joy, relaxation, etc.).

[0249] Specifically, the system takes into account the character's current dance context when generating the virtual character's voice. For example, in a vibrant dance performance scene, the system may adjust the voice's pitch and speed to make the voice express excitement and enthusiasm; while in a relaxed rehearsal scene, the voice will be calmer and more soothing. By adjusting the emotional expression of the voice in real time, the system ensures that the virtual character's voice performance is consistent with the dance scene and the user's emotional experience, thereby enhancing the game's immersion and the character's emotional depth.

[0250] It can be understood that by combining the user's timbre model with text-to-speech conversion technology, dynamically generating a virtual character voice that is consistent with the user's voice characteristics, and generating a matching virtual character image through timbre feature mapping, the system can provide the user with a highly personalized virtual character that closely matches the user's voice characteristics. Therefore, the user's immersion and emotional connection in the game are significantly enhanced, and the visual and auditory performance of the virtual character is also highly coordinated, improving the overall quality of the gaming experience.

[0251] In addition, the embodiment of the present invention also proposes a computer-readable storage medium, which can be any one or any combination of a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, etc. The computer-readable storage medium includes a somatosensory dance game program 10 for generating a virtual character based on user voice information. The specific implementation of the computer-readable storage medium of the present invention is roughly the same as the specific implementation of the somatosensory dance game method for generating a virtual character based on user voice information and the server 1, which will not be repeated here.

[0252] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0253] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0254] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0255] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0256] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0257] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for generating a virtual character based on user voice information, characterized in that: include: When creating a virtual character for a somatosensory dance game, activate the preset voice assistant to interact with the user; Collecting first voice data of the user communicating with the voice assistant, and sending a reading request to the user to read aloud the specified text when the first voice data meets a preset condition; Collecting second voice data when the user reads the specified text; Based on the second voice data, using a pre-trained machine learning model to analyze the user's biological characteristics and personality characteristics, and generate a timbre model that matches the user's timbre; generating an image of the virtual character based on the biological characteristics and personality characteristics; The voice of the virtual character is generated based on the voice model.

2. The method for generating a virtual character based on user voice information according to claim 1, characterized in that: The voice assistant communicates with the user in a preset tone and preset text, and the preset tone and the preset text are designed for the purpose of building user trust and / or eliminating the user's vigilance; The preset condition refers to judging that the user is in a relaxed state based on the first voice data, and the judgment of the relaxed state includes at least one of the following characteristics: The user's speaking speed is maintained within the predetermined speaking speed range; The user's pitch is maintained within the preset pitch range; The user's tone of voice changes at a frequency lower than a preset tone threshold; The user's voice rhythm is maintained within the preset voice range; The emotion recognition algorithm detects that the emotional characteristics in the user's voice meet the preset standards for a calm or pleasant state.

3. The method for generating a virtual character based on user voice information according to claim 1, characterized in that: Analyzing the user's biometric features using a pre-trained machine learning model based on the second voice data includes: Extracting pitch features, formant features and spectrum features from the second speech data; Normalizing and reducing the dimension of the pitch features, formant features and spectrum features; The pitch features, formant features and spectrum features after normalization and dimension reduction are fused into the first feature vector; The first feature vector is input into a pre-trained gender recognition model to predict the gender feature of the user.

4. The method for generating a virtual character based on user voice information according to claim 3, characterized in that: Analyzing the user's biometric features using a pre-trained machine learning model based on the second voice data includes: Calculating the total number of syllables and the duration of voice activity in the second voice data; Calculating speech rate features according to the total number of syllables and the duration of speech activity; Extracting fundamental frequency features from the speech data, and calculating intonation range, intonation rising mode, intonation falling mode, intonation mean and intonation standard deviation according to the fundamental frequency features; The intonation range, the intonation rising pattern, the intonation falling pattern, the intonation mean and the intonation standard deviation are integrated to obtain the intonation feature; The speech rate feature, intonation feature, pitch feature, formant feature and spectrum feature are integrated into a second feature vector; The second feature vector is input into a pre-trained age recognition model to predict the age feature of the user.

5. The method for generating a virtual character based on user voice information according to claim 4, characterized in that: When collecting the second voice data of the user reading the specified text, using a microphone array consisting of at least two microphones distributed at different heights and positions to collect the second voice data when the user reads the specified text; Analyzing the user's biometric features using a pre-trained machine learning model based on the second voice data, further comprising: Inputting the second feature vector into a pre-trained region recognition model to predict the region where the user is located; According to the gender characteristics, age characteristics and the region where the user is located, the corresponding statistical height of the population is obtained from a preset height database; Predicting the user's preliminary height based on the demographic height of the population and a pre-built statistical model; Calculate the user's head position based on the time difference between the user's voice reaching different microphones; Calculate the user's physical height based on the user's head position; Merging the physical height with the preliminary height to obtain a final height of the user; The final height is used as the height feature of the user.

6. The method for generating a virtual character based on user voice information according to claim 1, characterized in that: The specified text includes specified lines in a preset plot content, and the preset plot content is displayed to the user when creating a virtual character of a somatosensory dance game. When the preset plot content is displayed to the user, the preset plot content is divided into a plurality of plot segments, and the plurality of plot segments are displayed in sequence according to a preset arrangement strategy, wherein the arrangement strategy is designed based on a preset personality test method to stimulate the user's expression in a plurality of personality dimensions through multi-dimensional situation changes; Based on the second voice data, analyzing the personality characteristics of the user using a pre-trained machine learning model includes: Extracting corresponding speech features from the speech data of each plot segment according to a preset character dimension analysis requirement, wherein the character dimension analysis requirement of each plot segment is determined according to a preset arrangement strategy; According to different personality dimensions, select an appropriate machine learning model to analyze the extracted voice features to generate the user's personality characteristics under each personality dimension; Based on the personality characteristics of each personality dimension, the overall personality type of the user is generated as the personality characteristics of the user.

7. The method for generating a virtual character based on user voice information according to claim 6, characterized in that: According to different personality dimensions, select an appropriate machine learning model to analyze the extracted voice features to generate the user's personality characteristics under each personality dimension, including: When analyzing the personality dimension of the current plot segment, voice features related to the current personality dimension are extracted from the voice data of other plot segments; Perform weighted integration on the speech features extracted from the current plot segment and the related speech features extracted from other plot segments; The weighted integrated voice features are input into the corresponding machine learning model to generate the user's personality characteristics under each personality dimension.

8. The method for generating a virtual character based on user voice information according to any one of claims 1 to 7, characterized in that: include: Generating the voice of the virtual character based on the voice model includes: Through text-to-speech conversion technology, the text or game script input by the user is combined with the timbre model to generate a virtual character voice that matches the user's timbre characteristics; Dynamically adjust the emotional expression of the virtual character's voice based on the virtual character's current situation and game scene. Generating a virtual character matching the user according to the biological characteristics and personality characteristics, including: According to the biological characteristics and personality characteristics of the user, a virtual character template corresponding to the biological characteristics and personality type is selected from a preset template library; Mapping the user's biometric characteristics to the virtual character's physical attributes, and mapping the personality characteristics to the virtual character's appearance attributes; Generate a virtual character that matches the user based on the mapping results.

9. A somatosensory dance game device for generating a virtual character based on user voice information, characterized in that: The invention comprises a memory, a processor and a somatosensory dance game program for generating a virtual character based on user voice information, which is stored in the memory and can be run on the processor. When the processor executes the somatosensory dance game program for generating a virtual character based on user voice information, a somatosensory dance game method for generating a virtual character based on user voice information as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a somatosensory dance game program for generating a virtual character based on user voice information. When the somatosensory dance game program for generating a virtual character based on user voice information is executed by a processor, a somatosensory dance game method for generating a virtual character based on user voice information as described in any one of claims 1-8 is implemented.

Images