Virtual human driving method and device based on multivariate perception, equipment and storage medium

By extracting and encoding user voice and face images, and generating rendered RGB images, the problem of insufficient perception and understanding of virtual people is solved, and the interaction and emotional expression ability of virtual people in virtual social environments is improved.

CN119091008AInactive Publication Date: 2024-12-06福建联政信息技术有限公司

Patent Information

Application Number
CN202411081931.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-12-06
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing virtual human-driven technologies focus on a single perception and interaction mode, resulting in limited ability of virtual humans to perceive and understand user behavior, emotions and environment in virtual social platforms and virtual reality social environments.

Method used

The artificial intelligence technology based on deep learning is used to extract and encode the user's voice and face images, and the rendered RGB images are generated through the processing of speech feature vectors and facial feature vectors, improving the multi-dimensional perception and understanding ability of virtual people.

Benefits of technology

By leveraging the diverse information of users, virtual people can more accurately identify and respond to users’ emotional needs, enhance interactive experience, and achieve a more realistic and personalized user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091008A_ABST
    Figure CN119091008A_ABST
Patent Text Reader

Abstract

The invention relates to the field of virtual human driving, and particularly discloses a virtual human driving method and device based on multi-element perception, equipment and a storage medium, which are used for performing feature extraction and coding on voice and face images of a user by using an artificial intelligence technology based on the deep learning field to obtain an RGB image rendered by a virtual human. In this way, by means of the multi-information of the user, the ability of the virtual human to perceive and understand the behavior, emotion and environment of the user is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of virtual human driving, and more specifically to a virtual human driving method and apparatus, device, and storage medium based on multi-sensory perception. Background Art

[0002] Virtual human refers to a virtual entity created through computer technology and artificial intelligence technology, which has human appearance, language, behavior and interaction capabilities. Virtual humans can simulate human appearance and behavior in computer programs, interact with users, and show certain intelligence and emotions. At present, common virtual human drivers focus on a single perception and interaction method, such as expression or voice, which results in virtual humans' ability to perceive and understand user behavior, emotions and environment in virtual social platforms, virtual meetings or virtual reality social environments.

[0003] Therefore, an optimized virtual human driving method is expected. Summary of the invention

[0004] In order to solve the above technical problems, this application is proposed. The embodiments of this application provide a virtual human driving method and device based on multi-perception, equipment, and storage medium, which uses artificial intelligence technology based on the field of deep learning to extract and encode the user's voice and face image to obtain an RGB image rendered by the virtual human. In this way, by utilizing the user's multi-dimensional information, the virtual human's ability to perceive and understand the user's behavior, emotions, and environment is improved.

[0005] According to one aspect of the present application, a virtual human driving method based on multi-sensory perception is provided, which includes:

[0006] Obtain user voice and user face image;

[0007] Processing the user's speech to obtain a speech feature vector;

[0008] Processing the user's facial image to obtain a facial feature vector;

[0009] The speech feature vector and the facial feature vector are processed to obtain a rendered RGB image.

[0010] Optionally, processing the user voice to obtain a voice feature vector includes:

[0011] Performing speech recognition on the user speech to obtain a text of the user speech recognition;

[0012] The text of the user speech recognition is segmented and passed through a converter-based context encoder including an embedding layer to obtain a speech feature vector.

[0013] Optionally, the text of the user speech recognition is segmented and then passed through a converter-based context encoder including an embedding layer to obtain a speech feature vector, including:

[0014] Performing word segmentation processing on the text of the user speech recognition to obtain a plurality of speech words;

[0015] Passing the plurality of spoken words through an embedding layer to convert each of the plurality of spoken words into a speech word embedding vector to obtain a sequence of speech word embedding vectors, wherein the embedding layer uses a learnable embedding matrix to embed the each speech word;

[0016] Inputting the sequence of speech word embedding vectors into the converter-based context encoder to obtain the plurality of speech semantic feature vectors;

[0017] The multiple speech semantic feature vectors are cascaded to obtain the speech feature vector.

[0018] According to another aspect of the present application, a virtual human driving device based on multi-sensory perception is also provided, which includes:

[0019] A user information acquisition module, used to acquire user voice and user face image;

[0020] A user voice encoding module, used for processing the user voice to obtain a voice feature vector;

[0021] A user facial feature extraction module, used for processing the user's facial image to obtain a facial feature vector;

[0022] The rendered image generation module is used to process the speech feature vector and the facial feature vector to obtain a rendered RGB image.

[0023] According to another aspect of the present application, a virtual human driving device based on multi-sensing is also provided, which runs on a baseboard management controller BMC, including:

[0024] Memory for storing computer programs;

[0025] A processor is used to implement the steps of the virtual human driving method based on multi-sensory perception when executing the computer program.

[0026] According to another aspect of the present application, a storage medium containing computer executable instructions is also provided, characterized in that the computer executable instructions implement the virtual human driving method based on multi-perception when executed by a computer processor.

[0027] Compared with the prior art, the virtual human driving method and device, equipment, and storage medium provided by the present application based on multi-dimensional perception use artificial intelligence technology based on the field of deep learning to extract and encode the user's voice and face image to obtain an RGB image rendered by the virtual human. In this way, by utilizing the user's multi-dimensional information, the virtual human's ability to perceive and understand the user's behavior, emotions, and environment is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0029] Figure 1 The figure is a flowchart of a virtual human driving method based on multi-perception according to an embodiment of the present application.

[0030] Figure 2 The present invention is a flowchart of obtaining a speech feature vector by performing word segmentation processing on the text of the user speech recognition in the virtual human driving method based on multi-perception according to an embodiment of the present application and then passing it through a converter-based context encoder including an embedding layer.

[0031] Figure 3 The present invention is a flowchart of inputting the sequence of speech word embedding vectors into the converter-based context encoder to obtain the multiple speech semantic feature vectors in the virtual human driving method based on multi-perception according to an embodiment of the present application.

[0032] Figure 4 4 is a block diagram of a virtual human driving device based on multi-sensory perception according to an embodiment of the present application.

[0033] Figure 5 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0035] Figure 1 FIG. 1 is a flow chart of a virtual human driving method based on multi-sensory perception according to an embodiment of the present application. Figure 1 As shown, the virtual human driving method based on multi-perception according to the embodiment of the present application includes: S110, acquiring user voice and user face image; S120, processing the user voice to obtain a voice feature vector; S130, processing the user face image to obtain a facial feature vector; and S140, processing the voice feature vector and the facial feature vector to obtain a rendered RGB image.

[0036] In step S110, the user's voice and the user's face image are obtained. As mentioned in the above background technology, common virtual human driving focuses on a single perception and interaction mode, such as expression or voice, which results in the virtual human's ability to perceive and understand user behavior, emotion and environment being limited in a virtual social platform, virtual meeting or virtual reality social environment. Therefore, an optimized virtual human driving method is expected.

[0037] In response to the above technical problems, a virtual human driving method based on multi-perception is proposed, which uses artificial intelligence technology based on deep learning to extract and encode the user's voice and face image features to obtain the RGB image rendered by the virtual human. In this way, by utilizing the user's multi-dimensional information, the virtual human's ability to perceive and understand the user's behavior, emotions and environment is improved.

[0038] At present, deep learning and neural networks have been widely used in computer vision, natural language processing, speech signal processing and other fields. In addition, deep learning and neural networks have also shown a level close to or even beyond that of humans in image classification, object detection, semantic segmentation, text translation and other fields.

[0039] In recent years, the development of deep learning and neural networks has provided new solutions and plans for virtual human driving methods based on multi-perception.

[0040] Specifically, first, the user's voice and the user's face image are obtained. In order to achieve multi-dimensional perception and emotional understanding of the user, the user's voice and face image are obtained, thereby improving the interactive experience between the virtual person and the user. This multi-dimensional perception method can give the virtual person stronger interactive ability and emotional expression ability, thereby achieving a more realistic and personalized user experience. Voice and facial expressions are important ways to express emotions. By obtaining the user's voice and facial image, the virtual person can more accurately identify the user's emotional state, thereby better responding to the user's emotional needs. Voice and facial information can provide clues about the user's situation. For example, the user's voice tone and facial expression can reflect their current emotional state, context, and interactive intentions, which helps the virtual person better understand the user's needs and intentions.

[0041] In step S120, the user voice is processed to obtain a voice feature vector. Considering that the voice signal is a high-dimensional and complex signal, there are some challenges and limitations in directly using the original voice signal for analysis and processing. The user voice can be processed to obtain a voice feature vector. By extracting the voice feature vector, the characteristics of the voice signal can be more effectively characterized, thereby achieving more accurate voice recognition, emotion recognition, and speech synthesis tasks. Considering that the voice signal is usually high-dimensional and contains a lot of information, but much of the information may not be very important for a specific task. By extracting the voice feature vector, the dimensionality reduction processing of the voice signal can be achieved, and the key information therein can be extracted, so as to better represent the characteristics of the voice. At the same time, the extraction of the voice feature vector can help enhance the robustness to noise and other interferences. The original voice signal may be affected by factors such as environmental noise and microphone quality, and these interferences can be better dealt with by feature extraction. The voice feature vector is usually more compact than the original voice signal, which is convenient for storage and processing, and can reduce the calculation and storage overhead of the system.

[0042] Specifically, in the above-mentioned virtual human driving method based on multi-perception, the user voice is processed to obtain a voice feature vector, including: performing voice recognition on the user voice to obtain a text of user voice recognition; and performing word segmentation on the text of user voice recognition and then passing it through a converter-based context encoder including an embedding layer to obtain a voice feature vector.

[0043] Specifically, in order to make voice information easier to process, understand and apply, the user's voice is subjected to voice recognition to obtain the text of the user's voice recognition. Converting voice to text makes it easier for the computer to understand and process the user's voice instructions and conversations. Once the voice is converted to text, natural language processing technology can be applied to understand the user's intentions, answer questions or execute instructions.

[0044] More specifically, in order to convert text information into a vector representation with more semantic information, the text of user speech recognition can be segmented, and a context encoder based on a transformer including an embedding layer can be used to obtain a speech feature vector, which provides better input for subsequent natural language processing and machine learning tasks. Through the word segmentation and embedding layers, the text can be converted into a vector representation, which can better capture the semantic relationship between words. This helps to convert text information into a vector representation with more semantic information. The transformer-based context encoder can understand the contextual relationship between words in a sentence, thereby better capturing the overall semantics of the sentence. This helps to ensure that the speech feature vector can contain more contextual information. By converting text into a speech feature vector, the speech information can be converted into a form that can be understood and processed by the computer, thereby providing input for subsequent speech recognition, sentiment analysis or other natural language processing tasks. The obtained speech feature vector can be used as input to a deep learning model for training and inference. This vector representation can help the model better understand and process speech information. At the same time, the obtained speech feature vector usually has a lower dimension, which helps to reduce the complexity and computational cost of the data while retaining most of the useful semantic information. In this way, it provides better input for subsequent natural language processing and machine learning tasks.

[0045] Figure 2 The flowchart is a method for driving a virtual human based on multi-sensory perception according to an embodiment of the present application, wherein the text of the user speech recognition is segmented and then passed through a context encoder based on a converter including an embedding layer to obtain a speech feature vector. Figure 2 As shown, the text of the user speech recognition is segmented and then passed through a converter-based context encoder including an embedding layer to obtain a speech feature vector, including: S10, segmenting the text of the user speech recognition to obtain a plurality of speech words; S20, passing the plurality of speech words through an embedding layer to convert each of the plurality of speech words into a speech word embedding vector to obtain a sequence of speech word embedding vectors, wherein the embedding layer uses a learnable embedding matrix to embed-code each speech word; S30, inputting the sequence of speech word embedding vectors into the converter-based context encoder to obtain the plurality of speech semantic feature vectors; and S40, cascading the plurality of speech semantic feature vectors to obtain the speech feature vector.

[0046] Figure 3 The present invention is a flowchart of inputting the sequence of speech word embedding vectors into the converter-based context encoder to obtain the multiple speech semantic feature vectors in the virtual human driving method based on multi-sensory perception according to an embodiment of the present application. Figure 3As shown, S31, arranging the sequence of speech word embedding vectors as an input vector; S32, converting the input vector into a query vector and a key vector respectively through a learnable embedding matrix; S33, calculating the product between the query vector and the transposed vector of the key vector to obtain a self-attention association matrix; S34, normalizing the self-attention association matrix to obtain a standardized self-attention association matrix; S35, inputting the standardized self-attention association matrix into a Softmax activation function for activation to obtain a self-attention feature matrix; and S36, multiplying the self-attention feature matrix with each speech word embedding vector in the sequence of speech word embedding vectors to obtain the multiple speech semantic feature vectors.

[0047] In step S130, the user's facial image is processed to obtain a facial feature vector. Facial images contain rich information, including facial contours, expressions, eyes, mouths and other features. By processing facial images and extracting facial feature vectors, the user's facial features, such as facial expressions, eye contact, etc., can be better characterized, thereby providing a more accurate basis for the expression synthesis and emotion recognition of virtual humans. By processing facial images and extracting facial feature vectors, virtual humans can more naturally imitate the facial expressions and movements of real people, making the interaction more natural and vivid. At the same time, processing facial images can help the system adapt to the facial features of different users and factors such as lighting and angles in different environments, thereby improving the robustness and adaptability of the system.

[0048] Specifically, in order to extract representative facial features from an image, the user's face image can be passed through a convolutional neural network model as a facial feature extractor to obtain a facial feature vector. Convolutional neural networks (CNNs) have excellent capabilities in processing images and can extract various features in images, including edges, textures, shapes, etc. Through the CNN model, representative facial features can be extracted from face images. Facial feature vectors usually have a lower dimension. Compared with the original image data, this vector representation is more compact, which helps to reduce data complexity and computational cost while retaining most of the useful feature information.

[0049] Specifically, in the above-mentioned virtual human driving method based on multi-perception, the user face image is processed to obtain a facial feature vector, including: passing the user face image through a convolutional neural network model as a facial feature extractor to obtain a facial feature vector.

[0050] Specifically, in the above-mentioned virtual human driving method based on multi-perception, the user face image is passed through a convolutional neural network model as a facial feature extractor to obtain a facial feature vector, including: using each layer of the convolutional neural network model as a facial feature extractor to perform the following on the input data in the forward pass of the layer: convolution processing on the input data to obtain a convolutional feature map; pooling processing of the local feature matrix along the channel dimension of the convolutional feature map to obtain a pooled feature map; and nonlinear activation of the pooled feature map to obtain an activated feature map; wherein the output of the last layer of the convolutional neural network model as a facial feature extractor is the facial feature vector, and the input of the first layer of the convolutional neural network model as a facial feature extractor is the user face image.

[0051] In step S140, the speech feature vector and the facial feature vector are processed to obtain a rendered RGB image. Considering that the speech feature vector and the facial feature vector represent speech information and facial expression information respectively. Combining them can realize the fusion of cross-modal information, thereby more comprehensively representing the user's emotional state and interaction intention. By combining speech features and facial features, the user's emotional state and interaction intention can be more vividly expressed, making the performance of the virtual person more vivid and natural. The rendered RGB image can better simulate the facial expressions and speech expressions of real people, thereby improving the naturalness and realism of the interaction. At the same time, combining speech and facial features can realize personalized customization of the virtual human image, and generate a virtual human image that meets the user's characteristics according to the user's speech and facial expression characteristics.

[0052] Specifically, in the above-mentioned virtual human driving method based on multi-perception, the speech feature vector and the facial feature vector are processed to obtain a rendered RGB image, including: based on the speech feature vector, the facial feature vector is subjected to viewpoint metric perception based on multi-viewpoint quantization between features to obtain an optimized facial feature vector, and the optimized facial feature vector and the speech feature vector are fused to obtain a speech-face association feature matrix; the speech-face association feature matrix is ​​passed through a bidirectional attention mechanism to obtain an enhanced speech-face association feature matrix; and, the enhanced speech-face association feature matrix is ​​passed through a generator to obtain a rendered RGB image.

[0053] Specifically, the speech feature vector and the facial feature vector represent different perceptual modalities. In order to capture the correlation and connection between speech and facial expressions, the correlation feature matrix between the two can be calculated to achieve the integration and association of cross-modal information. In particular, in the technical solution of the present application, it is considered that the speech feature vector is extracted based on text data (text processed by speech recognition and word segmentation), while the facial feature vector is extracted based on image data (face image). These two data types essentially reflect different information, which may lead to semantic inconsistency of feature vectors. Specifically, the speech feature vector extracts features through a converter-based context encoder, focusing on the semantic information of the text; while the facial feature vector extracts features through a convolutional neural network model, focusing on the visual information of the image. These two different feature extraction methods may lead to differences in the expression of feature vectors. When the speech feature vector and the facial feature vector are associated and fused, if the fusion strategy does not fully consider the characteristic characteristics and weights of the two, it may lead to the lack of posterior-prior causal association. In order to solve this problem, in the technical solution of the present application, based on the speech feature vector, the facial feature vector is further subjected to viewpoint metric perception based on multi-viewpoint quantization between features to obtain an optimized facial feature vector, and then the optimized facial feature vector and the speech feature vector are fused to obtain a speech-face association feature matrix.

[0054] Specifically, in the above-mentioned virtual human driving method based on multi-perception, based on the voice feature vector, the facial feature vector is subjected to viewpoint measurement perception based on multi-viewpoint quantization between features to obtain an optimized facial feature vector, and the optimized facial feature vector and the voice feature vector are fused to obtain a voice-face association feature matrix, including: calculating the positional difference between the facial feature vector and the voice feature vector to obtain a first viewpoint measurement representation vector; calculating the positional point multiplication between the facial feature vector and the voice feature vector to obtain a second viewpoint measurement representation vector; calculating the positional sum between the facial feature vector and the voice feature vector to obtain a third viewpoint measurement representation vector; calculating the first viewpoint measurement The gradient amplitude of each position in the representation vector, the second viewpoint metric representation vector and the third viewpoint metric representation vector is calculated to obtain the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector; the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector are input into the backward machine learning reverse multi-viewpoint perception optimization module to obtain the reverse multi-viewpoint perception optimization representation vector; the positional point multiplication between the reverse multi-viewpoint perception optimization representation vector and the facial feature vector is calculated to obtain the optimized facial feature vector; the vector product between the optimized facial feature vector and the transposed vector of the speech feature vector is calculated to obtain the speech-face association feature matrix.

[0055] More specifically, the reverse multi-viewpoint perceptual optimization representation vector is calculated using the following perceptual optimization formula, wherein the perceptual optimization formula is:

[0056]

[0057] in, and represents the eigenvalues ​​of the i-th position of the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector, respectively, and μ represents the three eigenvalues ​​of the corresponding positions in the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector , σ is the standard deviation of the three eigenvalues ​​at corresponding positions in the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector, represents the maximum value of the three eigenvalues ​​at the corresponding positions in the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector, ReLU represents a linear rectification function, softmax represents a normalized exponential function, and v ri Represents the eigenvalue of the i-th position of the inverse multi-view perceptual optimization representation vector.

[0058] That is, since the speech feature vector and the facial feature vector may not be sufficiently correlated in the high-dimensional feature space, before the two are correlated and fused, first, based on the speech feature vector, the facial feature vector is subjected to viewpoint metric perception based on multi-viewpoint quantization between features to obtain an optimized facial feature vector. Specifically, the optimization process includes: first, by multi-view analysis, the positional difference, dot multiplication and sum between the facial feature vector and the speech feature vector are calculated to obtain three viewpoint metric representation vectors, and the relationship between the features is quantified from different angles. Then, the gradient amplitude of each position in these viewpoint metric representation vectors is calculated by backward error perception to identify key change points to provide a basis for feature optimization.

[0059] Then, the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector are input into the backward machine learning reverse multi-view perception optimization module, wherein the backward machine learning reverse multi-view perception optimization module captures the significant comprehensive ability factor of the feature importance perception (Feature Importance Perception) of the feature sub-dimension distribution of the corresponding positions in the first to third viewpoint metric perception vectors by adopting a squeeze-incentive mechanism composed of -functions. Then, the position point multiplication between the reverse multi-view perception optimization representation vector and the facial feature vector is calculated to perform backward optimization perception (BackwardOptimization Perception) on the facial feature vector to adjust the facial feature vector so as to better associate with the speech feature vector and enhance the causal relationship between the facial feature vector and the speech feature vector.

[0060] More specifically, in order to better utilize the key features in the speech-face association feature matrix, the speech-face association feature matrix can be enhanced through a bidirectional attention mechanism to better integrate cross-modal information. The association between speech and facial expressions is complex and diverse. Through the bidirectional attention mechanism, the information of speech and facial expressions can be better integrated, making the association feature matrix more comprehensive and accurate. At the same time, through the attention mechanism, the degree of association between speech and facial features can be dynamically weighted, so that the model can pay more attention to key speech-face association features, improving the expressiveness and discrimination of the features. The bidirectional attention mechanism helps to capture the contextual information between speech and facial expressions, so that the association feature matrix can better reflect the temporal and spatial relationship between speech and facial expressions. The enhanced speech-face association feature matrix can improve the accuracy of emotion recognition and synthesis, so that the system can better understand and express the speaker's emotional state.

[0061] Specifically, in the above-mentioned virtual human driving method based on multi-perception, the voice-face association feature matrix is ​​subjected to a bidirectional attention mechanism to obtain an enhanced voice-face association feature matrix, including: pooling the voice-face association feature matrix along the horizontal direction and the vertical direction respectively to obtain a first-direction pooling vector and a second-direction pooling vector; associatively encoding the first-direction pooling vector and the second-direction pooling vector to obtain a bidirectional association matrix; inputting the bidirectional association matrix into a Sigmoid activation function to obtain a bidirectional association weight matrix; and calculating the positional point multiplication between the bidirectional association weight matrix and the voice-face association feature matrix to obtain the classification feature matrix.

[0062] Furthermore, by processing the speech feature vector and the facial feature vector, the enhanced speech-face correlation feature matrix is ​​passed through a generator to obtain a rendered RGB image, which reflects the correlation between the user's speech and facial features.

[0063] Specifically, in the above-mentioned virtual human driving method based on multi-perception, the model of the generator can be a generative adversarial network model or a variational autoencoder model, which trains the enhanced speech-face association feature matrix to map the enhanced speech-face association feature matrix to a representation of the rendered RGB image.

[0064] In summary, in the technical solution of the present application, first, the user voice and the user face image are obtained. Then, the user voice is subjected to voice recognition to obtain the text of the user voice recognition. Then, the text of the user voice recognition is subjected to word segmentation processing and then passed through a converter-based context encoder containing an embedding layer to obtain a voice feature vector. Then, the user face image is passed through a convolutional neural network model as a facial feature extractor to obtain a facial feature vector. Then, based on the voice feature vector, the facial feature vector is subjected to viewpoint metric perception based on multi-viewpoint quantization between features to obtain an optimized facial feature vector, and the optimized facial feature vector and the voice feature vector are fused to obtain a voice-face association feature matrix. Then, the voice-face association feature matrix is ​​passed through a bidirectional attention mechanism to obtain an enhanced voice-face association feature matrix. Furthermore, the enhanced voice-face association feature matrix is ​​passed through a generator to obtain a rendered RGB image.

[0065] In summary, the virtual human driving method based on multi-perception according to the embodiment of the present application has been explained, which uses artificial intelligence technology based on the field of deep learning to extract and encode the user's voice and face image to obtain an RGB image rendered by the virtual human. In this way, by utilizing the user's multi-dimensional information, the virtual human's ability to perceive and understand the user's behavior, emotions and environment is improved.

[0066] Based on the same inventive concept, the embodiments of the present application also provide a virtual human driving device based on multi-perception, which can be used to implement the method described in the above embodiment, as described in the following embodiments. Since the principle of solving the problem by the virtual human driving device based on multi-perception is similar to the above method, the implementation of the virtual human driving device based on multi-perception can refer to the implementation of the above method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0067] Figure 4 FIG. 1 is a block diagram of a virtual human driving device based on multi-sensory perception according to an embodiment of the present application. Figure 4 As shown, according to the embodiment of the present application, the virtual human driving device 100 based on multi-perception includes: a user information acquisition module 110, used to acquire user voice and user face image; a user voice encoding module 120, used to process the user voice to obtain a voice feature vector; a user facial feature extraction module 130, used to process the user face image to obtain a facial feature vector; a rendering image generation module 140, used to process the voice feature vector and the facial feature vector to obtain a rendered RGB image.

[0068] Here, those skilled in the art can understand that the specific functions and operations of the various units and modules in the virtual human driving device 100 based on multi-sensing have been described in the above reference. Figures 1 to 3 The description of the virtual human driving method based on multi-perception has been introduced in detail, and therefore, its repeated description will be omitted.

[0069] In summary, the virtual human driving device based on multi-sensing according to the embodiment of the present application has been explained, which uses artificial intelligence technology based on the field of deep learning to extract and encode the user's voice and face image to obtain an RGB image rendered by the virtual human. In this way, by utilizing the user's multi-sensing information, the virtual human's ability to perceive and understand the user's behavior, emotions and environment is improved.

[0070] As described above, the virtual human driving device 100 based on multi-sensory perception according to the embodiment of the present application can be implemented in various terminal devices, such as a server driven by a virtual human based on multi-sensory perception, etc. In one example, the virtual human driving device 100 based on multi-sensory perception according to the embodiment of the present application can be integrated into the terminal device as a software module and / or a hardware module. For example, the virtual human driving device 100 based on multi-sensory perception can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the virtual human driving device 100 based on multi-sensory perception can also be one of the many hardware modules of the terminal device.

[0071] Alternatively, in another example, the virtual human driving device 100 based on multi-perception and the terminal device may also be separate devices, and the virtual human driving device 100 based on multi-perception may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0072] The devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is an electronic device, and the electronic device may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0073] In a typical example, the electronic device specifically includes: a memory for storing a computer program; and a processor for implementing the steps of the virtual human driving method based on multi-sensory perception when executing the computer program.

[0074] Figure 5 FIG. 1 is a block diagram of an electronic device according to an embodiment of the present application. Figure 5As shown, a virtual human driving method based on multi-sensory perception, this specification also provides an electronic device, including: a memory for storing a computer program; a processor for implementing the steps of the virtual human driving method based on multi-sensory perception when executing the computer program. The processor can take the form of a computer-readable medium, logic gate, switch, application specific integrated circuit (Application Specific Integrated Circuit, ASIC), programmable logic controller and embedded microcontroller, etc., such as a microprocessor or processor and a computer-readable program code (such as software or firmware) that can be executed by the (micro) processor. The memory includes but is not limited to random access memory (Random Access Memory, RAM), read-only memory (Read-Only Memory, ROM), cache (Cache), hard disk (Hard Disk Drive, HDD) or memory card (Memory Card). The electronic device can be a server of the method described in the above embodiment, etc. Of course, the electronic device can also include module structures such as input devices and transmission modules. In this embodiment, the functions and effects specifically implemented by the electronic device can be explained in comparison with other embodiments, and will not be repeated here.

[0075] Furthermore, an embodiment of the present specification also provides a storage medium containing computer executable instructions, characterized in that the computer executable instructions implement the virtual human driving method based on multi-sensory perception when executed by a computer processor. The storage medium may include a physical device for storing information, which is usually to digitize the information and then store it in a medium using electrical, magnetic or optical methods. The storage medium may include: devices that store information using electrical energy, such as various types of memories, such as RAM, ROM, etc.; devices that store information using magnetic energy, such as hard disks, floppy disks, magnetic tapes, magnetic core memories, bubble memories, and U disks; devices that store information optically, such as CDs or DVDs. Of course, there are other types of readable storage media, such as quantum memories, graphene memories, and so on.

[0076] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0077] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0078] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0079] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0080] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A virtual human driving method based on multi-sensory perception, characterized in that: include: Obtain user voice and user face image; Processing the user's speech to obtain a speech feature vector; Processing the user's facial image to obtain a facial feature vector; The speech feature vector and the facial feature vector are processed to obtain a rendered RGB image.

2. The virtual human driving method based on multi-sensory perception according to claim 1, characterized in that: Processing the user voice to obtain a voice feature vector includes: Performing speech recognition on the user speech to obtain a text of the user speech recognition; The text of the user speech recognition is segmented and passed through a converter-based context encoder including an embedding layer to obtain a speech feature vector.

3. The virtual human driving method based on multi-sensory perception according to claim 2 is characterized in that: The text of the user speech recognition is segmented and passed through a converter-based context encoder including an embedding layer to obtain a speech feature vector, including: Performing word segmentation processing on the text of the user speech recognition to obtain a plurality of speech words; Passing the plurality of spoken words through an embedding layer to convert each of the plurality of spoken words into a speech word embedding vector to obtain a sequence of speech word embedding vectors, wherein the embedding layer uses a learnable embedding matrix to embed the each speech word; Inputting the sequence of speech word embedding vectors into the converter-based context encoder to obtain the plurality of speech semantic feature vectors; The multiple speech semantic feature vectors are cascaded to obtain the speech feature vector.

4. The virtual human driving method based on multi-sensory perception according to claim 3 is characterized in that: Processing the user's face image to obtain a facial feature vector includes: The user's face image is passed through a convolutional neural network model as a facial feature extractor to obtain a facial feature vector.

5. The virtual human driving method based on multi-sensory perception according to claim 4, characterized in that: Processing the speech feature vector and the facial feature vector to obtain a rendered RGB image includes: Based on the speech feature vector, performing viewpoint metric perception based on multi-viewpoint quantization between features on the facial feature vector to obtain an optimized facial feature vector, and fusing the optimized facial feature vector and the speech feature vector to obtain a speech-face correlation feature matrix; The speech-face correlation feature matrix is ​​subjected to a bidirectional attention mechanism to obtain an enhanced speech-face correlation feature matrix; The enhanced speech-face association feature matrix is ​​passed through a generator to obtain a rendered RGB image.

6. The virtual human driving method based on multi-sensory perception according to claim 5, characterized in that: Based on the speech feature vector, the facial feature vector is subjected to viewpoint metric perception based on multi-viewpoint quantization between features to obtain an optimized facial feature vector, and the optimized facial feature vector and the speech feature vector are fused to obtain a speech-face correlation feature matrix, including: Calculating a position-wise difference between the facial feature vector and the speech feature vector to obtain a first viewpoint metric representation vector; Calculate the position point multiplication between the facial feature vector and the speech feature vector to obtain a second viewpoint metric representation vector; Calculating a position-based sum between the facial feature vector and the speech feature vector to obtain a third viewpoint metric representation vector; Calculating the gradient magnitude of each position in the first viewpoint metric representation vector, the second viewpoint metric representation vector, and the third viewpoint metric representation vector to obtain a first viewpoint metric perception vector, a second viewpoint metric perception vector, and a third viewpoint metric perception vector; Inputting the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector into a backward machine learning inverse multi-view perception optimization module to obtain a reverse multi-view perception optimization representation vector; Calculating the position point multiplication between the inverse multi-view perception optimization representation vector and the facial feature vector to obtain the optimized facial feature vector; The vector product between the optimized facial feature vector and the transposed vector of the speech feature vector is calculated to obtain the speech-face associated feature matrix.

7. The virtual human driving method based on multi-sensory perception according to claim 6, characterized in that: The speech-face correlation feature matrix is ​​subjected to a bidirectional attention mechanism to obtain an enhanced speech-face correlation feature matrix, comprising: Pooling the voice-face correlation feature matrix along the horizontal direction and the vertical direction respectively to obtain a first-direction pooling vector and a second-direction pooling vector; Performing association coding on the first-direction pooled vector and the second-direction pooled vector to obtain a bidirectional association matrix; Input the bidirectional association matrix into a Sigmoid activation function to obtain a bidirectional association weight matrix; The classification feature matrix is ​​obtained by calculating the position point multiplication between the bidirectional association weight matrix and the voice-face association feature matrix.

8. A virtual human driving device based on multi-sensory perception, characterized in that: include: A user information acquisition module, used to acquire user voice and user face image; A user voice encoding module, used for processing the user voice to obtain a voice feature vector; A user facial feature extraction module, used for processing the user's facial image to obtain a facial feature vector; The rendered image generation module is used to process the speech feature vector and the facial feature vector to obtain a rendered RGB image.

9. A virtual human driving device based on multi-sensory perception, characterized in that: Runs on the baseboard management controller BMC, including: Memory for storing computer programs; A processor, used to implement the steps of the virtual human driving method based on multi-perception as described in any one of claims 1 to 7 when executing the computer program.

10. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, the virtual human driving method based on multi-perception as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Material packaging production line monitoring platform and method based on industrial internet of things

    CN118968419A

  • Remote monitoring alarm system and method for intelligent safety box

    CN119152626A

  • Content automatic generation system and method based on user data

    CN119226607A

  • Tunnel bridge monitoring management system and method

    CN119251753A

Cited By

  • Material packaging production line monitoring platform and method based on industrial internet of things

    CN118968419A