14008results about "Animation" patented technology

The invention belongs to the technical field of visual special effects, and discloses an AI visual special effect dynamic generation system fusing multi-modal perception. Through cooperative work of six core modules of multi-modal input processing, importance analysis, parameter configuration, parameterized rendering, parameter optimization and rendering and output, deep fusion of special effects and contents is realized. An image sequence, audio data and scene parameters can be processed at the same time, a multi-modal feature data set is constructed, a scene key visual area is scientifically recognized, a visual importance distribution map is generated, and the special effect parameter configuration and rendering process is guided. An improved neural radiation field technology and a physical constraint model are adopted to ensure the consistency and reality of the special effect at different visual angles; and a multi-dimensional quality evaluation and parameter automatic optimization mechanism is introduced, so that the visual expressive force and the artistic value of the special effect are guaranteed. The creation threshold is remarkably reduced, the cooperative expression ability of the special effect and the content is improved, and the special effect becomes a powerful tool for enhancing the narration and enhancing the emotion.

The invention provides an intelligent real-time interactive question-answering system based on a virtual digital human, and belongs to the technical field of voice signal processing and voice recognition, and the system comprises a data acquisition module which receives a voice or text interaction request input by a user, collects the expression dynamic parameter sequence and limb movement sequence data of the user in real time, and transmits the data to a user interaction module; obtaining a standardized voice feature vector and structured text data; the cross-modal fusion module is used for constructing an interactive feature matrix; the behavior decision module outputs a decision instruction set; the knowledge retrieval module is used for generating an answer text with emotional adaptability and voice features; and the voice generation module is used for generating a mouth shape animation key frame, a micro expression parameter sequence and a limb action track of the virtual digital human, generating a voice response in combination with the answer text and the voice characteristics, and pushing the voice response to the user terminal. According to the method, the interaction experience and adaptability of the virtual digital human are remarkably improved.

In some embodiments, a computer system facilitates movement, including rotation, of a virtual object in a three-dimensional environment. In some embodiments, a computer system facilitates movement of a virtual object in a three-dimensional environment toward a movement boundary. In some embodiments, a computer system facilitates dynamic scaling of a virtual object in a three-dimensional environment based on movement of the virtual object in the three-dimensional environment. In some embodiments, a computer system facilitates inertial movement of a virtual object in a three-dimensional environment. In some embodiments, a computer system facilitates converging offsets between a portion of a user and a virtual object. In some embodiments, a computer system facilitates rotation of a volumetric virtual object in a three-dimensional environment.

In some embodiments, a computer system changes a level of detail with which a respective environment is being displayed. In some embodiments, a computer system applies a neutralization adjustment to generate a representation of the physical environment. In some embodiments, a computer system displays a user interface object in a three-dimensional environment with a three-dimensional stereoscopic effect. In some embodiments, a computer system facilitates light blending. In some embodiments, a computer system transitions between three-dimensional environments using visual effects. In some embodiments, a computer system detects a movement in a viewpoint of a user while displaying a portal to a virtual environment, and either maintains or ceases display of the portal based on the amount of movement and / or a direction in which the portal opens. In some embodiments, a computer system outputs a different sound effect when initiating display of different virtual three-dimensional environments.

Disclosed are an emotional evolution method and terminal for a virtual avatar in educational metaverse. By collecting expression data and audio data of the virtual avatar, and performing emotional feature extraction based on the expression data and the audio data; fusing an extracted sound emotional feature with an extracted expression emotional feature by using an emotional feature fusion model, and performing emotion recognition on a multi-modal emotional feature fusion result obtained by fusion to obtain an emotional category corresponding to the multi-modal emotional feature fusion result; labeling the multi-modal emotional feature fusion result based on a semantic vector of the emotional category to generate an emotional evolution sequence; and driving the virtual avatar to perform emotional expression according to a target emotional evolution pattern extracted from the emotional evolution sequence, a unified and united multi-modal emotional feature fusion result is formed, the emotional self-evolution of the virtual avatar is achieved.

An example operation may include one or more of generating an image of an object based on execution of a generative artificial intelligence (GenAI) model and displaying the image via a user interface of a software application, receiving inputs via the user interface, determining that a new feature of the software application has been activated by a user account of the software application based on the received inputs, and in response to the activation of the new feature, adding additional content to the image of the object based on execution of the GenAI model on information associated with the new feature and refreshing a display of the image of the object within the user interface of the software application.

A computer system that customizes an output provided by a pretrained neural network is described. During operation, the computer system may receive a prompt (or input) associated with a first individual. Then, the computer system may access stored information (in memory) associated with a persona associated with a second individual. For example, the second individual may be different from the first individual. However, in some embodiments, the second individual may be the same as the first individual. Note that the stored information may include memories associated with one or more different sessions of interacting with the pretrained neural network than a current session. Moreover, the computer system may modify the prompt based at least in part on the persona associated with the second individual. Next, the computer system may generate, using the pretrained neural network, the output based at least in part on the modified prompt.

The present invention relates to a digital twin processing method and system, and a cloud platform. The method comprises: acquiring production system elements, carrying out abstraction definition and parameterization description on the production system elements by means of digital-to-analog knowledge fusion of a mathematical model and a mechanistic model, so as to construct a digital twin ontology model; analyzing and reconstructing model data to obtain a mapping model of which object variables can be directly accessed and operated by a collective motion control method, so that the model is visualized at the cloud; and using an external data source to drive parameter update and operation matching of the model by means of a motion control method, so as to complete cooperative deployment and synchronous evolution of an actual physical device and the model in the production process on a cloud server. According to the present invention, a model is constructed by means of digital-to-analog knowledge fusion of a mathematical model and a mechanistic model, the model is mapped to achieve motion visualization, model parameter update and operation matching on the cloud are achieved, and then cooperative deployment and synchronous evolution of a physical device and the model are completed.

The invention discloses an intelligent multi-modal virtual digital human interaction system based on an AI language large model. The system comprises a high-authenticity face generation module; the high-authenticity face generation module uses an AdaAN network, based on adaptive feature fusion and voice driving and time sequence modeling of voice features, feature information related to voice is extracted, the extracted voice features are processed through a deep neural network, it is ensured that the voice and facial expressions are highly aligned in time and space, and the face recognition accuracy is improved. Collecting a bio-electricity signal, mapping the signal to facial muscle movement, generating a final facial expression, and interacting with a user; the system further comprises an intelligent interaction module, a training optimization and efficient generation module, an efficient integration module, a multi-modal data acquisition module, an AI large model core processing module, a digital human image generation and driving module, an interaction scene adaptation module and a feedback optimization module. The invention further discloses a multi-mode digital human interaction method which has wide application value.

Aspects of the subject technology provide for human subject Gaussian splatting using machine learning. A method includes receiving a video input having a scene and a subject. The method also includes obtaining a three-dimensional (3D) reconstruction of the subject and the scene from the video input. The method includes generating a 3D Gaussian representation of each of the scene and the subject. The method also includes generating a deformed 3D Gaussian representation of the subject by adapting the 3D Gaussian representation of the subject to the 3D reconstruction of the subject. The method includes rendering a visual output comprising an animatable avatar of the subject and the scene using differentiable Gaussian rasterization based at least in part on the deformed 3D Gaussian representation of the subject and the 3D Gaussian representation of the scene.

An example operation may include one or more of receiving context of a user during an inquiry of a feature via a software application, executing a waiting period via the software application, during the waiting period, selecting an animation to display via the software application based on the context of the user and the feature inquiry wherein the animation provides contextual data associated with the feature, wherein the contextual data is based on a determined need of the user, displaying the animation via the software application during the waiting period, and determining if the user has accepted the feature via the software application. At least one portion of the example operation: integrates with an artificial intelligence (AI) chatbot, interacts with the AI chatbot, is performed by the AI chatbot, and / or is associated with an AI model.

The invention belongs to the technical field of virtual person generation, and particularly relates to a digital person video generation method based on a multi-modal large model, and the method comprises the following steps: 1, constructing a multi-modal data system; 2, multi-modal large model training and adaptation are carried out; 3, constructing a digital human three-dimensional model; step 4, performing semantic analysis and modal mapping; 5, generating a time sequence action and a mouth shape; step 6, building and rendering a virtual scene; step 7, audio and video synchronous rendering and synthesis; step 8, quality optimization and defect repair; and step 9, performing user interaction and iterative optimization. Through technical innovation and engineering, the core pain point in digital human video generation is solved, efficient, vivid and customizable content production capacity is provided for virtual anchors, intelligent customer service, enterprise training and other scenes, and the AI digital human technology is promoted to be applied to large-scale business from experiments.

The present disclosure provides embodiments for joint twist generation for animation. The system can utilize a neural network, also referred to as a deep neural network, which utilizes machine learning processes in order to create animation data that are more life-like and realistic. The system can obtain a set of axis vectors for a rig of a virtual character model; obtain a twist model for the rig; input the set of axis vectors to the twist model to obtain a set of twist vectors; and determine animation data based on the set of axis vectors and the set of twist vectors.

A multi-modal articulation system translates human gestures into control commands for digital and automation devices. The system integrates a camera module, which captures facial and hand movements, with an image processing unit that extracts facial features, such as eyebrow movement, lip curvature, eyelid motion, and eye trajectory. These extracted features are analyzed in real time by an articulation recognition module that compares them against a database to recognize specific gestures. The system tracks and processes air-drawn hand gestures using a gesture trajectory tracking unit, which converts the motions into digital representations and classifies them using deep learning algorithms. The recognized gestures are converted into machine-readable control signals and executed on various connected devices via a relay control unit. The system incorporates adaptive algorithms to improve accuracy, account for environmental conditions, and stabilize hand tremors, providing a gesture-based control interface for applications ranging from smart home systems to multimedia devices.

In some embodiments, a computer system displays user interfaces for enrolling one or more features of a user of a computer system. In some embodiments, a computer system displays visual effects associated with a virtual avatar in an XR environment. In some embodiments, a computer system displays objects having different visual characteristics in an XR environment. In some embodiments, a computer system switches between different presentation modes associated with a user represented in an XR environment. In some embodiments, a computer system displays a virtual avatar in an XR environment.

Methods of assisting individuals with communication are provided. In the disclosed methods, cortical activity from a region of the brain associated with movement, speech production, and / or language perception is recorded while an individual attempts to perform an action (e.g., to say words, express an emotion, perform a movement, etc.). Deep learning computational models are used to detect and decode the attempted action from the recorded brain activity. Decoding of actions from brain activity may be aided by the use of self-supervised machine learning techniques, which discretize each action into one or more action representations that serve as an effective intermediary for decoding neural activity patterns and features into meaningful action outputs. Methods for synthesizing decoded actions into audio and / or visual stimuli are also provided, allowing for more naturalistic and expressive communication for individuals who are unable to speak or experience other mobility limitations that inhibit full embodied communication.

In various examples, systems and methods are disclosed relating to generation of four-dimensional (4D) content models, such as 4D content models to render realistic sequences of frames of 3D data. The systems can initialize a 3D component of the 4D content model, such as a 3D Gaussian splatting representation, based at least on a prompt for the 4D content. The system can configure motion and / or dynamics for the sequence of frames by evaluating frames rendered from the 4D content model using one or more latent diffusion models (LDMs), including a video LDM. The system can perform operations such as autoregressive generation of frames to create long sequences of content, motion amplification to facilitate realistic, dynamic motion generation, and regularization to facilitate generation of complex dynamics.

The invention provides a rehabilitation training action evaluation method and device based on multi-view vision. According to the method, a multi-view camera is adopted to synchronously collect a rehabilitation training image sequence, two-dimensional coordinates of key points of a human body are extracted by improving an HRNet deep learning model, three-dimensional reconstruction is performed in combination with a Gaussian process, motion feature data are extracted, and rehabilitation training quality is evaluated. Accurate motion capture without wearing mark points by the patient is realized, the training constraint feeling of the patient is effectively reduced, the rehabilitation evaluation accuracy is improved, and a personalized rehabilitation scheme can be generated.

In various examples, systems and methods are disclosed relating to generating animatable characters or avatars. The system can assign a plurality of first elements of a three-dimensional (3D) model of a subject to a plurality of locations on a surface of the subject in an initial pose. Further, the system can assign a plurality of second elements to the plurality of first elements, each second element of the plurality of second elements having an opacity corresponding to a distance between the second element and the surface of the subject. Further, the system can update the plurality of second elements based at least on a target pose for the subject and one or more attributes of the subject to determine a plurality of updated second elements. Further, the system can render a representation of the subject based at least on the plurality of updated second elements.

The invention discloses a digital human real-time interaction system and method. The interaction system comprises terminal equipment and a cloud server. The terminal device receives voice input by a user and transmits the voice to the cloud server. The cloud server comprises a video acquisition and processing module, an intelligent question and answer and voice generation module, an audio processing module and an image rendering module. A video capture and processing module captures original video data indicative of an object of a digital figure and generates successive image frames. The intelligent question and answer and voice generation module converts the voice into a text; performing semantic analysis on the text, understanding the intention of the user, and generating a corresponding reply text; the reply text is also converted into a reply voice. The audio processing module processes the reply voice and the continuous image frames to generate first video data. And the image rendering module fuses the first video data with the virtual background, and performs real-time processing on the fused video to generate a response video. And the terminal equipment outputs the response video.

Methods for displaying and organizing user interfaces in a computer-generated environment provide for an efficient and intuitive user experience. In some embodiments, user interfaces can be grouped together into a container. In some embodiments, user interfaces can be added to a container, removed from a container, or moved from one location in the container to another location. In some embodiments, a visual indication is displayed before a user interface is added to a container. In some embodiments, a user interface can replace an existing user interface in a container. In some embodiments, when moving a user interface in a computer-generated environment, the transparency of a user interface that is obscured can be modified.

Systems and methods of the present disclosure include animating virtual avatars or agents according to input audio and one or more selected or determined emotions and / or styles. For example, a deep neural network can be trained to output motion or deformation information for a character that is representative of the character uttering speech contained in audio input. The character can have different facial components or regions (e.g., head, skin, eyes, tongue) modeled separately, such that the network can output motion or deformation information for each of these different facial components. During training, the network can use a transformer-based audio encoder with locked parameters to train an associated decoder using a weighted feature vector. The network output can be provided to a renderer to generate audio-driven facial animation that is emotion-accurate.

The present disclosure generally relates to user interface for capturing and managing media (e.g., photo media, video media). In some examples, user interfaces for managing the file format of media (e.g., photo, video media) are described. In some examples, user interfaces for storing media (photo media (e.g., a sequences of image, a single still image), video media) are described.

Techniques for rendering visual content, in response to one or more utterances, are described. A device receives one or more utterances that define a parameter(s) for desired output content. A system (or the device) identifies natural language data corresponding to the desired content, and uses natural language generation processes to update the natural language data based on the parameter(s). The system (or the device) then generates an image based on the updated natural language data. The system (or the device) also generates video data of an avatar. The device displays the image and the avatar, and synchronizes movements of the avatar with output of synthesized speech of the updated natural language data. The device may also display subtitles of the updated natural language data, and cause a word of the subtitles to be emphasized when synthesized speech of the word is being output.

Embodiments described herein provide a video generation framework built on a decoupled multimodal cross-attention module to simultaneously condition the generation on both an input image and a text input. The video generation may thus be conditioned on the visual appearance of a target object reflected in the input image. In this way, zero-shot video generation may be achieved with little fine-tuning efforts.

The invention relates to the technical field of data processing, and discloses a training method and system for generating a 5D digital human based on AIGC, and a medium. The method comprises the following steps: preprocessing real person feature data to obtain multi-dimensional preprocessed data; performing multi-level processing on the multi-dimensional preprocessing data to obtain 5D digital human generation data; performing cross-modal coordination processing on the 5D digital human generation data through a progressive multi-task learning algorithm to obtain to-be-processed digital human data; performing dynamic knowledge integration processing on the to-be-processed digital human data to obtain adaptive digital human data; carrying out personalized fine tuning and style migration processing on the adaptive digital human data to obtain target digital human data; and performing real-time rendering and intelligent interaction processing on the target digital human data to obtain target 5D digital human data, and generating a digital human training scheme according to the target 5D digital human data. According to the method and the device, the training efficiency and accuracy of generating the 5D digital human based on the AIGC are improved.

Embodiments of the present invention provide a server, a display device, and a digital human processing method. The method comprises: receiving image data; inputting the image data into a shared encoder to obtain a first depth encoding feature; inputting the first depth encoding feature into a segmentation decoder to obtain a segmentation mask; inputting the segmentation mask and the depth encoding feature into a guided flow architecture module to obtain a refined feature; inputting the depth encoding feature and the refined feature into an image matting decoder to obtain a face image having undergone image matting; performing image quality enhancement processing on said face image to obtain a face enhanced image; determining face keypoint data; and generating standard data. According to the embodiments of the present invention, a segmentation mask is obtained by means of a segmentation decoder, and the segmentation mask provides, for an image matting decoder, semantic information obtained by means of segmentation, such that the accuracy of an image matting result is higher, the portrait edge segmentation is cleaner, the quality and precision of standard data are improved, and human assistance information is not needed.

A method for controlling an artificial intelligence (AI) device for implementing a digital avatar can include receiving an audio signal corresponding to a user query, converting, by a speech-to-text neural network model, the audio signal into a text query, inputting the text query into a large language gesture instruction model to generate high level movement instructions, inputting the text query and the high level movement instructions into an information retrieval model to generate a text response including at least one sentence and digital avatar control information, and inputting the text response into a text-to-speech neural network model to generate an audio response. Also, the method can include inputting the audio response into an audio-to-facial animation model and an audio-to-conversational gesture model to generate updated digital avatar control information including gesture information, and outputting the audio response and the updated digital avatar control information for controlling the digital avatar.