7949results about "Animation" patented technology

The invention belongs to the technical field of visual special effects, and discloses an AI visual special effect dynamic generation system fusing multi-modal perception. Through cooperative work of six core modules of multi-modal input processing, importance analysis, parameter configuration, parameterized rendering, parameter optimization and rendering and output, deep fusion of special effects and contents is realized. An image sequence, audio data and scene parameters can be processed at the same time, a multi-modal feature data set is constructed, a scene key visual area is scientifically recognized, a visual importance distribution map is generated, and the special effect parameter configuration and rendering process is guided. An improved neural radiation field technology and a physical constraint model are adopted to ensure the consistency and reality of the special effect at different visual angles; and a multi-dimensional quality evaluation and parameter automatic optimization mechanism is introduced, so that the visual expressive force and the artistic value of the special effect are guaranteed. The creation threshold is remarkably reduced, the cooperative expression ability of the special effect and the content is improved, and the special effect becomes a powerful tool for enhancing the narration and enhancing the emotion.

The invention provides an intelligent real-time interactive question-answering system based on a virtual digital human, and belongs to the technical field of voice signal processing and voice recognition, and the system comprises a data acquisition module which receives a voice or text interaction request input by a user, collects the expression dynamic parameter sequence and limb movement sequence data of the user in real time, and transmits the data to a user interaction module; obtaining a standardized voice feature vector and structured text data; the cross-modal fusion module is used for constructing an interactive feature matrix; the behavior decision module outputs a decision instruction set; the knowledge retrieval module is used for generating an answer text with emotional adaptability and voice features; and the voice generation module is used for generating a mouth shape animation key frame, a micro expression parameter sequence and a limb action track of the virtual digital human, generating a voice response in combination with the answer text and the voice characteristics, and pushing the voice response to the user terminal. According to the method, the interaction experience and adaptability of the virtual digital human are remarkably improved.

The present invention relates to a digital twin processing method and system, and a cloud platform. The method comprises: acquiring production system elements, carrying out abstraction definition and parameterization description on the production system elements by means of digital-to-analog knowledge fusion of a mathematical model and a mechanistic model, so as to construct a digital twin ontology model; analyzing and reconstructing model data to obtain a mapping model of which object variables can be directly accessed and operated by a collective motion control method, so that the model is visualized at the cloud; and using an external data source to drive parameter update and operation matching of the model by means of a motion control method, so as to complete cooperative deployment and synchronous evolution of an actual physical device and the model in the production process on a cloud server. According to the present invention, a model is constructed by means of digital-to-analog knowledge fusion of a mathematical model and a mechanistic model, the model is mapped to achieve motion visualization, model parameter update and operation matching on the cloud are achieved, and then cooperative deployment and synchronous evolution of a physical device and the model are completed.

The invention discloses an intelligent multi-modal virtual digital human interaction system based on an AI language large model. The system comprises a high-authenticity face generation module; the high-authenticity face generation module uses an AdaAN network, based on adaptive feature fusion and voice driving and time sequence modeling of voice features, feature information related to voice is extracted, the extracted voice features are processed through a deep neural network, it is ensured that the voice and facial expressions are highly aligned in time and space, and the face recognition accuracy is improved. Collecting a bio-electricity signal, mapping the signal to facial muscle movement, generating a final facial expression, and interacting with a user; the system further comprises an intelligent interaction module, a training optimization and efficient generation module, an efficient integration module, a multi-modal data acquisition module, an AI large model core processing module, a digital human image generation and driving module, an interaction scene adaptation module and a feedback optimization module. The invention further discloses a multi-mode digital human interaction method which has wide application value.

An example operation may include one or more of receiving context of a user during an inquiry of a feature via a software application, executing a waiting period via the software application, during the waiting period, selecting an animation to display via the software application based on the context of the user and the feature inquiry wherein the animation provides contextual data associated with the feature, wherein the contextual data is based on a determined need of the user, displaying the animation via the software application during the waiting period, and determining if the user has accepted the feature via the software application. At least one portion of the example operation: integrates with an artificial intelligence (AI) chatbot, interacts with the AI chatbot, is performed by the AI chatbot, and / or is associated with an AI model.

The invention belongs to the technical field of virtual person generation, and particularly relates to a digital person video generation method based on a multi-modal large model, and the method comprises the following steps: 1, constructing a multi-modal data system; 2, multi-modal large model training and adaptation are carried out; 3, constructing a digital human three-dimensional model; step 4, performing semantic analysis and modal mapping; 5, generating a time sequence action and a mouth shape; step 6, building and rendering a virtual scene; step 7, audio and video synchronous rendering and synthesis; step 8, quality optimization and defect repair; and step 9, performing user interaction and iterative optimization. Through technical innovation and engineering, the core pain point in digital human video generation is solved, efficient, vivid and customizable content production capacity is provided for virtual anchors, intelligent customer service, enterprise training and other scenes, and the AI digital human technology is promoted to be applied to large-scale business from experiments.

Methods for displaying and organizing user interfaces in a computer-generated environment provide for an efficient and intuitive user experience. In some embodiments, user interfaces can be grouped together into a container. In some embodiments, user interfaces can be added to a container, removed from a container, or moved from one location in the container to another location. In some embodiments, a visual indication is displayed before a user interface is added to a container. In some embodiments, a user interface can replace an existing user interface in a container. In some embodiments, when moving a user interface in a computer-generated environment, the transparency of a user interface that is obscured can be modified.

Some embodiments of a method may include: obtaining Avatar JSON Interchange File (AJIF) data; decoding the AJIF data to generate avatar data; obtaining animation parameters from the avatar data; obtaining user movement data corresponding to a movement of a user; generating updated avatar data based on the user movement data; and sending the updated avatar data to a client device. For some embodiments, a method may include: generating Avatar JSON Interchange File (AJIF) data corresponding to an avatar; sending, to an application server, the AJIF data; obtaining user movement data corresponding to a movement of a user; sending, to the application server, the user movement data corresponding to the movement of the user; receiving, from the application server, scene update data, wherein the scene update data includes updated avatar data corresponding to the movement of the user; and rendering a scene update based on the scene update data.

A neural end-to-end system is provided for the face and voice preserving translation of videos. The system is a pipeline of multiple models that produces a video of the original speaker speaking in the target language with modified lip movement to match the target speech, while preserving emphases and prosody of the original speech, and voice characteristics of the original speaker. The pipeline starts with automatic speech recognition including emphasis detection, followed by the translation model. The translated text is then synthesized by a Text-to-Speech model that recreates the original emphases in the target sentence. The resulting synthetic speech is then converted back to the original speakers' voice using a voice conversion model. Finally, to synchronize the lips of the speaker with the translated audio, a generative model generates frames of adapted lip movements which are combined with the audio to produce the final output. The disclosure further describes several use-cases and configurations that apply these techniques to video conferencing, dubbing, low-bandwidth transmission, speech enhancement and assistive technology for the hearing impaired.

Provided in the embodiments of the present application are a video generation method and interaction method based on a digital human, and a device, a storage medium and a program product. In the embodiments of the present application, text-to-speech processing is performed on the basis of voice features of a user and an emotion label, speech-to-expression processing is performed on the basis of a mapping relationship between the voice features of the user and expression coefficients, and a digital human model is rendered on the basis of speech signals and the expression coefficients, so as to obtain video data of the digital human model. Thus, voice features of a user are accurately simulated, so as to ensure that a speech output of a digital human sounds natural and is also highly personalized, thereby realizing personalized driving of the digital human, and improving the realism of the digital human in terms of voice and dynamic images. Thus, the user experience is improved, and the interactivity of the digital human and the authenticity and immersion are enhanced.

The invention discloses an AI-based animation split script automatic generation and visual preview method and system, and the method comprises the following steps: 1, receiving a natural language script text inputted by a user, the natural language script text comprising scene description, role action, dialogue and shot indication information; step 2, performing semantic analysis and structured analysis on the script text based on a natural language processing technology, and identifying and extracting key narrative elements; by introducing an artificial intelligence technology, end-to-end automatic generation and interactive optimization from a character script to a dynamic split rehearsal video are realized, the system can deeply understand scenes, actions, role emotions and shot languages in the script, corresponding visual elements are automatically matched and generated, and the dynamic split rehearsal effect is improved. And the timeline and the rhythm conforming to the film and television grammar are constructed, so that the efficiency and the consistency of the split creation are greatly improved, and the professional threshold and the manufacturing cost are reduced.

The embodiment of the invention discloses an intelligent surveying and mapping method and system based on AI and BIM fusion. The method comprises the steps that an unmanned aerial vehicle platform carrying a laser radar, an RGB camera and a positioning system is used for scanning ancient building cultural relics and surroundings in a multi-angle flight mode, point cloud data, multi-view image data and position and attitude data are synchronously collected, and the three are associated through timestamps; after the point cloud data and the multi-view image data are preprocessed, cross-modal registration is completed through feature matching and pose estimation in combination with the position and pose data, and a registration data set is obtained; semantic segmentation is carried out on the point cloud data and the image data in the registration data set, and semantic segmentation results are fused based on the incidence relation; classifying and aggregating the original point cloud components according to category labels, constructing a topological relation reasoning assembly relation, calling corresponding BIM template instantiation model components based on the assembly relation, and hooking a segmentation result to generate a semantic enhanced BIM model; and integrating the BIM model and the GIS base map to form a fusion model so as to plot the historic building cultural relics.

The invention provides a game cartoon character display method based on a virtual reality technology. The method comprises the following steps: acquiring three-dimensional model data of a game cartoon character in a target display area through a user interaction terminal; the virtual reality content management platform determines a dynamic rendering precision level according to the model data, and generates a real-time rendering strategy including a model patch reduction coefficient, a texture compression rate and a skeleton animation updating frequency in combination with terminal performance parameters; the strategy is sent to a virtual reality supervision platform and a user interaction terminal, and a virtual reality rendering engine platform is instructed to execute real-time rendering; and the virtual reality supervision platform monitors the frame rate fluctuation data of the head-mounted display device, calculates the scene rendering stability, sends a rendering optimization instruction if the scene rendering stability is lower than a threshold value, and adjusts the model data acquisition frequency and the video memory cleaning period to optimize the performance. The rendering efficiency and the system stability can be improved, and the hardware load and the frame rate fluctuation are reduced.

The invention belongs to the field of digital delivery, and particularly relates to a digital delivery topology mapping method and system for multi-source real-time data fusion, and the method comprises the steps: obtaining factory building distribution, equipment distribution and operation control logic and preset function block operation logic, and constructing a hierarchical clustering function mapping space through combining an association analysis and clustering algorithm; in response to a target function demand, obtaining a layered response mapping path in combination with a deep search algorithm; layered synchronous response and distributed node anomaly monitoring are realized based on the path, the three-dimensional simulation model and the display system equipment performance and the network state. Tracing abnormities based on a monitoring result in combination with a hidden Markov algorithm and a forward reasoning model, performing iterative verification after conflict resolution until the function is free of abnormities, and updating a mapping space; and adjusting the demand repeating steps to obtain a complete and updated mapping space, and realizing accurate function and picture collaboration under multi-source data fusion.

Methods, systems, and computer programs are presented for audio-visual speech generation with multi-modal latent diffusion models. One method includes encoding raw audio signals and video frames into respective latent spaces using audio and visual autoencoders. A text transcript is processed into phoneme sequences using a text transcript processor. The audio and visual latent spaces are conditioned using the text transcript and a conditioning variable. Joint distributions of the visual and audio latent spaces, text transcript, and conditioning variable are learned using a multi-modal latent diffusion model. The model adds noise to the latent audio-visual representations and predicts the noise through denoising neural networks. An inverted diffusion process is utilized to generate diverse speech content and speaker characteristics, resulting in realistic audio-visual speech. The technology presented provides a novel approach to conditional speech generation with potential applications in speech synthesis, voice conversion, and speech recognition.

The invention provides a virtual human design and application platform, method and device based on artificial intelligence, and relates to the technical field of virtual digital humans. The method comprises the steps of performing local anonymization on multi-modal input data on user equipment, encoding generated anonymized multi-modal features to obtain a multi-modal feature vector, and inputting the multi-modal feature vector into an emotion calculation model to obtain a user emotion intensity quantized value; inputting the multi-modal feature vector into a context sensing model, and generating a user intention vector after context correction in combination with a knowledge graph; generating an updated personality parameter matrix according to the user emotion intensity quantized value and the user intention vector; and outputting voice waveform data, facial muscle motion parameters and skeleton joint coordinate data based on the personality parameter matrix, and driving the virtual digital human three-dimensional model to perform real-time rendering. According to the scheme, the naturalness, emotional resonance and long-term user retention rate of virtual digital human interaction can be improved, and user privacy data security is protected.

Certain aspects of the disclosure generally relate to devices, systems, applications, and / or objects of applications, and may be generally directed to artificial learning and / or use of artificial knowledge. Other aspects of the disclosure generally relate to consciousness, and may be generally directed to learning and / or implementing one or more purposes. One or more purposes may drive the use of artificial knowledge in implementing the one or more purposes. Therefore, in some aspects, a conscious device, system, application, and / or object of application may include one or more purposes and artificial knowledge so that the device, system, application, and / or object of application can act upon a world in implementing the one or more purposes. The disclosure also describes other functionalities.

The invention relates to the field of commodity visualization, in particular to a commodity display interaction visualization method and device. The method comprises the following steps: collecting a multi-azimuth image of a commodity, carrying out three-dimensional texture modeling, and constructing a three-dimensional texture mapping model; performing material light rendering on the three-dimensional texture mapping model to generate a material rendering result; collecting an environment detection image of a commodity display environment, and performing environment illumination adaptation compensation on a material rendering result to obtain an illumination compensation rendering commodity; carrying out attribute information visual layout on the illumination compensation rendering commodity to obtain a commodity visual space; and carrying out interaction response animation analysis according to the commodity visualization space, carrying out multi-target parallel rendering, and executing commodity interaction visualization operation. The form and surface details of the commodity in the real world are accurately restored, the visual reality sense is improved, and the interactive experience feeling of browsing the commodity by a user is enhanced.

Some embodiments of a method may include: obtaining a MPEG Avatar Representation Format (MARF)-based file; parsing the MARF-based file into a data structure; selecting an asset from the data structure; verifying the asset comprises a controller set property; obtaining controller set data corresponding to the controller set property; obtaining container data using the controller set data; determining a mime type using the controller set data; decoding the container data using the mime type; obtaining a mesh and a skeleton using the container data; and animating an avatar corresponding to the mesh and the skeleton.

The invention discloses a virtual-real fusion exhibition display interaction system and a multi-mode perception method, and belongs to the technical field of exhibition display interaction. The system collects audience eyeball fixation points, gesture actions and ambient light data through AR / VR equipment, analyzes coordinates of a region of interest through an eyeball fixation point attention mechanism, a gesture space-time encoder and a multi-modal fusion unit, triggers holographic projection explanation and virtual exhibition stand light and shadow dynamic adjustment (including illumination intensity, color, Gaussian blur and the like) based on a threshold value, and performs real-time display on the virtual exhibition stand. And multi-user collaborative interaction is realized through federal learning. According to the method, reinforcement learning is adopted to optimize an event-driven threshold value, and virtual and real visual splitting is eliminated in combination with ambient light adaptive mapping. The problems of low participation degree, insufficient single-mode interaction information and multi-user cooperation of traditional exhibition are solved, interest analysis accuracy is improved through multi-mode fusion, personalized experience is enhanced through dynamic interaction, the method is suitable for multiple scenes such as museums and science and technology museums, and exhibition intellectualization, immersion and group interaction efficiency are effectively improved.

The invention discloses a personalized digital human generation method based on a single video, and relates to the field of virtual digital human modeling and driving, and the method aims at the video and voice data of a target person, through introducing a multi-modal alignment constrained voice driving synchronization mechanism and combining semantic understanding and an emotion label expression generation model, a personalized digital human model is generated. High synchronization, nature and vividness of digital human facial expressions and voice contents are realized. According to the method, a self-supervised style consistency constraint is added in model training to ensure that the style of a generated character image is stable, and a 3D semantic mask is added in an image fusion stage to improve the fusion precision and realistic effect of a synthetic facial expression and a reference face. According to the method, the digital human can be rapidly cloned and driven to synthesize the expression through the voice only through a single video sample, the generation process is efficient, and the obtained digital human video has excellent sense of reality and interactivity.

The invention discloses a three-dimensional attitude high-precision acquisition and reconstruction system for exercise training, which relates to the technical field of three-dimensional attitude acquisition and reconstruction and comprises a camera calibration module, an image acquisition module, a key point extraction module, a three-dimensional reconstruction module, an attitude optimization module, a data fusion module and a feedback generation module. The camera calibration module is used for performing internal and external parameter calibration on a multi-view camera assembly by adopting a self-adaptive joint calibration method to obtain a projection matrix parameter and a transformation matrix of each camera; the image acquisition module is used for synchronously acquiring multi-view video images of the athlete in the training process based on the calibration parameters, and performing timestamp alignment processing to obtain an original image sequence; and the key point extraction module is used for detecting and identifying a human body image by adopting a deep learning model, extracting two-dimensional coordinates of key human body joint points of each frame from an original image sequence, and obtaining a two-dimensional key point set under each view angle.

The invention discloses a non-perpetual culture immersive interactive experience device and system based on a virtual digital human, and relates to the technical field of intangible cultural heritage intelligent interaction, and the device comprises a multi-mode sensing module which collects data such as user actions and voices through a depth camera and the like; the non-perpetual knowledge base module stores a non-perpetual knowledge graph and a case library; the digital human modeling engine fuses the inheritor characteristics and the user data to generate a virtual digital human; the immersion interaction engine constructs a cross-platform rendering environment to realize five-sense fusion experience; and the adaptive learning module analyzes the interaction sequence to optimize the digital human behavior, and all the modules cooperate to realize the intelligent interaction experience of non-genetic culture. Through the multi-modal perception module, the digital human modeling module and the like, the immersive interactive experience of the non-perpetual culture is realized, the user behavior can be accurately captured, the characteristic virtual digital human is generated, the five-sense fusion experience is provided, the content can be optimized based on the user interaction, the inheritance and propagation of the non-perpetual culture are promoted, and the participation degree and the sense of identity of the user are improved.

The present disclosure generally relates to creating and editing avatars, and navigating avatar selection interfaces. In some examples, an avatar feature user interface includes a plurality of feature options that can be customized to create an avatar. In some examples, different types of avatars can be managed for use in different applications. In some examples, an interface is provided for navigating types of avatars for an application.

In some embodiments, a computer system receives data representing a pose of at least a first portion of a user and causes presentation of an avatar that includes a respective avatar feature corresponding to the first portion of the user and presented having a variable display characteristic that is indicative of a certainty of the pose of the first portion of the user. In some embodiments, a computer system receives data indicating current activity of one or more users is activity of a first type and, in response, updates a representation of a user having a first appearance based on a first appearance template. The system receives second data indicating current activity of the one or more users and, in response, updates the appearance of the representation of the first user based on the current activity of the one or more users using the first or a second appearance template.

The invention relates to a multi-modal fusion real-time digital human driving method based on unified behavior vector mapping, and belongs to the technical field of human-computer interaction. The method comprises the following steps of: 1) extracting multi-modal features of voice, action and vision, and generating input features by adopting language feature-emotion decoupling, staged action modeling and macroscopic-micro expression flow analysis; (2) cross-modal collaborative alignment and conflict correction are carried out, and high-precision fusion is realized through knowledge graph constraint time sequence decomposition, particle-by-particle interactive fusion and a situation-driven modal dominant strategy; and 3) constructing a three-dimensional behavior vector space, mapping the multi-modal features to a unified coordinate by utilizing honeycomb grid projection, and driving the digital human to output in combination with a coordinate-action mapping table. According to the method, the problems of difficulty in multi-modal time sequence alignment, feature isomerism and out-of-control conflict are solved, real-time interaction of naturalization, emotional and scenarized is realized, and the expressive force and adaptability of the digital human are remarkably improved.

The invention relates to the technical field of subtitle translation, in particular to a multi-modal time sequence alignment AI video translation method and system, and the method comprises the steps: 1, carrying out the multi-modal analysis of a to-be-translated video, and obtaining audio separation data, voiceprint feature data and visual time sequence data; 2, performing cross-language translation and context optimization on the basis of the voice of the audio separation data to generate a target language text, and synthesizing target language voice retaining the original voice color in combination with the voiceprint feature data and the target language text; generating a mouth shape animation matched with the target language voice based on the lip key point data and the limb action time sequence data; and step 3, performing four-dimensional alignment on the target language voice, the translated text, the mouth shape animation and the limb action sequence through a cross-modal time sequence encoder, and dynamically adjusting the layout of the bilingual subtitles to adapt to a video picture. According to the method and the device, multi-mode synchronization can be taken into consideration during video translation, so that the body actions such as voice, subtitles and mouth shapes are kept aligned.

The invention discloses a multi-mode driven virtual digital human face animation generation method and system, and relates to the field of computer graphics, and the method comprises the steps: obtaining voice input and text input, and extracting voice features and text features; dynamically fusing the two features through an attention fusion model to generate facial expression and head posture control parameters, and dynamically adjusting contribution weights of a voice mode and a text mode to the control parameters by adopting a driving strategy of differentiation of facial upper expression and facial lower expression; and performing local deformation on the virtual digital human face image based on the control parameter to generate an initial animation frame, and performing refining processing on the initial animation by using a generative adversarial network to obtain a refined face animation. By means of the technical scheme, the natural and vivid virtual human face animation matched with the voice content and the text semantics can be generated.

The invention provides a dance video generation method based on multi-mode music driving and frequency domain-space double-flow decomposition, and the method comprises the steps: extracting multi-granularity music features through a composite encoder (Librosa + Jukebox), employing a beat gating attention mechanism, enabling the key actions such as dancing hand raising, kicking and the like to be strictly aligned with a music re-beat point, and enabling the synchronization error to be reduced to 118 ms through the verification of a test data set; for the problem of visual detail loss, a frequency domain-space double-flow decomposition architecture is provided, a Butterworth filter bank is used to decouple a reference image into a low-frequency energy diagram and a high-frequency residual error, and a double-flow diffusion mechanism is used to optimize a global attitude and local details respectively; a joint confidence prediction module is introduced for the generation stability in a shielding scene, and the motion trail of an abnormal joint point is dynamically corrected through a time domain sliding window weighted fusion strategy, so that a reasonable action conforming to ergonomics can still be generated under a 50% limb shielding rate.

The invention discloses an intelligent coordination control method and system for power plant equipment based on multi-source heterogeneous data, and belongs to the technical field of intelligent manufacturing and industrial automation, and the method comprises the steps: deploying a multi-mode sensor network in the power plant equipment, collecting the multi-source heterogeneous data in real time, and carrying out the real-time data preprocessing through an edge calculation node; carrying out collaborative modeling on the preprocessed data by adopting a hybrid analysis framework, predicting an equipment state trend, identifying a fault propagation path, positioning a root cause and optimizing a maintenance decision scheme; the equipment failure probability is evaluated through a fault diagnosis result, a grading early warning mechanism is triggered, and a rule base is updated and optimized in combination with a dynamic knowledge base; a three-dimensional model is constructed by using a digital twinning technology to carry out virtual simulation and remote control, and maintenance guidance is carried out through an augmented reality auxiliary technology. According to the method, efficient real-time monitoring and fault prediction are achieved, the fault diagnosis time and the operation and maintenance cost are remarkably reduced by combining the fault tree model and the digital twinning technology, and the equipment operation safety and reliability are improved.