The invention provides an intelligent real-time interactive question-answering system based on a virtual digital human, and belongs to the technical field of voice signalprocessing and voice recognition, and the system comprises a data acquisition module which receives a voice or text interaction request input by a user, collects the expression dynamic parameter sequence and limb movement sequence data of the user in real time, and transmits the data to a user interaction module; obtaining a standardized voice feature vector and structured text data; the cross-modal fusion module is used for constructing an interactive feature matrix; the behavior decision module outputs a decision instruction set; the knowledge retrieval module is used for generating an answer text with emotional adaptability and voice features; and the voice generation module is used for generating a mouth shapeanimationkey frame, a micro expression parameter sequence and a limb action track of the virtual digital human, generating a voice response in combination with the answer text and the voice characteristics, and pushing the voice response to the user terminal. According to the method, the interaction experience and adaptability of the virtual digital human are remarkably improved.
Various implementations disclosed herein include devices, systems, and methods that associate user activities with user interface (UI) elements within a 3D environment. Some implementations do so in ways that account for inaccuracies (i.e., fuzziness) in sensor-based detection of the user activities, e.g., inaccuracy in sensor data-based gaze tracking or sensor data-based hand / joint position. Some implementations use a sampling technique to associate user activity in a 3D environment with an appropriate portion of a UI positioned within a 3D environment. For example, a sampling technique may be used to identify sample locations within the 3D environment (e.g., sample locations around a gaze direction) to evaluate and ultimately select from to associate with a user activity, e.g., associating a particular UI button with a gaze direction.
In some embodiments, a computer system changes a visual prominence of a respective virtual object in response to detecting a threshold amount of overlap between a first virtual object and a second virtual object. In some embodiments, a computer system changes a visual prominence of a respective virtual object based on a change in spatial location of a first virtual object with respect to a second virtual object. In some embodiments, a computer system applies visual effects to representations of physical objects, virtual environments, and / or physical environments. In some embodiments, a computer system changes a visual prominence of a virtual object relative to a three-dimensional environment based on display of overlapping objects of different types in the three-dimensional environment. In some embodiments, a computer system changes a level of opacity of a first virtual object overlapping a second virtual object in response to movement of the first virtual object.
A knowledge fusion multi-modalinteraction method and apparatus based on an improved alignment method. The method comprises: acquiring multi-modal data (S101); constructing a query transformation model (S102); on the basis of a multi-streamfeature extraction method, performing improved processing on a vision transformer to obtain a multi-scale vision transformer (S103); on the basis of a joint text method, training the multi-scale vision transformer to obtain a target vision transformer (S104); freezing the target vision transformer, and on the basis of the target vision transformer, performing vision-text alignment training processing on the query transformation model to obtain a target query transformation model (S105); connecting the target query transformation model to a language model to obtain a knowledge fusion multi-modalinteraction model (S106); and inputting the multi-modal data into the knowledge fusion multi-modal interaction model to obtain an interaction result (S107). In the embodiments of the present invention, the knowledge fusion multi-modal interaction model aligns visual representation with text representation, so that the mutual information between the visual representation and the text representation can be maximized, and the present invention can be widely applied to the technical field of artificial intelligence.
In some embodiments, a computer system displays a navigation user interface that includes a first representation of a first location experience and a first portion of a navigation user interface element with a first orientation that includes a first representation of a first point of interest associated with the first location experience. In some embodiments, the computer system changes the display of the navigation user interface element from the first orientation to a second orientation corresponding to the first location experience. In some embodiments, the computer system displays a collection of content associated with a physical location in response to user input corresponding to a request to generate the collection of content associated with the physical location.
Some examples of the disclosure are directed to systems and methods for displaying and interacting with a representation of a user interface of an electronic device in a three-dimensional environment. While presenting a first user interface, a first electronic device presents a representation of a second user interface of a second electronic device based on display data representing the second user interface. In response to detecting a respective event corresponding to user input, if a gaze of a user of the first electronic device is directed to the representation of the second user interface, the first electronic device causes the second electronic device to perform a first operation directed to the second user interface based on the respective event. Alternatively, if the gaze is directed to the first user interface, the first electronic device performs a second operation directed to the first user interface based on the respective event.
Some examples of the disclosure are directed to systems and methods for changing a level of zoom of displayed content. In some examples, an electronic device displays visual content. In some examples, in response to detecting a change in position and / or orientation of the user of the electronic device, in accordance with a determination that the position and / or orientation of the user satisfies the one or more criteria, the electronic device increases the size of the content and / or zooms in on the content.
A computer system displays representations of second computer systems included in a communication session and / or content in an environment in some embodiments. For example, the computer system displays the representations of the second computer systems in a first region while content is not shared in the communication session and displays the representations of the second computer systems in a second region different from the first region concurrently with content that is shared in the communication session. In some embodiments, the computer system displays the representations of the second computer system (e.g., a representation of a shared content item) such that it partially occludes a representation of the one or more participants of the video communication session when the representation of the one or more participants is a non-spatial representation.
An extended reality environment includes one or more virtual objects. A virtual object can be selected for manipulation by orienting a pointing device towards the virtual object and performing a selection input. In some embodiments, if the selection input is a first type of selection gesture received at the pointing device, the virtual object is selected for movement operations. In some embodiments, if the selection input is a second type of selection gesture received at the pointing device, the virtual object is selected for rotation operations. While the virtual object is selected for movement or rotation operations, the virtual object moves or rotates in accordance with the movement or rotation of the user's hand, respectively. In some embodiments, the virtual object has a first visual characteristic when the pointing device is pointing towards the virtual object and has a second visual characteristic when the virtual object is selected.
Some examples of the disclosure are directed to systems and methods for scrolling a user interface element. A user interface element including a plurality of selectable options scrollable along a first axis can be presented in an environment by a device. In some examples, the device detects an input corresponding to a request to scroll the user interface element. In accordance with a determination that the input satisfies one or more first criteria, including a criterion that is satisfied when the input includes movement of the device that corresponds to movement of a head of a user that exceeds a threshold amount of movement over a period of time, the device scrolls the user interface element in a first mode of operation. In accordance with a determination that the input satisfies one or more second criteria, the device scrolls the user interface element in a second mode of operation.
A system and method for creating complex, immersive, and interactive digital content is disclosed. The system integrates advanced artificial intelligence, multi-modal input processing, cloud-based shared environments, and immersive hardware to generate, optimize, and deliver rich interactive experiences. The platform supports content mashups, custom scenario generation, and adaptive AI behaviors, enabling the creation of unique and engaging digital environments across various media formats.
Some examples of the disclosure are directed to systems and methods for communicating image sensor data to a computer system using one or more media links that allow for image sensor data to be displayed within a three-dimensional environment. In some examples, while presenting a three-dimensional environment that includes an electronic device that is communicatively coupled to an image sensor (e.g., a camera, or some other imaging device) within the three-dimensional environment, the electronic device utilizes a visual code displayed on the electronic device to establish a communication link with the image sensor receiving an indication to display image sensor data from the image sensor that is communicatively coupled to the electronic device. In some examples, while presenting a three-dimensional environment that includes a first pre-defined location within the three-dimensional environment, the electronic device facilitates the user to choose an image sensor to display at the pre-defined location.
In some embodiments, a computer system changes a visual prominence of a respective virtual object in response to detecting a threshold amount of overlap between a first virtual object and a second virtual object. In some embodiments, a computer system changes a visual prominence of a respective virtual object based on a change in spatial location of a first virtual object with respect to a second virtual object. In some embodiments, a computer system applies visual effects to representations of physical objects, virtual environments, and / or physical environments. In some embodiments, a computer system changes a visual prominence of a virtual object relative to a three-dimensional environment based on display of overlapping objects of different types in the three-dimensional environment. In some embodiments, a computer system changes a level of opacity of a first virtual object overlapping a second virtual object in response to movement of the first virtual object.
A vehicle dashcam may be configured to execute one or more neural networks (and / or other artificial intelligence), such as based on input from one or more of the cameras and / or other sensors associated with the dashcam, to intelligently detect safety events in real-time. Detection of a safety event may trigger an in-cab alert to make the driver aware of the safety risk. The dashcam may include logic for determining which asset data to transmit to a backend server in response to detection of a safety event, as well as which asset data to transmit to the backend server in response to analysis of sensor data that did not trigger a safety event. The asset data transmitted to the backend server may be further analyzed to determine if further alerts should be provided to the driver and / or to a safety manager.
The invention relates to the technical field of knowledge graph and virtual reality fusion, in particular to a knowledge association learning method and system based on a knowledge graph and virtual reality, and the method comprises the following steps: obtaining multi-source data including text data, image data, video data and three-dimensional model data; constructing an interdisciplinary knowledge graph based on the multi-source data; mapping the interdisciplinary knowledge graph into a virtual reality environment; generating a personalized learning path based on knowledge representation in a virtual reality environment; interactive knowledge exploration is realized in a virtual reality environment; updating the interdisciplinary knowledge graph based on the user interaction data; the virtual reality environment is synchronously updated to reflect the change of the interdisciplinary knowledge graph, and the understanding ability of a learner to complex concepts is greatly improved by visualizing an abstract knowledge structure in the virtual reality environment; and the learner can more easily find and understand the relation among different discipline knowledge.