An interaction method and device, an electronic device, and a medium

Abstract

The present disclosure provides an interaction method and device, electronic equipment and medium, and relates to the technical field of Internet of Vehicles. The method comprises: in response to obtaining material data, outputting a virtual object, the virtual object having a visual correlation with the material data; and in response to receiving a trigger event, generating response data of the virtual object based on the trigger event and context information, to control the virtual object to perform an interaction behavior corresponding to the trigger event. By generating a virtual object having a visual correlation with the material data, the problem of fixed image and inability to customize the virtual object is solved, and the connection between the user and the virtual object is improved. Meanwhile, the response data is generated in real time in response to the trigger event, the individualization, the scene and the real-time of the virtual object behavior are realized, and the naturalness and the immersion of the interaction are improved.

Description

Technical Field

[0001] This disclosure relates to the field of vehicle networking technology, and in particular to an interaction method and device, electronic device and medium. Background Technology

[0002] Vehicle-to-everything (V2X) technology is evolving from basic information services to intelligent cockpits, in-vehicle human-machine interaction, virtual objects, audio-visual generative interaction, and multimodal interaction, and human-machine interaction can be achieved through audio-visual generation. Summary of the Invention

[0003] This disclosure provides an interaction method, apparatus, electronic device, and medium, and provides a solution for achieving personalized human-computer interaction by generating personalized virtual objects.

[0004] The first aspect of this disclosure proposes an interaction method, including: in response to acquiring material data, outputting a virtual object, wherein the virtual object has a visual association with the material data; in response to receiving a trigger event, generating response data of the virtual object based on the trigger event and context information, so as to control the virtual object to perform the interactive behavior corresponding to the trigger event.

[0005] The disclosed solution generates visually related virtual objects by acquiring user materials, solving the problem of fixed images and lack of personalization of virtual objects, and enhancing the emotional connection between users and virtual objects. At the same time, it generates response data in real time based on trigger events and context information, enabling the interactive behavior of virtual objects to dynamically adapt to the current scene, avoiding the rigid interaction problem caused by preset animations and fixed responses, realizing the personalization, contextualization and real-time nature of virtual object behavior, and improving the naturalness and immersion of the interaction.

[0006] In this disclosure, outputting a virtual object in response to acquiring material data includes: generating a virtual object in response to acquiring material data; displaying the virtual object through a visual output component of an interactive device; and / or outputting the initial audio of the virtual object through an audio output component of an interactive device.

[0007] The disclosed solution enables virtual objects to establish initial interaction with users through visual and auditory dual channels after generation by using visual display of visual output components and initial audio output of audio output components. This enriches the presentation dimensions of virtual objects and enhances users' perception and acceptance of virtual objects.

[0008] In this disclosure, in response to obtaining material data, generating a virtual object includes: determining initial state parameters of the virtual object based on the material data; and determining the image features of the virtual object based on the initial state parameters, so as to generate the virtual object in a visual output component.

[0009] The disclosed solution determines the initial state parameters of the virtual object based on the material data, and generates image features based on the parameters, so that the virtual object presents an image in the visual output component that is consistent with the user-uploaded material, thereby enhancing the personalization and identity recognition of the virtual object, and enabling the user to continuously feel the connection with their own material in subsequent interactions.

[0010] In this disclosure, determining the initial state parameters of a virtual object based on source data includes: outputting at least one visual style through an interactive device; and, in response to a user triggering a target visual style among the at least one visual style, determining the initial state parameters of the virtual object based on the target visual style and the source data.

[0011] The disclosed solution presents users with multiple visual style options and responds to their selections, combining material data with the target visual style to generate the initial state parameters of virtual objects. While ensuring personalized images, it empowers users with active choices, enhancing the operability of the interaction and user participation. At the same time, different style selections allow the same material to generate diverse virtual images (i.e., images of virtual objects), expanding the applicable scenarios of the solution.

[0012] In this disclosure, in response to receiving a trigger event, response data of a virtual object is generated based on the trigger event and context information to control the virtual object to perform the interactive behavior corresponding to the trigger event, including: generating at least one of audio data, video data, and text data of the virtual object based on the trigger event and context information, wherein the response data includes at least one of audio data, video data, and text data; and controlling the virtual object to perform the interactive behavior corresponding to the trigger event through an interactive device based on the response data.

[0013] The disclosed solution generates at least one of audio data, video data, and text data as response data based on triggering events and contextual information, so that the interactive behavior of virtual objects is no longer limited to a single modality (such as only voice or only text), but can comprehensively convey information to users through multiple channels of hearing, vision, and text, thereby improving the richness of interaction and the efficiency of information transmission.

[0014] In this disclosure, based on response data, controlling a virtual object to perform interactive behaviors corresponding to a triggering event includes: based on video data, controlling the virtual object to perform interactive actions and / or present facial expressions in the visual output component of the interactive device, and outputting audio data through the audio output component of the interactive device; and / or outputting text data through the visual output component.

[0015] The disclosed solution drives virtual objects to perform interactive actions and / or display expressions in the visual output component through video data, while outputting audio data through the audio output component, realizing a multimodal interactive experience with synchronized audio and video. This makes the responses of virtual objects more vivid and natural, enhancing the user's sense of companionship and immersion. The output of text data provides users with information acquisition channels other than hearing, adapting to different usage scenarios.

[0016] In this disclosure, based on video data, a virtual object is controlled to perform interactive actions and / or present facial expressions in the visual output component of an interactive device, and audio data is output through the audio output component of the interactive device, including: based on video data, controlling the virtual object to perform interactive actions and / or present facial expressions in the visual output component from a first-person perspective; and outputting audio data of the virtual object from a first-person perspective through the audio output component.

[0017] The disclosed solution controls virtual objects to perform interactive actions and / or display facial expressions from a first-person perspective, and outputs audio data from a first-person perspective, enabling virtual objects to communicate with users as "I," significantly enhancing the anthropomorphism and emotional companionship of virtual objects. In scenarios involving interconnected home devices, virtual objects describe the current state of targets in the home in a first-person perspective, allowing users to feel the consistency of identity between virtual objects and physical entities in the home as the same entity. This solves the problem of distance caused by existing solutions that only provide objective notifications from a third-person perspective, and enhances the user's emotional connection and companionship experience.

[0018] In this disclosure, based on response data, controlling a virtual object to perform interactive behavior corresponding to a trigger event through an interactive device includes: outputting at least one of the following through the visual output component of the interactive device: the generation progress of response data; the target visual style; the status information of the virtual object performing interactive behavior; and the completion status of the trigger event, wherein the status information includes the action state and emotion of the virtual object.

[0019] The disclosed solution makes the previously invisible generation process transparent and visualized by displaying the generation progress, visual style, action state and emotion of virtual objects, and completion status of triggering events in real time in the visual output component. This allows users to clearly understand the current state of virtual objects and the system's processing progress, eliminating the uncertainty of waiting in a black box in traditional generational solutions and enhancing users' sense of control and trust in the interaction.

[0020] In this disclosure, in response to a user triggering a target visual style in at least one visual style, the initial state parameters of a virtual object are determined based on the target visual style and material data, including: performing verification processing on the material data; in response to the verification processing passing, determining the initial state parameters of the virtual object based on the target visual style and material data, and adding material identifiers to the initial state parameters.

[0021] The disclosed solution ensures the validity and security of the materials by verifying the material data, avoiding system anomalies caused by illegal or unusable materials entering the generation process. After successful verification, initial state parameters are generated and material identifiers are added, providing a data foundation for the continuous maintenance of virtual object identity characteristics and material traceability in subsequent interactions, thereby improving the stability and maintainability of the system.

[0022] In this disclosure, the triggering event includes at least one of the following: a touch operation on the interactive device; an instruction input through the interactive device; an update of state parameters associated with the mobile device, wherein the mobile device and the interactive device are associated; and a trigger signal from at least one terminal device associated with the mobile device.

[0023] The solution disclosed herein defines the triggering event as at least one of touch operation, input command, update of mobile device status parameters, and trigger signal of associated terminal device, so that the interactive behavior of virtual objects can be triggered by events from multiple sources, including user-initiated interaction and system-initiated interaction, thereby upgrading the interaction mode from passive response to active companionship and covering a wider range of application scenarios.

[0024] In this disclosure, in response to receiving a trigger event, response data for a virtual object is generated based on the trigger event and context information to control the virtual object to perform the interactive behavior corresponding to the trigger event, including: in response to receiving a trigger event, obtaining state data associated with the virtual object; generating response data based on the state data, the trigger event, and context information; and controlling the virtual object to perform the interactive behavior corresponding to the trigger event based on the response data.

[0025] The solution disclosed herein obtains the state data associated with virtual objects and uses the state data, trigger events, and context information together as the basis for generating response data. This enables the behavior of virtual objects to have continuity and consistency, avoids the discontinuity of role behavior caused by separate responses in each round of interaction, realizes the continuous output of virtual objects, and enhances the realism and coherence of the interaction.

[0026] In this disclosure, response data is generated based on state data, triggering events, and context information, including: generating response data through at least one generation strategy based on state data, triggering events, and context information, wherein the response data includes at least one of audio data, video data, and text data.

[0027] The solution disclosed herein generates response data by employing at least one generation strategy, making the generation of audio, video, and text data modular and scalable. Different generation strategies can be flexibly combined or replaced, providing an architectural foundation for subsequent system upgrades and model iterations, and avoiding the monotony of a generation mode bound to a single model.

[0028] In this disclosure, response data is generated based on state data, triggering events, and context information through at least one generation strategy, including: generating interaction planning data of a virtual object based on state data, triggering events, and context information through a planning generation strategy, wherein the interaction planning data includes at least one of text data, initial action labels, and emotion labels; and generating audio data and / or video data based on the interaction planning data through at least one response generation strategy, wherein the at least one generation strategy includes a planning generation strategy and at least one response generation strategy.

[0029] The disclosed solution first generates interaction planning data through a planning generation strategy, and then generates audio and / or video data based on the interaction planning data through a response generation strategy. This decouples response content planning from multimodal content generation, making the generation process structured and controllable. The introduction of interaction planning data provides clear semantic guidance for subsequent audio and video generation, improving the quality and consistency of the generated content and avoiding the uncontrollable content problem caused by direct end-to-end generation.

[0030] In this disclosure, audio data and / or video data are generated based on interaction planning data and through at least one response generation strategy, including: determining a target label through preprocessing based on initial action labels and / or emotion labels in the interaction planning data; determining a target response generation strategy in at least one response generation strategy based on the target label; and generating audio data and / or video data through the target response generation strategy based on at least one of the target label, text data in the interaction planning data, and image features of virtual objects.

[0031] The disclosed solution preprocesses the initial action tags and / or emotion tags in the interaction planning data and maps them to a preset set of target tags, thus constraining the freely generated action semantics within a stable and generateable range. This effectively reduces the risks of character identity drift, action distortion, and unstable images during video generation. Based on the target tags, a target generation strategy is determined, allowing different action types to be matched with the most suitable generation method, thereby improving the quality and efficiency of video generation.

[0032] In this disclosure, interaction planning data for virtual objects is generated based on state data, trigger events, and context information, including: responding to a touch operation on an interactive device in response to a trigger event, and determining initial action labels in the interaction planning data from a preset action label set based on state data, the touch operation corresponding to the trigger event, and context information.

[0033] When the triggering event is a touch operation, the solution disclosed herein determines the initial action label from the preset action label set based on state data, touch operation type, and context information. This makes touch actions such as patting the head, poking, and waving no longer simple instructions that trigger fixed animations, but rather semantic control signals that enter the interaction planning data generation process. Together with the current state and context, they determine the next round of actions and responses of the virtual object, realizing a generative response for touch interaction, which is different from the mechanical feedback mode where touch only triggers preset animations.

[0034] In this disclosure, audio data and / or video data are generated based on interactive planning data and through at least one response generation strategy, including: generating first audio data based on interactive planning data and through at least one response generation strategy according to a preset generation cycle; and generating first video data based on the first audio data and through at least one response generation strategy.

[0035] The disclosed solution generates first audio data according to a preset generation cycle, and then generates first video data based on the first audio data. This allows the audio data to be ready first, and the video data to be generated synchronously based on the audio data. This achieves the phased generation of audio and video, provides a data foundation for streaming playback, shortens the waiting time for users to perceive the first feedback from virtual objects, and improves the real-time performance and smoothness of the interaction.

[0036] In this disclosure, based on response data, controlling a virtual object to perform interactive behavior corresponding to a triggering event includes: outputting first audio data through the audio output component of the interactive device; and outputting first video data through the visual output component of the interactive device.

[0037] The disclosed solution immediately plays audio and video data through the audio output component and visual output component after a cycle of audio and video data is generated, realizing a synchronized audio and video streaming experience. Users do not need to wait for the complete data to be generated before they can hear the virtual object's response voice and actions, which significantly reduces the sense of waiting for interaction in the in-vehicle scenario and improves driving safety and user experience.

[0038] In this disclosure, the method further includes: pushing phased events to the interactive device based on the generation progress of response data, wherein the phased events are used to control the virtual object to perform interactive behavior through the interactive device; the phased events include at least one of the following: a visual style progress event; an interactive planning data planning event; a video generation submission event; a video generation progress event; a video generation completion event; an audio generation completion event; a response data storage event; a response data generation failure event; a preset event; and a trigger event completion status event.

[0039] The solution disclosed herein pushes phased events to the interactive device during the generation of response data, enabling the visual output component to update visualization information such as generation progress, planning status, and video generation status in real time, thus achieving full-link transparency of the generation process. At the same time, it supports storage and replacement of media after it is ready, providing a standardized interface foundation for the subsequent access of streaming generation models, thereby improving the scalability of the system and the user experience.

[0040] In this disclosure, the method further includes: in response to failure to generate response data, or in response to triggering a preset event, controlling the virtual object to perform a preset interactive behavior through an interactive device.

[0041] The solution disclosed herein controls the virtual object to perform preset interactive behaviors when response data generation fails or a preset event is triggered, providing the system with degradation and fault tolerance mechanisms. This ensures that basic interaction can still be maintained under abnormal conditions such as network instability, unavailability of model services, or insufficient computing power, avoiding user confusion and negative experience caused by the system being completely unresponsive, and improving the robustness and availability of the system.

[0042] In summary, the interaction method disclosed herein achieves the creation of personalized virtual companions by generating visually related virtual objects from user materials; by generating response data such as audio, video, and text in real time based on trigger events and contextual information, the interactive behavior of virtual objects differs from traditional preset animations and fixed responses, realizing real-time and personalized interactive content; by incorporating multiple trigger sources such as touch operations, device status updates, and signals from associated terminal devices into the generation basis, it achieves an upgrade from passive response to active companionship in the interaction mode; and by streaming generation and phased event push, it achieves visualization and rapid feedback of the generation process, reducing the feeling of waiting for interaction, improving the immersion and smoothness of the user experience, and enhancing the stability and scalability of the system.

[0043] A second aspect of this disclosure discloses an interactive device comprising: a first interactive module and a second interactive module. The first interactive module is configured to output a virtual object in response to acquiring material data, wherein the virtual object has a visual association with the material data. The second interactive module is configured to generate response data of the virtual object based on the trigger event and context information in response to receiving a trigger event, so as to control the virtual object to perform interactive behavior corresponding to the trigger event.

[0044] In this disclosure, the first interactive module is used to: generate a virtual object in response to acquiring material data; display the virtual object through the visual output component of the interactive device; and / or output the initial audio of the virtual object through the audio output component of the interactive device.

[0045] In this disclosure, the first interaction module is used to: determine the initial state parameters of the virtual object based on the material data; and determine the image features of the virtual object based on the initial state parameters, so as to generate the virtual object in the visual output component.

[0046] In this disclosure, the first interaction module is used to: output at least one visual style through an interaction device; and, in response to a user triggering a target visual style among the at least one visual style, determine the initial state parameters of a virtual object based on the target visual style and material data.

[0047] In this disclosure, the second interaction module is used to: generate at least one of audio data, video data, and text data of the virtual object based on the triggering event and context information, and the response data includes at least one of audio data, video data, and text data; and control the virtual object to perform the interactive behavior corresponding to the triggering event through the interaction device based on the response data.

[0048] In this disclosure, the second interaction module is used to: control a virtual object to perform interactive actions and / or present facial expressions in the visual output component of the interaction device based on video data, and output audio data through the audio output component of the interaction device; and / or output text data through the visual output component.

[0049] In this disclosure, the second interaction module is used to: control the virtual object to perform interactive actions and / or present expressions in the visual output component from a first-person perspective, based on video data; and output the audio data of the virtual object from a first-person perspective through the audio output component.

[0050] In this disclosure, the second interaction module is used to output at least one of the following through the visual output component of the interaction device: the generation progress of response data; the target visual style; the status information of the virtual object performing interactive behavior; and the completion status of the triggering event, wherein the status information includes the action status and emotion of the virtual object.

[0051] In this disclosure, the second interaction module is used to: perform verification processing on the material data; in response to the verification processing passing, determine the initial state parameters of the virtual object based on the target visual style and the material data, and add material identifiers to the initial state parameters.

[0052] In this disclosure, the triggering event includes at least one of the following: a touch operation on the interactive device; an instruction input through the interactive device; an update of state parameters associated with the mobile device; or a trigger signal from at least one terminal device associated with the mobile device.

[0053] In this disclosure, the second interaction module is used to: in response to receiving a trigger event, obtain state data associated with the virtual object; generate response data based on the state data, the trigger event, and context information; and control the virtual object to perform the interactive behavior corresponding to the trigger event based on the response data.

[0054] In this disclosure, the second interaction module is used to: generate response data based on state data, triggering events, and context information, through at least one generation strategy, wherein the response data includes at least one of audio data, video data, and text data.

[0055] In this disclosure, the second interaction module is used to: generate interaction planning data of a virtual object based on state data, triggering events, and context information, through a planning generation strategy, wherein the interaction planning data includes at least one of text data, initial action labels, and emotion labels; and generate audio data and / or video data based on the interaction planning data through at least one response generation strategy, wherein the at least one generation strategy includes a planning generation strategy and at least one response generation strategy.

[0056] In this disclosure, the second interaction module is used to: determine a target label based on the initial action label and / or emotion label in the interaction planning data through preprocessing; determine a target response generation strategy in at least one response generation strategy based on the target label; and generate audio data and / or video data based on at least one of the target label, text data in the interaction planning data, and image features of the virtual object through the target response generation strategy.

[0057] In this disclosure, the second interaction module is used to: in response to a trigger event being a touch operation on the interactive device, determine the initial action label in the interaction planning data from a preset action label set based on state data, the touch operation corresponding to the trigger event, and context information.

[0058] In this disclosure, the second interaction module is used to: generate first audio data according to a preset generation cycle, based on interaction planning data, and through at least one response generation strategy; and generate first video data based on the first audio data and through at least one response generation strategy.

[0059] In this disclosure, the second interactive module is used to: output first audio data through the audio output component of the interactive device; and output first video data through the visual output component of the interactive device.

[0060] In this disclosure, the second interaction module is used to: push phased events to the interaction device based on the generation progress of response data. The phased events are used to control the virtual object to perform interactive behavior through the interaction device. The phased events include at least one of the following: a visual style progress event; an interaction planning data planning event; a video generation submission event; a video generation progress event; a video generation completion event; an audio generation completion event; a response data storage event; a response data generation failure event; a preset event; and a trigger event completion status event.

[0061] In this disclosure, the second interaction module is used to: control the virtual object to perform a preset interactive behavior through the interaction device in response to the failure to generate response data or in response to the triggering of a preset event.

[0062] In summary, this device enables the creation of personalized virtual companions; it distinguishes the interactive behavior of virtual objects from traditional preset animations and fixed responses, achieving real-time and personalized interactive content; and it realizes the personalization, contextualization, and real-time nature of virtual object behavior, enhancing the naturalness and immersion of the interaction.

[0063] A third aspect of this disclosure provides an electronic device comprising: a processor and a memory for storing a computer program capable of running on the processor, wherein, when the processor is used to run the computer program, it performs the method described in any embodiment of the first aspect of this disclosure.

[0064] A fourth aspect of this disclosure provides a computer-readable storage medium storing computer instructions for causing a computer to perform the methods described in any embodiment of the first aspect of this disclosure.

[0065] The fifth aspect of this disclosure provides a program product including computer instructions for causing a computer to perform the methods described in any embodiment of the first aspect of this disclosure.

[0066] In summary, the interaction method proposed in this disclosure, in response to acquiring material data, outputs a virtual object, which has a visual association with the material data; in response to receiving a trigger event, it generates response data for the virtual object based on the trigger event and context information to control the virtual object to execute the interactive behavior corresponding to the trigger event. By acquiring user materials and generating a visually associated virtual object, the problem of fixed virtual object images and lack of personalization is solved, enhancing the emotional connection between the user and the virtual object; simultaneously, the real-time generation of response data based on trigger events and context information enables the interactive behavior of the virtual object to dynamically adapt to the current scene, avoiding the rigid interaction problems caused by preset animations and fixed responses, achieving personalization, contextualization, and real-time nature of virtual object behavior, and improving the naturalness and immersion of the interaction.

[0067] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0068] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0069] Figure 1 This is one of the flowcharts illustrating the interaction method in an embodiment of this disclosure; Figure 2 This is a second schematic flowchart illustrating the interaction method in an embodiment of this disclosure; Figure 3 This is a third schematic flowchart illustrating the interaction method in an embodiment of this disclosure; Figure 4A This is a typical application scenario for an interaction method; Figure 4B A system block diagram for an interaction method; Figure 4C This is a flowchart illustrating an interaction scheme. Figure 4D A flowchart illustrating the process of multi-device interaction; Figure 5 This is a schematic diagram of the structure of an interactive device according to an embodiment of the present disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0070] Embodiments of this disclosure are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting this disclosure.

[0071] In related technological fields, smart cockpits have gradually evolved from traditional instruments, central navigation, and voice assistants into comprehensive interactive spaces combining multi-screen linkage, voice interaction, personalized services, and emotional companionship. In-vehicle systems typically support voice control for functions such as navigation, music, messaging, phone calls, and air conditioning. Simultaneously, the introduction of virtual assistants, voice avatars, digital humans, or virtual pets enhances the sense of companionship within the cockpit. However, most related technical solutions focus on preset images, preset voices, and fixed function entry points, which have the following problems: Virtual images (or images of virtual objects) are mostly pre-set by manufacturers, making it difficult for users to customize in-vehicle virtual companions; voice assistants mainly output text or voice results, lacking integrated expressions such as personalized short videos, emotions, actions, and touch feedback generated based on the current system status; screen touch actions often only trigger local fixed animations or fixed responses, without truly entering the next round of multimodal generation links, making it difficult to generate actions and responses in real time based on current semantics, user pet identity, vehicle status, family status, and historical dialogues; although vehicle context can be used for navigation and reminders, it is rarely integrated with the virtual image's expressions, actions, voice messages, and video generation; generated content is usually a black box waiting for results, and most solutions require waiting for the complete video to be generated before playback, lacking the ability to generate and play simultaneously, provide rapid feedback on the first frame, have audio and video segments ready, and gradually replace media; although home cameras may recognize pets eating, resting, or approaching their food bowls, they usually only display alarms or video clips in mobile applications, failing to convert this state into digital analog image actions and first-person companion feedback on the in-vehicle screen.

[0072] Therefore, there is a need for a sustainable and iterative virtual companion interaction solution for mobile devices, which enables users to create virtual partners or virtual objects from personal materials, and allows text, voice, touch, vehicle context, family pet status and historical conversations to drive the virtual partner to generate personalized voiceovers, subtitles, emotions, actions and audio and video outputs in real time.

[0073] To address the aforementioned technical issues, this disclosure proposes an interaction method that generates personalized virtual objects from user-uploaded materials. This data, combined with triggering events and contextual information associated with the mobile device, generates response data to drive the virtual objects to perform corresponding interactive behaviors. This achieves a personalized, real-time feedback interaction scheme, enabling users to perceive and interact with the virtual objects through multiple senses, including sight and hearing. This method overcomes the shortcomings of templated and fixed interactions in related technical solutions.

[0074] The interaction method provided in this application will be described in detail below with reference to the accompanying drawings.

[0075] The interaction method proposed in this disclosure can be applied to the field of mobile devices, including but not limited to smart cars, robots, etc., for intelligent interaction on mobile devices, such as smart cockpits, in-vehicle human-machine interaction, virtual digital humans, virtual pets, audio-visual generative interaction, and multimodal interaction.

[0076] Optionally, the interaction method proposed in this disclosure can be executed by a mobile device, specifically, by an in-vehicle human-machine interaction system, enabling the mobile device to generate personalized virtual objects and interact with the user in a multimodal manner, thereby realizing the combination of smart cockpit and artificial intelligence / AI.

[0077] Optionally, the interaction method proposed in this disclosure can also be executed by a cloud server, or by a mobile device in collaboration with a cloud server.

[0078] Optionally, in the interaction method proposed in this disclosure, the mobile device can be interconnected with multiple terminals to exchange data through communication links, thereby achieving a collaborative in-vehicle human-machine interaction solution. For example, a smart cockpit can form a human-vehicle-home linkage scenario with devices such as home cameras, smart feeders, home central control systems, and mobile terminals.

[0079] The interactive method proposed in this disclosure is applied to mobile devices. The interactive device associated with the mobile device can be set on the mobile device or on other devices associated with the mobile device, and this disclosure does not limit this. The interactive device includes a visual output component and an audio output component. The visual output component is, for example, a screen or display, and the audio output component is, for example, a speaker or sound system, which are used to display the visual behavior of the virtual object and play the audio content of the virtual object, respectively.

[0080] For example, visual output components can be the in-vehicle cockpit central control screen, instrument cluster screen, rear entertainment screen, or vehicle-mounted simulation terminal of a smart cockpit.

[0081] The interactive method disclosed herein can generate a personalized virtual companion avatar based on user-uploaded materials, and generate virtual companion audio and video behaviors in real time based on text, voice, screen touch, vehicle context, and home camera / smart home signals.

[0082] The interactive method disclosed herein can be extended to the following areas: intelligent terminal companion interaction, such as virtual companion avatars in mobile phones, tablets, and home control screens; virtual home butlers or child companions in smart home scenarios; robots, digital human screen terminals, and in-vehicle rear entertainment screens; product scenarios that require user materials to generate characters and engage in continuous interaction, such as enterprise customer service, brand IP virtual avatars, and virtual characters in car owner communities; multimodal generative interactive systems that employ streaming audio and video generation, lightweight edge video generation, audio-driven video generation, frame-by-frame video generation, real-time web communication / transmission streaming, or cloud-based video generation services; and human-vehicle-home ecosystem linkage scenarios, such as synchronizing pet status from home cameras, smart feeders, door sensors, and temperature and humidity sensors to the vehicle cabin.

[0083] The interaction method proposed in this disclosure is based on user authorization for all operations related to user privacy or security, such as obtaining user voice, obtaining image information, taking screenshots, and granting operation authorization. It also strictly complies with relevant laws and regulations on privacy and security, and protects consumers' personal data privacy.

[0084] In some embodiments, the interaction method proposed in this disclosure can be applied to an interaction system, which includes a mobile device and a terminal. The terminal may be a mobile phone, tablet computer, smartwatch, IoT device, home camera, smart feeder, door and window management device (e.g., door magnet), temperature and humidity sensor, etc. The mobile device and the terminal can communicate with each other.

[0085] Optionally, the interactive system may also include a cloud server, which can communicate with mobile devices and terminals.

[0086] Figure 1 This is one of the flowcharts illustrating an interaction method in an embodiment of this disclosure. This method can be applied to the aforementioned interaction system, such as... Figure 1 As shown, the method includes the following steps.

[0087] Step 101: In response to obtaining the material data, output a virtual object.

[0088] In some embodiments, virtual objects have a visual association with material data.

[0089] In some embodiments, the material data may be user-provided raw data used to generate virtual objects, and may include image-type data and / or descriptive-type data capable of presenting the visual features of the target object.

[0090] In some embodiments, the material data may be images or videos uploaded by users to mobile devices, such as photos of people, cartoon characters, pet photos, brand images, etc. The material data may be a reference for the personalized image set by the user for the virtual object, so that the generated virtual object can form a visual association with the material data in terms of identity characteristics.

[0091] In some embodiments, the material data may be image data, such as a static image or video frame containing the visual appearance of the target object; or descriptive data, such as textual descriptions of the characteristics of the target object; or authorized data, such as authorized materials obtained from a user account, a material library, or a third-party platform.

[0092] For example, the source data is a pet photo. A user uploads a front-facing photo of their cat via the car's central control screen or a mobile terminal linked to the vehicle. This photo, as source data, contains visual information such as the pet's fur color, face shape, eye features, ear shape, body characteristics, and distinctive markings. The system can generate a virtual object based on this source data, which retains these identity features in subsequent interactions, allowing the user to recognize that the virtual object and the pet are the same entity.

[0093] For example, the source data consists of photos of people, such as a user uploading a front-facing photo of themselves or a family member. The system generates a virtual digital avatar with a visual association with the user's appearance based on the photo, which can be used for in-car companionship or navigation broadcasts.

[0094] For example, the material data can be a brand IP image. For instance, a user authorizes and selects a brand IP image from the brand image library as the material data, and the system generates a virtual in-vehicle object based on that IP image. Alternatively, the material data can be user-defined material. For example, a user uploads a hand-drawn character illustration, an anime character illustration, or a cartoon character image saved from a social media platform, and the system generates a corresponding virtual in-vehicle object based on that material.

[0095] In some embodiments, the source data can be images from multiple angles. For example, to generate a three-dimensional virtual object, a user can upload photos of people or pets, IP characters, cartoons, etc., from multiple angles, and the system can generate a three-dimensional virtual object.

[0096] In some embodiments, when images are unavailable, users can describe the characteristics of the target object through text, such as an orange shorthair cat, white chest fur, green eyes, and a triangular pattern on its left ear. The system then generates a corresponding virtual object based on the text description.

[0097] In some embodiments, the visual association between the virtual object and the material data can be that the visual features of the virtual object are the same as or similar to the visual features of the target object in the material data. For example, if a user uploads a photo of a pet cat, the virtual object generated by the system will be the image of that pet cat. Or if a user uploads a photo of a person, the virtual object generated by the system will be the image of that person. This is to enable users to visually perceive that the mobile device is a personalized virtual image generated based on the material data, rather than a fixed preset image, thereby improving the user's visual perception experience.

[0098] In some embodiments, the virtual object may be two-dimensional or three-dimensional, and may be determined according to user needs and the hardware configuration of the mobile device; however, this disclosure does not limit it.

[0099] In some embodiments, in response to acquiring material data, a virtual object can be output through an interactive device associated with the mobile device. In other words, when a user uploads material data via a mobile device, it can trigger the display of an image that has a visual relationship with the material data on the interactive device associated with the mobile device.

[0100] For example, the in-vehicle terminal or a mobile terminal linked to the vehicle receives image materials uploaded by the user. Uploaded materials can be in image formats such as JPEG, PNG, and WebP; they can also be authorized materials from the user's account material library, mobile phone photo album, car owner community, or brand image library.

[0101] In some embodiments, in response to acquiring material data, outputting a virtual object includes: generating a virtual object in response to acquiring material data; outputting the virtual object through a visual output component of the interactive device; and / or outputting the initial audio of the virtual object through an audio output component of the interactive device.

[0102] In some embodiments, when a user uploads material data to a mobile device, the mobile device can generate a corresponding virtual object in real time in response to the material data, thereby realizing the generation of a personalized virtual avatar based on the user.

[0103] In some embodiments, after generating a virtual object, the system can output the virtual object through the interactive device of a mobile device, allowing the user to perceive the generation effect of the virtual object. Specifically, this can be achieved by displaying the image of the virtual object through a visual output component; or by outputting preset initial audio for the virtual object through an audio output component; or by simultaneously displaying the image of the virtual object through a visual output component and outputting the initial audio through an audio output component.

[0104] In some embodiments, the initial audio may be pre-set audio output during the initial generation stage for the virtual object, used to inform the user that the virtual object has been generated and that they can interact with it subsequently; or, it may be used to remind the user that the virtual object has been generated, and the user can manually verify whether the virtual object meets the user's needs. If it does not meet the user's needs, the user can re-upload the material data, and the system can regenerate it.

[0105] In the above embodiments, through the visual display of the visual output component and the initial audio output of the audio output component, the virtual object can establish initial interaction with the user through both visual and auditory channels after it is generated, which enriches the presentation dimension of the virtual object and enhances the user's perception and acceptance of the virtual object.

[0106] In some embodiments, in response to acquiring material data, generating a virtual object includes: determining initial state parameters of the virtual object based on the material data; and determining the image features of the virtual object based on the initial state parameters, so as to generate the virtual object in a visual output component.

[0107] In some embodiments, determining the initial state parameters of a virtual object based on source data can be done by determining the initial state parameters of the virtual object based on the identity features or descriptions of the identity features in the source data. The initial state parameters can be a set of data used to define the basic attributes and visual presentation of the virtual object when it is first generated.

[0108] In some embodiments, the initial state parameters may include at least one of the following dimensions: Identity dimension: used to uniquely identify the correspondence between the virtual object and the material data, including source material identifier, material type tag, user identifier, etc.; Visual feature dimension: used to describe the main identity features of the virtual object, including the target object's category (cat / dog / person / cartoon, etc.), fur / skin color, face shape, facial features, body shape, distinctive patterns, texture, etc.; Style parameter dimension: used to define the visual presentation style of the virtual object, such as hand-drawn style, realistic style, 3D cartoon style, etc.; Initial state dimension: used to define the default state of the virtual object when it is first displayed, including initial emotion, such as calm or happy; initial action, such as waiting and breathing; initial orientation, etc.

[0109] In some embodiments, the visual features of a virtual object can be visual attribute data used to actually render and display the virtual object in the visual output component, including texture, color, shape, posture, animation skeleton, etc. The visual features are the specific representation of the initial state parameters in the visual output component, used to transform the abstract initial state parameters into displayable visual effects.

[0110] In some embodiments, the initial state parameters of the virtual object can be determined by feature extraction from the material data. Based on the initial state parameters, parameter transformation can be performed to convert the initial state parameters into executable image features. Based on the image features, the virtual object is generated in the visual output component, thereby achieving the effect of displaying the virtual image in the visual output component.

[0111] For example, the initial state parameters include: the subject is a "cat", the fur color is "orange", the face shape is "round", the eyes are "green", the markings are "white chest fur", and the style is "3D cartoon". The system inputs the above parameters into the character generation model to generate a 3D cartoon-style orange cat model: a round head, short orange fur texture, large green eyes, and a white halo effect on the chest fur. This 3D model is displayed as a virtual object in the visual output component and supports subsequent motion-driven and facial expression changes.

[0112] In the above embodiments, by determining the initial state parameters of the virtual object based on the material data and generating image features based on the parameters, the virtual object presents an image in the visual output component that is consistent with the user-uploaded material, thereby enhancing the personalization and identity recognition of the virtual object and enabling the user to continuously feel the connection with their own material in subsequent interactions.

[0113] In some embodiments, determining the initial state parameters of a virtual object based on source data includes: outputting at least one visual style through an interactive device; and, in response to a user triggering a target visual style among the at least one visual style, determining the initial state parameters of the virtual object based on the target visual style and the source data.

[0114] In some embodiments, when the system acquires material data, it can output at least one visual style through an interactive device to stylize the material data, thereby determining the initial state parameters of the virtual object.

[0115] In some embodiments, at least one visual style may be one or more visual styles determined according to the type of material data, or it may be a visual style configured in the full output system for users to make personalized choices.

[0116] In some embodiments, at least one visual style may be displayed as a style option via a visual output component of the interactive device, or the style may be output via an audio output component; alternatively, at least one visual style may be displayed via a visual output component and the style may be output via an audio output component. Users may determine a target visual style by touching the style option on the visual output component, or they may directly issue a voice instruction for the target visual style; this is not limited in this respect.

[0117] In some embodiments, the system can determine the initial state parameters of a virtual object based on the target visual style and material data triggered by the user, and the initial state parameters include the target visual style.

[0118] For example, the system displays multiple cockpit presentation style options to the user, such as healing hand-drawn, realistic, 3D cartoon, retro film, pixelated companion, and watercolor illustration. After the user selects a style, the system inputs the source materials and style parameters into the stylized image generation module to obtain a stylized virtual companion image and assigns it a stylized material identifier.

[0119] In the above embodiments, by presenting users with multiple visual style options and responding to their selections, the material data is combined with the target visual style to generate the initial state parameters of the virtual object. While ensuring a personalized image, it gives users the right to actively choose, which improves the operability of the interaction and the user's sense of participation. At the same time, different style selections allow the same material to generate diverse virtual images, expanding the applicable scenarios of the solution.

[0120] Step 102: In response to receiving a trigger event, based on the trigger event and context information, generate response data for the virtual object to control the virtual object to perform the interactive behavior corresponding to the trigger event.

[0121] In some embodiments, the triggering event can be an event that triggers interaction with the virtual object, and the triggering event can be customized according to the actual scenario or needs. For example, when a user performs a touch operation on a visual output component, or when a user issues a command, etc. The type of triggering event is not limited in this disclosure.

[0122] In some embodiments, receiving a triggering event can be a signal or data received by the system through an interactive device or other device associated with a mobile device, indicating that an event has occurred that requires a response from the virtual object. The triggering event can be an interactive behavior initiated by the user or a state change passively triggered by the system or environment.

[0123] In some embodiments, receiving a trigger event can be achieved by the system detecting the signal of a trigger event in real time, thereby determining the corresponding trigger event. It is understood that the system can determine the trigger event through active detection or passive reception, and this disclosure does not limit this.

[0124] In some embodiments, context information can be various types of information associated with the mobile device that describe the current scene state. Context information can be obtained from the mobile device itself or from its associated system, and the context information associated with the mobile device provides information about the specific generated scene for the virtual object's response.

[0125] In some embodiments, the context information may include: external environment information of the mobile device, such as weather conditions, temperature, and geographic location; internal status information of the mobile device, such as cabin temperature, driving speed, and remaining battery power; trip or task information associated with the mobile device, such as navigation routes, schedules, and destinations; and status information of other devices bound to the mobile device, such as the content of images recognized by home cameras and the working status of smart home devices.

[0126] In this embodiment, the system uses both the triggering event and context information as the basis for generation, which enables the virtual object's response to both the user's or system's event input and the current scene environment, thereby giving the response scene adaptability and personalized features.

[0127] In this embodiment, the response data can be single-modal response data or multi-modal response data, that is, it contains a variety of different information presentation forms, so as to convey information to the user from multiple sensory channels, including auditory channel (voice output), visual channel (dynamic picture output) and text channel (text output), etc.

[0128] In this embodiment, the system generates response data based on the triggering event and context information, rather than directly selecting a ready-made audio file, video file, or text template from a preset material library for playback. Instead, it creates this data dynamically and in real time based on the current triggering event and context information. Each generation is independent and specific to the current scenario; therefore, the same type of triggering event may generate completely different response data at different times and in different contexts.

[0129] In the above embodiments, in response to receiving a trigger event, the system can generate corresponding response data for the trigger event, so as to respond to the trigger event by controlling the virtual object to perform interactive behavior, thereby achieving the purpose of responding to the trigger event. This realizes a complete generation chain from the occurrence of the event to the virtual object's response, and this response is not preset or fixed, but generated in real time based on the specific content of the trigger event and the current scene context.

[0130] In some embodiments, in response to receiving a trigger event, response data of a virtual object is generated based on the trigger event and context information to control the virtual object to perform the interactive behavior corresponding to the trigger event, including: generating at least one of audio data, video data, and text data of the virtual object based on the trigger event and context information, wherein the response data includes at least one of audio data, video data, and text data; and controlling the virtual object to perform the interactive behavior corresponding to the trigger event through an interactive device based on the response data.

[0131] In some embodiments, based on triggering events and contextual information, the system can generate response data containing at least one of audio data, video data, and text data. This allows the system to control the virtual object to output at least one type of response data through an interactive device, enabling the user to perceive the interactive behavior performed by the virtual object through multiple sensory channels. In other words, the user can perceive the image of the virtual object through the output virtual object and perceive the virtual object's feedback to the triggering event through the response data.

[0132] In some embodiments, response data can be generated using a pre-trained model based on the triggering event and contextual information. For example, a single model can simultaneously generate audio, video, and text data, or different models can be used to generate different types of response data.

[0133] In some embodiments, the response data generated based on the triggering event and context information can be generated locally by the mobile device or sent to a cloud server, where it is generated and fed back to the mobile device; this disclosure does not limit the specific method. When generated by the mobile device, the model is deployed on the mobile device; when generated by the cloud server, the model is deployed on the cloud server, thereby reducing the memory footprint on the device side and the resource consumption of the generation process, achieving multimodal response data generation through edge-cloud collaboration.

[0134] In some embodiments, video data can be video frames, which are videos generated based on the visual characteristics of virtual objects, enabling visual perception of the virtual objects' actions, emotions, etc. Audio data can be audio streams, i.e., audio corresponding to the lip movements of the virtual objects in the video data. Text data can be text associated with the audio and video data, such as subtitles.

[0135] In some embodiments, audio and video data can be generated and output synchronously, such as a spoken segment of a virtual object, i.e., the behavior or content of a virtual object speaking. The spoken segment includes the voice content, i.e., the specific words spoken by the virtual object, such as "The weather is so nice today" or "We'll be there in 2 minutes"; and the voice expression, i.e., the speaking speed, tone, volume, and intonation, so that the voice not only conveys textual information but also conveys emotions and attitudes.

[0136] In some embodiments, based on response data, controlling the virtual object to perform interactive behavior corresponding to the trigger event through the interactive device can be that the system uses the generated response data as control instructions or playback content to drive the virtual object to produce specific, user-perceptible behavioral performance on the interactive device.

[0137] In some embodiments, the interactive behavior corresponding to the triggering event can be a semantically related behavior that matches the content of the triggering event. For example, if a user asks "How's the weather today?", the corresponding interactive behavior is for a virtual object to announce the weather information and display relevant expressions; if a user "pokes" a virtual object, the corresponding interactive behavior is for the virtual object to perform a reaction action and make a voice statement after being poked.

[0138] In this embodiment, the correspondence between interactive behaviors and triggering events is not a fixed mapping, but is determined in real time by the system based on the specific content during the generation process. In other words, the response data (audio, video, text, etc.) generated by the system is used to drive virtual objects, enabling the virtual objects to respond on the interactive device in the form of sound, images, and text, in accordance with the content of the triggering event.

[0139] In the above embodiments, at least one of audio data, video data, and text data is generated as response data based on the triggering event and context information, so that the interactive behavior of virtual objects is no longer limited to a single modality (such as only voice or only text), but can comprehensively convey information to users through multiple channels of hearing, vision and text, thereby improving the richness of interaction and the efficiency of information transmission.

[0140] In some embodiments, based on response data, controlling a virtual object to perform interactive behaviors corresponding to a triggering event through an interactive device includes: based on video data, controlling the virtual object to perform interactive actions and / or present facial expressions in the visual output component of the interactive device, and outputting audio data through the audio output component of the interactive device; and / or outputting text data through the visual output component.

[0141] In some embodiments, based on response data, virtual objects can be driven to perform corresponding interactive behaviors through interactive devices to respond to triggering events, thereby generating and feeding back interactions associated with the triggering events in real time.

[0142] In some embodiments, after the system generates video and audio data, it can control the virtual object to perform interactive actions and / or display facial expressions in the visual output component, and output audio data through the audio output component. That is, the user can both visually perceive the virtual object's actions or facial expressions and hear what the virtual object says, thereby achieving a vivid interactive experience.

[0143] In some embodiments, the system can generate text data while generating audio data, and output the text data through a visual output component. The text data and audio data have a corresponding relationship, so that the user can both hear what is said and see what is said.

[0144] In the above embodiments, video data drives virtual objects to perform interactive actions and / or display expressions in the visual output component, while audio data is output through the audio output component, realizing a multimodal interactive experience with synchronized audio and video. This makes the virtual object's responses more vivid and natural, enhancing the user's sense of companionship and immersion. The output of text data provides users with information acquisition methods other than hearing, adapting to different usage scenarios.

[0145] In some embodiments, based on video data, controlling a virtual object to perform interactive actions and / or present facial expressions in the visual output component of the interactive device, and outputting audio data through the audio output component of the interactive device, includes: based on video data, controlling a virtual object to perform interactive actions and / or present facial expressions in the visual output component from a first-person perspective; and outputting audio data of the virtual object from a first-person perspective through the audio output component.

[0146] In some embodiments, the first-person perspective can be expressed by a virtual object as "I" as the subject of discourse and action. During the interaction, the virtual object does not act as an objective describer, such as "He is eating" or "The cat is waiting for you at the door," but rather as the subject of action, such as "I am eating" or "I am waiting for you at the door."

[0147] In some embodiments, the first-person perspective can be a first-person perspective at the language level, that is, the use of first-person pronouns such as "I" and "my" in spoken text and subtitles; a first-person perspective at the behavioral level, that is, the actions performed and expressions presented by the virtual object are "its own actions" rather than "describing the actions of others"; or a first-person perspective at the identity level, that is, the virtual object acts as the subject of the action based on its own identity.

[0148] In some embodiments, the first-person perspective in video can be manifested in the following ways: Subjectivity of action, meaning the video content shows the virtual object itself performing actions. For example, in the video, the virtual object is "wagging its tail," "nodding," or "jumping," and the initiator and performer of the actions are both the virtual object itself; Subjectivity of expression, meaning the video content shows the virtual object's own facial expression changes. For example, in the video, the virtual object's own lips are upturned, and its own eyes are curved, and the presenter of the expression is the virtual object itself; Consistency of visual focus, meaning that in the video frame, the virtual object is always the core subject of the image, and its actions and expressions always revolve around "self," rather than showcasing the behavior or state of other objects.

[0149] In the above embodiments, by performing actions and presenting expressions from a first-person perspective, the virtual object visually presents the perception that I am an independent subject, allowing users to naturally understand the virtual object as a companion with initiative and independent personality when watching.

[0150] In some embodiments, the first-person perspective is specifically manifested at the audio level as follows: the audio content uses first-person pronouns such as "I," "my," and "we." For example, the audio content is "I'm eating," rather than "It's eating" or "Your pet is eating"; in the audio content, the virtual object always expresses itself as the speaker and actor, rather than as a relayer of objective information. For example, the audio content is "I'm waiting for you at the door," rather than "It / the pet is waiting for you at the door"; and the tone is consistent with the identity, meaning the tone of the audio content matches the identity characteristics of the virtual object. For example, for pet-type virtual objects, the audio tone can be lively and affectionate; for teacher-type virtual objects, the audio tone can be gentle and patient.

[0151] For example, the response planning module does not use a third-person perspective, as if it were the cat or another avatar, but instead generates first-person voice prompts, such as "I'm eating," "The food bowl is in good condition," "I'm going back to my bed to rest," "You can drive safely," or "My cat food is almost gone, remember to refill it when you get home." Meanwhile, the action control module maps family behavior categories to digitally simulated actions; for example, eating corresponds to looking down, chewing, and wagging its tail; resting corresponds to squinting, curling up, and breathing softly; and low food levels correspond to looking at the user and raising a paw as a reminder.

[0152] In the above embodiments, by controlling the virtual object to perform interactive actions and / or display facial expressions from a first-person perspective, and outputting audio data from a first-person perspective, the virtual object communicates with the user from the perspective of "I," significantly enhancing the anthropomorphism and emotional companionship of the virtual object. Especially in the scenario of home device linkage, the virtual object describes the current state of the target in the home from a first-person perspective, allowing the user to feel the consistency of identity between the virtual object and the physical objects in the home as the same entity. This solves the problem of distance caused by the existing solution's objective notification from a third-person perspective, and enhances the user's emotional connection and companionship experience.

[0153] In some embodiments, based on response data, controlling a virtual object to perform interactive behavior corresponding to a trigger event through an interactive device includes: outputting at least one of the following through the visual output component of the interactive device: the generation progress of response data; the target visual style; the state information of the virtual object performing interactive behavior; and the completion status of the trigger event, wherein the state information includes the action state and emotion of the virtual object.

[0154] In some embodiments, while generating response data, the system can output the interaction status in real time via an interactive device, such as the progress of response data generation. By displaying the generation progress in real time, users can understand that the system is generating the data, rather than facing a static, unresponsive interface, thereby reducing uncertainty and anxiety during the waiting process.

[0155] For example, a progress bar can be output through the vehicle's central control screen, such as displaying a progress bar that reads "Video generation in progress... A%", which gradually fills up as the generation progresses; or a stage reminder can be displayed, such as displaying stage text prompts like "Generating audio...", "Generating video...", "Coming soon"; or a status indicator can be displayed, such as displaying a status label like "Generating" or "Completed".

[0156] In the above embodiments, by making the originally invisible black-box generation process transparent and visible, users can clearly understand the system's processing progress, eliminating the uncertainty of waiting for the black box in related generative solutions, and enhancing users' sense of control and trust in interaction.

[0157] In some embodiments, outputting the target visual style through the visual output component can enhance the user's perception and confirmation of the virtual object style through visualization, thereby improving the user's sense of control and satisfaction with personalized settings.

[0158] In some embodiments, outputting status information of the virtual object's interactive behavior through a visual output component allows users to understand what the virtual object is doing and its current emotional state, thereby better understanding the virtual object's expressive intent and emotional attitude. For example, displaying action descriptions such as "wagging its tail," "nodding its head," or "jumping" in the interactive interface; or displaying the current action through dynamic icons or symbols, such as displaying status labels like "voicing," "standby," or "action being performed."

[0159] In some embodiments, the form in which emotions are displayed is not limited in this disclosure. It may be displaying emotional text labels such as "happy", "calm", "curious", "caring" in the interactive interface; or displaying emotional icons; or displaying emotional values; or displaying emotional animations.

[0160] In the above embodiments, by displaying the action state and emotions of virtual objects, users can perceive that virtual objects are not mechanical execution devices, but virtual images with dynamic states and emotional changes, thereby enhancing the realism of the interaction and emotional connection.

[0161] In some embodiments, the completion status of a triggered event output by the visual output component can indicate whether the interactive behavior corresponding to the current triggered event has been completed. When a user issues a triggered event, they need to know whether the system has finished processing it and whether the virtual object has responded. Feedback on the completion status allows the user to clearly understand the stage boundaries of the current interaction. The output format can be displaying status labels such as "Completed" or "Replied," or displaying a time indicator.

[0162] In the above embodiments, by displaying the generation progress, visual style, action state and emotion of virtual objects, and completion status of triggering events in real time in the visual output component, the originally invisible generation process is made transparent and visualized, enabling users to clearly understand the current state of virtual objects and the system processing progress. This eliminates the uncertainty of waiting in a black box in traditional generative solutions and enhances users' sense of control and trust in interaction.

[0163] In summary, the interaction method proposed in this disclosure solves the problem of fixed virtual images and lack of personalization by generating visually related virtual objects from user materials, thereby enhancing the emotional connection between users and virtual objects. At the same time, it generates response data in real time based on trigger events and contextual information, enabling the interactive behavior of virtual objects to dynamically adapt to the current scene. This avoids the rigid interaction problem caused by preset animations and fixed responses, and realizes the personalization, contextualization, and real-time nature of virtual object behavior, thereby improving the naturalness and immersion of the interaction.

[0164] Figure 2 This is a second schematic flowchart illustrating the interaction method in an embodiment of this disclosure, based on Figure 1 The embodiment shown, Figure 2 right Figure 1 Step 101 in the text will be further explained, such as Figure 2 As shown, the steps include the following.

[0165] Step 201: Verify the source data.

[0166] In some embodiments, during the process of generating virtual objects based on material data, the material data can be validated first to ensure that the material data uploaded by the user meets the system processing requirements, and to avoid system processing abnormalities, generation failures or security risks caused by material format incompatibility, excessively large files or non-compliant content.

[0167] In some embodiments, the verification process may include at least one of the following: format verification, size verification, security verification, integrity verification, etc. The type of verification process can be customized according to actual needs or scenarios, and this disclosure does not limit it.

[0168] In some embodiments, format verification can be used to check whether the file format of the source data is a system-supported format. This ensures that subsequent image decoding, feature extraction, and stylization processing can be performed correctly.

[0169] In this embodiment, supported formats include, but are not limited to, image formats such as JPEG, PNG, WebP, BMP, and GIF, and video formats such as MP4 and AVI.

[0170] In this embodiment, the system compares the format of the acquired material data with a preset set of formats: if the format of the material data is in the preset set of formats, the format verification is determined to be successful; if the format of the material data is not in the preset set of formats, the format verification is determined to be unsuccessful, and a prompt message indicating that the format is not supported is output through the interactive device.

[0171] In some embodiments, size verification can be used to check whether the file size and image dimensions of the source data exceed the upper limit of the system's processing capacity, so as to avoid problems such as excessively long upload times, excessive memory usage, model processing timeouts, or generation failures caused by excessively large files.

[0172] In this embodiment, the file size of the source data is compared with a preset size threshold. If the file size does not exceed the preset size threshold, the size verification is deemed successful; if the file size exceeds the preset size threshold, the size verification is deemed unsuccessful, and a message indicating that the file is too large is output through the interactive device.

[0173] In one implementation, while returning a prompt message, the system can also guide the user to compress the materials and re-upload them.

[0174] In some embodiments, security verification may be used to check whether the material data contains illegal, sensitive or malicious content, so as to ensure that the generated content of the virtual object complies with laws and regulations and public order and good morals, and to prevent malicious code or harmful content from being injected into the system through the material data.

[0175] In this embodiment, security verification may include at least one of the following methods: content security detection, malicious code detection, and facial / biometric compliance detection. If the security verification fails, a prompt may be issued to the user via an interactive device.

[0176] In one implementation, the system may also prompt the user to confirm that they have obtained authorization and consent from the relevant parties in the material when the user uploads the material data.

[0177] In some embodiments, integrity verification can be used to check whether the source data is complete or corrupted. For example, it can determine whether an image file is complete and readable by verifying the file's hash value or checking the file header information. If the image file is corrupted or incomplete, a message indicating that the source data is corrupted is output via an interactive device.

[0178] For example, after the vehicle-mounted terminal or the mobile terminal bound to the vehicle receives the image material uploaded by the user, the system performs format verification, size verification, and security verification on the image.

[0179] In the above embodiments, through verification processing, the system can filter out unqualified materials in advance before entering the stylization generation process, thereby improving the system's stability, security, and processing efficiency.

[0180] Step 202: In response to the successful verification process, the initial state parameters of the virtual object are determined based on the target visual style and material data, and material identifiers are added to the initial state parameters.

[0181] In some embodiments, if the verification process passes, it means that the material data can be used to generate virtual objects. The system determines the initial state parameters based on the target visual style selected by the user and the material data, and adds a material identifier. The material identifier is used to uniquely associate the virtual object with the material data uploaded by the user.

[0182] In some embodiments, identity features are extracted from the source data and fused with style parameters of the target visual style to generate structured initial state parameters. The initial state parameters can be a set of parameters stored in the form of structured data, including at least: source identifier, style identifier, identity features, and initial state (e.g., initial emotion: calm; initial action: standby breathing).

[0183] In the above embodiments, by verifying the material data, the validity and security of the materials are ensured, and system anomalies caused by illegal or unusable materials entering the generation process are avoided. After the verification is passed, initial state parameters are generated and material identifiers are added, which provides a data foundation for the continuous maintenance of the identity characteristics of virtual objects and the traceability of materials in subsequent interactions, thereby improving the stability and maintainability of the system.

[0184] Figure 3 This is the third flowchart illustrating the interaction method in this embodiment of the disclosure, based on Figures 1-2 The embodiment shown, Figure 3 right Figure 1 Step 102 in the text will be further explained, such as Figure 3 As shown, the steps include the following.

[0185] Step 301: In response to the detection of the triggering event, obtain the state data associated with the virtual object.

[0186] In some embodiments, the triggering event includes at least one of the following: a touch operation on the interactive device; an instruction input through the interactive device; an update of state parameters associated with the mobile device; or a trigger signal from at least one terminal device associated with the mobile device.

[0187] In some embodiments, touch operations on an interactive device can be user-triggered operations on a visual output component. For example, if options are displayed on the visual output component, and the user triggers one of the options with their finger or a touch device, the system detects the touch operation and then triggers subsequent state data acquisition and multimodal response data generation.

[0188] In some embodiments, the instructions output by the interactive device may be text instructions input by the user through the visual output component, or audio instructions input by the user through the audio acquisition component, or gesture instructions input through the visual acquisition component of the interactive device, etc. The specific instruction type is not limited in this disclosure and can be customized according to the hardware configuration and scenario of the mobile device.

[0189] In some embodiments, the update of the state parameters associated with the mobile device may be a change in the state parameters of the external environment in which the mobile device is located, such as changes in weather, changes in external ambient temperature, or the appearance of vehicles cutting in front of the driving route; or a change in the state parameters of the internal environment of the mobile device, such as changes in the cabin environment, changes in the route selected by the vehicle navigation system, changes in vehicle speed, or the approaching date set by the user.

[0190] In some embodiments, the trigger signal of at least one terminal device associated with the mobile device may be a trigger signal sent to the mobile device when the detection data of the terminal device reaches a trigger threshold set for the terminal device. The type of terminal device is not limited in this disclosure, nor is the trigger threshold set for the terminal device. The mobile device can interconnect with any communicable terminal device to receive the trigger signal sent by the terminal device.

[0191] In some embodiments, the trigger signal may be, for example, a home camera recognizing that a pet is eating or resting in its bed; or a feeder detecting that the pet's remaining food is low; or a home camera recognizing that a child is watching television or an elderly person in the home is suddenly unwell; or a smartwatch worn by a family member detecting a decline in that person's health. The type of trigger signal is not limited in this disclosure.

[0192] In some embodiments, the trigger signal is, for example, a home status packet. When the system receives the pet behavior recognition result from the home camera, the home ecosystem signal access module converts the raw event into a standardized home status packet. The home status packet may include device identifier, event time, pet identity, behavior category, confidence level, duration, location, and device description. For example, if the camera detects that a pet is near a food bowl and stays there, it can be interpreted as the pet eating; if the camera detects that a pet has entered a cat bed and remains still, it can be interpreted as the pet resting in the bed; if a smart feeder detects that there is not enough food left, it can be interpreted as the food being low.

[0193] For example, the system receives at least one interactive event from the user: text input, such as "Will there be traffic on the road today?"; voice input, which is text obtained after speech recognition; quick prompts, such as recommending in-car music based on the weather, or reminding me to be home by 8:45 PM; screen touch actions, such as clicking on the virtual companion, patting its head, poking it, waving, or feeding it; vehicle status change events, such as route updates, upcoming schedules, or changes in in-car temperature; home ecosystem signals, such as a home camera detecting that the pet is eating in its food bowl area, the pet returning to its bed to rest, the pet lingering at the door, or the smart feeder detecting that the remaining food is low; example signals for the current project include a camera showing the physical cat eating, a feeder signal indicating low food, or a camera showing the physical cat resting in its bed.

[0194] In some embodiments, the state data associated with the virtual object may be the state of the virtual object in the system at the moment the triggering event is detected, including at least one of the following: the virtual object's material identifier, action, emotion, visual style, and conversation identifier; and historical dialogue.

[0195] In some embodiments, by acquiring real-time state data of the virtual object upon detecting a triggering event, the real-time nature, validity, and continuity of subsequent response data generation can be ensured.

[0196] Step 302: Generate response data based on state data, triggering events, and context information.

[0197] In some embodiments, generating response data based on state data, triggering events, and context information includes: generating response data based on state data, triggering events, and context information through at least one generation strategy, wherein the response data includes at least one of audio data, video data, and text data.

[0198] In some embodiments, response data is generated based on state data, triggering events, and context information through at least one generation strategy. This can be achieved by packaging the state data, triggering events, and context information as input data to the generation strategy to obtain the output response data.

[0199] In some embodiments, at least one generation strategy may be at least one generation model, which may be a pre-trained model used to generate response data.

[0200] It should be noted that at least one generative model can be a single model or a combination of multiple models. This disclosure does not limit this, and includes, for example, large language models, rule-based models, graph-generated video models, speech-driven video models, cloud-based diffusion models, streaming video models, planning-based generative models, etc.

[0201] In the above embodiments, by employing at least one generation strategy to generate response data, the generation of audio, video and text data becomes modular and scalable. Different generation strategies can be flexibly combined or replaced, providing an architectural foundation for subsequent system upgrades and model iterations, and avoiding the monotony of a generation mode bound to a single model.

[0202] In some embodiments, response data is generated based on state data, triggering events, and context information through at least one generation strategy, including: generating interaction planning data of a virtual object based on state data, triggering events, and context information through a planning generation strategy, wherein the interaction planning data includes at least one of text data, initial action labels, and emotion labels; and generating audio data and / or video data based on the interaction planning data through at least one response generation strategy, wherein the at least one generation strategy includes a planning generation strategy and at least one response generation strategy.

[0203] In some embodiments, interactive planning data for virtual objects is generated based on state data, triggering events, and context information through a planning generation strategy. This can be achieved by using state data, triggering events, and context information as input to the planning generation strategy and outputting interactive planning data. The planning generation strategy can be a pre-trained planning generation model, which may be the same large model as the model that generates the response data; this disclosure does not limit its scope.

[0204] In some embodiments, interaction planning data can be a response plan or response scheme that guides the generation of response data. Interaction planning data is a planning description of what the virtual object should do and how it should do it, including text data, i.e. what to say; initial action labels, i.e. what action to take; emotion labels, i.e. what emotion to use, etc.

[0205] In some embodiments, the initial action tag may be a semantic tag that describes what action the virtual object should perform when performing this interaction.

[0206] In one implementation, the initial action tags include, but are not limited to, the following types: head actions: nodding, shaking head, tilting head, looking at the user, turning head, etc.; body actions: jumping, turning around, walking closer, backing away, bowing, waving, etc.; tail actions (for pets): wagging tail, tucking tail, wagging tail, etc.; mouth actions: mouthing, smiling with mouth closed, surprised with mouth open, etc.; combined actions: jumping + wagging tail, nodding + smiling, etc.

[0207] In some embodiments, the emotion tag may be a semantic tag used to describe what emotional state the virtual object should present when performing this interaction.

[0208] In one implementation, the emotion labels include, but are not limited to, the following types: calm, happy, curious, relaxed, excited, concerned, surprised, and serious.

[0209] For example, the response planning module generates a structured audio and video response plan in real time based on the context packet. This plan is not selected from a fixed response library, but rather generated using LLM in conjunction with the current system state. It can include: short responses suitable for virtual companions to speak; captions displayed on the cockpit screen; tone of voice, such as warm, playful, reminding, or surprised; emotions, such as calm, happy, curious, relaxed, or excited; action tags, such as speaking, nodding, wagging a tail, jumping, looking at the user, or subtle prompts; voice generation prompts or tone constraints; video generation visual cues; negative prompts to avoid identity drift, image distortion, aggressive expressions, etc.; and the duration of the current video or speaking segment.

[0210] In some embodiments, based on interactive planning data, audio data and / or video data are generated through at least one response generation strategy. This can be achieved by the system using at least one response generation strategy to perform the generation task, whereby the response generation strategy converts the interactive planning data into specific audio or video data.

[0211] In one implementation, the response generation strategy includes at least one of the following types: Audio generation strategy: used to convert text data in the interaction planning data into playable audio data. Image-based video generation strategy: used to combine the image features of virtual objects with action tags and emotion tags in the interaction planning data to generate dynamic videos containing virtual objects performing actions and / or displaying facial expressions. Audio-driven video generation strategy: used to input audio data into an audio-driven video model to generate lip-sync and facial expression videos synchronized with the audio. Real-time rendering strategy on the device: used to drive the skeletal animation and facial expressions of virtual objects in real time based on action tags and emotion tags in the interaction planning data using a 2D / 3D character engine on the device, generating dynamic visuals.

[0212] In the above embodiments, the two-stage architecture of planning generation strategy and response generation strategy realizes a complete generation chain from raw input to interaction planning and then to response output, solving the problems of the lack of personalization in preset response methods; the lack of intermediate planning process in single-stage generation methods, making it difficult to control the generation of actions and expressions; and single model binding. This makes the generation process controllable, interpretable, and scalable.

[0213] In the above embodiments, interactive planning data is first generated by planning the generation strategy, and then audio and / or video data is generated based on the interactive planning data through the response generation strategy. This decouples the response content planning from the multimodal content generation, making the generation process structured and controllable. The introduction of interactive planning data gives subsequent audio and video generation clear semantic guidance, improves the quality and consistency of the generated content, and avoids the uncontrollable content problem caused by direct end-to-end generation.

[0214] In some embodiments, audio data and / or video data are generated based on interaction planning data and through at least one response generation strategy, including: determining a target label through preprocessing based on initial action labels and / or emotion labels in the interaction planning data; determining a target response generation strategy in at least one response generation strategy based on the target label; and generating audio data and / or video data through the target response generation strategy based on at least one of the target label, text data in the interaction planning data, and image features of virtual objects.

[0215] In some embodiments, the system may preprocess the initial action labels and / or emotion labels to avoid situations such as duplicate generated labels or exceeding the controllable range of virtual objects, which could lead to the inability to generate multimodal response data.

[0216] In some embodiments, preprocessing may include cleaning, cropping, rule mapping, etc. The preprocessing method can be customized according to specific scenarios or needs, and this disclosure does not limit it.

[0217] In some embodiments, the target labels obtained after preprocessing are valid labels that can drive the response generation strategy to generate audio data or video data. Based on the target labels, a strategy for generating response data can be determined in at least one response generation strategy. This can be either determining a single target response strategy or determining target response generation strategies for generating video data and audio data separately, which is not limited in this disclosure.

[0218] In some embodiments, audio data and / or video data are generated based on at least one of the following: target labels, text data in interaction planning data, and visual features of virtual objects, through a target response generation strategy. This can be achieved by inputting the target labels, text data, and visual features into the target response generation strategy, which then simultaneously generates and outputs the audio and video data. Alternatively, the data can be generated in segments, or audio data can be generated first, followed by video data generation driven by the audio data, resulting in a synchronized correspondence between audio and video.

[0219] In the above embodiments, by preprocessing the initial action tags and / or emotion tags in the interaction planning data and mapping them to a preset target tag set, the freely generated action semantics are constrained within a stable and generateable range, effectively reducing the risks of character identity drift, action distortion, and unstable images during video generation; the target generation strategy is determined based on the target tags, so that different action types are adapted to the most suitable generation method, improving the quality and efficiency of video generation.

[0220] In some embodiments, generating interaction planning data for a virtual object based on state data, triggering events, and context information includes: in response to a touch operation on an interactive device caused by a triggering event, determining an initial action label in the interaction planning data based on state data, the touch operation corresponding to the triggering event, and context information from a preset action label set.

[0221] In some embodiments, the system can pre-set a set of preset action tags, which are associated with triggering events, state data, and context information. The system can select a set of initial action tags from the preset action tag set according to the association relationship as action tags in the interaction planning data.

[0222] In some embodiments, the action tags in the preset action tag set may be action tags corresponding to touch operations, action tags corresponding to input commands, action tags corresponding to the current state of virtual objects, etc., which will not be listed here, and the system can customize the settings.

[0223] For example, the system pre-sets a set of actions that can stably drive video or animation generation, such as: lip-syncing, breathing in standby mode, nodding, wagging tail, jumping, looking at the user, and giving a subtle prompt. If the current interaction is screen touch, the system prioritizes mapping action combinations based on the touch action. For example, if the in-vehicle screen displays touch options such as "tap," "pat head," "wave," and "feed," the user clicking "tap" is mapped to lip-syncing + jumping + wagging tail; the user clicking "pat head" is mapped to lip-syncing + nodding + wagging tail; the user clicking "wave" is mapped to lip-syncing + looking at the user + wagging tail; and the user clicking "feed" is mapped to lip-syncing + wagging tail + jumping.

[0224] For example, if the user input contains semantics such as route, going home, traffic jam, reminder, plan, etc., the system can add actions such as looking at the user, giving a light prompt, nodding, etc.

[0225] In some embodiments, the preset action tags can be used as degraded action tags when the network is abnormal, the model is unavailable, or the computing power is insufficient, or they can be used as skeletal animation templates, pose priors, or hybrid generated materials, etc., which are not limited in this disclosure.

[0226] In the above embodiments, when the triggering event is a touch operation, the initial action label is determined from the preset action label set based on the state data, touch operation type and context information. This makes touch actions such as patting the head, poking, waving and so on no longer simple instructions that trigger fixed animations, but rather semantic control signals that enter the process of generating interaction planning data. Together with the current state and context, they determine the next round of actions and responses of the virtual object, realizing a generative response of touch interaction, which is different from the mechanical feedback mode of touch that only triggers preset animations.

[0227] In some embodiments, based on interactive planning data, audio data and / or video data are generated through at least one response generation strategy, including: generating first audio data based on interactive planning data and through at least one response generation strategy according to a preset generation cycle; and generating first video data based on the first audio data and through at least one response generation strategy.

[0228] In some embodiments, the system can generate audio and video data in real-time streaming. For example, it can first generate text and audio data, and then generate the first frame, short video, or complete video data; or, it can generate audio streams and video frames simultaneously. In other words, the system can generate multimodal response data according to a preset generation cycle.

[0229] In some embodiments, the preset generation cycle can be a pre-set streaming generation cycle, that is, according to a fixed cycle, the data within one cycle is generated first and then pushed to the interactive device for output, while the data for the next cycle is generated. After the generation is completed, it is pushed to the interactive device for output. The difference is that in this embodiment, the data is generated and then output immediately, without waiting for the complete response data to be generated before outputting. This allows the front-end interactive device to play part of the content before the complete audio and video are generated.

[0230] In some embodiments, the method of generating the first audio data based on the interaction planning data and through at least one response generation strategy is as described in the above embodiments, and will not be repeated here; the generation of the first video data based on the first audio data and through at least one response generation strategy may be to generate corresponding video data according to the first audio data.

[0231] For example, audio generation can take one of the following approaches: generating a video with native audio tracks using a video generation module; generating spoken audio using a standalone TTS module and playing it simultaneously with the video; preserving subtitles and a silent video when audio is unavailable; or directly generating lip-synced video using an audio-driven video generation module. Video generation can take one of the following approaches: generating short videos based on stylized images and video cues; generating audio-driven videos based on stylized images, speech, and action tags; segmented video generation based on the current frame, action, and context; outputting virtual companion actions, lip movements, and audio segment by segment or frame by frame using a streaming audio and video generation module; rendering in real-time using a 2D / 3D character engine on the device based on action tags, then overlaying spoken audio generated from the cloud; or driving digital avatar actions based on pet behavior recognition results from a home camera, such as eating, resting, approaching the door, or exhibiting unusual activity.

[0232] In the above embodiments, first audio data is generated according to a preset generation cycle, and then first video data is generated based on the first audio data. This allows the audio data to be ready first, and the video data to be generated synchronously based on the audio data. This realizes the phased generation of audio and video, provides a data foundation for streaming playback, shortens the waiting time for users to perceive the first feedback from virtual objects, and improves the real-time performance and smoothness of the interaction.

[0233] Step 303: Based on the response data, control the virtual object to execute the interactive behavior corresponding to the triggered event.

[0234] In some embodiments, the system can generate complete response data before controlling the virtual object to perform corresponding interactive behaviors according to the response data. The specific process can be as follows: Figure 1 As shown in step 102.

[0235] In some embodiments, based on response data, controlling a virtual object to perform interactive behavior corresponding to a triggering event includes: outputting first audio data through the audio output component of the interactive device; and outputting first video data through the visual output component of the interactive device.

[0236] In some embodiments, the system can control the virtual object to perform corresponding interactive behaviors after generating response data within a preset generation cycle; after generating response data for the next cycle, it can control the virtual object to perform the next set of interactive behaviors, and so on, to achieve real-time streaming generation of audio and video clips.

[0237] In some embodiments, audio and video synchronization of virtual objects can be achieved by simultaneously outputting audio and video data, meaning that users can see the virtual object speaking and hear what the virtual object is saying.

[0238] In the above embodiments, audio and video data are played immediately after a cycle of data generation through the audio output component and the visual output component, achieving a synchronized audio and video streaming experience. Users do not need to wait for the complete data generation to hear the virtual object's response voice and actions, significantly reducing the sense of waiting in the in-vehicle scenario and improving driving safety and user experience.

[0239] In some embodiments, the method further includes: pushing phased events to the interactive device based on the generation progress of response data, wherein the phased events are used to control the virtual object to perform interactive behavior through the interactive device; the phased events include at least one of the following: a visual style progress event; an interactive planning data planning event; a video generation submission event; a video generation progress event; a video generation completion event; an audio generation completion event; a response data storage event; a response data generation failure event; a preset event; and a trigger event completion status event.

[0240] In some embodiments, during the generation of response data, the system may push phased events to the interactive device based on the generation progress, so as to trigger the virtual objects in the interactive device to execute the corresponding phased events.

[0241] In some embodiments, phased events can be used to update output data in the interactive device, that is, by pushing phased events, virtual objects are triggered to perform interactive behaviors at each stage.

[0242] In some embodiments, the progress event of the visual style can be a progress of a virtual object generated based on the target visual style and material data.

[0243] In some embodiments, the planning events for interactive planning data can be the content and progress that generate the interactive planning data.

[0244] In some embodiments, a video generation submission event can be an event that triggers a response generation strategy to generate video data; a video generation progress event can be the progress of video data generation in response to the generation strategy; a video generation completion event can be the completion of video data generation; an audio generation completion event can be the completion of audio data generation; a response data storage event can be the storage status of the generated response data, such as existing stored audio / video data; a response data generation failure event can be the failure of response data generation, thereby triggering the interactive device to output a failure prompt or other interactive behavior; a preset event can be an event type that the system detects; and a trigger event completion status event can be the completion of the interactive behavior of the virtual object triggering the event.

[0245] In some embodiments, the preset event can be a pre-defined event that triggers the system to execute a preset interactive behavior, such as a network error, service unavailability, operation timeout, insufficient computing power, security events, etc., which are not limited in this disclosure. The system pre-defines the corresponding multimodal response data under the preset event to control the virtual object to execute the corresponding interactive behavior.

[0246] In some embodiments, the storage event of response data can be that the system stores the response data in local memory, and the data generated first can be pushed first, so that the interactive device can output local data, thereby improving the smoothness of data output.

[0247] For example, the system pushes phased events to the front end during the generation process. These include: stylized progress events; audio / video response plan events; video task submission events; video generation progress events; video ready events; audio ready events; media saving events; generation failure or degradation events; and task completion events. The cockpit stage updates subtitles, moods, action states, progress bars, video URLs, audio URLs, audio on / off switches, generation logs, frame numbers, and delays based on these events. If the remote video returns first, the front end can immediately play it; if the back end subsequently completes local saving, it switches to the local playback address via the media ready event.

[0248] In the above embodiments, by pushing phased events to the interactive device during the generation of response data, the visual output component can update visualization information such as generation progress, planning status, and video generation status in real time, realizing full-link transparency of the generation process; at the same time, it supports storage and replacement after media is ready, providing a standardized interface foundation for the subsequent access of streaming generation models, and improving the scalability of the system and user experience.

[0249] In some embodiments, the method further includes: in response to a failure to generate response data, or in response to triggering a preset event, controlling the virtual object to perform a preset interactive behavior through an interactive device.

[0250] In some embodiments, the preset interactive behavior can be an interactive behavior controlled by pre-set audio data and / or video data, used to control the virtual object to perform the interactive behavior in the event of data generation failure or triggering a preset event, so as to prevent the user from being unable to perceive the current interactive state and to prevent the system from entering an unresponsive state.

[0251] For example, preset interactive behaviors could include displaying a message like "Network connection error, I'm trying to restore..."; controlling the virtual object to output preset voice messages like "My signal isn't very good, but I'm still here"; displaying a progress status like "Reconnecting..."; controlling the virtual object to display concerned emotions and expressions; controlling the virtual object to output preset voice messages like "It's okay, I'll wait with you for a while"; controlling the virtual object to perform gentle, reassuring actions; displaying a "Retry?" or "Try again later" selection button; controlling the virtual object to output alternative suggestions like "Why don't you listen to some music first?"; stopping video playback and switching to audio-only mode; and outputting safety reminders like "Focus on driving."

[0252] For example, if a session fails, the system can rebuild the session based on the user's current context and retain or restore the available virtual partner state, allowing the user to continue interacting without having to re-upload materials.

[0253] For example, in a network interruption scenario, the system detects "response data generation failed" and displays the subtitle "The network signal is weak, I cannot display the screen for the time being, but I can still chat with you" on the interactive interface; it plays the preset voice message "The signal is not very good, but I can still talk to you" through the audio output module; and it controls the virtual object to remain in standby mode and continue to be displayed on the interactive interface.

[0254] For example, in a scenario where the user actively stops the process, the system detects a "preset event" (user-initiated stop event), immediately interrupts the current response data generation process, controls the virtual object to stop the currently generated video and audio content, outputs the preset voice message "Okay, call me again if needed," and the virtual object returns to standby mode.

[0255] In the above embodiments, when response data generation fails or a preset event is triggered, the virtual object is controlled to perform preset interactive behavior, providing a degradation and fault tolerance mechanism for the system. This ensures that basic interaction can still be maintained under abnormal circumstances such as network instability, unavailability of model services, or insufficient computing power, avoiding user confusion and negative experience caused by the system being completely unresponsive, and improving the robustness and availability of the system.

[0256] In the above embodiments, by acquiring the state data associated with the virtual object and using the state data, triggering events, and context information together as the basis for generating response data, the behavior of the virtual object is made continuous and consistent, avoiding the discontinuity of role behavior caused by separate responses in each round of interaction, realizing the continuous output of the virtual object, and improving the realism and coherence of the interaction.

[0257] In summary, the interaction method proposed in this disclosure achieves the creation of personalized virtual companions by generating visually related virtual objects from user materials; it distinguishes the interactive behavior of virtual objects from traditional preset animations and fixed responses by generating response data such as audio, video, and text in real time based on trigger events and contextual information, thus realizing real-time and personalized interactive content; by incorporating multiple trigger sources such as touch operations, device status updates, and signals from associated terminal devices into the generation basis, it upgrades the interaction mode from passive response to active companionship; and it achieves visualization and rapid feedback of the generation process through streaming generation and phased event push, reducing the feeling of waiting for interaction, improving the immersion and smoothness of the user experience, and enhancing the stability and scalability of the system.

[0258] based on Figures 1-3 The embodiments shown below provide a specific implementation method, and the execution method shown in the following implementation method can be used as... Figures 1-3 One specific implementation method of the embodiment shown is described.

[0259] In this embodiment, an application scenario is as follows: a user uploads a personal image, such as a pet photo, brand image, authorized image of a family member, cartoon character, or user-defined companion image, while the vehicle is stationary or in a safe condition. The system then generates a virtual companion image suitable for the cabin display based on the user's selected style. Subsequently, this virtual companion acts as an in-vehicle companion, responding to the driver's or passenger's text input, voice input, and screen touch actions.

[0260] The system can obtain contextual information such as weather, location, schedule, cabin temperature, speed, and route from the vehicle or in-vehicle services. It can also receive pet behavior recognition results from home cameras, smart feeders, or home IoT platforms. During each interaction, the system integrates user input, touch actions, vehicle context, pet status, current virtual companion appearance, and historical dialogue status to stream and generate the next segment of the virtual companion's speech, subtitles, emotions, actions, audio stream, video frames, and audio-visual content in real time. This content is then generated and played simultaneously or in segments on the in-vehicle screen.

[0261] This embodiment can be deployed as an in-vehicle application or implemented using an in-vehicle user interface + cloud-based generation service. In subsequent implementations, it is necessary to achieve lower first-frame latency, more stable identity retention, stronger real-time interaction, and more natural audio-visual synchronization.

[0262] Figure 4A This is a typical application scenario for an interactive method. Users create a virtual companion and initiate interaction through the in-vehicle central control screen. The vehicle context service provides information related to the driving scenario, and the virtual companion interaction generation system outputs audio and video behaviors facing the cockpit screen, which are ultimately displayed on the virtual companion stage.

[0263] Figure 4B This is a system block diagram for an interaction method, which may include the following modules: **Material Access Module:** Receives image materials uploaded by users and generates source material identifiers. Materials can be pets, characters, cartoon characters, brand IPs, or other user-authorized images. **Stylized Image Generation Module:** Transforms the source materials into stylized virtual companion images based on the user-selected cockpit presentation style. Styles can include hand-drawn, realistic, 3D, retro, pixel, watercolor, etc., and can be expanded by the car manufacturer or user. **Cockpit Stage Display Module:** Displays the virtual companion's image, video, audio controls, emotions, and action states on the in-vehicle screen. **Multimodal Input Module:** Receives text input, voice recognition results, shortcut prompts, screen clicks, stroking, poking, waving, feeding, and other touch actions. **Vehicle Context Access Module:** Obtains scene information such as weather, temperature, location, schedule, cockpit temperature, vehicle speed, and route. Home Ecosystem Signal Access Module: This module receives events from home IoT devices such as home cameras, smart feeders, door and window sensors, and temperature and humidity sensors. In pet scenarios, it receives structured statuses such as pet eating, pet resting in its bed, pet approaching the door, and low food levels in the food bowl. Virtual Companion Status Management Module: This module maintains source material identifiers, stylized material identifiers, selected styles, subject identity characteristics, current mood, current action, recent touch actions, historical dialogues, family pet status, media generation status, and current session identifier. Response Planning Module: This module generates structured audio and video response plans based on user input, touch actions, vehicle context, family pet status, historical dialogues, media status, and virtual companion status. The response plan includes at least personalized spoken text, subtitles, tone, mood, action tags, voice prompts, visual cues, and duration. Action Control Module: This module restricts freely generated action semantics to a preset action set and combines touch actions, user / pet identity, current system status, or route reminders to generate stable action combinations in real time, rather than simply playing fixed actions by pressing a button. The streaming audio and video generation module generates the next virtual partner video, audio stream, or audio / video segment based on stylized appearance, actions, emotions, narration meaning, and visual cues. The event stream and media management module pushes events such as stylization progress, response plans, first audio package, video tasks, video progress, first frame ready, segment ready, video ready, audio ready, media saving, and media replacement to the front end, and manages media resources such as source images, stylized images, audio, video, streaming segments, and local cache. The degradation and fault tolerance module maintains basic interaction when the model service is unavailable, generation fails, or the network is interrupted, such as preserving the original image, playing local animations, displaying subtitles, logging, or using simulated responses.

[0264] Figure 4C A flowchart illustrating an interaction scheme, such as... Figure 4C As shown, the steps include the following.

[0265] S1: Receive user-uploaded materials and establish source material records. The vehicle-mounted terminal or a mobile terminal bound to the vehicle receives image materials uploaded by the user. The system performs format verification, size verification, and security verification on the images. After passing these checks, a source material identifier is assigned to the image. The source material identifier is used for subsequent stylization, video generation, and historical image management.

[0266] In one implementation, the uploaded materials can be image formats such as JPEG, PNG, and WebP; or they can be authorized materials from user account material libraries, mobile phone photo albums, car owner communities, or brand image libraries.

[0267] S2: Receives the cockpit presentation style and generates a stylized virtual companion avatar. The system displays multiple cockpit presentation style options to the user, such as soothing hand-drawn, realistic, 3D cartoon, retro film, pixelated companion, and watercolor illustration. After the user selects a style, the system inputs the source materials and style parameters into the stylized avatar generation module to obtain a stylized virtual companion image and assigns it a stylized material identifier.

[0268] The stylization result of this step is not an isolated image, but rather it is registered as the identity basis for subsequent in-vehicle virtual companion interactions and directly enters the cockpit video stage. The system can further save the subject's identity feature descriptions, such as: orange cat, white chest, round eyes, short fur, pattern on the left ear, etc., so that subsequent voice-over videos and action generation remain consistent with the user's own pet.

[0269] S3: Initialize the virtual companion state and cockpit video stage. The system displays a stylized virtual companion avatar on the cockpit screen and initializes state objects. State objects may include: source material identifier; stylized material identifier; selected style; current emotion; current action; recent touch action; conversation identifier; recent rounds of dialogue history; and current media playback status. The cockpit video stage displays the current avatar or video, along with the current style, generation progress, action status, frame number, delay, and emotion reading, allowing the user to observe the virtual companion's generation and interaction state.

[0270] S4: Receive multimodal interaction events. The system receives at least one interaction event from the user: text input, such as "Will there be traffic on the road today?"; voice input, which is text obtained after speech recognition; quick prompts, such as recommending in-car music based on the weather, or reminding me to be home by 8:45 PM; screen touch actions, such as clicking on the virtual companion, patting its head, tapping it, waving, or feeding it; vehicle status change events, such as route updates, upcoming schedules, or changes in in-car temperature; and home ecosystem signals, such as a home camera detecting that the pet is eating in its food bowl area, the pet is resting in its bed, the pet is lingering at the door, or the smart feeder detects that the remaining food is low.

[0271] For touch actions, the system records the action identifier, action label, suggested emotion, and suggested action. For example, a tap corresponds to excitement and actions such as jumping and wagging the tail; a pat on the head corresponds to happiness and actions such as nodding and gently wagging the tail.

[0272] S5: Integrating Vehicle Context and Virtual Partner Status. The system acquires the current vehicle context, including weather, temperature, location, schedule, cabin temperature, speed, and route; simultaneously, it reads the current status of the virtual partner and recent dialogue history. The system generates a context package for this round of generation, which includes at least: user input or touch events; interaction type; screen action information; current virtual partner avatar status; current vehicle context; and a summary of recent dialogues. Through this context package, the virtual partner can reflect information such as current location, weather, schedule, route reminders, in-vehicle status, what the user just touched or tapped, what the pet is doing at home, and what was discussed in the previous round in its responses, instead of generating generic responses unrelated to the cabin or calling pre-set scripts.

[0273] S6: Generate a structured audio and video response plan. The response planning module generates a structured audio and video response plan in real time based on the context packet. This plan is not selected from a fixed response library, but is generated using LLM in combination with the current system state. It can include: short responses suitable for virtual partners to speak; subtitles displayed on the cockpit screen; tone of voice, such as warm, playful, reminder, surprise; emotions, such as calm, happy, curious, relaxed, excited; action tags, such as speaking, nodding, wagging tail, jumping, looking at the user, and subtle prompts; voice generation prompts or tone constraints; video generation visual cues; negative prompts to avoid identity drift, image distortion, aggressive expressions, etc.; and the duration of the current video or speaking segment.

[0274] In this embodiment, the response planning module can be implemented using rules, traditional natural language processing models, large language models, multimodal models, or edge-cloud collaborative models.

[0275] S7: Map action semantics to stable action labels. The action control module cleans, trims, and maps action labels in the response plan according to rules. The system presets a set of actions that can stably drive video or animation generation, such as: lip-syncing, standby breathing, nodding, tail wagging, jumping, looking at the user, and subtle prompts. If the current interaction is screen touch, the system prioritizes mapping action combinations based on touch actions. For example: tap: lip-syncing + jump + tail wagging; pat head: lip-syncing + nodding + tail wagging; wave: lip-syncing + looking at the user + tail wagging; feed: lip-syncing + tail wagging + jump.

[0276] If user input includes semantics such as route, going home, traffic jam, reminder, or plan, the system can add actions such as looking at the user, giving a subtle prompt, or nodding. This mechanism ensures that actions respond to semantics while remaining within a stable and generateable range, reducing the risk of video instability and character identity drift. After each interaction, the system instantly calculates action tags, action combinations, action duration, tone, and visual cues based on the current input, context, historical state, and character identity, then drives subsequent video generation, streaming generation, or animation compositing. Pre-set action clips can be used as degraded material in case of network anomalies, model unavailability, or insufficient computing power, or as skeletal animation templates, pose priors, or hybrid generation materials.

[0277] In one implementation, motion generation can be achieved using any of the following methods or combinations thereof: outputting motion tags from a large language model or rule model, then inputting them into a graph-generated video model to generate short clips; using a voice-driven video model to synchronously generate lip movements and head actions based on the spoken audio; using an edge-side skeletal animation or Live2D / 3D rig controller to drive the character in real time based on motion tags; generating video frames segment by segment using a cloud-based diffusion model, Transformer video model, or streaming video model; mixing preset basic motion clips with real-time generated spoken audio, facial expressions, subtitles, particle effects, and background states; mapping the physical pet's motion categories recognized by the camera to the vehicle's digital avatar motion, and then combining this with real-time spoken audio to generate video.

[0278] S7A: Converts the pet status from a home camera into a digital analog image in the car.

[0279] When the system receives pet behavior recognition results from a home camera, the home ecosystem signal access module converts the raw event into a standardized home status package. This package can include device identification, event time, pet identity, behavior category, confidence level, duration, location, and a textual description of the device. For example, if a camera detects a pet approaching and remaining near a food bowl, it can be interpreted as the pet eating; if a camera detects a pet entering a cat bed and remaining still, it can be interpreted as the pet resting in its bed; if a smart feeder detects insufficient food, it can be interpreted as the food level being low.

[0280] The system binds the aforementioned family status package to the status of the in-vehicle virtual companion, making the virtual companion on the car's screen a digital representation of the physical pet inside the vehicle. Subsequently, the response planning module does not use a third-person perspective (like the cat or another clone) to provide prompts, but instead generates first-person voice prompts, such as "I'm eating," "The food bowl is normal," "I'm going back to my bed to rest," "You can drive safely," or "My cat food is almost gone, remember to give me some more when you get home." Simultaneously, the action control module maps family behavior categories to digitally simulated actions; for example, eating corresponds to lowering the head, chewing, and wagging the tail; resting corresponds to squinting, curling up, and breathing softly; and low food levels correspond to looking at the user and raising a paw as a reminder.

[0281] This step transforms the results of home security cameras capturing pet activity from simple notifications into a digitally simulated image, real-time voiceover, subtitles, emotions, and actions visible on the in-vehicle screen. This transformation can rely on existing pet behavior recognition capabilities on the camera itself, or it can involve secondary recognition of screenshots, short video clips, detection boxes, and event tags in the cloud or on-vehicle.

[0282] S8: Real-time streaming generation of audio and video clips. The system generates audio and video in real-time streaming based on the response plan and action tags. The system can first generate spoken text and an initial audio packet, then generate the first frame, short clips, or a complete video; alternatively, it can simultaneously generate audio streams and video frames, allowing the front-end to play partial content before the complete media is generated. Audio generation can be achieved using one of the following methods: generating video with native audio tracks using the video generation module; generating spoken audio using a separate TTS module, then playing it simultaneously with the video; retaining subtitles and a silent video when audio is unavailable; or, in subsequent implementations, having the audio drive the video generation module to directly generate lip-synced video.

[0283] Video generation can be achieved in one of the following ways: generating short videos based on stylized images and video cues; generating audio-driven videos based on stylized images, speech, and action tags; segmented video generation based on the current frame, action, and context; outputting virtual companion actions, lip movements, and audio segment by segment or frame by frame by a streaming audio and video generation module; rendering in real-time by a 2D / 3D character engine on the device based on action tags, and then overlaying the spoken audio generated by the cloud; or driving digital avatar actions based on pet behavior recognition results from a home camera, such as eating, resting, approaching the door, or exhibiting abnormal behavior.

[0284] S9: Pushes generation phases and updates the cockpit stage via event stream. The system pushes phased events to the front end during the generation process. Examples include: stylization progress events; audio / video response plan events; video task submission events; video generation progress events; video ready events; audio ready events; media saving; generation failure or degradation events; and task completion events.

[0285] The cockpit stage updates subtitles, emotions, action status, progress bar, video URL, audio URL, audio on / off, log generation, frame number, and delay based on events. If the remote video returns first, the frontend can play the remote video immediately; if the backend subsequently completes local saving, it switches to the local playback address via a media ready event.

[0286] S10: Update conversation history and virtual partner status. After this round of interaction, the system writes user input, virtual partner's verbal responses, emotions, actions, media results, etc., into the conversation history and updates the current virtual partner status. The next round of interaction will continue to use these statuses, thus forming a continuous companionship experience.

[0287] If a session fails, the system can rebuild the session based on the user's current context and retain or restore the available virtual partner status, allowing the user to continue interacting without having to re-upload materials.

[0288] Figure 4D A flowchart illustrating multi-device interaction, such as... Figure 4D As shown, the in-vehicle user uploads images / voice messages / touchscreens / views home signal to the in-vehicle front-end; the in-vehicle front-end submits interaction requests and status to the back-end service; home devices report to the back-end service that the camera has detected eating / resting / low food reserves, etc.; the back-end service integrates vehicle context, home status, and history to push a response plan and action tags to the in-vehicle front-end; the back-end service submits real-time actions / voiceover / video generation to the generation module; the generation module returns remote video or streaming clips to the back-end service; the back-end service pushes the video to the in-vehicle front-end for playback; the back-end service downloads and saves the media to media storage; and the media storage returns the local playback address to the back-end service.

[0289] In summary, the solution proposed in this embodiment embodies a three-tiered mechanism: personalized avatar, personalized audio and video content, and non-pre-built actions and responses. The virtual companion avatar is not a pre-set character uniformly provided by the manufacturer, but rather has a perceptible connection with user-uploaded materials. Each time a response is generated, the system not only reads the user's input for that round but also considers the overall system state to achieve personalized audio and video content. After each round of interaction, the system combines or generates spoken text, subtitles, actions, emotions, audio, and video in real time based on the current state; pre-built materials can only be used as degraded resources, basic action templates, gesture priors, or safety fallbacks. This allows the same touch action to trigger different results in real time due to different current system states, reflecting its non-pre-built and personalized nature. It enhances media playback robustness, supporting multiple path combinations such as native video audio tracks, independent TTS audio, muted video subtitles, priority playback of remote video, and subsequent replacement of local media, improving usability under different generation conditions.

[0290] Figure 5 This is a schematic diagram of the structure of an interactive device 500 according to an embodiment of this disclosure. Figure 5 As shown, the device includes: a first interaction module 510 and a second interaction module 520.

[0291] The first interaction module is used to respond to the acquired material data and output a virtual object, which has a visual relationship with the material data; the second interaction module is used to respond to the received trigger event and generate response data for the virtual object based on the trigger event and context information, so as to control the virtual object to perform the interactive behavior corresponding to the trigger event.

[0292] In some embodiments, the first interaction module is configured to: generate a virtual object in response to acquiring material data; display the virtual object through the visual output component of the interaction device; and / or output the initial audio of the virtual object through the audio output component of the interaction device.

[0293] In some embodiments, the first interaction module is used to: determine the initial state parameters of the virtual object based on the material data; and determine the image features of the virtual object based on the initial state parameters, so as to generate the virtual object in the visual output component.

[0294] In some embodiments, the first interaction module is configured to: output at least one visual style through an interaction device; and, in response to a user triggering a target visual style among the at least one visual style, determine the initial state parameters of the virtual object based on the target visual style and material data.

[0295] In some embodiments, the second interaction module is configured to: generate at least one of audio data, video data, and text data of the virtual object based on the triggering event and context information, wherein the response data includes at least one of audio data, video data, and text data; and control the virtual object to perform the interactive behavior corresponding to the triggering event through the interaction device based on the response data.

[0296] In some embodiments, the second interaction module is used to: control a virtual object to perform interactive actions and / or present facial expressions in the visual output component of the interaction device based on video data, and output audio data through the audio output component of the interaction device; and / or output text data through the visual output component.

[0297] In some embodiments, the second interaction module is used to: control the virtual object to perform interactive actions and / or present expressions in the visual output component from a first-person perspective, based on video data; and output the audio data of the virtual object from a first-person perspective through the audio output component.

[0298] In some embodiments, the second interaction module is configured to: output at least one of the following through the visual output component of the interaction device: the generation progress of response data; the target visual style; the status information of the virtual object performing interactive behavior; and the completion status of the triggering event, wherein the status information includes the action status and emotion of the virtual object.

[0299] In some embodiments, the second interaction module is used to: perform verification processing on the material data; in response to the verification processing passing, determine the initial state parameters of the virtual object based on the target visual style and the material data, and add material identifiers to the initial state parameters.

[0300] In some embodiments, the triggering event includes at least one of the following: a touch operation on the interactive device; an instruction input through the interactive device; an update of state parameters associated with the mobile device; or a trigger signal from at least one terminal device associated with the mobile device.

[0301] In some embodiments, the second interaction module is configured to: in response to receiving a trigger event, obtain state data associated with the virtual object; generate response data based on the state data, the trigger event, and context information; and control the virtual object to perform the interactive behavior corresponding to the trigger event based on the response data.

[0302] In some embodiments, the second interaction module is used to: generate response data based on state data, triggering events, and context information, through at least one generation strategy, wherein the response data includes at least one of audio data, video data, and text data.

[0303] In some embodiments, the second interaction module is configured to: generate interaction planning data for a virtual object based on state data, triggering events, and context information, through a planning generation strategy, wherein the interaction planning data includes at least one of text data, initial action labels, and emotion labels; and generate audio data and / or video data based on the interaction planning data through at least one response generation strategy, wherein the at least one generation strategy includes a planning generation strategy and at least one response generation strategy.

[0304] In some embodiments, the second interaction module is configured to: determine a target label based on initial action labels and / or emotion labels in the interaction planning data through preprocessing; determine a target response generation strategy in at least one response generation strategy based on the target label; and generate audio data and / or video data based on at least one of the target label, text data in the interaction planning data, and image features of the virtual object, through the target response generation strategy.

[0305] In some embodiments, the second interaction module is configured to: in response to a trigger event being a touch operation on an interactive device, determine an initial action label in the interaction planning data based on state data, the touch operation corresponding to the trigger event, and context information from a preset action label set.

[0306] In some embodiments, the second interaction module is configured to: generate first audio data based on interaction planning data according to a preset generation cycle and through at least one response generation strategy; and generate first video data based on the first audio data and through at least one response generation strategy.

[0307] In some embodiments, the second interaction module is configured to: output first audio data through the audio output component of the interaction device; and output first video data through the visual output component of the interaction device.

[0308] In some embodiments, the second interaction module is used to: push phased events to the interaction device based on the generation progress of response data, the phased events being used to control the virtual object to perform interactive behavior through the interaction device; the phased events include at least one of the following: a visual style progress event; an interaction planning data planning event; a video generation submission event; a video generation progress event; a video generation completion event; an audio generation completion event; a response data storage event; a response data generation failure event; a preset event; and a trigger event completion status event.

[0309] In some embodiments, the second interaction module is used to: control the virtual object to perform a preset interaction behavior through the interaction device in response to a failure to generate response data or in response to triggering a preset event.

[0310] In summary, this device enables the creation of personalized virtual companions; it distinguishes the interactive behavior of virtual objects from traditional preset animations and fixed responses, achieving real-time and personalized interactive content; and it realizes the personalization, contextualization, and real-time nature of virtual object behavior, enhancing the naturalness and immersion of the interaction.

[0311] Figure 6 This is a schematic diagram of the structure of an electronic device 600 for implementing the above-described interaction method, according to an exemplary embodiment.

[0312] Reference Figure 6 The electronic device 600 may include one or more of the following components: a processing component 602, a memory 604, a power supply component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 614, and a communication component 616.

[0313] Processing component 602 typically controls the overall operation of electronic device 600, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 602 may include one or more processors 620 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 602 may include one or more modules to facilitate interaction between processing component 602 and other components. For example, processing component 602 may include a multimedia module to facilitate interaction between multimedia component 608 and processing component 602.

[0314] Memory 604 is configured to store various types of data to support the operation of electronic device 600. Examples of such data include instructions for any application or method operating on electronic device 600, contact data, phonebook data, messages, pictures, videos, etc. Memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0315] Power supply component 606 provides power to various components of electronic device 600. Power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 600.

[0316] Multimedia component 608 includes a screen that provides an output interface between electronic device 600 and user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 608 includes a front-facing camera and / or a rear-facing camera. When electronic device 600 is in an operating mode, such as a shooting mode or video mode, the front-facing camera and / or rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0317] Audio component 610 is configured to output and / or input audio signals. For example, audio component 610 includes a microphone (MIC) configured to receive external audio signals when electronic device 600 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 604 or transmitted via communication component 616. In some embodiments, audio component 610 also includes a speaker for outputting audio signals.

[0318] I / O interface 612 provides an interface between processing component 602 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0319] Sensor assembly 614 includes one or more sensors for providing state assessments of various aspects of electronic device 600. For example, sensor assembly 614 may detect the on / off state of electronic device 600, the relative positioning of components such as the display and keypad of electronic device 600, changes in position of electronic device 600 or a component of electronic device 600, the presence or absence of user contact with electronic device 600, orientation or acceleration / deceleration of electronic device 600, and temperature changes of electronic device 600. Sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 614 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 614 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0320] Communication component 616 is configured to facilitate wired or wireless communication between electronic device 600 and other devices. Electronic device 600 can access wireless networks based on communication standards, such as WiFi, 2G or 3G, 4G LTE, 5G NR (NewRadio), or combinations thereof. In one exemplary embodiment, communication component 616 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 616 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0321] In an exemplary embodiment, the electronic device 600 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0322] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, which can be executed by a processor 620 of an electronic device 600 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0323] In some embodiments, the electronic device is a mobile device such as a vehicle or robot. In other embodiments, the electronic device is a terminal such as a mobile phone or computer.

[0324] Embodiments of this disclosure also provide a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the interactive methods described in the above embodiments of this disclosure.

[0325] Embodiments of this disclosure also provide a computer program product including computer instructions for causing a computer to perform the interactive methods described in the above embodiments of this disclosure.

[0326] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses, systems, and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0327] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0328] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain.

[0329] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processing module, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (control method), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0330] It should be understood that various parts of the embodiments of this disclosure can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0331] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0332] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The aforementioned storage medium can be a read-only memory, a hard disk, or an optical disk, etc.

[0333] Although embodiments of the present disclosure have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present disclosure. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present disclosure.

Claims

1. An interaction method, characterized in that, The method includes: In response to acquiring material data, a virtual object is output, wherein the virtual object has a visual relationship with the material data; In response to receiving a trigger event, response data for the virtual object is generated based on the trigger event and context information, so as to control the virtual object to perform the interactive behavior corresponding to the trigger event.

2. The method according to claim 1, characterized in that, The response to acquiring material data and outputting a virtual object includes: In response to acquiring the material data, the virtual object is generated; The virtual object is displayed through the visual output component of the interactive device; and / or the initial audio of the virtual object is output through the audio output component of the interactive device.

3. The method according to claim 2, characterized in that, The step of generating the virtual object in response to obtaining the material data includes: Based on the material data, determine the initial state parameters of the virtual object; Based on the initial state parameters, the visual characteristics of the virtual object are determined in order to generate the virtual object in the visual output component.

4. The method according to claim 3, characterized in that, The step of determining the initial state parameters of the virtual object based on the material data includes: At least one visual style is output through the interactive device; In response to a user triggering a target visual style in at least one of the visual styles, the initial state parameters of the virtual object are determined based on the target visual style and the material data.

5. The method according to claim 1, characterized in that, The step of responding to a received trigger event by generating response data for the virtual object based on the trigger event and context information, in order to control the virtual object to perform the interactive behavior corresponding to the trigger event, includes: Based on the triggering event and the context information, at least one of the following is generated for the virtual object: audio data, video data, and text data. The response data includes at least one of the following: audio data, video data, and text data. Based on the response data, the virtual object is controlled to perform the interactive behavior corresponding to the triggered event through the interactive device.

6. The method according to claim 5, characterized in that, The step of controlling the virtual object to perform the interactive behavior corresponding to the triggering event through the interactive device based on the response data includes: Based on the video data, the virtual object is controlled to perform interactive actions and / or display facial expressions in the visual output component of the interactive device, and the audio data is output through the audio output component of the interactive device; and / or The text data is output through the visual output component.

7. The method according to claim 6, characterized in that, Based on the video data, controlling the virtual object to perform interactive actions and / or display facial expressions in the visual output component of the interactive device, and outputting the audio data through the audio output component of the interactive device, includes: Based on the video data, the virtual object is controlled to perform the interactive actions and / or present expressions in the visual output component from a first-person perspective. The audio output component outputs the audio data of the virtual object from a first-person perspective.

8. The method according to claim 5, characterized in that, The step of controlling the virtual object to perform the interactive behavior corresponding to the triggering event through the interactive device based on the response data includes: The visual output component of the interactive device outputs at least one of the following: The response data generation progress; the target visual style; the completion status of the triggering event; and the status information of the virtual object executing the interactive behavior, the status information including the virtual object's action status and emotion.

9. The method according to claim 4, characterized in that, In response to a user triggering a target visual style among the at least one visual styles, the initial state parameters of the virtual object are determined based on the target visual style and the material data, including: The material data is then validated. In response to the successful verification process, based on the target visual style and the material data, the initial state parameters of the virtual object are determined, and material identifiers are added to the initial state parameters.

10. The method according to any one of claims 1 to 9, characterized in that, The triggering event includes at least one of the following: For touch operation on interactive devices; Instructions input via an interactive device; Update the status parameters associated with the mobile device; A trigger signal from at least one terminal device associated with a mobile device.

11. The method according to any one of claims 1 to 9, characterized in that, The step of responding to a received trigger event by generating response data for the virtual object based on the trigger event and context information, in order to control the virtual object to perform the interactive behavior corresponding to the trigger event, includes: In response to receiving the trigger event, obtain the state data associated with the virtual object; The response data is generated based on the state data, the triggering event, and the context information; Based on the response data, the virtual object is controlled to perform the interactive behavior corresponding to the triggering event.

12. The method according to claim 11, characterized in that, The step of generating the response data based on the state data, the triggering event, and the context information includes: Based on the state data, the triggering event, and the context information, the response data is generated through at least one generation strategy. The response data includes at least one of audio data, video data, and text data.

13. The method according to claim 12, characterized in that, The step of generating the response data based on the state data, the triggering event, and the context information, through at least one generation strategy, includes: Based on the state data, the triggering event, and the context information, an interaction planning data for the virtual object is generated through a planning generation strategy. The interaction planning data includes at least one of the text data, initial action labels, and emotion labels. Based on the interactive planning data, the audio data and / or the video data are generated through at least one response generation strategy, wherein the at least one generation strategy includes the planning generation strategy and the at least one response generation strategy.

14. The method according to claim 13, characterized in that, The step of generating the audio data and / or the video data based on the interaction planning data and through the at least one response generation strategy includes: Based on the initial action labels and / or emotion labels in the interaction planning data, the target labels are determined through preprocessing; Based on the target label, a target response generation strategy is determined among the at least one response generation strategy; Based on at least one of the target label, the text data in the interaction planning data, and the image features of the virtual object, the audio data and / or the video data are generated through the target response generation strategy.

15. The method according to claim 13, characterized in that, The step of generating interaction planning data for the virtual object based on the state data, the triggering event, and the context information includes: In response to the triggering event being a touch operation on the interactive device, based on the state data, the touch operation corresponding to the triggering event, and the context information, the initial action label in the interaction planning data is determined from the preset action label set.

16. The method according to claim 13, characterized in that, The step of generating the audio data and / or the video data based on the interaction planning data, through at least one response generation strategy, includes: According to the preset generation cycle, based on the interaction planning data, the first audio data is generated through the at least one response generation strategy; Based on the first audio data, first video data is generated using the at least one response generation strategy.

17. The method according to claim 16, characterized in that, The step of controlling the virtual object to execute the interactive behavior corresponding to the triggering event based on the response data includes: The first audio data is output through the audio output component of the interactive device; and, The first video data is output through the visual output component of the interactive device.

18. The method according to claim 11, characterized in that, The method further includes: Based on the generation progress of the response data, a phased event is pushed to the interactive device. The phased event is used to control the virtual object to perform the interactive behavior through the interactive device. The phased events include at least one of the following: visual style progress event; interactive planning data planning event; video generation submission event; video generation progress event; video generation completion event; audio generation completion event; the storage event of the response data; the failure event of the generation of the response data; a preset event; and the completion status event of the triggering event.

19. The method according to claim 11, characterized in that, The method further includes: In response to the failure to generate the response data, or in response to the triggering of a preset event, the virtual object is controlled to perform a preset interactive behavior through the interactive device.

20. An interactive device, characterized in that, The interactive device includes a first interactive module and a second interactive module. The first interaction module is used to output a virtual object in response to the acquisition of material data, wherein the virtual object has a visual association with the material data; The second interaction module is used to respond to a received trigger event, and based on the trigger event and context information, generate response data for the virtual object, so as to control the virtual object to perform the interactive behavior corresponding to the trigger event; The interactive device is configured to perform the method of any one of claims 1 to 19.

21. An electronic device, characterized in that, include: A processor and a memory for storing a computer program capable of running on the processor, wherein the processor, when running the computer program, performs the method of any one of claims 1 to 19.

22. A computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 19.

23. A program product, characterized in that, Includes computer instructions for causing a computer to perform the method of any one of claims 1 to 19.

Images