A method, device and storage medium for generating and presenting video

Abstract

A method, device and storage medium for generating and presenting a video, the method comprising: acquiring video data corresponding to a story through an electronic device; presenting a portion of the video data corresponding to a selected storyline path of the story through a display interface; receiving a user request to switch from a 2D video streaming mode to a 3D exploration mode through an input interface; and in response to the user request to switch from the 2D video streaming mode to the 3D exploration mode, acquiring, through a processor, a 3D video scene with an exploration option for an avatar, the 3D video scene matching a current story state and currently presented video data; and presenting the 3D video scene with the exploration option through the display interface.

Description

Technical Field

[0001] The present invention relates to the technical field of content display, and in particular to a device and method for interactive video presentation. Background Art

[0002] Television has been the most influential entertainment device for decades. Many technologies and innovations have been deployed in the field of television to enhance the television experience. For example, based on the assumption that television is a relaxing experience that requires as few user interactions as possible, the frequency of user interactions and / or clicks on remote control buttons is considered as a basic indicator for evaluating television performance. In some other examples, sensing-based automatic user recognition learning methods are studied and personalized recommendations are proposed to speed up the user interaction process in front of the TV. Some applications explore the layout structure and organization of TV content and provide scalable user interaction mechanisms to achieve faster content search and selection processes. In some other applications, target hierarchical access and interaction tasks in the process of watching TV are studied to enable users to handle experiences such as TV shopping and information retrieval in real time.

[0003] With the advent of interactive movies, the story format is different from the traditional one, where the protagonist is periodically faced with two or three options, each of which leads to more options and then to one of multiple endings. The results show that the introduced unpredictability encourages users to watch repeatedly. However, the frequent interaction requirements for users in front of the TV make it disadvantageous for users who prefer a more relaxed style.

[0004] Therefore, some methods and devices are needed to improve the video viewing experience and strike a balance between maintaining the beauty of unpredictability and not disturbing users excessively. Summary of the invention

[0005] In one aspect, the present invention provides a method for interactive video presentation. The method comprises: acquiring video data corresponding to a story through an electronic device; presenting a portion of the video data corresponding to a selected storyline path of the story through a display interface; receiving a user request to switch between a two-dimensional (2D) video streaming mode and a three-dimensional (3D) exploration mode through an input interface; and in response to the user request to switch from the 2D video streaming mode to the 3D exploration mode: acquiring a 3D video scene with an exploration option of an avatar through the processor, the 3D video scene matching the current story state and the currently presented video data; and presenting the 3D video scene with the exploration option through the display interface.

[0006] In some embodiments of the method, acquiring the 3D video scene with the exploration option includes: generating a motion path of the avatar in a 3D space; and acquiring the 3D environment scene, the 3D environment scene changing according to the motion path.

[0007] In some embodiments of the method, before generating the motion path of the avatar in the 3D space, acquiring the 3D video scene with exploration options further comprises: receiving a user instruction to operate the avatar in the 3D space; and generating the motion path according to the user instruction.

[0008] In some embodiments of the method, after acquiring the 3D environment scene that changes according to the motion path, acquiring the 3D video scene with exploration options further includes: generating and demonstrating animation content corresponding to the interaction content between the objects in the 3D space and the avatar.

[0009] In certain embodiments of the method, the interactive content between the subject and the avatar includes a conversation, and the subject expresses one or more of: a legal opinion, a confusing opinion, a wise opinion, an educational opinion, or a random opinion during the conversation.

[0010] In certain embodiments of the method, neural network processing is used to generate the 3D video scene with exploration options based on a trained model of character responses during the dialogue.

[0011] In certain embodiments of the method, the story is a super story including multiple storyline paths, and the super story also includes multiple character states corresponding to multiple story characters at multiple time points in the story; and the method also includes: receiving user input of a preferred story development through an input interface; and based on the preferred story development, identifying a selected storyline path from the multiple storyline paths of the super story through the processor.

[0012] In certain embodiments of the method, the user input for the preferred story development includes voice input or gesture input; and the method also includes translating the user input to determine the preferred story development, including: performing a natural language processing process to convert the user input into an intended command; or performing gesture analysis to determine a user selection from a menu displayed on the display interface.

[0013] In certain embodiments of the method, identifying the selected storyline path based on the preferred story development includes: obtaining, by the processor, a directed acyclic graph (DAG) matrix of size M×N for representing the DAG space of the super story, M is the number of all character state combinations of multiple story characters, N is the number of time points, and each vertex of the matrix represents a character state combination at a time point; identifying, by the processor, a starting vertex in the matrix corresponding to a current time point and a current character state combination; executing, by the processor, a minimization process of a willingness-to-realize distance (WRD) function to identify a connection path connecting multiple vertices in the DAG space, the connection path starting from the starting vertex and ending at a result vertex, wherein, based on the degree of matching between the willingness and the realization result, the story quality and the video quality of the realization process, a delay between the willingness and the realization result, one or more of the potential user exploration opportunities in the minimization process, the minimization process dynamically measures and minimizes the distance between the preferred story development and the realization process, the realization process represented by the connection path.

[0014] In certain embodiments of the method, the minimization process of the WRD function includes: calculating, by the processor, a degree of match between the preferred story development and the result of the connection path; calculating, by the processor, a delay factor, the delay factor being used to describe the minimum duration of all paths between the starting vertex and the result vertex; calculating, by the processor, a chance factor, the chance factor being used to describe the maximum total number of scenes for all paths between the starting vertex and the result vertex; calculating, by the processor, a quality factor, the quality factor being used to describe the quality of the lowest quality step among all steps in the connection path; calculating, by the processor, the WRD function as a weighted sum of the degree of match, the delay factor and the chance factor, wherein the weighted sum includes a first weighting coefficient of the degree of match, a second weighting coefficient of the delay factor and a third weighting coefficient of the chance factor, the first weighting coefficient of the degree of match being ≤0, the second weighting coefficient of the delay factor being ≥0, and the third weighting coefficient of the chance factor being ≥0; and identifying the connection path by finding a path corresponding to a minimum value of the WRD function while keeping the quality factor above a threshold.

[0015] In some embodiments, before presenting the portion of the video data corresponding to the selected storyline path, the method further includes: before presenting the portion of the video data corresponding to the selected storyline path, notifying the delay of the implementation result and the confidence level of the implementation result through the display interface.

[0016] Another aspect of the present invention discloses a device for performing computer-aided diagnosis (CAD) based on medical images. The device includes: a memory storing computer-executable instructions; and a processor coupled to the memory, when executing the computer-executable instructions, the processor is further configured to: obtain video data corresponding to a story; display a portion of the video data corresponding to a selected storyline path of the story through a display interface; receive a user request to switch between a two-dimensional (2D) video streaming mode and a three-dimensional (3D) exploration mode; and in response to the user request to switch from the 2D video streaming mode to the 3D exploration mode: obtain a 3D video scene with an avatar's exploration option, the 3D video scene matching the current story state and the currently displayed video data; and present the 3D video scene with the exploration option through the display interface.

[0017] In some embodiments of the electronic device, the processor is further configured to: generate a motion path of the avatar in a 3D space; and acquire a 3D environment scene, the 3D environment scene changing according to the motion path.

[0018] In some embodiments of the electronic device, the processor is further configured to: receive a user instruction to operate the avatar in the 3D space; and generate the motion path according to the user instruction.

[0019] In some embodiments of the electronic device, the processor is further configured to: generate and present animation content corresponding to the interaction content between the object in the 3D space and the avatar.

[0020] In some embodiments of the electronic device, the interactive content between the object and the avatar includes a conversation, and one or more of legal opinions, confusing opinions, wise opinions, educational opinions, and random opinions expressed by the object during the conversation.

[0021] In certain embodiments of the electronic device, the story is a super story, including multiple storyline paths, and the super story also includes multiple character states corresponding to multiple story characters at multiple time points in the story; and the processor is also configured to: receive user input of a preferred story development through an input interface; and identify a selected storyline path from the multiple storyline paths of the super story based on the preferred story development.

[0022] In certain embodiments of the electronic device, the user input for the preferred story development includes voice input or gesture input; and the processor is further configured to: perform a natural language processing process to convert the user input into an intended command; or perform gesture analysis to determine a user selection from a menu displayed on the display interface.

[0023] In certain embodiments of the electronic device, the processor is further configured to: obtain a directed acyclic graph (DAG) matrix of size M×N for representing the DAG space of the super story, where M is the number of all character state combinations of the multiple story characters, N is the number of the time points, and each vertex of the matrix represents a character state combination at a time point; identify a starting vertex in the matrix corresponding to the current time point and the current character state combination; perform a minimization process of a willingness-to-realize distance (WRD) function to identify a connection path connecting multiple vertices in the DAG space, the connection path starting from the starting vertex and ending at a result vertex, wherein the minimization process dynamically measures and minimizes the distance between the preferred story development and the realization process represented by the connection path, while considering one or more of the following: the degree of match between the willingness and the realization result, the story and video quality of the realization process, the delay between the proposed willingness and the realization result, and the potential user exploration opportunities in the process.

[0024] In certain embodiments of the electronic device, the processor is further configured to: calculate the degree of match between the preferred story development and the result of the connection path; calculate a delay factor, the delay factor being used to describe the minimum duration of all paths between the starting vertex and the result vertex; calculate a chance factor, the chance factor being used to describe the maximum total number of scenes of all paths between the starting vertex and the result vertex; calculate a quality factor, the quality factor being used to describe the quality of the lowest quality step among all steps in the connection path; calculate the WRD function as a weighted sum of the degree of match, the delay factor and the chance factor, wherein the weighted sum includes a first weighting coefficient of the degree of match, a second weighting coefficient of the delay factor and a third weighting coefficient of the chance factor, the first weighting coefficient of the degree of match being ≤0, the second weighting coefficient of the delay factor being ≥0, and the third weighting coefficient of the chance factor being ≥0; and identify the connection path by finding a path corresponding to a minimum value of the WRD function while keeping the quality factor above a threshold.

[0025] In some embodiments of the electronic device, the processor is further configured to: before presenting the portion of the video data corresponding to the selected storyline path, notify the delay of the implementation result and the confidence level of the implementation result through the display interface.

[0026] Another aspect of the present invention provides a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium stores a plurality of instructions, wherein when the processor executes the plurality of the instructions, the processor performs the following operations: obtaining video data corresponding to a story; displaying a portion of the video data corresponding to a selected storyline path of the story through a display interface; receiving a user request to switch between a two-dimensional (2D) video streaming mode and a three-dimensional (3D) exploration mode; and in response to the user request to switch from the 2D video streaming mode to the 3D exploration mode: obtaining a 3D video scene with an exploration option for an avatar, the 3D video scene matching a current story state and the currently displayed video data; and displaying the 3D video scene with the exploration option through the display interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly describes the drawings used in the description of the embodiments. It is obvious that the drawings in the following description are only some embodiments of the present invention. Based on these drawings, ordinary technicians in this field can obtain other drawings.

[0028] Figure 1A An example of an interactive video presentation experience is shown;

[0029] Figure 1B Another example of an interactive video presentation experience is shown;

[0030] Figure 1C Another example of an interactive video presentation experience is shown;

[0031] Figure 2 shows switching between different modes of interactive video presentation according to certain embodiments of the present invention;

[0032] Figure 3 shows functional units according to some embodiments of the present invention;

[0033] Figure 4 shows the hardware implementation of the IDEO device according to some embodiments of the present invention;

[0034] Figure 5 shows an example of a DAG representing a super story according to some embodiments of the present invention;

[0035] Fig. 6AA story segment showing a specific story according to some embodiments of the present invention;

[0036] Figure 6B shows a DAG representation Fig. 6A The story shown in;

[0037] Figure 7 A reinforcement learning method is shown;

[0038] Figure 8 An example of functional connection in a story is shown;

[0039] Fig. 9 shows a DAG diagram representing a super story according to some embodiments of the present invention;

[0040] Fig.10 An interactive video presentation method implemented by the IDEO system and device is shown. DETAILED DESCRIPTION

[0041] The technical solutions of the embodiments of the present invention are described below with reference to the accompanying drawings. The described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] The present invention provides a device and method for interactive video presentation. In particular, the method provided by the present invention extends the user's interactive capabilities from only expressing willingness outside the TV to being able to experience (for example, through their own avatar) and explore inside a three-dimensional (3-Dimensional, 3D) story scene. The functions provided by the method can bring more value to the user, such as understanding the background, obtaining clues, observing visual details, and receiving suggestions. The interaction can be completed through an input device, for example, by hand (if a touch screen or gesture recognition mechanism is used), by voice commands, by head movement (if a head-mounted device is used), by a remote control or mouse, by a mobile phone and mobile device, or by an Internet of Things device such as a toy. The TV can recognize the user's interactive content and manage the avatar representing the user in the 3D story scene.

[0043] It is worth noting that the general term "television" used in the present invention is not limited to televisions. "Television" can refer to all video entertainment systems, including projection-based, PC-based, STB / STB-based and mobile-based devices, etc.

[0044] Figure 1A –1C shows an example of a 3D exploration scene. Figures 1A-1C As shown, Figure 1BThe avatar in the 3D exploration scene in the video may operate (e.g., speak) according to the user's intention. Figure 1A ) in the 3D scene ( Figure 1C The objects can become alive in the 3D scene and can interact with each other or with the avatar. These objects can be story-intelligent and can provide story context, clues and suggestions to the user (through interaction with the avatar). With the user's permission, the avatar can explore the 3D scene on its own and interact with the living objects, in which case the exploration experience can be returned to the user's viewing experience.

[0045] like Figure 2 As shown, some embodiments of the present invention allow users to switch between three scenes, namely, watching TV in 2D space, watching the avatar automatically exploring in 3D space, and driving the avatar to explore in 3D space, to switch freely between 2D and 3D space and between passive and active modes. This interactive experience can be called interactive video (IDEO), which includes arbitrary switching between 2D video frames to 3D story scenes and vice versa, collaborative exploration experience with multiple 3D story smart objects and avatars, and guiding the future direction of the story at any time through interaction (e.g., expressing wishes).

[0046] According to some embodiments, the IDEO system may provide one or more of the following features:

[0047] 1) Switching between 2D and 3D space through simple interactions (e.g., clicking a remote button, through gestures, or through voice commands);

[0048] 2) Manage 2D video data and 3D scene data simultaneously to make them seamlessly match (i.e. find the associated 3D scene at any video moment), and support 2D video playback when 3D data is not available;

[0049] 3) Once the user switches from 2D to 3D, the 3D scene associated with the current 2D video frame can be rendered in real time, thereby providing a collaborative computing solution with support from multiple devices and / or cloud computing resources;

[0050] 4) training story smart objects and user avatars with story knowledge gained from usable stories;

[0051] 5) Dynamically guide story smart objects and avatars to interact interactively;

[0052] 6) guiding the user avatar to automatically explore the 3D scene when desired (e.g., when the user sets the avatar to automatic mode); and

[0053] 7) Implement user wishes by dynamically adjusting the path selection of story branches within the super story network.

[0054] Figure 3 1 shows functional units according to some embodiments of the present invention. Figure 3 As shown, the user can interact with the TV using a simple interaction model, that is, whenever the user expresses a desire through interaction, the TV can recognize the user's immediate intention, then dynamically guide the storytelling engine to meet his / her expectations, and inform the user through feedback messages about when his / her desire can be realized (short-term or long-term). The user can switch between experience modes at any time (such as Figure 2 As shown), the TV will transmit 2D video or rendered 3D scene accordingly. In 3D mode, the story intelligent dialogue engine can enable dialogue between smart objects and avatars, and the avatar path planner can guide the avatar's automatic navigation in the scene (if the user chooses a non-self-driving avatar).

[0055] exist Figure 3 In the diagram, the main functional units of IDEO are shown in the dotted boxes. It is worth noting that this is a conceptual diagram, and not all components need to be in the physical TV box, but rather the components can be executed in the TV software, or as a cloud service, or a service provided by another device that can be accessed and used by the TV software. Figure 3 In the diagram, the modules on the left are classified as "online modules", which require the system to provide immediate computing resources and generate immediate responses. The modules on the right are classified as "offline modules", which allow offline computing or processes that do not require immediate results. The classification is based on a conceptual point of view and does not need to be strictly followed in implementation. In other words, in the actual implementation of the IDEO system, whether a module is online or offline can be determined based on the actual conditions and relevant requirements of the system settings.

[0056] Figure 4 1 shows an example of a hardware implementation of the IDEO device 100 according to some embodiments of the present invention. Figure 4As shown, the IDEO device 100 can be configured to interactively present 2D and 3D video scenes. In some embodiments, the IDEO device 100 may include a computing device, the computing device including a processor 102 and a storage medium 104. The IDEO device 100 may also include a display 106 or other output unit, a communication module 108, and additional peripherals 112. Some devices may be omitted, and other devices may be included. The processor 102 may include any suitable processor. In some embodiments, the processor 102 may include multiple cores for multi-threading or parallel processing. The processor 102 may execute a sequence of computer program instructions to perform various processes, such as a neural network processing program. The storage medium 104 may be a non-transitory computer-readable storage medium, and may include a memory module, such as a ROM, RAM, a flash memory module, an erasable and rewritable memory, and a large-capacity storage device, such as a CD-ROM, a USB flash drive, and a hard disk. When executed by the processor 102, the storage medium 104 may store computer programs and instructions for performing various processes, thereby causing the processor to perform various steps of generating and presenting 2D and 3D video scenes. The communication module 108 may include a network device for establishing a connection over a network. The display 106 may include any suitable type of display device or electronic device display (e.g., a CRT or LCD based device, a touch screen, a VR headset, etc.). The peripherals 112 may include additional I / O devices such as a keyboard, a mouse, a camera, a microphone, etc. The processor 102 may be configured to execute instructions stored on the storage medium 104 and perform various operations related to the IDEO method as detailed in the following description.

[0057] See also Figure 3 , an extensible super story generator is configured to generate a storyline according to the user's wishes. A key difference between a super story and a regular story is that the super story contains many stories, and each story has a unique path, which depends on the options created by the system and the choices that the characters may take. The super story can be generated manually or automatically by a computing device. At different stages of the storyline, the story may have different quality levels and therefore be suitable for different groups or numbers of audiences. The advantage of using an extensible super story generator is that the extensible super story generator allows different branches of the super story to be at various quality levels, and the quality can be improved from time to time in an extensible manner.

[0058] As the name implies, a hyper-story refers to a story network. In some previous work, the functions of 31 characters were extracted and abstracted as the basic structural components of fairy tales. In other words, any story can be represented as a sequence of serialized functions.

[0059] In the present invention, a directed acyclic graph (DAG) may be used to represent a super story. Figure 5 An example of a DAG representing a super story is shown. Each vertex in the DAG represents a combination of states of all the main characters, and the directed edges from one vertex to another represent events that cause the states of these characters to change. Figure 5 In the super story DAG shown, there is a single start vertex because all stories contained in the super story set the same starting point, but there are multiple result vertices because multiple endings of various stories may be generated during the story creation process. Obviously, any of the multiple paths from the start vertex to the result vertex is a possible story as long as the basic criteria for a good story are met. Edges from one vertex to multiple vertices are considered multiple branches of the story graph.

[0060] The well-known movie / story "The Lion King" can be used as an example to illustrate Figure 5 DAG structure. There are 3 main characters in the story, namely Mufasa, Scar and Simba. The story touches on two levels of Maslow's hierarchy of needs theory to a large extent, kingship and life / death. In an example, kingship=0 can mean not considering kingship. Kingship=1 can mean desiring to obtain kingship. Kingship=2 can mean obtaining kingship. Life=0 can mean dead, 1 means in danger, and 2 means life without immediate danger. With these quantifications, the status of all 3 characters can be represented by 6 tuples, which are composed of the kingship and life values ​​of all 3 characters including Mufasa, Scar and Simba. Fig. 6A A story segment of The Lion King according to the above scheme is shown. Fig. 6A As shown, a portion of a story with 7 segments (or steps) can be represented by a 6-tuple consisting of the kingship and life values ​​of all 3 characters.

[0061] Figure 6B Shows Fig. 6A The DAG representation of the story shown in . Figure 6BAs shown, the DAG representation of the above story can be represented by a solid line path, wherein each vertex represents the state in each step, and each edge represents the transition from one state to another state. By adding vertices and edges with dotted lines, a new story can be formed as an example. Starting from the state transition, the storyline can be completed by the following steps. For example, the edge from (221201) to (001202) can refer to Mufasa saving Simba, but losing his life. The edge from (001202) to (001222) can refer to Simba's development as the new king moving forward. It can be imagined that the space for generating new stories in the DAG structure can be very large, which is related to the number of steps of the story. In addition, even if the number of steps of the story is fixed, the duration of the story can also vary, which is highly related to the actual duration of the editing and cutting processing according to these steps and the video post-processing stage.

[0062] See also Figure 3 In order to enable the story in text format to be delivered to the user in the format of a movie or video animation, an automatic (or semi-automatic) workflow is needed to generate a video from the script or other format of the story. Automatic cinematography technology has been shown to achieve this goal. The video scene generator module takes input from the super story generator and creates a video accordingly, which is then stored in a database. At the same time, a 3D scene associated with the video can also be stored so that the user can enter the 3D scene at any time during the video playback (through an avatar).

[0063] In order to best serve the user, the system needs to understand the user's preferences through his / her interaction patterns with the TV. The User Profile Engine processes the user's behavior in front of the TV (e.g., how often they make an offer, how many 3D explorations they complete, etc.), builds a profile for each user, and models their behavior and preferences.

[0064] In order to make the image and behavior of the objects in the 3D scene vivid, an artificial intelligence (AI)-based method can be used to support human-like dialogue and indicate story intelligence features. On the other hand, when the avatar is allowed to navigate the scene by itself, an AI engine is needed to guide its movement and interaction with surrounding objects. The environmental AI engine is configured to perform these tasks. Training can be performed offline, and the trained model can be stored in a database for use by online modules such as the story intelligence dialogue engine and the avatar path planner.

[0065] Stories usually include transitions that progress through multiple scenes. In traditional movies, viewers may enjoy watching projected 2D images of story scenes, because current movie settings (if not 3D movies) do not support users entering 3D scenes while watching videos. However, according to surveys, many users like the idea of ​​being able to enter video scenes to explore in 3D space, and they have a very positive attitude towards living objects (e.g., humanoid objects) and want to interact with the objects through dialogue and guidance. Most users may like to receive information related to the story, and they may also be interested in character information and small talk for more realism and variety. Many users also indicated that story clues would be practical.

[0066] A Story Smart Object can be an object in a scene that is personified in the context of a storyline or scene. A Story Smart Object in a 3D environment can contain many different characters or viewpoints, for example, expressing:

[0067] 1) Legitimate Opinion – Story Smart Objects represent the voice of legitimate characters who respect authority, honor tradition, and judge those who do not fulfill their responsibilities;

[0068] 2) Chaotic Opinion – Story Smart Object represents the voice of a chaotic character who follows his or her conscience, does not like to be told what to do, and prefers new ideas over tradition;

[0069] 3) Wise Opinion – Story Smart Objects represent the voice of a Wise Character who has better insight into the future and has more clues to unseen possibilities;

[0070] 4) Educational Opinion – Story Smart Objects represent the voice of an insider who knows the background and inherent knowledge / information of all characters and story-related objects; and

[0071] 5) Random Opinions – Story Smart Object provides information related or unrelated to the story and sometimes sounds absurd and funny to the audience.

[0072] In some embodiments, when a user accesses a 3D scene through an avatar, certain objects can have a two-way conversation with the user or the user's avatar. The user can guide the avatar's direction of movement, or authorize the avatar to move around and take actions automatically. The automation of the avatar and these objects can be achieved through AI algorithms.

[0073] According to certain embodiments, an ambient AI engine is deployed to support smart objects and avatars in the following tasks:

[0074] 1) Avatar navigation in 3D scenes; and

[0075] 2) Intelligent dialogue between the smart object and the avatar about the story.

[0076] The training of AI models can be done offline, and the trained models can be stored in a database for use by other online modules. For avatar autonomous navigation, manual tasks can include:

[0077] 1) The avatar (representing the user) walks around in the 3D space, inspects some details of interest (e.g., a close-up view of a map), and interacts / communicates with smart objects, and after navigation is complete, returns to the 2D video viewing experience; and

[0078] 2) The user watches the avatar's performance in a leisurely mode, but may provide guidance to the avatar from time to time (eg, checking content of interest).

[0079] The problem to be solved can be defined as: given a 3D scene floor plan containing multiple objects / details to be explored (categories, location information, etc.) and the user's exploration history for each experience (explored objects and time information), determine the exploration sequence of the avatar of the selected K objects that meets the user's expectations.

[0080] like Figure 7 As shown, reinforcement learning methods can be applied to solve the problem. Figure 7 The environment in is a 3D scene with objects / details that the agent can verify. The possible actions of the agent are movement in 9 directions (assuming that the interaction will automatically occur when the avatar is within the surrounding box of the object / detail) and switching (switching to 2D video playback mode). The state of the agent exploring the environment is S t The reward mechanism is designed as follows:

[0081] (1) Each action a t will receive a negative reward of -1.0.

[0082] (2) After a period of time (the average exploration time of a typical user), each movement will receive a negative reward of -2.0.

[0083] (3) Based on the user's preference (which can be derived from historical data), the agent verifies the new object / detail and obtains various positive rewards. The higher the preference (the guidance / request from the user is considered the highest preference), the higher the reward R the agent will receive. t The higher.

[0084] The model can be implemented using a Deep Q-network (DQN). After multiple iterations of training, the trained model can guide the avatar to quickly determine a good routine to explore the categories of objects / details that the user may prefer within good timing control. The variable time for the conversation between the avatar and the smart object can be jointly considered in the framework to make the result more attractive.

[0085] For storytelling, IDEO can include 3 types of dialogues, including:

[0086] 1) Type A: One-way annotation (e.g., annotation from smart objects);

[0087] 2) Type B: answers to questions (e.g., smart objects answering user questions); and

[0088] 3) Type C: Two-way conversation (e.g., the avatar asks a question and the smart object answers).

[0089] When considering “story intelligence,” all three types of conversations can be condensed to story-related topics. For Type A, one can expect IDEO to derive story trends from the current stage of story development based on knowledge of story common sense or existing meta-stories. Figure 8 shows an example of function connections in a story. Figure 8 As shown, the functions in the story have an interrelated pattern, for example, including a donor preparation function and a transfer form of a magic potion, wherein the donor preparation function includes a test D 1 , interrogation D 2 , a dying man 3 , for mercy and freedom 4,5 , used to divide D 6 , other D 7 , attempting to annihilate D 8 , small conflict D 9 , exchange proposal D 10 The transfer form of the magic agent includes F 1 Transfer, F 2 Instructions, F 3 Prepare, F 4 Sell, F 5 Discovery, F 6 Appearance, F 7 Swallow, F 8 Seize, F 9 Provides a service. Thus, given the current function found in the current state of the story, a rule-based mechanism can be used to derive possible next functions. In the previous description, the character states represented by 6-tuples clearly support the same logic, where a rule-based mechanism can be used to derive possible next states for the character. Rules can be established through existing knowledge and analysis, learned from existing movies / stories, or directly from the current super-story, or a combination of all of these sources. Typically, once a guess for the next step information is available, a template-based language generation method can be used to form a sentence to express it.

[0090] For Type B, the required intelligence is higher than Type A because the system needs to understand the story (or part of the story), understand the problems that include what, who, why, how, where, which, how much, when, etc., and be able to solve these problems. This problem can be solved with various training data sets and artificial intelligence algorithms.

[0091] The difference between Type B and Type C is that Type C requires the avatar to ask reasonable, human-like questions. Indeed, research has recently begun on question generation, where AI algorithms can generate reasonable questions. By combining question generation and question answering, IDEO can handle Type C conversations through the combined AI algorithm.

[0092] See also Figure 3 ,The module characterized by online modules processes the interactive content with the user to receive requests and deliver results.,The online modules utilize the data and models stored in the database to,perform instant intelligent operations such as activating the performance and dialogue of objects,,and the self-navigation of avatars, etc., thus generating IDEO’s novel experience.

[0093] The user intent recognizer receives user interaction content and translates it into the current intention from the user. If the user uses voice input to clearly express his / her intention, such as "I hope Susie can survive", the recognizer can use NLP (natural language processing) to understand the who, when, and content in this instruction and convert it into a template of the intention command, such as "Susie (who) + survive (future state)". If the user expresses his / her intention using gestures or by operating remote control buttons (with the help of menu selection), for example, selecting a character (such as John) on the screen and then selecting "jail" in the pop-up menu to express the intention of "putting John in jail", the recognizer can immediately obtain his / her intention. There are many ways to express their wishes, and the task of the user intent recognizer is to understand the user's immediate needs and send the immediate needs to the storytelling engine.

[0094] The storytelling engine is the core of the online system because it determines how to respond to the user's wishes based on the existing availability of stories in the hyperstory. A new workflow can be deployed in the system to balance multiple factors such as the delay in achieving the result (i.e., the duration between proposing a wish and realizing the display of the video), the quality of the story, the confidence of the response to the wish, the diversity of 3D scenes for exploration, etc., to find the best story branch that meets the user's expectations and the best way to get there. After making a decision, the storytelling engine can inform the user with confidence how long it will take to realize his / her wish. Depending on the availability of the plot branches, the IDEO system can try to find the best solution and inform the user even if the confidence is not high enough sometimes.

[0095] The main responsibilities of the storytelling engine include:

[0096] 1) Find the best solution (i.e., the future path of stories in the hyperstory network) that satisfies the user’s immediate wishes based on the availability of story branches and the user’s preferences observed in his / her behavior patterns; and

[0097] 2) Feed story trends to the Storytelling Intelligent Conversation Engine so that smart objects and avatars will be able to demonstrate visionary cues and actions under the guidance of the AI ​​model trained by the Ambient AI Engine.

[0098] Integer N represents Fig. 6A and Figure 6B The number of steps in the superstory shown. If we only consider the key developments of the character's state changes, N may be a small number, but if we zoom in on the character's actions and emotional changes and consider more factors in the character's state, N may also be a large number. For example, when N is large enough, the movie is divided into N steps, and each step takes at most one minute (or even less) on average, then we are able to transform the continuous time world into a discrete step world. On the other hand, it is reasonable to say that the number of all possible character states is finite. The integer M represents the number of all combinations of character states, so Fig. 6A and Figure 6B The DAG space shown in can be represented by Fig. 9 The element V is represented by a vertex matrix of size M×N. i,j (i<N,j<M) means Fig. 9 The vertex in the i-th column and j-th row of the matrix. Fig. 9 As shown, the problem of the storytelling engine can be expressed as: given the current will generated around the time of step x and the related current vertex V i,j , you need to choose V from all possible situations rx,ry Vertex and bridge V x,y and V rx,ry Path. Fig. 9 In the figure, point A is marked as an example vertex corresponding to the current desired point, vertices B / C / D are possible realization points, and possible paths from A to B, C, and D are marked with bold lines. The ultimate goal is to determine a point from B, C, and D, and there is only one path from A to the determined point.

[0099] The present invention provides a mechanism called Wish-Realization-Distance (WRD) function, which can dynamically measure the distance between the wish and the realization process reflected by the transmission from one vertex to another in the DAG space. WRD takes into account the degree of match between the wish and the realization result, the quality of the story and video in the realization process, the delay between the proposal of the wish and the realization result, the potential user exploration opportunities in the realization process, etc. Using the WRD function, the above problem can be converted into an easier problem of finding the minimum value for the distance list (such as AB, AC, AD) in the DAG.

[0100] If P(V x,y ,V rx,ry ) represents the path of selected edges with the minimum WRD among all possible choices, then ideally it simultaneously represents the best match between user intention and implementation path, the best quality process to achieve the result, the lowest delay between proposing intention and achieving the result, and the richest exploration opportunities. However, in practice, P(V x,y ,V rx,ry ) is very difficult to win in all these components, so the joint cost function of WRD is used to make P(V x,y ,V rx,ry ) wins more reasonably when weighing between these components.

[0101] If w represents the intent delivered from the user intent recognizer module, which is derived from the user's current intention, M(w, V x,y , V rx,ry ) represents the degree of match between the user's intention and the achieved result. The value of the M function is in the range of [0.0, 1.0], and the larger the value, the better the match. Usually, the user's intention is to change or prevent the change of the character's status, which is related to actions such as punishment, rescue, forgiveness, winning, escape, and search. Therefore, V can be used effectively. x,y and V rx,ry The state changes between the two are compared with the user intention to calculate M(w, V x,y , V rx,ry ). In order to simplify the system, the M function only considers the starting and ending states of the character, and does not consider using the state change trajectory of the character on the P(Vx,y,Vrx,ry) path to determine its alignment with the intention (such as Figure 6B Although this may be a good direction to explore, the degree of fit is indeed the key factor in choosing the implementation path.

[0102] For the delay factor, T(V x,y ,V rx,ry ) indicates V x,y and V rx,ryThe minimum video time period of all paths between x,y ,V x+1,yy ) is the connected vertex pair V x,y and V x+1,yy The video session time of the path between them, then T(V x,y ,V rx,ry ) can be recursively calculated as:

[0103] For all vertices V x,y Directly connected vertices V x+1,yy , T(V x,y ,V rx,ry )=Minimum[T(V x,y ,V x+1,yy )+T(V x+1,yy ,V rx,ry )] (1)

[0104] If rx=x+1, then T(V x+1,yy , V rx,ry )=0,.

[0105] For the exploration opportunity factor, the number of scenes included in the story can be considered, assuming that the more new scenes, the more opportunities for users to explore. To simplify the system, repeated scenes can be allowed, and the system can calculate all scenes in and out during the story. S(V x,y , V rx,ry ) indicates that V x,y and V rx,ry The maximum total number of scenarios introduced by the steps between , so it can be calculated as:

[0106] For all vertices V x,y Directly connected vertices V x+1,yy ,S(V x,y ,V rx,ry )=Maximum[S(V x,y ,V x+1,yy )+S(V x+1,yy ,V rx,ry )] (2)

[0107] If rx = x + 1, then S(V x+1 ,yy,V rx,ry )=0.

[0108] For the quality factor, we need to evaluate P(V x,y ,V rx,ry ) and ensure that its quality is higher than a predefined standard (such as Q 阀值 ). If Q(P(V x,y ,V rx,ry)) represents the path P(V x,y ,V rx,ry ), it can be calculated as:

[0109] For P(V x,y ,V rx,ry ) in any edge (V1,V2), Q(P(V x,y ,V rx,ry ))=Minimum[q(V1,V2) (3)

[0110] The q(V1,V2) function measures the quality of a step, which means that any low-quality step will degrade the evaluation of all longer video clips containing this step.

[0111] Now, WRD can be defined as a combination of functions of M, T and S. Its calculation process is as follows: WRD(w,V x,y ,V rx,ry )=a*[1.0-M(w,V x,y ,V rx,ry )]+b*T(V x,y ,V rx,ry )-(1–a–b)*S(V x,y ,V rx,ry ), (4)

[0112] Where a and b are in the range of [0.0, 1.0] and are parameters used by the system to adjust the weights of components M, T, and S; therefore, in order to minimize WRD, maximize the matching degree M and exploration opportunity S, and minimize the delay T.

[0113] Given V x,y , considering {V rx,ry} and all possible paths, we can solve the problem of minimizing WRD(w,V x,y ,V rx,ry ) so that Q(P(V x,y ,V rx,ry ))≥Q 阀值 .

[0114] The storytelling engine works as follows:

[0115] 1. Once there is a new story, the videos of related stories can be obtained from the scalable hyper-story generator and video generator, and the functions T and S in equations (1) and (2) are recalculated and stored in all vertex combinations in the DAG space.

[0116] 2. Once you are close to the vertex V x,y Time to receive the will w:

[0117] a. Consider all vertices V in the next L steps rx,ry (L is a system configurable parameter) and use P(V x,y , V rx,ry ) The path selects a vertex that satisfies equation (5);

[0118] b. If S(w, V x,y , V rx,ry ) is below a configurable confidence level, the user will be notified that "the wish cannot be fulfilled in the current super story";

[0119] c. Otherwise, with confidence M(w, V x,y , V rx,ry ) to notify the user of the delay in achieving the result T(V x,y , V rx,ry ).

[0120] The output video scheduler may be configured to dynamically schedule video streams based on decisions made by the storytelling engine and deliver the desired video segments to be displayed on a television display.

[0121] It is worth noting that the IDEO framework is also applicable to single-ending stories, because the single-ending story with only one path is the smallest extreme case of the super story. Although some users' wishes may not be effectively resolved due to the limitations of the branches contained in the story, the free switching between viewing and exploration modes and the interactive content of avatars / objects provide users with attractive features.

[0122] Figure 3 The current design of the IDEO framework shown can be used to handle single-ending stories without any changes, because single-ending stories can be considered the smallest extreme of a hyperstory.

[0123] See also Figure 3 , the experience mode switcher is configured to handle Figure 2 Once the storytelling engine receives the requested mode from the experience mode switcher, it can decide which branches in the super story will be used to complete the goal. For 2D and 3D presentations, different modules can be called to generate different outputs to the user.

[0124] The storytelling intelligent dialogue engine uses a deep neural network model trained by the environmental AI engine to provide dialogue capabilities for smart objects and avatars. When the user allows the avatar to explore the 3D scene on its own and the user switches to 3D exploration mode, the avatar path planner can be activated. The learning model trained by the environmental AI engine can be used to guide the avatar to move within the scene and interact with adjacent smart objects. The 3D rendering module renders the 3D scene for the user following the guidance of the storytelling intelligent dialogue engine, the avatar path planner, and the 3D data transmitted from the database.

[0125] use Figure 3 In the IDEO system shown, the user can use various interactive mechanisms to express his / her wishes at any time, and the system can respond to his / her wishes instantly, and inform the user's wishes how soon and at what confidence level they can be realized in the storytelling. On the other hand, the user can switch freely between TV viewing and 3D environment exploration experience at any time. Smart objects and avatars with AI functions can effectively expand the original 2D video presentation of the story into a new interactive experience in an immersive 3D scene. By combining the user's willingness mechanism that can change the original story to a completely different story with an experience switching mechanism that gives the user a sense of immersion and interaction, the IDEO experience can attract users to repeat the story with different wishes and a richer curated journey at different times. IDEO can implicitly transform the traditional TV viewing experience into an exploration experience in which users spontaneously participate. 3D space can be regarded as an optional companion layer for 2D video, so if there are any conditions or constraints that limit the successful presentation of the 3D space exploration experience, IDEO can be used as a simple 2D video.

[0126] Fig.10 According to some embodiments, Figure 3 and Figure 4 The interactive video presentation method 200 implemented by the IDEO system and device shown in FIG. The interactive video presentation method 200 may include the following steps. Fig.10 As shown, the method may include the following steps.

[0127] Step 210 is to obtain video data corresponding to the super story. According to some embodiments, the super story can be machine-generated. According to some other embodiments, the super story can be generated by the story author and input to the electronic device. The super story can include multiple story line paths, and the story also includes multiple character states corresponding to multiple story characters at multiple time points in the story. After obtaining the super story, the electronic device can create video data based on the super story and store the video in a storage device. In some embodiments, a 3D scene associated with the video can also be stored so that the user can enter the 3D scene at any time during the video playback (via an avatar).

[0128] Step 220 is receiving user input for a preferred story development. According to some embodiments, the user input for a preferred story development includes voice input or gesture input. The user input is translated to determine the preferred story development. For example, if the user input is voice input, the electronic device can perform a natural language processing process to convert the user input into a desired command. If the user input is a gesture, the electronic device 100 can perform gesture analysis to determine the user's selection from a menu displayed by the displayed interface.

[0129] Step 230 is to identify a selected storyline path from a plurality of storyline paths of a super story according to a preferred story development. According to some embodiments, a super story may be represented by a DAG matrix of size M×N, wherein the DAG matrix of size M×N represents the DAG space of the super story. M is the number of all combinations of character states of a plurality of story characters. N is the number of time points. Each vertex of the matrix represents a combination of character states at a certain time point. A minimization process of a willingness-to-realize distance (WRD) function is performed to identify a connection path connecting a plurality of vertices in the DAG space, wherein the connection path starts from a starting vertex and ends at a result vertex, wherein the minimization process dynamically measures and minimizes the distance between the preferred story development and the realization process represented by the connection path, while taking into account one or more of the following: the degree of match between the proposed intention and the realization result, the story and video quality during the realization process, the delay between the proposed intention and the realization result, and the potential user exploration opportunities during the process. The minimization process of the WRD function may be performed according to equations (1)-(5). For example, the minimization process of the WRD function may include: calculating, by a processor, the degree of match between the preferred story development and the result of the connection path; calculating, by a processor, a delay factor for describing the minimum duration of all paths between the starting vertex and the result vertex; calculating, by a processor, an opportunity factor for describing the maximum total number of scenes for all paths between the starting vertex and the result vertex; calculating, by a processor, a quality factor for the lowest quality of all steps in the connection path; calculating, by a processor, a weighted sum of the degree of match, the delay factor and the opportunity factor of the WRD function, the weighted sum including a first weighting coefficient of the degree of match, a second weighting coefficient of the delay factor and a third weighting coefficient of the opportunity factor, wherein the first weighting coefficient of the degree of match is ≤0, the second weighting coefficient of the delay factor is ≥0, and the third weighting coefficient of the opportunity factor is ≥0; while keeping the quality factor above a threshold, identifying the connection path by finding the path corresponding to the minimum value of the WRD function.

[0130] Step 240 is to notify the user of the delay in achieving the result and the confidence level of achieving the result. According to some embodiments, when the user expresses his / her intention through interaction at any time during the story, the electronic device can recognize the user's immediate intention and then dynamically guide the storytelling engine to meet his / her expectations. Depending on the availability of plot branches, the IDEO system can try to find the best solution and notify the user even if the confidence level is not high enough sometimes. The electronic device can also notify the user how long and how confident his / her intention is.

[0131] Step 250 is to present the portion of video data corresponding to the selected storyline path. In some embodiments, after acquiring the super story, the electronic device can create video data based on the super story and store the video in a storage device. After the selected storyline path is identified, the electronic device can identify the portion of video data corresponding to the selected storyline path and stream the portion of video data.

[0132] Step 260 is to receive a user request to switch from a two-dimensional (2D) video streaming mode to a three-dimensional (3D) exploration mode. Step 270 is to obtain a 3D video scene with an exploration option for an avatar in response to the user request to switch from the 2D video streaming mode to the 3D exploration mode. The 3D video scene matches the current story state and the currently displayed video data. According to some embodiments, the 3D video scene with the exploration option may include a 3D environment scene change according to the movement path of the avatar. In some embodiments, the movement path of the avatar can be automatically generated without user input. In some other embodiments, the avatar can be operated by the user according to user instructions. In some embodiments, the animation content can be generated corresponding to the interactive content between the object and the avatar in the 3D space. The interactive content between the object and the avatar may include a dialogue. During the dialogue, the object may express one or more: a legal opinion, a confusing opinion, a wise opinion, an educational opinion, or a random opinion. In some embodiments, a 3D video scene with an exploration option is generated using a neural network processing based on a training model of the character response during the dialogue. In some embodiments, the method further comprises switching from the 3D exploration mode back to the 2D video streaming mode upon user request.

[0133] The method and device provided in the present invention are described in detail above according to the embodiments. The principle and implementation mode provided in the present invention are described herein by specific examples. The description of the above embodiments is only used to help understand the method provided in the present invention. Meanwhile, those skilled in the art will change the specific embodiments and application scope according to the ideas provided by the present invention. In a word, the contents of this specification should not be interpreted as limiting the present invention.

[0134] This invention contains material that is protected by copyright. The copyright is the property of the copyright owner. The copyright owner has no objection to the reproduction of the patent document or disclosure of the patent in the official records and files of the Patent and Trademark Office.

Claims

1. A method for generating and demonstrating a video, characterized in that: The method for generating and presenting a video is applied to an electronic device having a processor, and includes: Acquiring video data corresponding to the story through the electronic device; displaying, via a display interface, a portion of the video data, the portion of the video data corresponding to a selected storyline path of the story; receiving, via the input interface, a user request to switch from a 2D video streaming mode to a 3D exploration mode; In response to the user request to switch from the 2D video streaming mode to the 3D exploration mode, a 3D video scene with exploration options is obtained by the processor, wherein the exploration options are exploration options for an avatar, and the 3D video scene matches a current story state and currently displayed video data; and the 3D video scene with the exploration options is displayed by the display interface.

2. The method for generating and demonstrating a video according to claim 1, characterized in that: The obtaining of a 3D video scene with an exploration option comprises: generating a motion path of the avatar in 3D space; The 3D environment scene is acquired, and the 3D environment scene changes according to the motion path.

3. The method for generating and demonstrating a video according to claim 2, characterized in that: Before generating the motion path of the avatar in the 3D space, the acquiring of the 3D video scene with the exploration option further comprises: receiving user instructions to operate the avatar in the 3D space; and The motion path is generated according to the user instruction.

4. The method for generating and demonstrating a video according to claim 3, characterized in that: After acquiring the 3D environment scene that changes according to the motion path, acquiring the 3D video scene with exploration options further includes: Animation content corresponding to the interaction content between the object in the 3D space and the avatar is generated and presented.

5. The method for generating and demonstrating a video according to claim 4, characterized in that: The interactive content between the subject and the avatar includes a conversation, and one or more of a legal opinion, a confusing opinion, a wise opinion, an educational opinion, and a random opinion expressed by the subject during the conversation.

6. The method for generating and demonstrating a video according to claim 5, characterized in that: A neural network process is used to generate the 3D video scene with exploration options based on a trained model of the character's responses during the dialogue.

7. The method for generating and presenting a video according to claim 1, characterized in that: The story is a super story, the super story includes a plurality of storyline paths, and the super story also includes a plurality of character states corresponding to a plurality of story characters at a plurality of time points in the story; and The method of generating and demonstrating a video also includes: Receiving user input for optimizing story development via an input interface; and Based on the preferred story development, a selected storyline path is identified by the processor from a plurality of the storyline paths of the super story.

8. The method for generating and demonstrating a video according to claim 7, characterized in that: The user input of the preferred story development comprises a voice input or a gesture input; and The method of generating and presenting a video also includes interpreting the user input to determine the preferred story development, including: performing a natural language processing process to convert the user input into an intended command; or Gesture analysis is performed to determine a user selection from a menu displayed on the display interface.

9. The method for generating and presenting a video according to claim 7, characterized in that: The identifying the selected storyline path according to the preferred story development comprises: The size obtained by the processor is M×N The DAG matrix is ​​used to represent the DAG space of the super story. M is the number of all character state combinations of multiple story characters, N is the number of the time points, and each vertex of the DAG matrix represents a combination of role states at a time point; identifying, by the processor, a starting vertex in the DAG matrix corresponding to a combination of a current time point and a current character state; The processor executes a minimization process of the WRD function to identify a connection path connecting multiple vertices in the DAG space, wherein the connection path starts from the starting vertex and ends at the result vertex, wherein a delay between the intention and the implementation result is proposed based on the degree of matching between the intention and the implementation result, the story quality and the video quality of the implementation process, one or more of the potential user exploration opportunities in the minimization process, and the minimization process dynamically measures and minimizes the distance between the preferred story development and the implementation process, and the implementation process is represented by the connection path.

10. The method for generating and presenting a video according to claim 9, characterized in that: The minimization process of the WRD function includes: Calculating, by the processor, a degree of match between the preferred story development and the result of the connection path; calculating, by the processor, a delay factor, the delay factor being used to describe a minimum duration of all paths between the starting vertex and the result vertex; calculating, by the processor, a chance factor, the chance factor being used to describe a maximum total number of scenarios for all paths between the starting vertex and the result vertex; calculating, by a processor, a quality factor for describing the quality of a lowest quality step among all steps in the connection path; Calculating the WRD function by the processor, the WRD function being a weighted sum of the matching degree, the delay factor and the opportunity factor, the weighted sum comprising a first weighting coefficient of the matching degree, a second weighting coefficient of the delay factor and a third weighting coefficient of the opportunity factor, wherein the first weighting coefficient of the matching degree is ≤ 0, the second weighting coefficient of the delay factor is ≥ 0, and the third weighting coefficient of the opportunity factor is ≥ 0; and The connection path is identified by finding a path corresponding to a minimum value of the WRD function while maintaining the quality factor above a threshold.

11. The method for generating and presenting a video according to claim 9, characterized in that: Prior to presenting the portion of the video data corresponding to the selected storyline path, the method further includes: The delay of the implementation result and the confidence level of the implementation result are notified through the display interface.

12. A device for generating and presenting a video, characterized in that: include: a memory storing computer executable instructions; and A processor coupled to the memory, when executing the computer executable instructions, is further configured to: Get the video data corresponding to the story; displaying, via a display interface, a portion of the video data, the portion of the video data corresponding to a selected storyline path of the story; receiving a user request to switch between a 2D video streaming mode and a 3D exploration mode; and In response to the user request to switch from the 2D video streaming mode to the 3D exploration mode: obtaining a 3D video scene with exploration options, wherein the exploration options are exploration options for an avatar, and the 3D video scene matches a current story state and currently displayed video data; and displaying the 3D video scene with the exploration options through the display interface.

13. The device according to claim 12, characterized in that The processor is further configured to: generating a motion path of the avatar in 3D space; and The 3D environment scene is acquired, and the 3D environment scene changes according to the motion path.

14. The device according to claim 13, characterized in that The processor is further configured to: receiving user instructions to operate the avatar in the 3D space; and The motion path is generated according to the user instruction.

15. The device according to claim 14, characterized in that The processor is further configured to: Animation content corresponding to the interaction content between the object in the 3D space and the avatar is generated and presented.

16. The device according to claim 12, characterized in that: The story is a super story, the super story includes a plurality of storyline paths, and the super story also includes a plurality of character states corresponding to a plurality of story characters at a plurality of time points in the story; and The processor is further configured to: Receiving user input for optimizing story development via an input interface; and Based on the preferred story development, a selected storyline path is identified from a plurality of the storyline paths of the super story.

17. The device according to claim 16, characterized in that The user input of the preferred story development comprises a voice input or a gesture input; and The processor is further configured to: performing a natural language processing process to convert the user input into an intended command; or Gesture analysis is performed to determine a user selection from a menu displayed on the display interface.

18. The device according to claim 16, characterized in that The processor is further configured to: Get the size M×N The DAG matrix is ​​used to represent the DAG space of the super story. M is the number of all character state combinations of multiple story characters, N is the number of the time points, and each vertex of the DAG matrix represents a combination of role states at a time point; Identifying a starting vertex in the DAG matrix corresponding to a combination of a current time point and a current character state; A minimization process of the WRD function is performed to identify a connection path connecting multiple vertices in the DAG space, the connection path starting from the starting vertex and ending at the result vertex, wherein a delay between the intention and the implementation result is proposed based on the degree of matching between the intention and the implementation result, the story quality and the video quality of the implementation process, one or more of the potential user exploration opportunities in the minimization process, and the minimization process dynamically measures and minimizes the distance between the preferred story development and the implementation process, and the implementation process is represented by the connection path.

19. The device according to claim 18, characterized in that The processor is further configured to: Calculating the degree of match between the preferred story development and the result of the connection path; Calculating a delay factor, the delay factor being used to describe a minimum duration of all paths between the starting vertex and the result vertex; Calculating a chance factor, the chance factor being used to describe a maximum total number of scenarios for all paths between the starting vertex and the resulting vertex; calculating a quality factor, the quality factor being used to describe the lowest quality among all steps in the connection path; Calculating the WRD function, wherein the WRD function is a weighted sum of the matching degree, the delay factor, and the opportunity factor, wherein the weighted sum includes a first weighting coefficient of the matching degree, a second weighting coefficient of the delay factor, and a third weighting coefficient of the opportunity factor, wherein the first weighting coefficient of the matching degree is ≤ 0, the second weighting coefficient of the delay factor is ≥ 0, and the third weighting coefficient of the opportunity factor is ≥ 0; and The connection path is identified by finding a path corresponding to a minimum value of the WRD function while maintaining the quality factor above a threshold.

20. A storage medium, the storage medium being a non-transitory computer-readable storage medium storing a plurality of instructions, characterized in that: When the processor executes the plurality of instructions, the processor is caused to perform the following operations: Get the video data corresponding to the story; displaying, via a display interface, a portion of the video data, the portion of the video data corresponding to a selected storyline path of the story; receiving a user request to switch between a 2D video streaming mode and a 3D exploration mode; and In response to the user request to switch from the 2D video streaming mode to the 3D exploration mode: obtaining a 3D video scene with exploration options, wherein the exploration options are exploration options for an avatar, and the 3D video scene matches a current story state and currently displayed video data; and displaying the 3D video scene with the exploration options through the display interface.

Images