Video playing interaction method and apparatus, and electronic device
By generating virtual images and interactive information matching users, the problem of insufficient interactivity during video playback is solved, and user experience and movie satisfaction are improved.
Patent Information
- Application Number
- CN202411143471.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-12-06
AI Technical Summary
The chat system is not interactive during the existing video playback, and the reply information is difficult to resonate with the user, and lacks timely feedback.
By generating viewing preference information based on historical playback videos and playback progress, creating virtual avatars that match users, and driving virtual avatars to interact based on interactive information, providing a personalized interactive experience.
It improves the interactivity and user experience during video playback, avoids repeated recommended videos and spoilers, provides personalized interactions synchronized with users, and enhances user viewing satisfaction.
Smart Images

Figure CN119094855A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to an interactive method, device and electronic device for video playback. Background Art
[0002] Watching movies and TV shows is a popular form of entertainment, and chat systems that interact with users during video playback have also emerged. In the chat systems of related technologies, the replies are basically uniform in content, which makes it difficult to resonate with users, lacks timely feedback, and has poor interactivity. Summary of the invention
[0003] Embodiments of the present application provide a method, device, electronic device, and storage medium for interactive video playback.
[0004] According to the first aspect, an embodiment of the present application provides an interactive method for video playback, the method comprising: generating viewing preference information based on video information and playback progress of historically played videos; generating a virtual image based on the viewing preference information; generating interactive information based on the video information and playback progress of historically played videos and the viewing preference information, and driving the virtual image to interact based on the interactive information.
[0005] According to the second aspect, an embodiment of the present application provides an interactive device for video playback, including: a first generation module, a second generation module and a driving interaction module, the first generation module is configured to generate viewing preference information based on video information and playback progress of historically played videos; the second generation module is configured to generate a virtual image based on the viewing preference information; the driving interaction module is configured to generate interaction information based on the video information and playback progress of historically played videos and the viewing preference information, and drive the virtual image to interact based on the interaction information.
[0006] According to the third aspect, an embodiment of the present application provides an electronic device, which includes one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the first aspect.
[0007] According to a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.
[0008] The interactive method for video playback of the present application generates viewing preference information through video information and playback progress of historically played videos, generates a virtual image based on the viewing preference information, generates interactive information based on the video information and playback progress of historically played videos and the viewing preference information, and drives the virtual image to interact based on the interactive information. The virtual image is related to the user's viewing preference information, the virtual image is more compatible with the user, and it also helps to optimize the interactivity between the virtual image and the user during video playback.
[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0011] Figure 2 is a flow chart of an embodiment of an interactive method for video playback according to the present application;
[0012] Figure 3 is a flow chart of another embodiment of the interactive method for video playback according to the present application;
[0013] Figure 4 is a schematic diagram of application scenario 2 of the interactive method for video playback according to the present application;
[0014] Figure 5 This is a flowchart of application scenario 4 of the interactive method for video playback according to the present application;
[0015] Figure 6 is a schematic diagram of an embodiment of an interactive device for video playback according to the present application;
[0016] Figure 7 It is a structural diagram of a computer system suitable for implementing a server of an embodiment of the present application. DETAILED DESCRIPTION
[0017] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0018] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0019] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the relevant laws and regulations and do not violate public order and good morals.
[0020] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of the interactive method for video playback of the present application can be applied.
[0021] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105, and between the terminal devices. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.
[0022] Users can use terminal devices 101, 102, 103 to interact with other terminal devices or server 105 through network 104 to receive or send messages, etc. Client application software, such as video playback application software, communication application software, etc., can be installed on terminal devices 101, 102, 103.
[0023] The terminal devices 101, 102, 103 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices, including but not limited to televisions, laptops, tablet computers, mobile phones, and projection devices. When the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules, or as a single software or software module. No specific limitation is made here.
[0024] The server 105 may be a server that provides various services, for example, obtaining video preference information based on historically played videos, obtaining a virtual image based on the video preference information, and obtaining interaction information based on video play information and video preference information.
[0025] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules (for example, for providing interactive services for video playback), or it can be implemented as a single software or software module. No specific limitation is made here.
[0026] It should be noted that the interactive method for video playback provided in the embodiments of the present disclosure can be executed by the server 105, or by the terminal devices 101, 102, 103, or by the server 105 and the terminal devices 101, 102, 103 cooperating with each other. Accordingly, the various parts (such as various units, sub-units, modules, sub-modules) included in the interactive device for video playback can be all set in the server 105, or all set in the terminal devices 101, 102, 103, or respectively set in the server 105 and the terminal devices 101, 102, 103.
[0027] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0028] Figure 2 A flowchart 200 of an embodiment of an interactive method for video playback that can be applied to the present application is shown. The interactive method for video playback includes the following steps:
[0029] Step 201: generating viewing preference information based on video information and playback progress of historically played videos.
[0030] In this embodiment, the execution subject (for example, Figure 1 The server 105 or terminal devices 101, 102, 103) can obtain the video information and playback progress of the historically played video locally or from a remote server with a storage function.
[0031] Here, historically played videos can be understood as videos that have been played in the target account, or videos that have been marked as watched in the target account. The target account can be understood as the account of any user. Historically played videos can include but are not limited to all videos that have been played, videos played within a preset time period, and a preset number of videos that have been played. The preset time period can be understood as the period from the current moment to a certain historical moment. The preset time period can be one month, three months, six months, one year, three years, etc. before the current moment. The preset time period is not strictly limited and can be set as needed. The preset number can be understood as a number of videos that are played backwards from the currently played video, such as 10, 50, or 100 videos before the currently played video.
[0032] The video information of the historically played videos may be understood as one or more of the video name, the video type, the video duration, and the video subtitles.
[0033] In some other cases, the video information of the historically played videos may also include, but is not limited to: videos marked as fast-forward, videos marked as skipped, and videos marked as viewed multiple times.
[0034] The video marked as fast forward can be understood as a video clip played faster than the normal playback speed, for example, the user clicks fast forward, drags the progress bar, etc. The video marked as fast forward can be understood as a video clip of a period of time in a single episode, or can be understood as several episodes.
[0035] Videos marked as skipped can be understood as videos that are determined not to be watched by the user by clicking or dragging the progress bar, etc., and can be considered as videos marked as skipped.
[0036] Videos marked as viewed multiple times can be understood as videos that have been played twice or more.
[0037] The playback progress of historically played videos can be understood as the playback progress of a single episode played in history, and / or the comprehensive playback progress of two or more related episodes played in history. Two or more related episodes can be understood as two or more episodes of a TV series, or two or more episodes of a documentary, or two or more episodes of a variety show. Among them, a single episode can be understood as, but not limited to, an episode of a movie, a TV series, an episode of a documentary, or an episode of a variety show.
[0038] For example, the playback progress of historically played videos can be expressed as 80% of a single episode, or 30% of the combined playback progress of related episodes. A single episode playback progress of 80% can be understood as 80% of the total playback time of the episode, for example, the total duration of the episode is 40 minutes, and 32 minutes have been played; the combined playback progress of related episodes is 30%, and the total duration of a TV series is 60 hours, and 18 hours have been played.
[0039] Based on the video information and playback progress of historically played videos, viewing preference information is generated. It can be understood that among the historically played videos, a type of video with a longer playback progress is in line with the user's preference, and the video type of this type of video can be recorded as a preferred video; among the historically played videos, a type of video with a shorter playback progress can be recorded as a non-preferred video. Viewing preference information is used to characterize the user's preference for historically played videos in different dimensions. Viewing preference information may include, but is not limited to, preference type, preference degree, preference time period, etc. Preference types may include TV series, movies, variety shows, documentaries, etc.; preference types may also include animation, costume dramas, reality shows, etc. Preference degrees may include preference for TV series, average movies, dislike of variety shows, etc. Preference time periods can be divided into: morning, noon, and evening; preference type combined with preference time period: preference for English dramas in the morning, preference for TV series in the afternoon, and preference for variety shows in the evening.
[0040] Exemplarily, in step 201, the video information and playback progress of the collected historically played videos are used to generate the user's viewing preference information by using the user viewing preference analysis model. The user viewing preference model includes collaborative filtering, neural networks, deep learning, etc. For example, after the user controls a video to play for 10 minutes, exits the playback and selects other videos, it can be learned that the user is not very interested in the video, and this type of video recommendation can be reduced in the subsequent video recommendation. If the watched part of a certain theme series movie in the video resource library accounts for 80%, the remaining 20% of a certain theme series movie can be recommended in the subsequent video recommendation. For example, if the playback progress of animation videos is greater than 90%, it can be considered that the user prefers animation videos; if the playback progress of disaster movies is less than 20%, it can be considered that the user does not prefer disaster movies. Among them, the above-mentioned videos marked as fast-forward and the videos marked as skipped can be understood as types that are not preferred by the user; the videos marked as played multiple times can be understood as types that the user prefers.
[0041] The above-mentioned record of historically played videos may be stored in a data storage server, and the data stored in the data storage server may be processed to obtain video information and playback progress of the historically played videos.
[0042] The initialization state of the data storage server has a complete video resource library, and all videos in the video resource library are marked as unplayed by the user. For a single video, the initial state of the playback progress of the user playing the video is 0. Of course, the video resource library will update the videos in the video resource library as more videos are added to the market.
[0043] While the user is playing the video, the data storage server records the playback information, updates the data tags in the video resource library, distinguishes between watched videos and unwatched videos, and updates the playback progress of the video.
[0044] Step 202: Generate a virtual image based on the viewing preference information.
[0045] The avatar may include, but is not limited to, appearance, torso, and clothing.
[0046] In an exemplary embodiment, a virtual image is generated based on the viewing preference information. For example, if the user's viewing preference is animation videos, a cartoon-type virtual image is generated; if the user's viewing preference is ancient costume videos, a ancient costume-type virtual image is generated.
[0047] In some other embodiments, a virtual image is generated based on a target image. For example, if the target image is a landscape image, a tree-shaped or cloud-shaped virtual image can be generated based on features such as trees or clouds in the landscape image; if the target image includes a portrait, a virtual image can be generated based on the portrait in the target image. The target image can be an image input by the user (e.g., an image of the user's character, an image of a character that the user likes), the target image can also be an avatar extracted from the user's account or an image stored in the user's account, the target image can also be a randomly selected image, and so on.
[0048] The virtual image is generated based on image data selected by the user. For example, the image data may include, but is not limited to, facial image data of the user, or facial image data of a character in a video watched by the user, to generate the appearance of the virtual image.
[0049] In some other embodiments, a virtual image is generated based on viewing preference information and a target image. For example, if the user's viewing preference is anime videos and the target image includes a portrait image, a virtual image of a cartoon portrait can be generated; if the user's viewing preference is ancient costume videos and the target image is a cartoon image, a virtual image of an ancient costume cartoon can be generated. For example, image processing technology is used to extract content features of the target image, and a virtual image generation model is used to reorganize the content features of the target image and viewing preference information to construct a virtual image. The virtual image generation model can use the existing Artificial Intelligence Generated Content (AIGC) technology and make fine adjustments, which will not be described in detail here.
[0050] The virtual image obtained based on the viewing preference information and / or the target image better matches the user's preferences, which helps to improve the user experience.
[0051] Step 203: Generate interaction information based on the video information and playback progress of the historically played videos and the viewing preference information, and drive the virtual image to interact based on the interaction information.
[0052] Generate interactive information based on the played videos and viewing preference information to interact with users through interactive information. The content and style of the interactive information are related to the historical played videos and viewing preference information, which can avoid spoilers. Among them, the interactive information can be displayed through a display window, the interactive information can also be set on the upper layer of the video, and the interactive information can also be played through voice. The output method of interactive information is diverse. Among them, the interactive information can include but is not limited to film and television chat, film and television Q&A, film and television recommendations, etc. The interactive information can also be but is not limited to the generated summary.
[0053] In an exemplary embodiment, the video information and playback progress of historically played videos, as well as the viewing preference information generated by the viewing preference analysis model, are input into an interactive information generation model fine-tuned based on a large language model (LLM for short), and the LLM's super contextual understanding and reasoning capabilities are used to generate interactive information with the user.
[0054] The virtual image is driven to interact based on the interaction information, and the facial movements of the virtual image, the torso movements of the virtual image, the sounds made by the virtual image, etc. are driven according to the content of the interaction information.
[0055] In an exemplary embodiment, a language sentiment analysis model can be used to analyze the meaning of the interactive information with the user, and the actions of the virtual image can be controlled according to the emotional style of the text. For example, if the content expresses a sad emotion, the virtual image can lower its head or other actions to express sadness; if the content contains positive emotions, the virtual image can smile or wave to express it.
[0056] The embodiment of the present invention provides an interactive method for video playback. Through historical playback videos, the playback progress of historical playback videos can be synchronized in real time, and the viewing preference information of users can be collected to provide personalized interaction synchronized with users. A virtual image related to the user's historical playback videos and viewing preference information can be personalized and created. Interactive information related to the user's historical playback videos and viewing preferences can also be generated. When the user wants to express emotions or opinions about the plot, corresponding feedback can be obtained, thereby improving the user's interactivity and personalization during the viewing process. A data storage server for users can also be built, and the video information and playback progress of historical playback videos can be synchronously stored in the data storage server, which can avoid repeated recommendations of historical playback videos and avoid revealing unplayed plots.
[0057] In step 203, the interactive information is generated based on the video information and playback progress of the historically played videos and the viewing preference information, including:
[0058] In response to detecting that the user's attention switches from leaving the currently playing video to returning to the currently playing video at the current time, generating summary information of a preset video segment played before the current time;
[0059] Interaction information is generated based on video information and playback progress of historically played videos, viewing preference information, and summary information.
[0060] The currently playing video can be understood as the video being played, and can include the currently playing TV series, the currently playing movie, the currently playing documentary, etc.
[0061] The current time can be understood as the time when the user's attention switches from the state of leaving the currently playing video to the state of returning to the state of watching the currently playing video. The current time can be determined by detection or by information input by the user, and the method for determining the current time is not limited.
[0062] The preset video clip can be understood as a video clip of a period of time before the current time. The preset video clip can be a video clip of a time period input by the user, a video clip of a preset time period before the current time, or a video clip of a time period during which the user's attention shift is detected. The preset time period can be input by the user or set in advance. For example, the preset time period is 10 minutes, 30 minutes, 80 minutes, etc., and the preset video clip can be a video clip 10 minutes, 30 minutes, or 80 minutes before the current time.
[0063] In some embodiments, in response to detecting that the user's attention switches from leaving the currently playing video to returning to the currently playing video at the current time, generating summary information of a preset video segment played before the current time includes:
[0064] In response to detecting that the user's attention is away from the currently playing video, determining a first playing progress of the currently playing video;
[0065] In response to detecting that the user's attention returns to the currently playing video, determining a second playing progress of the currently playing video;
[0066] Summary information is generated based on a video segment between a first playback progress and a second playback progress.
[0067] By detecting the first playback progress corresponding to the first time when the user's attention leaves the currently playing video, and the second playback progress corresponding to the current time when the user's attention returns to the currently playing video, determining that the video segment between the first playback progress and the second playback progress is the preset video segment can simplify user operations and ensure that the user watches the video continuously. For example, the first time can be obtained through the attention monitoring system. In some other embodiments, if the first time is not collected, the preset video segment can be a video segment of a preset period before the current time, for example, a video segment 10 minutes, 30 minutes, etc. before the current time.
[0068] In some other embodiments, the preset video clip can be understood as a video clip of a set length between the user's inquiry time (e.g., a video clip from 30 minutes before the inquiry time to the inquiry time). The inquiry time can be understood as the time when the user's inquiry information is received, for example, when the user's voice or text inquiry is received. For example, the user spontaneously asks "what just happened", "what happened ten minutes ago", etc.
[0069] The summary information may include one or more of a text summary, a video summary, and a picture summary.
[0070] When it is determined that the user's attention has been away from the currently playing video, the summary information of the preset video clip before the current time is provided to the user through interactive information, which can solve the discontinuity of the plot caused by the user's attention shift and improve the user's viewing experience. Among them, the preset video clip can be flexibly selected according to the user's needs.
[0071] refer to Figure 3 As shown, when the summary information includes a text summary and a video summary, the step of generating summary information of a preset video segment played before the current time includes steps 301 to 304.
[0072] Step 301: extract subtitle information of a preset video segment, and obtain a first text summary based on the subtitle information and viewing preference information.
[0073] Based on the positioning of the preset video clip, the corresponding subtitle information is obtained from the video in the user's data storage server; and the subtitle information is filtered as needed (for example, advertisements, repeated content, etc. are filtered out), key information such as major events, characters, highlights, etc. are extracted from the subtitle information, and then the key information is organized into a first text summary based on the user's viewing preference information. The subtitle information includes text information and / or voice information in the video source. If there is no subtitle file for the video, the voice information is converted into text information through speech recognition technology (referred to as ASR technology), and the subtitle information is extracted from the text information.
[0074] The subtitle information is filtered according to the viewing preference information, and key information in the subtitle information is extracted to obtain a first text summary. For example, the subtitle information is filtered according to one or more of the role preference, actor preference, and scene preference in the viewing preference information. When the subtitle information includes information of three roles A, B, and C, and the viewing preference information records that the user prefers role A and dislikes role C, role A is highlighted and role C is weakened while ensuring that the semantic expression of the first text summary is accurate.
[0075] In some embodiments, before processing the subtitle information, the subtitle information is converted into a format that can be understood and processed (e.g., JSON format), such as {"title":"Movie and TV title","subtitles":["timestamp":"00:00:01","text":"Subtitle text 1"]}, and the timestamp information is retained during the processing to facilitate the superposition of subtitle information added in real time later.
[0076] Step 302: extract image frames from a preset video clip, and determine the importance of the image frames based on viewing preference information.
[0077] The video in the preset video clip is divided into multiple image frames, and an importance score is assigned to one or more image frames in combination with the user's viewing preference information.
[0078] Step 303: Based on the first text summary and the importance of the image frame, key frames in the image frame are determined and a first video summary is formed according to the key frames.
[0079] The image frames with higher scores can be used to obtain key frames. If the importance of the image frames is determined based on the viewing preference information, key frames are obtained based on the image frames with higher scores, and the first video summary is formed based on the first text summary and the key frames. When multiple image frames form a video segment, if the importance of the video segment is determined based on the viewing preference information, key segments are obtained based on the video segments with higher scores, and the first video summary is formed based on the first text summary and the key segments. It is necessary to ensure that the first video summary matches the subtitle information.
[0080] Step 304: align the first text summary and the first video summary according to time and content to obtain a text summary and a video summary.
[0081] The first text summary and the first video summary can be integrated based on the alignment model through timestamp alignment and content matching to obtain a text summary and a video summary. Among them, time alignment: ensure that the image frames in the video summary are aligned in time with the corresponding parts in the text summary (for example, timestamp alignment) and there are no large blank moments; content alignment: ensure that the video summary and the text summary are matched in content.
[0082] In some other embodiments, the interactive method of video playback further includes:
[0083] Based on the key frames and the viewing preference information, the first text summary is filtered to obtain a second text summary.
[0084] It can be understood that the extracted subtitle information is filtered twice or more times to optimize the accuracy of the second text summary.
[0085] The key frame may be understood as the key frame in step 303. The viewing preference information may be understood as the viewing preference information in step 201.
[0086] In step 304, the first text summary and the first video summary are aligned according to time and content to obtain a text summary and a video summary, including:
[0087] The second text summary is aligned with the first video summary in terms of time and content to obtain a text summary and a video summary.
[0088] The second text summary is obtained by filtering the subtitle information twice or more times. The text summary and video summary obtained by aligning the second text summary with the first video summary are also more concise and accurate.
[0089] The above are text summaries and video summaries, which can be displayed through a small window. The text summary and video summary can be displayed synchronously in the video window; the text summary can also be displayed alone through the text display window, and the video summary can be displayed alone through the video window; the video summary can also be played in a small window, and the virtual image can feedback the text summary and broadcast it as interactive content with the user. Of course, the virtual image can also be driven to act according to the content of the text summary.
[0090] In an exemplary embodiment, the interactive method of video playback further includes:
[0091] Save the subtitle information and timestamp of the currently playing video, and mark the playback progress of the currently playing video.
[0092] When the user is watching the currently playing video, the subtitle information and timestamp of the currently playing video are saved synchronously. The subtitle information needs to be synchronized with the timestamp in order to determine the playback progress and to locate based on the timestamp later.
[0093] If the subtitle file already has timestamp information (such as SRT or SSA format subtitle files), these timestamps can be used directly. If there is no timestamp, the application needs to match the subtitle text after each ASR recognition with the timestamp.
[0094] Use appropriate protocols and tools to transmit subtitle information with timestamps to the data storage server in real time, and synchronously update the playback progress of the currently playing video. If the currently playing video is finished, update the complete viewing completion tag.
[0095] Among them, the currently played video is called the historically played video after it is played. The video information of the historically played video mainly includes subtitle information and the corresponding timestamp. The playback progress is mainly divided into two dimensions. One dimension is for the playback progress of a movie, which is mainly obtained through the real-time uploaded timestamp, and the other dimension is based on whether the entire movie has been watched. For example, for a 20-episode TV series A, if the user has watched the first 10 episodes, the playback progress is 50% played and has not been fully played.
[0096] In an exemplary embodiment, the interactive method of video playback further includes:
[0097] Detect at least two of the following: whether the user is in a detection area of the playback device, the user's head posture, and the user's eye gaze information, and obtain at least two detection results;
[0098] Determine, according to the type of the playback device, the influencing factors corresponding to at least two detection results respectively;
[0099] Based on at least two detection results and an influencing factor, it is detected that the user's attention leaves or returns to the currently playing video.
[0100] Detecting whether the user is in the detection area of the playback device can be understood as detecting obstacles in the detection area of the playback device through images or distances. If a human-shaped obstacle is detected in the detection area of the playback device, it is considered that the user is in the detection area and the user's attention is likely to be on the playback device. If no obstacle is detected in the detection area of the playback device, it is considered that the user is not in the detection area and the user's attention is likely to be away from the playback device.
[0101] Detecting the user's head posture can be understood as detecting the direction of the head posture, such as obtaining the direction of the head posture through image analysis. When it is detected that the user's head is facing the playback device, the probability that the user's attention is on the currently playing video is high. When it is detected that the user's head is away from the playback device, the probability that the user's attention is on the currently playing video is low.
[0102] Detecting the user's eye gaze information can be understood as detecting the direction of the eyeball's attention, such as obtaining the eyeball's attention direction through image analysis. When it is detected that the user's eyes are looking at the playback device, the probability that the user's attention is on the currently playing video is high. When it is detected that the user's eyes are away from the playback device, the probability that the user's attention is on the currently playing video is low.
[0103] The types of playback devices include one or more of a television, a desktop computer, a laptop computer, a tablet computer, and a mobile phone. Different types of playback devices have different screen areas.
[0104] Adjusting the impact factors corresponding to the detection results of at least two detection schemes according to the type of playback device can also be understood as adjusting the impact factors corresponding to the detection results according to the screen area of the playback device. The larger the screen area, the greater the impact of the results of whether the user is in the detection area of the playback device and the user's head posture on the detection result, and the larger the corresponding impact factor; conversely, the smaller the screen area, the greater the impact of the user's eye gaze information on the detection result, and the larger the corresponding impact factor.
[0105] The size of the impact factor corresponding to different detection schemes can be set according to the type of playback device, and the detection result of the user's attention can be determined by combining the results of multiple detection schemes, which helps to improve the accuracy of the detection result.
[0106] And adjust the impact factor based on the playback device and playback scene. The impact factor can be understood as the degree of influence of different detection methods on the detection results. For example, in scenarios where the playback device is far away from the user, such as TVs and projectors, the impact factor of the user's head posture on the final detection result will be increased; in scenarios where the playback device is close to the user, such as mobile phones and tablets, the impact of the eye gaze information on the final detection result will be increased.
[0107] For example, if the playback device is a TV, the influence factor of the user in the detection area of the playback device and the influence factor of the user's head posture are greater than the influence factor of the eye gaze information. The influence factor of whether the user is in the detection area of the playback device on the detection result is 0.3; the influence factor of the user's head posture (towards or away from the TV) on the detection result is 0.6; the influence factor of the user's eye gaze information (towards or away from the TV) on the detection result is 0.1. For example, if the user is in the detection area of the playback device, the probability that the user's attention is on the currently playing video is 0.3, and the probability that the user's head posture is toward the TV and the user's attention is on the currently playing video is 0.6. It can be considered that the user's attention is on the currently playing video. Weakening the influence of the user's eye gaze information on the detection result can reduce the detection accuracy.
[0108] In an exemplary embodiment, the attention monitoring system detects whether the user is within the detection area of the playback device. The attention monitoring system is hardware-based and can use distance detection (such as ultrasonic distance detection, infrared distance detection) and image recognition technology to perform fusion monitoring in the "macro" and "micro" dimensions, locate the timestamp of the user's attention transfer, and the timestamp of the attention recovery. Detecting whether the user is within the detection area of the playback device can be understood as the macro dimension, and detecting the user's head posture and detecting the user's eye gaze information can be understood as the micro dimension.
[0109] The position of the human body can be monitored through ultrasound or images. For example, the playback device emits ultrasound, then receives the echo of the ultrasound through a microphone, measures the time it takes for the sound wave to reflect back, and thus calculates the distance between the human body and the sound source. When the human body leaves the detection area of the playback device (the detection area is such as the forward visual range of the playback device), the distance changes. When the human body leaves the detection area of the playback device, it is determined that the user has left, and the current video playback timestamp is marked as the first moment of attention transfer; then, if the user moves into the detection area, this moment is marked as the current moment when the user's attention is restored.
[0110] Detect eye gaze information and head posture through images: Involving the user's personal privacy, image recognition monitoring requires the user to choose whether to turn on the camera. If the camera is turned on, the eye gaze information and head posture are used to jointly determine whether the user's attention is on watching the film and television; if the user does not look directly at the playback device for a long time, the marking start time is the first time the user's attention is diverted; then, when the user does not look directly at the playback device for a long time, and the time exceeds the threshold, and then returns the gaze to the playback device, this moment is marked as the current time when the user's attention is restored. Human posture is mainly aimed at scenes where the playback device is far away from the user, such as TVs, projectors, etc.; eyeball monitoring is mainly aimed at situations where the distance to the playback device is close, such as mobile phones, tablets, etc.
[0111] The detection result of the attention monitoring system adopts the principle of taking the first time of attention transfer first and the current time of attention recovery last to calculate the time period of the user's attention loss and obtain the preset video clip corresponding to the time period.
[0112] In some other embodiments, the timestamp of when the user's attention is transferred and the timestamp of when the user's attention is restored are obtained, the subtitle information and video information within the time period are matched, and the video information and playback progress of the historically played videos and the viewing preference information are combined to generate interaction information between the virtual image and the user.
[0113] In some other embodiments, based on the viewing preference information, the language style of the virtual image can also be generated to enhance the user interaction experience. For example, if the user's viewing preference includes anime videos, the language style of the virtual image can include anime style; if the user's viewing preference includes English videos, the language style of the virtual image can include English or Chinese and English.
[0114] After the user's permission, the torso posture of the avatar is generated based on the user's personalized data. The personalized data may include favorite movie and TV images, the user's zodiac sign, etc.
[0115] The appearance of the virtual image (including appearance, clothing, etc.) is generated based on the character image data selected by the user (including the user's own facial data, or the facial data of the characters in the video being watched, or other image data); the language style for the virtual image to interact with the user is generated based on the user's personalized data and the viewing preference information generated by the viewing preference analysis model; the torso posture of the virtual image is generated based on the language information of the interaction information.
[0116] The interactive method based on the above video playback includes:
[0117] First, according to the historical playback video, a user's data storage memory is built, the data storage memory records the subtitle information and corresponding timestamps of the historical playback video, and integrates the data to obtain the video information and playback progress of the historical playback video;
[0118] Secondly, using the user viewing preference analysis model, the user's viewing preference information is generated based on the video information and playback progress of the historically played videos;
[0119] Secondly, based on the viewing preference information, generate a virtual image and interaction style;
[0120] Further, based on the video information and playback progress of the historically played videos and the viewing preference information, generating interactive information;
[0121] Furthermore, through the virtual image and interactive information, under appropriate or preset trigger conditions, the virtual image is driven to interact with the user.
[0122] The interactive method for video playback in an embodiment of the present invention provides personalized interaction synchronized with users, improves the interactive effect, solves the problem that unified replies cannot resonate with users, and avoids repeated video recommendations and spoilers.
[0123] When a user leaves the video midway through watching, or engages in other distracting behaviors that cause loss of attention, the missed video clips can be summarized to ensure the continuity of the plot, such as providing feedback on a text summary while playing the video summary in a small window.
[0124] Application scenario 1 of the interactive method of video playback: real-time feedback of plot information.
[0125] When the user is watching the currently playing video, the subtitle information and timestamp data are synchronously transmitted to the user's viewing data storage server to obtain the playback progress. Through the playback progress, the user's watched and unwatched content is distinguished to avoid the presence of information that the user has not watched in the interactive information.
[0126] At the 21st minute, the user has not watched the subsequent plot. When the user puts forward the viewpoint "I guess the next plot is... (ellipsis indicates the user's viewpoint)", the interactive information can be "Hahaha, is this the legendary intuition? But I won't spoil it~", so that the interactive information does not reveal that the user has not watched the plot. At the 40th minute, the user has watched the specific plot, and the interactive information can mention the plot information. The user puts forward the viewpoint "Sure enough, I guessed it right." The interactive information can be "Yes, it is... (ellipsis indicates the plot summary)".
[0127] Application scenario 2 of the interactive method for video playback: personalized adjustment of interactive information based on the user's personalized information, video information of historically played videos, and playback progress.
[0128] The avatar will match the person in the picture uploaded by the user, and different users will have different visual avatars. Figure 4 As shown, the virtual image corresponding to a 20-year-old female user A is different from the virtual image corresponding to a 50-year-old male user B. The user's personalized information is obtained with the user's consent.
[0129] The language style of the virtual image's interactive information will also be adjusted according to the progress of the video and the viewing preference information. For example, a 50-year-old user B who likes to watch news and legal programs and a 20-year-old girl A who likes to watch Korean dramas will have different language styles for the interactive information of their corresponding virtual images under the same plot.
[0130] For example, Figure 4 When two users A and B with different identities and personalities watch the same plot X, they will have different feelings due to their different life experiences. The language style will be automatically adjusted based on the user's personalized information, video information of historically played videos, etc., and more positive video feedback will be output.
[0131] Application scenario three of the interactive method for video playback: in response to the user's attention diversion during the video playback, the lost time period is captured, and based on the lost time period (the time period between the first time and the current time), interactive information is generated.
[0132] In one case, the user temporarily leaves the detection range of the playback device while watching a movie because of housework or other factors. In this scenario, ultrasonic monitoring will prioritize the first time the user's attention is diverted and the current time when the attention is restored. After the user returns to the detection range of the playback device, it will automatically ask whether an introduction is needed or whether the preset video clip that was missed is played. If the user needs an introduction to the preset video clip, a text summary and video summary are generated based on the subtitle information and preset video clip from the first time to the current time, and the video summary is played in a small window, and the virtual image synchronously feedbacks the text summary.
[0133] In other cases, the user lowers his head to play with his mobile phone or make a phone call during the movie without leaving the detection range of the playback device. In this scenario, image monitoring will prioritize the first time the user's attention is diverted and the current time when the attention is restored. After the user's attention is restored, it will not automatically ask whether to play the plot summary, but will automatically record the time period. If within the time threshold, the user spontaneously asks "what just happened", a text summary and video summary are generated based on the subtitle information from the first time to the current time and the preset video clips, and the video summary is played in a small window, and the virtual image synchronously feedbacks the text summary.
[0134] Application scenario 4 of the interactive method of video playback:
[0135] When the user is watching a movie, the attention monitoring system does not capture the time period of the user's attention transfer and recovery, or the last time the attention was recovered is more than the time threshold from the current time. At this time, the user asks "what just happened?". Based on the subtitle information 10 minutes before the inquiry time and the preset video clips, a text summary and a video summary can be generated, and the video summary can be played in a small window, and the virtual image will synchronously feedback the text summary.
[0136] refer to Figure 5 As shown, at the inquiry time T when the user inquiry information is received, it is first determined whether the time period of the user's attention transfer is obtained. When it is determined that the time period of the user's attention transfer is obtained, it is then determined whether the duration between the inquiry time T and the user's attention transfer time t1 exceeds the threshold. When it is determined that the duration between T and t1 exceeds the threshold, a summary of the duration between the user's attention transfer time t1 and the user's attention recovery time t2 is generated. When it is determined that the time period of the user's attention transfer is not obtained, a summary of the 10 minutes before T is generated. If it is determined that the duration between T and t1 does not exceed the threshold, a summary of the 10 minutes before T is generated.
[0137] Application scenario 5 of the interactive method of video playback: automatic monitoring and reminder for students watching online classes.
[0138] For students watching online classes, if the attention monitoring system detects that the user's attention has shifted, whether it is ultrasonic monitoring or image monitoring, when the student's attention is restored, it will automatically prompt "We just detected that your attention has been distracted. Do you need to briefly review the knowledge points you just missed?" If so, the video summary and text summary will be played, and the question "If it is not clear enough, do you need to jump to the detailed node?" will be asked again.
[0139] Application scenario 6 of the interactive method of video playback: personalized video recommendations based on the types of videos the user has watched.
[0140] In the scenario of film and television drama recommendations, the recommended videos will be determined based on the user's historical video playback and viewing preference information.
[0141] For example, a user has a history of watching animations that he or she has loved in the past year, and recently, an animation film X has been added to the film and television resource library. Then, the film and television work X can be recommended to the user in his or her spare time.
[0142] For example, a 20-year-old Gemini female user is watching a certain theme series movie for the first time and is about to finish watching it. The user gives a positive review of this type of movie. Then, after the user watches it, you can recommend a theme series movie of the same type that the user has not watched.
[0143] Application scenario seven of the interactive method for video playback: Generate a summary for fast-forward content.
[0144] If a user is not interested in a certain preset video segment during the viewing process and wants to skip it, the user can fast forward to a certain timestamp and choose whether to watch the summary of the skipped video segment, which can save the user's time in watching the drama.
[0145] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an interactive device for video playback, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0146] like Figure 6 As shown, the interactive device for video playback in this embodiment includes: a first generating module 601 , a second generating module 602 and a driving interaction module 603 .
[0147] The first generating module 601 may be configured to generate viewing preference information based on video information and playback progress of historically played videos.
[0148] The second generating module 602 may be configured to view the preference information and generate a virtual image.
[0149] The driving interaction module 603 may be configured to generate interaction information based on the video information and playback progress of the historically played videos and the viewing preference, and drive the virtual image to interact based on the interaction information.
[0150] The driving interaction module 603 may also be configured to generate summary information of a preset video segment played before the current time in response to detecting that the user's attention switches from leaving the currently playing video to returning to the currently playing video at the current time;
[0151] Interaction information is generated based on video information and playback progress of historically played videos, viewing preference information, and summary information.
[0152] The driver interaction module 603 may also be configured as follows:
[0153] In response to detecting that the user's attention is away from the currently playing video, determining a first playing progress of the currently playing video;
[0154] In response to detecting that the user's attention returns to the currently playing video, determining a second playing progress of the currently playing video;
[0155] Summary information is generated based on a video segment between a first playback progress and a second playback progress.
[0156] In an exemplary embodiment, the preset video segment is a video segment within a preset period of time before the current time.
[0157] The interactive device for video playback also includes:
[0158] A first obtaining module may be configured to extract subtitle information of a preset video segment and obtain a first text summary based on the subtitle information and viewing preference information;
[0159] A second obtaining module may be configured to extract image frames from a preset video segment and determine the importance of the image frames based on the viewing preference information;
[0160] A forming module may be configured to determine key frames in the image frames based on the first text summary and the importance of the image frames and form a first video summary according to the key frames;
[0161] The third generating module may be configured to align the first text summary and the first video summary in terms of time and content to obtain a text summary and a video summary.
[0162] The interactive device for video playback also includes:
[0163] A third obtaining module may be configured to filter the first text summary based on the key frame and the viewing preference information to obtain a second text summary;
[0164] The third generating module may also be configured to align the second text summary with the first video summary in terms of time and content to obtain a text summary and a video summary.
[0165] The interactive device for video playback also includes:
[0166] A detection module detects at least two of the following: whether the user is within the detection area of the playback device, the user's head posture, and the user's eye gaze information, and obtains at least two detection results;
[0167] A first determination module determines, according to the type of the playback device, the impact factors corresponding to the at least two detection results;
[0168] The second determination module determines whether the user's attention leaves or returns to the currently playing video based on at least two detection results and an influencing factor.
[0169] In an exemplary embodiment, the video information of the historically played videos further includes: one or more of videos marked as fast-forward, videos marked as skipped, and videos marked as viewed multiple times.
[0170] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.
[0171] like Figure 7 , is a block diagram of an electronic device according to the interactive method for video playback according to an embodiment of the present application.
[0172] The electronic device includes a display module, and the display module is configured to display a virtual image and interactive information. The display device can be understood as a display screen or a projection device.
[0173] Figure 7 is a block diagram of an electronic device according to an interactive method for video playback according to an embodiment of the present application. Figure 7 As shown, the electronic device includes: one or more processors 701, a memory 702, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 701 is taken as an example.
[0174] The memory 702 is a non-transient computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the interactive method for video playback provided in the present application. The non-transient computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the interactive method for video playback provided in the present application.
[0175] The memory 702 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the interactive method for video playback in the embodiment of the present application (for example, the attached Figure 6 The processor 701 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 702, that is, the interactive method of video playback in the above method embodiment is implemented.
[0176] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of an electronic device for historical video playback, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to an electronic device for performing an interactive method for video playback via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0177] The electronic device of the interactive method for video playback may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.
[0178] The input device 703 can receive input digital or character information, as well as user settings of the electronic device playing the video stream and key signal input related to function control, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 704 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.
[0179] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0180] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0181] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0182] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0183] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
[0184] According to the technical solution of the embodiment of the present application, the accuracy and effectiveness of fault location performed by the trained fault location model are effectively improved.
[0185] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.
[0186] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.
Claims
1. A video playback interactive method, the method include: Generate viewing preference information based on video information and playback progress of historically played videos; generating a virtual image based on the viewing preference information; Interaction information is generated based on the video information and playback progress of the historically played videos and the viewing preference information, and the virtual image is driven to interact based on the interaction information.
2. The method according to claim 1, in, The generating of interactive information based on the video information and the playback progress of the historically played video and the viewing preference information includes: In response to detecting that the user's attention switches from leaving the currently playing video to returning to the currently playing video at the current time, generating summary information of a preset video segment played before the current time; The interactive information is generated based on the video information and the playback progress of the historically played video, the viewing preference information, and the summary information.
3. The method according to claim 2, in, The summary information includes a text summary and a video summary, and the generating of the summary information of the preset video segment played before the current time includes: Extracting subtitle information of the preset video segment, and obtaining a first text summary based on the subtitle information and the viewing preference information; Extracting image frames from the preset video clip, and determining the importance of the image frames based on the viewing preference information; Based on the first text summary and the importance of the image frame, determining a key frame in the image frame and forming a first video summary according to the key frame; The first text summary and the first video summary are aligned in terms of time and content to obtain the text summary and the video summary.
4. The method according to claim 3, further comprising: include: filtering the first text summary based on the key frame and the viewing preference information to obtain a second text summary; Aligning the first text summary with the first video summary according to time and content to obtain the text summary and the video summary includes: The second text summary and the first video summary are aligned in terms of time and content to obtain the text summary and the video summary.
5. The method according to claim 2, in, In response to detecting that the user's attention switches from leaving the currently playing video to returning to the currently playing video at the current time, generating summary information of the preset video segment played before the current time includes: In response to detecting that the user's attention is away from the currently playing video, determining a first playing progress of the currently playing video; In response to detecting that the user's attention returns to the currently playing video, determining a second playing progress of the currently playing video; The summary information is generated based on a video segment between the first play progress and the second play progress.
6. The method according to claim 2, in, The preset video segment is a video segment within a preset period of time before the current time.
7. The method according to claim 2, further comprising: include: Detect at least two of the following: whether the user is within a detection area of a playback device, the user's head posture, and the user's eye gaze information, to obtain at least two detection results; Determining, according to the type of the playback device, the impact factors corresponding to the at least two detection results respectively; Based on the at least two detection results and the influencing factor, it is determined whether the user's attention leaves or returns to the currently playing video.
8. The method according to claim 1, in, The video information of the historically played videos also includes: one or more of videos marked as fast-forward, videos marked as skipped, and videos marked as viewed multiple times.
9. An interactive device for video playback, the device include: A first generating module is configured to generate viewing preference information based on video information and playback progress of historically played videos; a second generating module configured to generate a virtual image based on the viewing preference information; The driving interaction module is configured to generate interaction information based on the video information and playback progress of the historically played video and the viewing preference information, and drive the virtual image to interact based on the interaction information.
10. An electronic device, It is characterized in that include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 8.
11. The electronic device according to claim 10, further comprising: include: The display module is configured to display the virtual image and the interaction information.
12. A non-transitory computer-readable storage medium storing computer instructions, It is characterized in that The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 8.
Citation Information
Cited By
Video note generation method and device based on AI
CN120705353A
