Multi-mode interaction method and device based on television video information and equipment with screen

Abstract

The invention provides a multi-mode interaction method and device based on television video information and equipment with a screen, the method is applied to an interaction television end or a digital television end and used for controlling at least one second screen through a television, and the method comprises the following steps: receiving interaction video information, the interactive video information comprises standard video information and interactive information; determining a video interaction frame and extension information corresponding to the video interaction frame at least based on the interaction video information; receiving an interaction request of a target audience for the video interaction frame, and extracting the extension information corresponding to the video interaction frame in the interaction information; and controlling a second screen corresponding to the target audience to play the extension information, wherein the second screen comprises a mobile terminal or a wearable device. According to the invention, the audience can actively interact with the television end and obtain interactive content from the second screen while normal linear playing of the television end is not influenced, so that the television viewing experience of the audience is not easy to interrupt.

Description

Technical Field

[0001] The present invention relates to the technical field of interactive television and Internet television services, and particularly relates to a multi-mode interaction method, device, and screen-equipped device based on television video information. Background Art

[0002] Existing television programs based on the television terminal are usually presented in a linear playback form. As passive receivers, viewers cannot adjust the content in real time or explore it deeply. Although this one-way dissemination mode meets the basic entertainment needs of the public, there are obvious deficiencies in the depth, relevance, and interactivity of information acquisition. For example, when viewers have questions about the background of characters in a television program, the development of the plot, the technical and tactical movements or scoring rules of a sports game, etc., they often need to pause watching the TV and search for relevant information through external channels (such as search engines, social media, or professional websites). However, the playback process of television programs on the television terminal is continuous. During the process of searching for information or thinking, viewers may miss important subsequent plots or exciting moments, resulting in an interrupted and incomplete viewing experience. In addition, even if viewers choose to continue watching, the questions in their minds may still exist, affecting the understanding and appreciation of the program content.

[0003] The interactive system based on the smart TV platform can wake up the floating information window through voice and retrieve encyclopedia data, historical clips, or technical and tactical analysis videos in real time. However, the relevance between the window content and the currently linearly played television program depends on the accuracy of the user's question, rather than the television program content itself. The relevance between the floating information window content and the television program cannot be effectively guaranteed. In addition, the appearance of the above floating information window blocks part of the TV screen, and its voice content is also likely to affect the viewing experience of other viewers.

[0004] The content of video information on mobile terminals (such as mobile phones or tablets, etc.) has gradually been accepted by the public due to its ability to be paused at any time and its diverse and rich interactive information (such as bullet screens). How to utilize the unique direct / broadcast rights advantages of television programs and the increasingly developed intelligent level of interactive television / digital television to solve the bottlenecks in the television manufacturing, sales, and television program production industries while providing viewers with a richer viewing experience has become an urgent problem faced by the television industry. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the present invention proposes a multi-mode interaction method, device, and screen-equipped device based on television video information, which solves the defects in the prior art that during the playback of television programs on interactive television / digital television, the television programs are linearly played, resulting in obvious deficiencies in the depth, relevance, and interactivity of information acquisition, and thus easily interrupting the viewing experience of viewers and lacking understanding and appreciation of the program content.

[0006] The technical solution of the present invention is realized as follows: A multi-mode interaction method based on television video information, which is applied to an interactive television terminal or a digital television terminal and is used to control at least one second screen through the television. The method includes: Receiving interactive video information, where the interactive video information includes standard video information and interaction information; Determining at least based on the interactive video information a video interaction frame and extended information corresponding to the video interaction frame; Receiving an interaction request from a target audience for the video interaction frame, and extracting the extended information corresponding to the video interaction frame in the interaction information; Controlling the second screen corresponding to the target audience to play the extended information, where the second screen includes a mobile terminal or a wearable device.

[0007] In an embodiment of the present invention, by executing and controlling the second screen at the interactive television / digital television terminal, when the interactive television / digital television terminal receives interactive video information including standard video information and interaction information, at least based on the interactive video information, a video interaction frame and extended information corresponding to the video interaction frame are determined; an interaction request from a target audience for the video interaction frame is received, and the extended information corresponding to the video interaction frame in the interaction information is extracted; the second screen corresponding to the target audience is controlled to play the extended information. This enables the audience to actively interact with the television terminal while not affecting the normal linear playback of the television terminal, and obtain in-depth and interactive information of interest other than the standard video information on the second screen (such as the character background in a TV program, the plot development, the technical and tactical actions or scoring rules in a sports game, etc.). The above extended information (such as the above in-depth and interactive information) can be awakened on the second screen according to the needs of the audience, and the television viewing experience of the audience is not easily interrupted; when the audience does not want to be disturbed at present, the extended information can also be pushed later, improving the television viewing experience of the audience. At the same time, when there are multiple audiences corresponding to the same interactive television / digital television terminal, the interactive television / digital television terminal can also push personalized extended information corresponding to different audiences according to the characteristic information of different audiences or customized services in the Internet TV, meeting the television viewing needs of different audiences without affecting the viewing experience of other audiences.

[0008] In one of the embodiments, for the above one or more technical solutions, the technical defect that the existing television video information (standard video information) cannot provide interactive content, cannot closely associate the interactive content with the standard video information, and cannot be accurate to specific video interaction frames, this embodiment also provides an improvement solution. Specifically, the method for generating the interactive video information includes: Obtaining the characteristic information of the standard video information; Obtaining an interactive extension model or copyright extension information corresponding to the characteristic information; Parse the standard video information based on the interaction extension model or copyright extension information to generate extension sub-information and first anchor information corresponding to the extension sub-information; Generate extension information and interaction information based on at least the extension sub-information and the first anchor information corresponding to the extension sub-information; Generate interactive video information based on the interaction information and the standard video information.

[0009] The above steps of the embodiments of the present invention usually occur before obtaining the standard video information. It is easy to understand that the above steps can also occur when the audience watches a TV video by obtaining the feature information of the standard video information; obtaining the interaction extension model or copyright extension information corresponding to the feature information; parsing the standard video information based on the interaction extension model or copyright extension information to generate extension sub-information and first anchor information corresponding to the extension sub-information; it is possible to generate extension sub-information in real time or in advance and assign the first anchor information corresponding to the standard video information to the extension sub-information. Then, generate extension information and interaction information based on at least the extension sub-information and the first anchor information corresponding to the extension sub-information; generate interactive video information based on the interaction information and the standard video information. By generating extension information and interaction information before or when watching a TV video, the embodiments of the present invention realize the generation of interactive video information for both live signals and rebroadcast / replay signals, which is beneficial to the close association between interactive content and standard video information, and also facilitates the subsequent accurate positioning of extension information to specific video interaction frames, further enhancing the TV viewing experience of the audience.

[0010] In one of the embodiments, for the above one or more technical solutions, since the playback time of the extension information may overlap with the possible occurrence time of the wonderful content on the TV side, it is easy to cause the audience to miss the wonderful content on the TV side when reading the interaction information because they are distracted by the second screen, affecting the TV viewing experience of the audience. This embodiment also provides an improvement solution. Specifically, the step of at least determining the video interaction frame and the extension information corresponding to the video interaction frame based on the interactive video information includes: Determine the reading length of each extension sub-information; Generate the first distribution state information based on the extension information according to the reading length of the extension sub-information and the first anchor information corresponding to the extension sub-information; Obtain the interesting video segments according to the standard video information, and the second anchor information corresponding to each interesting video segment; Generate the second distribution state information based on the interesting video segments according to the video length of the interesting video segments and the second anchor information corresponding to each interesting video segment; Obtain the audience characteristic information and setting information of the second screen; Determine vector offset information corresponding to the first anchor information based on the first distribution status information, the second distribution status information, and the viewer characteristic information and setting information of the second screen. Determine a video interaction frame and extended sub-information corresponding to the video interaction frame according to the first anchor information and the vector offset information.

[0011] In this embodiment, based on the reading length of the extended sub-information and the first anchor information corresponding to the extended sub-information, first distribution status information based on the extended information is generated, and based on the video length of the interesting video segments and the second anchor information corresponding to each interesting video segment, second distribution status information based on the interesting video segments is generated. Then, based on the first distribution status information, the second distribution status information, and the viewer characteristic information and setting information of the second screen, vector offset information corresponding to the first anchor information is determined; such that the vector offset information, which is the playback offset time of the extended sub-information, can be clearly, efficiently, and personalized determined according to multiple information such as the first distribution status information, the second distribution status information, and the viewer characteristic information and setting information of the second screen, avoiding the overlap between the playback time of the extended information and the time of the wonderful content on the TV side, and preventing the situation where the viewer misses the wonderful content on the TV side because of being distracted by the second screen when reading the interactive information, thus enhancing the viewer's TV viewing experience. The vector offset information is used to determine a video interaction frame according to the first anchor information and the extended sub-information corresponding to the video interaction frame.

[0012] In one of the embodiments, for one or more of the above technical solutions, the interaction between the TV side and the viewer is rather rigid. For example, when the viewer temporarily leaves, chats with others, or plays with the mobile phone and misses some video content on the TV side, the TV side cannot adjust the recommendation priority of the extended information in real time according to the missed video content by the viewer, or the viewer cannot determine the more relevant extended information from numerous extended information, resulting in the defect that the content of the extended information (such as the causality of the plot explanation) cannot correspond before and after. This embodiment also provides an improvement solution. Specifically, the viewer characteristic information includes viewer preference information, viewer age information, and playback environment information, and the setting information includes the playback setting information of the extended information; the step of generating second distribution status information based on the interesting video segments according to the video length of the interesting video segments and the second anchor information corresponding to each interesting video segment includes: Determine viewer focus information according to the playback environment information and the usage information of the second screen. Determine unplayed second interesting video segments with high relevance according to the played first interesting video segments that the viewer is focused on and the viewer preference information. Generate second distribution status information based on the second interested video clip according to the video length of the second interested video clip and the second anchor information corresponding to each second interested video clip; Prioritize each piece of extended sub-information according to the second interested video clip; Update the first distribution status information according to the reading length of the extended sub-information after the prioritization and the corresponding first anchor information; The step of determining the vector information corresponding to the anchor information based on the first distribution status information, the second distribution status information, and the viewer characteristic information and setting information of the second screen includes: Determine the reading efficiency of the viewer based at least on the viewer age information and the playback environment information; Determine the vector offset information corresponding to the first anchor information according to the updated first distribution status information, the second distribution status information, the reading efficiency, and the playback setting information, where the playback setting information includes the playback form and the playback efficiency.

[0013] In the embodiment of the present invention, the first interested video clip that has been played and is concerned by the viewer is determined according to the viewer's focus information, and then the second interested video clip that has not been played and has a high plot / item relevance to the first interested video clip is determined in combination with the viewer preference information (such as favorite actors, characters, favorite sports items, and plot scenes). And prioritize each piece of extended sub-information according to the second interested video clip, and update the first distribution status information according to the reading length of the extended sub-information after the prioritization and the corresponding first anchor information; it is realized that when the viewer is interested in or relatively focused on the standard video information currently played on the TV, more extended information with relatively high priority can be pushed according to the priority, and when the viewer is relatively uninterested in or more distracted by the standard video information currently played on the TV, relatively less extended information can be pushed according to the priority. At the same time, it can ensure that there is a high coherence between the first interested video clip that the viewer has seen before and the second interested video clip and its corresponding extended sub-information, and the above adjustment has high real-time performance, which can ensure the viewing experience of the viewer in various behavior modes.

[0014] At the same time, in this embodiment, the reading efficiency of the viewer is determined based at least on the viewer age information and the playback environment information; and then the vector offset information corresponding to the first anchor information is determined according to the updated first distribution status information, the second distribution status information, the reading efficiency, and the playback setting information. Make the determination of the vector offset information more accurate, reliable and real-time, and highly relevant to the viewer's behavior mode.

[0015] In the previous embodiment, as an improvement to a technical solution for determining a video interaction frame on the TV side, the steps of receiving an interaction request from a target viewer for the video interaction frame and extracting the extended information corresponding to the video interaction frame in the interaction information include: Adding a priority identification flag to the standard video information according to the extended sub-information sorted by priority, and the priority identification flag corresponds to the video interaction frame; When the video plays to the video interaction frame corresponding to the extended information, display the priority identification flag corresponding to the current video interaction frame; Based on the selection operation of the priority identification flag in the video interaction frame by the target viewer, extract the extended sub-information corresponding to the video interaction frame in the interaction information.

[0016] The steps of controlling the second screen corresponding to the target viewer to play the extended information, where the second screen includes a mobile terminal or a wearable device, include: Controlling the second screen corresponding to the target viewer to play the extended sub-information corresponding to the video interaction frame during the video interaction frame.

[0017] In the embodiment of the present invention, the extended sub-information is sorted by priority through the priority identification flag, so that the viewer can select the extended sub-information with relatively higher priority according to their preferences to play on the second screen, avoiding the situation of missing the wonderful content on the TV side, and further improving the TV viewing experience of the viewer.

[0018] In one of the embodiments, in view of the problem that in the above one or more technical solutions, with the rise of the existing short video industry and the relative reduction of TV video practitioners, this embodiment also provides a method for obtaining copyright expansion information to provide stable and reliable copyright income for TV video practitioners and ensure the sustainability of the TV industry. Specifically, the steps of obtaining the copyright expansion information corresponding to the feature information include: When it is determined that the feature information includes replay information, determine the keyword information in the feature information, and the keyword information includes video name information, video category information, and video date information; Retrieve the interactive information resource library of the server according to the keyword information, and the interactive information resource library includes copyright author information and corresponding copyright expansion information; Obtain the viewer customization service information based on the viewer account, and the customization service information includes the resource usage period of the purchased copyright author; When the resource usage period has not expired, obtain the copyright expansion information corresponding to the standard video information of the purchased copyright author.

[0019] In this embodiment, when it is determined that the feature information of the standard video information includes replay information, the keyword information in the feature information is determined, and then the interactive information resource library of the server is retrieved according to the keyword information; then, the customized service information of the audience is obtained based on the audience account, and the copyright extension information corresponding to the standard video information of the purchased copyright author is obtained. This enables TV video practitioners (including but not limited to sports commentators, film critics, or plot analysts) to expand the standard video information according to their personal understanding, thereby obtaining extended information with different personal styles and using it to generate subsequent interactive video information. The audience can subscribe to the customized services of different copyright authors according to their own preferences, so as to obtain different personalized copyright extension information. After the copyright extension information is parsed, extended sub-information and the first anchor information corresponding to the extended sub-information are obtained for subsequent use. The embodiment of the present invention not only protects the intellectual property rights and interests of copyright authors, but also, since the copyright extension information of the copyright author will only be sent to the TV end together with the standard video information when the customized service information matches, it avoids the possibility that the copyright author is suspected of infringing on the standard video information, making the production activities of copyright authors on the TV end have a natural advantage over the short video platforms on mobile terminals.

[0020] The present invention also provides a multi-mode interaction device based on TV video information, which is applied to an interactive TV end or a digital TV end and is used to control at least one second screen through the TV. The device includes: A receiving module, configured to receive interactive video information, where the interactive video information includes standard video information and interactive information; A determining module, configured to determine at least a video interaction frame and extended information corresponding to the video interaction frame based on the interactive video information; An extraction module, configured to receive an interaction request of a target audience for the video interaction frame, and extract the extended information corresponding to the video interaction frame in the interactive information; A control module, configured to control the second screen corresponding to the target audience to play the extended information, where the second screen includes a mobile terminal or a wearable device.

[0021] The present invention also provides a TV platform for implementing the multi-mode interaction method described in any one of the above.

[0022] The present invention also provides a screen-equipped device for interacting with the TV end as a second screen, where the TV end executes the multi-mode interaction method based on TV video information described in any one of the above.

[0023] The present invention also provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the multi-mode interaction method based on TV video information described above is implemented.

[0024] The present invention also provides a computer storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the multi-mode interaction method based on TV video information described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0026] Figure 1 It is a flowchart of a multi-mode interaction method based on TV video information according to the first embodiment of the present invention; Figure 2 It is a flowchart of a multi-mode interaction method based on TV video information according to the second embodiment of the present invention; Figure 3 It is a detailed flowchart of S202 according to the second embodiment of the present invention; Figure 4 It is another detailed flowchart of S202 according to the second embodiment of the present invention; Figure 5 It is a detailed flowchart of S209 according to the second embodiment of the present invention; Figure 6 It is a detailed flowchart of S211 according to the second embodiment of the present invention; Figure 7 It is a detailed flowchart of S213 according to the second embodiment of the present invention Figure 8 It is a schematic structural diagram of a multi-mode interaction device based on TV video information according to the third embodiment of the present invention; Figure 9 It is a schematic internal structure diagram of a computer according to another embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Well-known modules, units, and their connections, links, communications, or operations are not shown or not described in detail. Moreover, the described features, architectures, or functions can be combined in any manner in one or more embodiments. Those skilled in the art should understand that the following various embodiments are only for illustration purposes and are not used to limit the protection scope of the present invention. It can also be easily understood that the modules, units, or processing methods in the embodiments described herein and shown in the drawings can be combined and designed in various different configurations. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the protection scope of the present invention.

[0028] For the definitions of various nouns or methods referred to in the following embodiments, unless it is logically impossible to hold, the nouns or methods generally refer to the broad concepts that can be implemented on the premise of the content disclosed in the embodiments. Under such an understanding, all specific lower-level specific definitions of the nouns or methods should be regarded as the content of the present invention, and should not be narrowly understood or prejudicially interpreted on the grounds that the specific definition is not disclosed in the specification. Exemplarily, when the present invention mentions a cloud platform, it includes not only virtual network servers but also real physical devices, which not only have the ability to store data but also can have the ability to perform data operations, intelligent analysis, and reasoning. Similarly, on the premise that it can be logically implemented, the order of the steps in the method is flexible and changeable, and all specific lower-level specific definitions in the broad concepts of various nouns or methods belong to the protection scope of the present invention.

[0029] First Embodiment: Please refer to Figure 1 As shown, the embodiments of the present invention disclose a multi-mode interaction method based on television video information, which is applied to an interactive television terminal or a digital television terminal and is used to control at least one second screen through the television. The method includes: S11, receiving interactive video information, where the interactive video information includes standard video information and interactive information.

[0030] The interactive television terminal or digital television terminal described in this embodiment includes large-screen television terminals such as stand-alone televisions, wall-mounted televisions, and projectors. Among them, the stand-alone television and the wall-mounted television can be LCD televisions, LED televisions, QLED televisions, MiniLED televisions, or OLED televisions, and the projector can be an LCD projector, a DLP projector, or an LCoS projector.

[0031] Interactive video information is usually sent by an Internet server, a television station or a cloud platform. The standard video information in this embodiment is usually a traditional live / broadcast / rebroadcast television video signal, including traditional advertising information, news information, documentaries, science and education films, TV dramas and movie information continuously played by a central station, a local station, an on-demand channel or a special channel. Correspondingly, the above interaction information is interaction information other than the standard video information and is deeply associated with the content of the standard video information.

[0032] As a specific solution rather than a limitation, the above interaction information may include extended information, somatosensory information or smart home control information. The somatosensory information includes, but is not limited to, vibration information, olfactory information and light and shadow information, and is used to control the corresponding somatosensory terminal to work and realize interaction with the TV terminal. The smart home control information is used to control other smart home terminals in the network to work. The above extended information is played on the second screen and is mainly used to provide in-depth and interactive information (such as the background of characters in a TV program, the development of the plot, the technical and tactical actions or scoring rules of a sports game, etc.). The technical and tactical actions referred to in this embodiment include both individual technical actions and team tactical actions. In some other embodiments, the extended information may also be accompanied by advertising information.

[0033] S12, determine at least the video interaction frame and the extended information corresponding to the video interaction frame based on the interactive video information.

[0034] The interactive video information in this embodiment includes interaction information, and the interaction information includes a video interaction frame and the extended information corresponding to the video interaction frame. Among them, the video interaction frame is used to represent the reference time frame of the corresponding extended information in the standard video information. It should be noted that the video interaction frame may be one frame. Usually, the video interaction frame may be dozens or hundreds of frames in a continuous video, so that the prompt of the extended information is for a period of time.

[0035] S13, receive an interaction request from a target audience for the video interaction frame, and extract the extended information corresponding to the video interaction frame in the interaction information.

[0036] In this embodiment, when the audience watches the video corresponding to the standard video information in real time and plays to the corresponding video interaction frame, a corresponding reminder will be made on the second screen or the screen of the TV terminal to inform the audience that they can obtain the extended information corresponding to the current standard video information. The reminder methods include, but are not limited to, the visual or vibratory prompt of the second screen, and the indication mark pushed on the screen of the TV terminal. The audience can activate the interaction request for the video interaction frame by pressing the remote control button, voice command or touch operation on the second screen. At this time, the TV terminal extracts the extended information corresponding to the current video interaction frame in the interaction information.

[0037] Both the audience and the second screen in this embodiment can be multiple. Based on this, the target audience referred to in this embodiment is the audience who sends an interaction request for the video interaction frame, used to distinguish other audiences who do not send interaction requests. Taking the remote control buttons as an example, by obtaining the idle buttons of the remote control when playing standard video information, and allocating the idle buttons to different audiences and prompting them on the TV screen / each second screen. When the target audience presses the remote control button corresponding to their own identity, the personalized identity of the target audience is determined. The personalized identity includes but is not limited to the age, education level of the target audience, or the degree of knowledge accumulation of the category of the currently played standard video information. Similarly, voice commands can distinguish different audiences by recognizing voice feature information such as the timbre and voiceprint of different audiences, so as to determine the target audience; in addition, the account information or hardware address information of different second screens can also be used to distinguish different audiences, so as to determine the target audience.

[0038] S14, control the second screen corresponding to the target audience to play the extended information, and the second screen includes a mobile terminal or a wearable device.

[0039] The above playback timing can respond to the interaction request of the audience in real time or can be delayed playback, which is not limited in this embodiment. The mobile terminals in this embodiment include but are not limited to mobile phones, tablets or laptop computers, etc., and the wearable devices include but are not limited to AR / VR / MR glasses, AR / VR / MR helmets or smart watches / bracelets, etc.

[0040] The connection methods between the second screen and the TV terminal include but are not limited to Wi-Fi or Bluetooth pairing, QR code scanning, NFC (Near Field Communication), and multi-terminal sharing of Apps. It should be noted that the number of second screens can be multiple to simultaneously push extended information to different audiences, or push customized extended information to different audiences in the corresponding video interaction frames.

[0041] After determining the personalized identity of the target audience, determine the second screen corresponding to the target audience through the "second screen - audience" matching information. The acquisition method of the "second screen - audience" matching information can be manual or automatic, which is not limited in this embodiment.

[0042] The control of the second screen referred to in the present invention includes, but is not limited to, controlling the second screen to play extended information, controlling the second screen to perform a prompt operation based on the appearance timing of video interaction frames, controlling the second screen to store and analyze, classify, and establish associations for the extended information. Exemplarily, when the second screen plays the second relevant extended information B2, the first relevant extended information B1 and the third relevant extended information B3 associated with the second relevant extended information B2 will be associated and stored in the memory of the second screen for the audience to retrieve specifically. The control referred to in the present invention includes both directly establishing a connection with the second screen to achieve direct control of the second screen by the TV side; and may also include establishing a connection with the second screen through a server to achieve indirect control of the second screen by the TV side through the server.

[0043] As a preferred solution rather than a limitation, corresponding to the implementation of obtaining the personalized identity of the target audience in the above S13, in this step S14, by obtaining the age, educational level of the target audience, or the degree of knowledge accumulation of the category of the standard video information currently being played, and further performing operations on the extended information such as content increase or decrease and adjustment of knowledge depth, updated extended information corresponding to the age, educational level of the target audience, or the degree of knowledge accumulation of the category of the standard video information currently being played is obtained, and the second screen corresponding to the target audience is controlled to play.

[0044] In the embodiment of the present invention, by executing and controlling the second screen at the interactive TV / digital TV end, when the interactive TV / digital TV end receives interactive video information including standard video information and interactive information / digital TV end executes and controls the second screen, at least determining a video interaction frame and the extended information corresponding to the video interaction frame based on the interactive video information; receiving an interaction request from the target audience for the video interaction frame, and extracting the extended information corresponding to the video interaction frame in the interactive information; controlling the second screen corresponding to the target audience to play the extended information. This enables the audience to actively interact with the TV end while not affecting the normal linear playback of the TV end, and obtain in-depth and interactive information of interest other than the standard video information on the second screen (such as the background of characters in a TV program, the plot development, the technical and tactical actions or scoring rules of a sports game, etc.). The above extended information (such as the above in-depth and interactive information) can be awakened on the second screen according to the needs of the audience, and the TV viewing experience of the audience is not easily interrupted; when the audience does not want to be disturbed at present, the extended information can also be pushed later, improving the TV viewing experience of the audience. At the same time, when there are multiple audiences corresponding to the same interactive TV / digital TV end, the interactive TV / digital TV end can also push personalized extended information corresponding to different audiences according to the characteristic information of different audiences or customized services in the Internet TV, meeting the TV viewing needs of different audiences without affecting the viewing experience of other audiences.

[0045] Second Embodiment: In view of the technical deficiencies of the existing TV video information (standard video information) in the above embodiments, which cannot provide interactive content, cannot closely associate interactive content with standard video information, and cannot be accurate to specific video interaction frames. At the same time, due to the overlapping of the playback time of extended information and the possible occurrence time of wonderful content on the TV side, it is easy to miss the wonderful content on the TV side when the audience reads interactive information because they are distracted by the second screen, affecting the audience's TV viewing experience. Please refer to Figures 2 to 7 As shown, this embodiment also provides a multi-mode interaction method based on TV video information, including S201 - S214, where: S201, obtain the feature information of the standard video information.

[0046] The standard video information in this embodiment is the same as that in the above first embodiment and will not be elaborated here. The feature information of the standard video information includes whether the standard video information is live / replay information, keyword or program category information of the standard video information, etc.

[0047] S202, obtain the interactive extension model or copyright extension information corresponding to the feature information.

[0048] In this embodiment, the interactive extension model is mainly used to parse the standard video information related to live sports. The steps of obtaining the interactive extension model corresponding to the feature information include S2021 - S2024, where: S2021, construct and train interactive extension models corresponding to different program categories; S2022, when it is determined that the feature information includes live information, determine the program category information in the feature information, and the program category information includes major category information and minor category information; S2023, determine the corresponding interactive extension model based on the program category information; S2024, based on the interactive extension model, parse the structured feature information in the standard video information in real time, and obtain the technical and tactical action annotation information corresponding to the structured feature information. The technical and tactical action annotation information includes at least basic information of technical and tactical actions, analysis information of technical and tactical actions, and evaluation information of technical and tactical actions.

[0049] The above steps S2021 - S2024 build interaction expansion models for different program categories (such as ball games, board games, or track and field), then intelligently determine the corresponding interaction expansion models according to the program category information including the major category information and the minor category information, parse the structured feature information in the standard video information in real time based on the interaction expansion models, and obtain the technical and tactical action annotation information corresponding to the structured feature information. Based on the basic information of the technical and tactical actions, the technical and tactical actions in the live standard video information are identified and basically explained in real time. Based on the technical and tactical action analysis information, the advantages and disadvantages of the live standard video information are analyzed in real time. Based on the technical and tactical action evaluation information, the standard degree of the technical and tactical actions in the live standard video information is analyzed. The above interaction expansion models usually apply dynamic frame sampling technology, fuse spatio-temporal features through a two-stream convolutional network (TSN), extract dynamic features with a three-dimensional residual network (3D-ResNet50), output skeletal coordinates with pose key point detection (HRNet-W48), model action sequences with a graph convolutional network (ST-GCN), fuse multi-modal features with an attention mechanism, and store the technical and tactical action database through a knowledge graph (Neo4j).

[0050] In this embodiment, the copyright expansion information is produced and uploaded by the copyright author of the network server according to the existing playback content. The steps of obtaining the copyright expansion information corresponding to the feature information include S2025 - S2028, where: S2025, when it is determined that the feature information includes replay information, determine the keyword information in the feature information, and the keyword information includes video name information, video category information, and video date information; S2026, retrieve the interactive information resource library of the server according to the keyword information, and the interactive information resource library includes copyright author information and corresponding copyright expansion information; S2027, obtain the viewer customization service information based on the viewer account, and the customization service information includes the resource usage period of the purchased copyright author; S2028, when the resource usage period has not expired, obtain the copyright expansion information of the purchased copyright author corresponding to the standard video information.

[0051] In the above steps S2025 - S2028, when it is determined that the characteristic information of the standard video information includes replay information, the keyword information in the characteristic information is determined, and then the interactive information resource library of the server is retrieved according to the keyword information; then, the viewer customization service information is obtained based on the viewer account, and the copyright extension information corresponding to the standard video information of the purchased copyright author is obtained. This enables television video practitioners (including but not limited to sports commentators, film critics, or plot analysts) to expand the standard video information according to their personal understanding, thereby obtaining extended information with different personal styles and using it for subsequent generation of interactive video information. Viewers can subscribe to the customization services of different copyright authors according to their own preferences, thereby obtaining different personalized copyright extension information. After parsing, the copyright extension information obtains extended sub - information and the first anchor information corresponding to the extended sub - information for subsequent use. The embodiment of the present invention not only protects the intellectual property rights and benefits of copyright authors, but also, since the copyright extension information of the copyright author will only be sent to the TV side together with the standard video information when the customization service information matches, it avoids the possibility of copyright authors being suspected of infringing on the standard video information, giving copyright authors a natural advantage in production activities on the TV side compared to short - video platforms on mobile terminals.

[0052] An interactive method based on television video information according to an embodiment of the present invention also provides a method for a copyright author to produce copyright extension information based on existing TV dramas / movies, including S20209 - S20219, where: S20209, obtaining a first set of target video segments of the standard video information according to a single video plot; S20210, obtaining a second set of target video segments of the standard video information according to visual / dramatic scenes; S20211, obtaining a third set of target video segments corresponding to different preference tags according to viewer preference tags. The first set of target video segments, the second set of target video segments, and the third set of target video segments may include the same video segments; S2012, generating a basic correlation coefficient for each first target video segment in the first set of target video segments based on the plot relevance; S2013, performing a deduplication operation based on the first set of target video segments, the second set of target video segments, and the third set of target video segments to remove the same video segments; S2014, combining the first set of target video segments, the second set of target video segments after the deduplication operation, and the third target video segments corresponding to the preference tags based on different preference tags to obtain two or more personalized video segment sets. The first set of target video segments, the second set of target video segments, and the third set of target video segments in the same personalized video segment set are defined as personalized video segments; S2015. Generate personalized correlation coefficients for each personalized video clip in each set of personalized video clips according to the said basic correlation coefficient. S2016. When a copyright author creates extended sub-information for a certain personalized video clip, select one or more personalized video clips with personalized correlation coefficients higher than a preset value from the corresponding set of personalized video clips for the copyright author to determine the interesting video clips corresponding to the said preference tags. S2017. In response to the copyright author's operation of confirming the plot supplement for the extended sub-information, push the interesting video clips corresponding to the current extended sub-information for the copyright author to create the plot supplement information corresponding to the current extended sub-information. S2018. In response to the copyright author's operation of determining the first anchor point information for the extended sub-information, determine the reading length corresponding to the extended sub-information and determine the first anchor point information between two interesting video clips based on the said reading length. S2019. Determine copyright extension information based at least on the said extended sub-information and the corresponding first anchor point information, where the copyright extension information includes plot supplement information.

[0053] The above steps S2029 - S20211 provide a method for a copyright author to create copyright extension information based on existing TV dramas / movies, and provide a solution for the copyright author to quickly create extended sub-information and generate the first anchor point information, so that viewers can obtain the copyright extension information of the copyright author from the second screen while enjoying the TV. During the continuous playback of the TV, viewers will neither miss the exciting plot of the interesting video nor can they interact with the second screen to obtain the corresponding explanatory information of the copyright author. At the same time, the above steps S2029 - S20211 generate personalized correlation coefficients, enabling the copyright author to add personalized explanations that are the same as the viewers' preferences (determined by the above preference tags) according to personal understanding. Exemplarily, the extended information corresponding to the personalized explanation can be developed with the characters in the drama as the protagonist, greatly enhancing the viewers' sense of immersion. Finally, by pushing the interesting video clips corresponding to the current extended sub-information for the copyright author to create the plot supplement information corresponding to the current extended sub-information, when viewers miss some interesting video clips, they can select the plot supplement information created by the copyright author on the page of the extended sub-information, enabling viewers to quickly complete the plot chain and further enhancing the viewers' viewing experience. The plot supplement information can include corresponding videos, pictures, and text information.

[0054] It should be noted that by reading the above content, those skilled in the art can also obtain the method for creating copyright extension information based on existing competition videos, etc. All improvements and evolutions based on the concept of the present invention fall within the protection scope of the present invention.

[0055] S203. Parse the standard video information based on the interactive extension model or copyright extension information to generate extension sub-information and first anchor information corresponding to the extension sub-information.

[0056] Corresponding to the above interactive extension model, the extension sub-information usually includes video, picture, and text information for identifying, explaining, the advantages and disadvantages, and the action standard of technical and tactical actions. The first anchor information is usually determined by the interactive extension model and is the insertion time information of each extension sub-information corresponding to the standard video information. The first anchor information can be located at the preparation time of the athlete in the live broadcast screen or the action interval time (such as the game pause time or rest time, etc.).

[0057] Corresponding to the above copyright extension information, the extension sub-information and the first anchor information are usually produced and determined by the copyright author and can also include corresponding video, picture, and text information.

[0058] S204. Generate extension information and interactive information based on at least the extension sub-information and the first anchor information corresponding to the extension sub-information.

[0059] The extension information is the set of all extension sub-information and the first anchor information. The interactive information also includes the somatosensory information or smart home control information described in the above first embodiment, which will not be elaborated here.

[0060] S205. Generate interactive video information based on the interactive information and the standard video information.

[0061] S206. Determine the reading length of each extension sub-information.

[0062] As a quantitative index, the reading length in this step includes the number of words and pictures in the extension sub-information, as well as the duration of the video content.

[0063] S207. Generate first distribution state information based on the extension information according to the reading length of the extension sub-information and the first anchor information corresponding to the extension sub-information.

[0064] The first distribution state information includes the reading length of the extension sub-information and the first anchor information corresponding to the extension sub-information, which is used to characterize the usage time distribution of the second screen by the audience.

[0065] S208. Obtain the interesting video segments according to the standard video information, and the second anchor information corresponding to each interesting video segment.

[0066] The interesting video segments in this embodiment can be the main / wonderful plot in a movie / TV drama, or the appearance / competition segments of athletes that the audience is concerned about in sports, etc., and usually correspond to the interactive video information of the broadcast or replay. The second anchor information in this embodiment is similar to the first anchor information, and can be determined by the interactive expansion model or by the copyright author, and is the playback start time information corresponding to the standard video information for the TV side.

[0067] S209. Generate the second distribution status information based on the interesting video segments according to the video length of the interesting video segments and the second anchor information corresponding to each interesting video segment.

[0068] The second distribution status information includes the video length of the interesting video segments and the second anchor information corresponding to each interesting video segment, and is used to characterize the viewing time distribution of the audience on the TV side.

[0069] S210. Obtain the audience characteristic information and setting information of the second screen.

[0070] As an example rather than a limitation, the above-mentioned audience characteristic information includes audience preference information, audience age information, and playback environment information, and the setting information includes the playback setting information of the extended information. The audience preference information and audience age information can be obtained through the audience account. The audience preference information includes the actors, characters, preferred sports events, and plot scenes that the audience likes. The above-mentioned playback environment information can be obtained through the camera / microphone of the second screen or the TV side. The playback environment information includes but is not limited to the audience departure information, the audience conversation status information, and the current time information.

[0071] It should be noted that when the number of second screens is multiple, it is necessary to determine the audience characteristic information and setting information of each audience.

[0072] The previous step S209 further includes S2091 - S2095, where: S2091. Determine the audience focus information according to the playback environment information and the usage information of the second screen.

[0073] In this step S2091, the audience departure information, the audience conversation status information, and the current time information are obtained through the camera / microphone of the second screen or the TV side, and the time period when the audience focuses on the interactive video information is comprehensively judged in combination with the usage information of the second screen, so as to obtain the audience focus information.

[0074] It should be noted that when the number of second screens is multiple, it is necessary to determine the focus information of each audience.

[0075] S2092. Determine unplayed second interesting video segments with high relevance based on the first interesting video segments that the audience has focused on and the audience preference information. In this step S2092, by further screening interesting video segments during the period when the audience focuses on the standard video information on the TV side, the first interesting video segments that the audience has focused on are obtained. Combining with the audience preference information, unplayed second interesting video segments with high relevance are determined, which has extremely high real-time performance and can adjust the second interesting video segments in real time according to the personalized needs of the audience. Exemplarily, the high relevance referred to in this embodiment can be that the second interesting video segments have high relevance to the plot or technical and tactical actions of the first interesting video segments.

[0076] It should be noted that when the number of second screens is multiple, it is necessary to determine the unplayed second interesting video segments of each audience according to the first interesting video segments that each audience focuses on and the preference information of each audience.

[0077] S2093. Generate second distribution state information based on the second interesting video segments according to the video lengths of the second interesting video segments and the second anchor information corresponding to each second interesting video segment.

[0078] Compared with the above S209, in this step S2093, by extracting the second interesting video segments and their corresponding second anchor information, the distribution interval of the second distribution state information is effectively reduced, so that when the audience is relatively uninterested or distracted by the standard video information currently played on the TV, the number of second interesting video segments is reduced, and the energy consumption of the audience on the TV side is reduced.

[0079] It should be noted that when the number of second screens is multiple, it is necessary to generate second distribution state information corresponding to different audiences.

[0080] S2094. Perform priority sorting on each extended sub-information corresponding to the second interesting video segments.

[0081] This step is used to perform priority sorting on each extended sub-information according to the period when the audience focuses on the standard video information on the TV side and the second interesting video segments. It should be noted that when performing priority sorting, the main line / exciting plot has a relatively high weighting value to avoid the situation where the audience misses it.

[0082] Exemplarily, when performing priority sorting on each extended sub-information, priority labels will also be assigned to each second interesting video segment, such as "main line", "special effects", "XX character / star / athlete", "XX technical and tactical action", etc., and different color assignments will be performed on the priority labels according to different priorities.

[0083] S2095. Update the first distribution status information according to the reading lengths of the extended sub-information sorted by priority and the corresponding first anchor information.

[0084] Compared with the above S207, in this step S2095, through the reading lengths of the extended sub-information sorted by priority and the corresponding first anchor information, preferably, the extended sub-information with higher priority is further screened, and the first distribution status information is updated according to the screened extended sub-information with higher priority, its corresponding reading length, and the first anchor information. This also effectively reduces the distribution range of the first distribution status information, so that when the audience is relatively uninterested in or distracted by the standard video information currently being played on the TV, the energy consumption of the audience on the second screen is reduced. At the same time, in this step S2095, the extended sub-information with higher audience attention can also be quickly screened out from numerous extended sub-information, reducing the situation where the first distribution status information and the second distribution status information intersect. This enhances the viewing experience of the audience.

[0085] It should be noted that when the number of second screens is multiple, the first distribution status information corresponding to different audiences needs to be generated. Thereby ensuring the subsequent generation of different vector offset information and pushing personalized extended sub-information to different audiences.

[0086] In the above steps S2091 - S2095, the first interesting video segment that the audience has watched is determined based on the audience's focus information, and then, in combination with the audience's preference information (such as favorite actors, characters, favorite sports events, and plot scenes), the unplayed second interesting video segment that has a high plot / item relevance to the first interesting video segment is determined. The priority of each extended sub-information is sorted according to the second interesting video segment, and the first distribution status information is updated according to the reading lengths of the extended sub-information sorted by priority and the corresponding first anchor information. It is realized that when the audience is interested in or relatively focused on the standard video information currently being played on the TV, more extended information with relatively higher priority can be pushed according to the priority, and when the audience is relatively uninterested in or distracted by the standard video information currently being played on the TV, relatively less extended information with higher priority can be pushed according to the priority. At the same time, it can ensure that the first interesting video segment that the audience has seen before has a high coherence with the second interesting video segment and its corresponding extended sub-information, and the above adjustment has high real-time performance, which can ensure the viewing experience of the audience in various behavior patterns.

[0087] S211. Determine the vector offset information corresponding to the first anchor information based on the first distribution status information, the second distribution status information, the audience characteristic information, and the setting information of the second screen.

[0088] When there is an intersection between the first distribution state information and the second distribution state information, the current extended sub-information can be played on the second screen in a delayed playback or segmented playback manner.

[0089] As a preferred solution rather than a limitation, this step S211 further includes S2111 - S2112, where: S2111, determine the reading efficiency of the audience at least according to the audience age information and the playback environment information.

[0090] In this step, by determining the audience departure information, the audience conversation status information, and the current time information in the playback environment information, the current state of the audience is judged, and at the same time, the reading efficiency of the audience is determined in combination with the audience age information.

[0091] S2112, according to the updated first distribution state information, the second distribution state information, the reading efficiency, and the playback setting information, determine the vector offset information corresponding to the first anchor point information, where the playback setting information includes the playback form and the playback efficiency.

[0092] The vector offset information in this embodiment is the offset time amount corresponding to the first anchor point information, which can be offset forward or backward. Usually, it is optimal that there is no intersection between the first distribution state information and the second distribution state information. The role of the vector offset information is on the one hand to ensure that there is no intersection between the first distribution state information and the second distribution state information, and on the other hand to adjust the position of the video interaction frame personalized according to the reading efficiency of the audience to avoid affecting the viewing experience of the audience.

[0093] S212, determine the video interaction frame and the extended sub-information corresponding to the video interaction frame according to the first anchor point information and the vector offset information.

[0094] S213, receive the interaction request of the target audience for the video interaction frame, and extract the extended sub-information corresponding to the video interaction frame in the interaction information.

[0095] As a preferred solution rather than a limitation, please refer to Figure 7 As shown, this step S213 can specifically include S2131 - S2133, where: S2131, add a priority recognition identifier to the standard video information according to the extended sub-information sorted by priority, and the priority recognition identifier corresponds to the video interaction frame; S2132, when the video plays to the video interaction frame corresponding to the extended information, display the priority recognition identifier corresponding to the current video interaction frame; Corresponding to the second paragraph of the above explanation of S2094, in this step S2132, the priority recognition identifier can be displayed at the position of the video interaction frame corresponding to the standard video information. The text content of the priority recognition identifier can be "main line", "special effect", "XX character / star / athlete", "XX technical and tactical action", etc., and different color identification operations are performed on the priority recognition identifier according to different priorities. As a preferred solution rather than a limitation, corresponding to the situation of multiple audiences and multiple second screens, in this step S2132, the text such as "Audience 1 / Audience 2 / Audience 3" can also be noted at the text content of the priority recognition identifier to indicate that the current video interaction frame is for the extended information of one or more of the audiences.

[0096] S2133. Based on the selection operation of the priority recognition identifier in the video interaction frame by the target audience, extract the extended sub-information corresponding to the video interaction frame in the interaction information.

[0097] The audience can complete the selection of the priority recognition identifier in the video interaction frame through remote control key presses, voice commands, or touch operations on the second screen. At this time, the TV terminal extracts the extended sub-information corresponding to the current video interaction frame in the interaction information.

[0098] S214. Control the second screen corresponding to the target audience to play the extended sub-information corresponding to the video interaction frame during the playback of the video interaction frame.

[0099] In this embodiment, the characteristic information of the standard video information is obtained; the interactive extension model or copyright extension information corresponding to the characteristic information is obtained; the standard video information is parsed based on the interactive extension model or copyright extension information to generate extension sub-information and first anchor information corresponding to the extension sub-information; the generation of the extension sub-information in real time or in advance can be realized, and the first anchor information corresponding to the standard video information is assigned to the extension sub-information. Then, based on at least the extension sub-information and the first anchor information corresponding to the extension sub-information, extension information and interactive information are generated; the interactive video information is generated based on the interactive information and the standard video information. By generating the extension information and the interactive information before or while watching the TV video, the embodiment of the present invention realizes the generation of the interactive video information for both the live signal and the rebroadcast / replay signal, which is beneficial to the close association between the interactive content and the standard video information, and also facilitates the subsequent accurate positioning of the extension information to the specific video interaction frame, further improving the TV viewing experience of the audience. At the same time, in this embodiment, the first distribution state information based on the extension information is generated according to the reading length of the extension sub-information and the first anchor information corresponding to the extension sub-information, and the second distribution state information based on the interesting video segment is generated according to the video length of the interesting video segment and the second anchor information corresponding to each interesting video segment. Then, based on the first distribution state information, the second distribution state information, the audience characteristic information and the setting information of the second screen, the vector offset information corresponding to the first anchor information is determined; so that the vector offset information serving as the playback offset time of the extension sub-information can be clearly, efficiently and personalized determined according to multiple information such as the first distribution state information, the second distribution state information, the audience characteristic information and the setting information of the second screen, avoiding the overlap between the playback time of the extension information and the time of the wonderful content on the TV side, and preventing the situation that the audience misses the wonderful content on the TV side because they are distracted by the second screen when reading the interactive information, thus improving the TV viewing experience of the audience. The vector offset information is used to determine the video interaction frame according to the first anchor information and the extension sub-information corresponding to the video interaction frame.

[0100] Third Embodiment: Please refer to Figure 8 As shown in the figure, the present invention also provides a multi-mode interaction device 100 based on TV video information, which is applied to an interactive TV terminal or a digital TV terminal and is used to control at least one second screen through the TV. The device includes: A receiving module 110, configured to receive interactive video information, where the interactive video information includes standard video information and interactive information; A determining module 120, configured to determine at least the video interaction frame and the extension information corresponding to the video interaction frame based on the interactive video information; An extraction module 130 is configured to receive an interaction request of a target audience for the video interaction frame, and extract the extended information corresponding to the video interaction frame in the interaction information. A control module 140 is configured to control a second screen corresponding to the target audience to play the extended information, where the second screen includes a mobile terminal or a wearable device.

[0101] The modules in this embodiment correspond one by one to the steps in the above first embodiment, and will not be elaborated here.

[0102] All the modules / units in this embodiment are the same as the corresponding steps in the above one or more method embodiments, and their logical relationships and working principles are also the same, which will not be elaborated here. Those skilled in the art can learn the corresponding virtual modules or units from the above method embodiments to make them correspond to the steps of the above method embodiments. The virtual modules / units not disclosed in this embodiment should also be regarded as part of the disclosure of the present invention.

[0103] In the embodiment of the present invention, by executing and controlling a second screen at an interactive TV / digital TV end, when the interactive TV / digital TV end receives interactive video information including standard video information and interaction information, the video interaction frame and the extended information corresponding to the video interaction frame are determined at least based on the interactive video information; an interaction request of a target audience for the video interaction frame is received, and the extended information corresponding to the video interaction frame in the interaction information is extracted; a second screen corresponding to the target audience is controlled to play the extended information. This enables the audience to actively interact with the TV end while not affecting the normal linear playback of the TV end, and obtain in-depth and interactive information of interest other than the standard video information on the second screen (such as the background of characters in a TV program, the development of the plot, the technical and tactical actions or scoring rules of a sports game, etc.). The above extended information (such as the above in-depth and interactive information) can be awakened on the second screen according to the needs of the audience, and the TV viewing experience of the audience is not easily interrupted; when the audience does not want to be disturbed at present, the extended information can also be pushed later, improving the TV viewing experience of the audience. At the same time, when there are multiple audiences corresponding to the same interactive TV / digital TV end, the interactive TV / digital TV end can also push personalized extended information corresponding to different audiences according to the characteristic information of different audiences or customized services in the Internet TV, meeting the TV viewing needs of different audiences without affecting the viewing experience of other audiences.

[0104] Those skilled in the art can clearly understand that for the convenience and conciseness of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working processes of the systems, devices, and units described above, reference can be made to the corresponding processes in the foregoing method embodiments, and details are not described herein again. An embodiment of the present invention further provides a television platform for implementing the multi-mode interaction method described in the first or second embodiment above.

[0105] An embodiment of the present invention further provides a screen-equipped device for interacting with a television terminal as a second screen, and the television terminal executes a multi-mode interaction method based on television video information described in the first or second embodiment above.

[0106] An embodiment of the present invention further provides a computer storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements a multi-mode interaction method based on television video information in each of the above embodiments.

[0107] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium, and when the computer program is executed, it can include the processes of the embodiments of a multi-mode interaction method based on television video information as described above. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include non-volatile and / or volatile memories. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0108] Alternatively, if the above integrated units of the present invention are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiments of the present invention, in essence or the part that contributes to the related art, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the methods of the various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as removable storage devices, RAM, ROM, magnetic disks, or optical discs.

[0109] Corresponding to the above computer storage medium, in one embodiment, a computer device is also provided. The computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. Among them, when the processor executes the program, it implements a multi-mode interaction method based on TV video information as in the above various embodiments.

[0110] The computer device can be a terminal, and its internal structure diagram can be as Figure 9 shown. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a multi-mode interaction method based on TV video information. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the outer shell of the computer device, or an external keyboard, a touchpad, or a mouse, etc.

[0111] In the embodiments of the present invention, the second screen is executed and controlled at the interactive TV / digital TV end. When the interactive TV / digital TV end receives interactive video information including standard video information and interactive information, the second screen is executed and controlled at the digital TV end, and at least the video interaction frame and the extended information corresponding to the video interaction frame are determined based on the interactive video information; the interaction request of the target audience for the video interaction frame is received, and the extended information corresponding to the video interaction frame in the interactive information is extracted; the second screen corresponding to the target audience is controlled to play the extended information. This enables the audience to actively interact with the TV end while not affecting the normal linear playback of the TV end, and obtain in-depth and interactive information of interest other than the standard video information on the second screen (such as the background of characters in a TV program, the development of the plot, the technical and tactical actions or scoring rules of a sports game, etc.). The above extended information (such as the above in-depth and interactive information) can be awakened on the second screen according to the needs of the audience, and the TV viewing experience of the audience is not easily interrupted; when the audience does not want to be disturbed at present, the extended information can also be pushed later, improving the TV viewing experience of the audience. At the same time, when there are multiple audiences corresponding to the same interactive TV / digital TV end, the interactive TV / digital TV end can also push personalized extended information corresponding to different audiences according to the characteristic information of different audiences or customized services in the Internet TV, meeting the TV viewing needs of different audiences without affecting the viewing experience of other audiences.

[0112] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0113] The above embodiments only represent several implementation manners of the present invention, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the invention patent should be subject to the appended claims.

Claims

1. A multi-mode interaction method based on television video information, characterized in that: Applied to an interactive television terminal or a digital television terminal, and used to control at least one second screen through the television, the method comprises: Receiving interactive video information, wherein the interactive video information includes standard video information and interactive information; Determine a video interaction frame and extended information corresponding to the video interaction frame based at least on the interactive video information; receiving an interaction request for the video interaction frame from a target audience, and extracting the extended information corresponding to the video interaction frame from the interaction information; The second screen corresponding to the target audience is controlled to play the extended information, where the second screen includes a mobile terminal or a wearable device.

2. The method according to claim 1, characterized in that The method for generating interactive video information comprises: Obtaining characteristic information of standard video information; Obtaining an interactive extension model or copyright extension information corresponding to the feature information; Parsing the standard video information based on the interactive extension model or the copyright extension information to generate extended sub-information and first anchor point information corresponding to the extended sub-information; generating extended information and interactive information based at least on the extended sub-information and the first anchor point information corresponding to the extended sub-information; Interactive video information is generated based on the interactive information and the standard video information.

3. The method according to claim 2, characterized in that The step of determining the video interaction frame and the extended information corresponding to the video interaction frame at least based on the interactive video information includes: Determine the reading length of each extended sub-information; generating first distribution state information based on the extended information according to the reading length of the extended sub-information and the first anchor point information corresponding to the extended sub-information; Acquire the video segments of interest and the second anchor point information corresponding to each video segment of interest according to the standard video information; generating second distribution state information based on the video segments of interest according to the video length of the video segments of interest and the second anchor point information corresponding to each video segment of interest; Obtaining audience feature information and setting information of the second screen; Determine vector offset information corresponding to the first anchor point information based on the first distribution state information, the second distribution state information, and the audience feature information and setting information of the second screen; The video interaction frame and the extended sub-information corresponding to the video interaction frame are determined according to the first anchor point information and the vector offset information.

4. The method according to claim 3, characterized in that The audience characteristic information includes audience preference information, audience age information and playback environment information, and the setting information includes playback setting information of the extended information; the step of generating the second distribution state information based on the video segments of interest according to the video length of the video segments of interest and the second anchor point information corresponding to each video segment of interest includes: Determine audience focus information based on playback environment information and second screen usage information; Determine a highly relevant but unplayed second video segment of interest based on the played first video segment of interest that the audience focuses on and the audience preference information; generating second distribution state information based on the second video segments of interest according to the video length of the second video segments of interest and the second anchor point information corresponding to each second video segment of interest; Prioritizing each extended sub-information according to the second interesting video segment; updating the first distribution state information according to the read length of the priority-sorted extended sub-information and the corresponding first anchor point information; The step of determining the vector information corresponding to the anchor point information based on the first distribution state information, the second distribution state information, and the audience feature information and setting information of the second screen comprises: Determine the audience's reading efficiency based on at least the audience's age information and the playback environment information; The vector offset information corresponding to the first anchor point information is determined according to the updated first distribution state information, the second distribution state information, the reading efficiency and the playback setting information, wherein the playback setting information includes the playback form and the playback efficiency.

5. The method according to claim 4, characterized in that The step of receiving an interaction request from a target audience for the video interaction frame and extracting the extended information corresponding to the video interaction frame from the interaction information comprises: Adding a priority identification mark to the standard video information according to the priority-sorted extended sub-information, wherein the priority identification mark corresponds to the video interaction frame; When the video is played to the video interaction frame corresponding to the extended information, the priority identification mark corresponding to the current video interaction frame is displayed; Based on the target audience's selection operation on the priority identification mark in the video interaction frame, the extended sub-information corresponding to the video interaction frame in the interaction information is extracted.

6. The method according to claim 5, characterized in that The step of controlling a second screen corresponding to the target audience to play the extended information, wherein the second screen includes a mobile terminal or a wearable device, comprises: The second screen corresponding to the target audience is controlled to play the extended sub-information corresponding to the video interaction frame in the video interaction frame.

7. The method according to claim 6, characterized in that The step of obtaining copyright extension information corresponding to the characteristic information includes: When it is determined that the characteristic information includes replay information, determining keyword information in the characteristic information, the keyword information including video name information, video category information and video date information; According to the interactive information resource library of the keyword information retrieval server, the interactive information resource library includes copyright author information and corresponding copyright extension information; Obtaining customized service information for viewers based on the viewer account, wherein the customized service information includes the resource usage period of the author of the purchased copyright; When the resource usage period has not expired, the copyright extension information of the purchased copyright author corresponding to the standard video information is obtained.

8. A multi-mode interactive device based on television video information, characterized in that: Applied to an interactive television terminal or a digital television terminal, and used to control at least one second screen through the television, the device comprises: A receiving module, used for receiving interactive video information, wherein the interactive video information includes standard video information and interactive information; A determination module, configured to determine a video interaction frame and extended information corresponding to the video interaction frame based at least on the interactive video information; An extraction module, configured to receive an interaction request from a target audience for the video interaction frame, and extract the extended information corresponding to the video interaction frame from the interaction information; The control module is used to control a second screen corresponding to the target audience to play the extended information, and the second screen includes a mobile terminal or a wearable device.

9. A device with a screen, characterized in that: Used to interact with a television end as a second screen, the television end executes a multi-mode interaction method based on television video information as described in any one of claims 1 to 7.

10. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a multi-mode interaction method based on television video information as described in any one of claims 1 to 7 is implemented.

Images