Method, content manager, and program for automatically generating an extension for AV content

Abstract

A method, content manager, and computer program for automatically generating enhancements to audiovisual (AV) content includes: analyzing data about consumer interactions with a processing unit, generating consumer insights about the original AV content, automatically associating the consumer insights with segments of the original AV content, automatically generating content for the segments according to the consumer insights, and injecting the generated content into the original AV content to create modified AV content. The method, content manager, and computer program may further receive data about consumer interactions with the original AV content through a network interface and automatically transmit the modified AV content.

Description

Technical Field

[0001] The present disclosure relates to creating and managing audiovisual (AV) content, and more particularly to automatically expanding AV content in response to viewer interaction with the AV content.

Background Art

[0002] The development of the EDVAC system in 1948 is often cited as the beginning of the computer age. Since then, computer systems have evolved into extremely complex devices. Today's computer systems typically include a combination of sophisticated hardware and software components, application programs, operating systems, processors, buses, memories, input / output devices, and the like. As performance has increased with advancements in semiconductor processing and computer architecture, more evolved computer software has evolved to take advantage of the higher performance of these functions, leading to the current computer systems that are much more powerful than just a few years ago.

[0003] A variety of content creators have come to utilize these improved functions via content streaming and podcasting. Content streaming generally refers to the transmission of audio or video or both files, typically over the Internet, from a server to a client, and is widely used to view video clips and movies on various media players such as televisions, computers, tablets, and smartphones. Typically, in content streaming, the viewer / end-user can start playing the content before the entire file has been transmitted. In contrast, podcasting also generally refers to the transmission of audio or video or both files, typically over the Internet, from a server to a client. However, in podcasting, the transmission generally occurs before the viewer starts playing the content.

[0004] Streaming and podcasting themselves have proven to be excellent platforms for things such as advertisements, product reviews, problem troubleshooting (e.g., information technology, home maintenance, industrial-related issues).

Summary of the Invention

[0005] According to an embodiment of the present disclosure, a method for automatically generating an extension for audiovisual (AV) content. One embodiment may include analyzing, by a processing unit, data about consumer interaction to generate consumer insights about the original AV content, automatically associating the consumer insights with segments of the original AV content, automatically generating content for the segments according to the consumer insights, and injecting the generated content into the original AV content to create modified AV content. The embodiment may further include receiving, by a network interface, data about consumer interaction with the original AV content and automatically transmitting the modified AV content.

[0006] According to an embodiment of the present disclosure, a content manager. One embodiment includes a content creation server communicatively coupled to a plurality of content consumer devices, the server including a processor coupled to a memory. The processor and the memory are configured to receive data about consumer interaction with original audiovisual (AV) content, analyze the data about consumer interaction to generate consumer insights about the original AV content, automatically associate the consumer insights with segments of the original AV content, automatically generate content for the segments according to the consumer insights, inject the generated content into the original AV content to create modified AV content, and automatically transmit the modified AV content.

[0007] According to an embodiment of the present disclosure, a computer program product for automatically generating an extension for audiovisual (AV) content. The computer program product in one embodiment may include a computer-readable storage medium having program instructions executable by a processor embodied therein. The program instructions may cause the processor to receive data about consumer interaction with original AV content, analyze the data about consumer interaction to generate consumer insights about the original AV content, automatically associate the consumer insights with segments of the original AV content, automatically generate content for the segments according to the consumer insights, inject the generated content into the original AV content to create modified AV content, and automatically transmit the modified AV content.

[0008] The above summary is not intended to describe every illustrative embodiment or every implementation of the present disclosure.

[0009] The drawings included in this application are incorporated herein and form a part thereof. These drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure. The drawings merely illustrate specific embodiments and do not limit the present disclosure.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Best Mode for Carrying Out the Invention

[0011] The present invention is subject to various modifications and alternative forms, and specific examples thereof are shown by way of illustration in the drawings and will be described in detail. However, it should be understood that the intention is not to limit the present invention to the specific embodiments described. On the contrary, it is intended to cover all modifications, equivalents, and alternative forms that fall within the spirit and scope of the present invention.

[0012] Aspects of the present disclosure relate to creating and managing audiovisual (AV) content, and more particularly to automatically extending AV content in response to consumer interactions with AV content such as comments and viewing behavior. The present disclosure is not necessarily limited to such applications, but various aspects of the present disclosure can be understood through discussion of various examples using this context.

[0013] Some embodiments of the present disclosure can provide an automated system for generating new versions of original content, such as videos or podcasts. This system can convert data about consumer interactions with content, such as general public comments and viewing / listening behavior (e.g., at which part of the video a viewer starts to interact or at which part of the video a user stops interacting), and external data such as trend search terms, into insights; use the insights to automatically generate content, or modify content, or both; input the generated, or modified, or both content to create a revised version of the content that meets the owner's constraints and goals (e.g., new video duration, external factors affecting video viewing time meet very demanding revisions); and minimize competing requests through comment creation and addition in the thread of requested content changes.

[0014] In this way, some embodiments can reduce the time and resources required to create various versions of AV content tailored to different viewer profiles, correct errors in the original content, supplement the original content, update the original content, maintain viewer engagement with the content, or a combination thereof. Specifically, some embodiments can enable automatic correction of minor but relevant details that may be missing or incorrect in the original version of the AV content, or automatic addition of information in response to trending topics (e.g., release of a new version of a smartphone), or both. Correction of these errors and deficiencies may lead to increased further viewing of the AV content, increased time spent on the AV content, and thus greater opportunities for branding and investment by the owner of the AV content.

[0015] In an exemplary operation, a content creator / owner can create an original version of a video and then upload the video to a streaming service. Next, the streaming service can both deliver the video to end users and provide a feedback mechanism such as a public bulletin board or a comment bulletin board. When uploading the original video to the streaming service, the content owner may specify that they desire, or permit, or both, an automatically generated extension for the video. Some content owners may also specify one or more restrictions for such an extension, such as "I want up to four versions of this video, and each version should be less than 130% of the duration of the original video."

[0016] Next, the content delivery service 455 (see FIG. 4, described in more detail below) can begin delivering the original video to the end user, and the end user can submit feedback to the streaming service in the form of text comments on the bulletin board / comment bulletin board. Examples of user comments can include "It would be perfect if it could be viewed at 1.25x speed", "It's difficult to view this URL in the browser at 12:54", and "Does this work on the newly released iPhone?". The streaming service can also enable users to react to comments submitted by others, such as by indicating that others "like (highly rate)" or "dislike (lowly rate)" the comment, or by submitting reaction comments to the original comment, or both (collectively user "reactions"). These reactions can be threaded using the base comment of the reaction so that, regardless of when and by whom two comments were submitted, a reaction stating "Since this is set to the variable PYTHON, this person is probably using 3.7" appears near the base comment "What version of Python was this tested with?".

[0017] The video creation service 460 may analyze the submitted comments and the submitted reactions to detect issues in the improvement of the original video, or areas therefor, or both. This may include using natural language processing (NLP) techniques, including sentiment analysis, to analyze and understand the meaning, tone, and direction of the comments and reactions. In some embodiments, this may include resolving conflicts between the submitted comments and reactions, such as preferring more recent comments, or comments by those with a higher online rating, or both. The video creation service 460 may then associate the resulting insights with specific segments of the original video. This may also include using NLP techniques, such as generating a time-indexed transcript of the original video. This may also include using object identification techniques to classify the content of the original video.

[0018] Next, in some embodiments, an optimal fix can be selected from a menu of potential actions in response to one or more of the comments or reactions or both. In some embodiments, this menu of potential actions may include, but is not limited to, increasing the broadcast speed of a portion of the video (e.g., from 1x speed to 1.25x speed), decreasing the broadcast speed of a portion of the video (e.g., from 1x speed to 0.8x speed), searching for and inserting supplementary external material, creating and inserting supplementary original material, and editing a portion of the original material. In some embodiments, this may include generating a plurality of candidates for addition and then making a selection from among the candidates using the preferences and limitations provided by the content owner.

[0019] In some embodiments, the supplementary material may include overlaying text material on an existing video segment, such as adding a URL where a particular product can be obtained, or a URL where further information can be found, or a URL with some additional explanatory text. In some embodiments, the supplementary material may be included to generate a realistic simulation of the voice or face or both of the content presenter / actor of the original video by using deep learning and adversarial generative network (GAN) techniques. In this way, for example, if the content or reaction indicates that the original content contains an error, it is possible to create a new version of the video using the correct facts, and in the new version, this fact appears as if the content presenter is actually stating the correct facts. Some embodiments may automatically issue the updated version, and some embodiments may request that the owner of the original video approve the change.

[0020] In some embodiments, the video creation service 460 may generate multiple different versions of the edit or the supplementary material or both to create multiple different revised versions of the original video, where each revised version targets a different viewer profile. Continuing with the above example, the video creation service 460 may create a first version of the video targeting an information technology (IT) specialization, in which an overlay is added and "Python version 3.7" is used, and a second version of the video reveals that it targets retail users of a particular brand of smartphone, and in this second version, GAN creates a new video segment announcing the compatibility and optimal settings for the latest version of the smartphone.

[0021] After issuing the updated video to the content delivery service 455, in some embodiments, the video production service 460 may then conduct A / B tests on various versions (e.g., original version vs. corrected version, new version A vs. new version B, etc.) to determine if there are additional areas for improvement and also to determine which version has a higher viewer engagement score. If the new version has a better score, the content delivery service 455 can then license the resulting production to the content owner. In some embodiments, the video production service 460 may also pay viewers who submitted comments and reactions to the original or corrected content used to create the new version of the video, as well as to external sources for any content used. In yet other embodiments, the content delivery service 455 may create a marketplace for video links (cross-reference), where different versions of the original content can be presented to different users based on their viewer profiles.

[0022] Cloud computing FIG. 1 illustrates a cloud environment consistent with some embodiments. Although this disclosure includes a detailed description of cloud computing, it should be understood that the implementations of the teachings set forth herein are not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0023] Cloud computing is a model for service delivery that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0024] The characteristics are as follows: · On-demand self-service: Cloud consumers can unilaterally provision computing capabilities such as server time and network storage automatically as needed, without requiring human interaction with the service provider. · Broad network access: The capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs). · Resource pooling: The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to demand. There is a sense of location independence in that consumers generally do not have control or information about the exact location of the provided resources, although location can sometimes be specified at a higher level of abstraction (e.g., country, state, or data center). · Rapid elasticity: The capabilities are provisioned rapidly and elastically, and in some cases, automatically, quickly scaled out, and rapidly released and quickly scaled in. For consumers, the capabilities available for provisioning often appear to be unlimited and can be purchased at any time in any quantity. · Service measurement: The cloud system automatically controls and optimizes resource usage by leveraging measurement capabilities at several levels of abstraction appropriate for the type of service (e.g., storage, processing, bandwidth, and active customer accounts). Resource usage is monitored, controlled, and reported, providing transparency to both the provider and consumer of the services being utilized.

[0025] The service model is as follows: · Software as a Service (SaaS): The functionality provided to the consumer is to use the provider's applications running on the cloud infrastructure. The applications are accessible from various client devices through a thin-client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, which includes the network, servers, operating systems, storage, or even individual application functionality, except for limited customer-specific application configuration settings in exceptional cases. · Platform as a Service (PaaS): The functionality provided to the consumer is to deploy consumer-created or acquired applications, created using programming languages and tools supported by the provider, onto the cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, which includes the network, servers, operating systems, or storage, but has control over the deployed applications and, in some cases, the applications hosting the environment configuration. · Infrastructure as a Service (IaaS): The function provided to consumers is to provision other basic computing resources such as processing, storage, network, and any software that consumers can deploy and run, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and in some cases limited control over selected networking components (e.g., host firewalls).

[0026] The deployment models are as follows: · Private cloud: The cloud infrastructure is operated only for a certain organization. It is managed by that organization or a third party and can exist on-premises or off-premises. · Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community with shared matters (e.g., mission, security requirements, policies, and compliance concerns). It is managed by an organization or a third party and can exist on-premises or off-premises. · Public cloud: The cloud infrastructure is made available to the general public or large industry groups and is owned by the organization selling the cloud services. · Hybrid cloud: The cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain as unique entities, but are joined by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0027] A cloud computing environment is service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0028] Next, referring to FIG. 1, an exemplary cloud computing environment 50 is depicted. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 that can communicate with local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, or an automotive computer system 54N or a combination thereof. The nodes 10 can communicate with each other. In one or more networks, such as the private, community, public, or hybrid clouds as described above in this specification, or combinations thereof, they can be physically or virtually grouped (not shown). Thereby, the cloud computing environment 50 can provide an infrastructure, platform, software, or a combination thereof as a service, such that a cloud consumer need not maintain resources on a local computing device. The types of computing devices 54A - N shown in FIG. 1 are merely intended to be exemplary, and it will be understood that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of computerized device on or via any type of network (e.g., using a web browser) or both.

[0029] Referring to FIG. 2, a set of functional abstraction layers provided by the cloud computing environment 50 (FIG. 1) is shown. It should be understood in advance that the components, layers, and functions shown in FIG. 2 are merely illustrative and the embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided.

[0030] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: mainframe 61, RISC (Reduced Instruction Set Computer) architecture-based server 62, server 63, blade server 64, storage device 65, and network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0031] The virtualization layer 70 provides an abstraction layer that provides the following examples of virtual entities: virtual server 71, virtual storage 72, virtual network 73 including a virtual private network, virtual applications and operating systems 74, and virtual clients 75.

[0032] In one example, the management layer 80 can provide the functions described below. Resource provisioning 81 provides for the dynamic procurement of computing resources and other resources utilized to perform tasks within a cloud computing environment. Metering and pricing 82 provides for cost tracking when resources are utilized within a cloud computing environment, and for charging or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides for the verification of identification information about cloud consumers and tasks, and for the protection of data and other resources. The customer portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides for the allocation and management of cloud computing resources such that the required service levels are met. Service level agreement (SLA) planning and enforcement 85 provides for the pre-negotiation of cloud computing resources for which future demands are expected to conform to the SLA, and for the procurement of cloud computing resources.

[0033] The workload layer 90 provides examples of functionality for which a cloud computing environment is utilized. Examples of workloads and functions resulting from this layer include: mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analysis processing 94, transaction processing 95, and content manager 96.

[0034] Data processing system FIG. 3 illustrates an embodiment of a data processing system (DPS) 300 suitable for use as a cloud computing node 10 within a cloud computing environment 50 consistent with some embodiments. In some embodiments, DPS 300 may be implemented as a processor incorporated in a personal computer, a server computer, a portable computer such as a laptop or notebook computer, a PDA (personal digital assistant), a tablet computer, or a smartphone, an automobile, an aircraft, a teleconferencing system, an appliance, or any other suitable type of electronic device. Moreover, there may be components other than or additional to those shown in FIG. 3, and the number, type, and configuration of such components may vary. Moreover, FIG. 3 depicts only representative and major components of DPS 300, and individual components may be much more complex than represented in FIG. 3.

[0035] The data processing system 300 of FIG. 3 may include a plurality of central processing units 310a - 310d (collectively referred to herein as processor 310 or CPU 310), and the plurality of central processing units are connected by a system bus 322 to a memory 312, a mass storage interface 314, a terminal / display interface 316, a network interface 318, and an input / output (I / O) interface 320. The mass storage interface 314 of this embodiment connects the system bus 322 to one or more mass storage devices such as a direct access storage device 340, a universal serial bus (USB) storage device 341, or a read / write optical disk drive 342. Through the network interface 318, the DPS 300a can communicate with other DPS 300b on the communication medium 306. The memory 312 may also include an operating system 324, a plurality of application programs 326, and program data 328.

[0036] The embodiment of the DPS300 in FIG. 3 may be a general-purpose computing device. Thus, the processor 310 may be any device capable of executing program instructions stored in the memory 312, and it may itself be constructed from one or more microprocessors or integrated circuits or both. In this embodiment, the DPS300 includes a plurality of processors or processing cores or both, as is common in large-scale, more high-performance computer systems. However, in other embodiments, the DPS300 may include a single-processor system, or a single processor designed to emulate a multi-processor system, or both. Furthermore, the processor 310 may be implemented using a plurality of heterogeneous DPS300s, in which system, the main processor exists on a single chip with secondary processors. As another illustrative example, the processor 310 may be a symmetric multi-processor system including a plurality of processors of the same type.

[0037] When the DPS300 is powered on, the associated processor 310 may first execute program instructions that configure an operating system 324 that manages the physical and logical resources of the DPS300. These resources may include the memory 312, the mass storage interface 314, the terminal / display interface 316, the network interface 318, and the system bus 322. Similar to the processor 310, some embodiments of the DPS300 may utilize a plurality of system interfaces 314, 316, 318, 320, and buses 322, thereby being able to include its own separate, fully-programmed microprocessors.

[0038] Instructions for an operating system, application, or program, or a combination thereof (collectively referred to as "program code", "computer-usable program code", or "computer-readable program code") may initially be disposed in mass storage devices 340, 341, 342 that communicate with processor 310 through system bus 322. The program code in various embodiments may be embodied in various physical or tangible computer-readable media such as system memory 312 or mass storage devices 340, 341, 342. In the illustrative example of FIG. 3, the instructions may be stored in a functional form of permanent storage on direct access storage device 340. These instructions may then be loaded into memory 312 for execution by processor 310. However, the program code may also optionally be disposed in a functional form on a computer-readable medium 342 that is removable, and may be loaded into or transferred to DPS 300 for execution by processor 310.

[0039] System bus 322 may be any device that facilitates communication between processor 310, memory 312, and interfaces 314, 316, 318, 320. Moreover, system bus 322 of this embodiment is a relatively simple single bus structure that provides a direct communication path between system bus 322 and other bus structures, including, but not limited to, point-to-point links in a hierarchical, star, or web configuration, multiple hierarchical buses, parallel and redundant paths, etc., and is consistent with the present disclosure.

[0040] Memory 312 and mass storage 340, 341, 342 can operate in cooperation to store the operating system 324, application programs 326, and program data 328. In an exemplary embodiment, memory 312 is a random access semiconductor device capable of storing data and programs. Although FIG. 3 conceptually depicts the device as a single monolithic entity, memory 312 may be in a more complex arrangement configuration, such as a hierarchical structure of caches and other memory devices, in some embodiments. For example, memory 312 can exist in multiple levels of cache, and these caches can be further divided by function such that one cache holds instructions while another cache holds non-instruction data used by one or more processors. Memory 312 can also be distributed and associated with different processors 310 or sets of processors 310, as is known in any of various so-called non-uniform memory access (NUMA) computer architectures. Moreover, some embodiments may utilize a virtual addressing mechanism that enables the DPS 300 to behave as if it has access to a large-scale single memory entity instead of access to multiple small memory entities such as memory 312 and mass storage devices 340, 341, 342.

[0041] The operating system 324, application program 326, and program data 328 are illustrated as being included within the memory 312, but some or all of these may be physically located on different computer systems, and in some embodiments, for example, may be accessed remotely via the communication medium 306. Thus, the operating system 324, application program 326, and program data 328 are illustrated as being included within the memory 312, but these elements need not all necessarily be contained simultaneously within exactly the same physical device, and may even be present in the virtual memory of other DPSs 300.

[0042] System interfaces 314, 316, 318, 320 support communication with various storage and I / O devices. The mass storage interface 314 may support attachment of one or more mass storages 340, 341, 342, which typically include storage devices such as rotating magnetic disk drives, solid state storage devices (SSDs) that use integrated circuit assemblies as memory for permanently storing data, typically using flash memory, or a combination of both. However, the mass storage devices 340, 341, 342 may also include other devices, such as an array of disk drives configured to appear to the host as a single large storage device (commonly referred to as a RAID array), or a hard disk drive, tape (e.g., mini DV), writable compact disk (e.g., CD-R and CD-RW), digital versatile disk (e.g., DVD, DVD-R, DVD+R, DVD+RW, DVD-RAM), holographic storage system, blue laser disk, archival storage media such as the IBM(R) Millipede device, or a combination thereof.

[0043] The terminal / display interface 316 may be used to directly connect one or more display units, such as monitor 380, to the DPS 300. These display units 380 may be non-intelligent (i.e., dumb) terminals, such as LED monitors, or may be fully programmable workstations used to enable IT administrators and customers to communicate with the DPS 300. However, note that while the display interface 316 is provided to support communication with one or more display units 380, the DPS 300 does not necessarily require a display unit 380, as all necessary interactions with customers and other processes may occur via the network interface 318.

[0044] The communication medium 306 can be any suitable network or combination of networks and can support any suitable protocol suitable for the communication of data or code or both to / from multiple DPS300s. Thus, the network interface 318 can be any device that facilitates such communication regardless of whether the network connection is made using modern analog or digital or both technologies or through some future networking mechanism. Suitable communication media 306 include, but are not limited to, "InfiniBand" or IEEE (Institute of Electrical and Electronics Engineers) 802.3x "Ethernet(R)" specifications; cellular transmission networks; wireless networks implementing one of IEEE802.11x, IEEE802.16, General Packet Radio Service ("GPRS"), FRS (Family Radio Service), or Bluetooth specifications; networks implemented using one or more of Ultra-Wideband ("UWB") technology as described in FCC02-48, etc. Those skilled in the art will understand that many different networks and transport protocols can be used to implement the communication medium 306. The Transmission Control Protocol / Internet Protocol ("TCP / IP") suite includes suitable network and transport protocols.

[0045] System Architecture Figure 4 depicts a system 400 architecture consistent with several embodiments. The system 400 embodiment of Figure 4 may include an owner / creator 410 of the original video 401, a person 420 captured in the original video 401, and a plurality of viewers 430a - 430d (collectively viewers 430) of the original video 401. Each of the owner 410 and the viewers 430 may have a DPS 411, 431a - 431d (such as a laptop, smartphone, tablet, etc., each of which may be the above-described DPS 300 in some embodiments) with a network interface capable of accessing one or more common computer networks (e.g., the Internet, a proprietary social network platform, etc.). In some embodiments, the system 400 may also include a cloud computing infrastructure 450 (such as the above-described cloud computing environment 50) that hosts a content manager 496, and the content manager 496 may include a content delivery service 455 and a video creation service 460.

[0046] During operation, the owner 410 of this embodiment may use their DPS 411 to upload the original version of the video 401 to the content delivery service 455. The content delivery service 455 may then store the original version of the video 401 in its computing resources 450 and may stream the original version of the video 401 to the DPS 431 associated with the viewers 430 upon request. In some embodiments, the group of viewers 430 may receive the stream substantially simultaneously (e.g., a live stream). In other embodiments, each of the viewers 430 may receive the stream at different times (e.g., an on-demand stream, a podcast, etc.).

[0047] In this embodiment, viewer 430 can consume the original video 401 from the content delivery service 455. Some of the viewers 430 may respond by posting comments or reactions or both to the content delivery service 455, thereby making these comments or reactions or both available to other viewers 430. Additionally, the content delivery service 455 may collect and use statistical values about the activities of viewers 430, such as how long they watched the video 401, which segment they started watching in, which segment they stopped watching in, which segments of the video 401 the viewer repeated (i.e., looped), whether the viewer 430 paused the video 401 in any segment, whether they searched for explanatory material from an external source, etc. In some embodiments, the video production service 460 may automatically analyze the collected statistical values, posted comments, or posted reactions or a combination thereof using natural language processing (NLP) techniques to generate insights about problems or potential areas for supplementation / improvement associated with the original video 401 or both. The video production service 460 may then calculate one or more preferred changes to the video 401 from a menu of potential actions and then automatically generate one or more modified videos 401a - 401d that include such changes. Some embodiments may further utilize conditions specified by the owner 410, such as limitations on the amount of computing power used when calculating the preferred changes, the total time of the newly created videos, the percentage increase in time of the new videos, the type of additional content (e.g., only child - appropriate language usage, only content without copyright), etc.

[0048] In some embodiments, the video creation service 460 further includes a related video segment analysis engine 462 for matching the calculated insights with corresponding segments of the video 401, a content modifier 464 for generating new content (e.g., modifying existing content, creating new content, creating new simulation content, or importing existing content from an external data source 465, or a combination thereof) and inserting the content into the video 401, a content selection module 466 for determining, according to constraints, which modifications to make, which modifications to include, or both, and a feedback engine 468 for analyzing viewer statistics, viewer comments, and viewer reactions to the video 401 using NLP.

[0049] Content Expansion Figures 5A - 5B (collectively Figure 5) illustrate one way to extend content 500, consistent with several embodiments. In operation 510, owner 410 of video 401 can define one or more video creation profiles. Each profile may include one or more preferences, such as the limit on the maximum number of modified videos to be created, the desirability weights of one or more user profiles for which modified videos are to be created, the limit on how frequently modified videos are to be created, the limit on how much time the modified version should indicate, what types of third - party content may be used, what types of editing operations are permitted, etc. Each profile may also include one or more restrictions, such as the limit on the amount of computing power to be used, the total playback time of any modified videos 401a - 401d, the percentage increase in playback time compared to the original video 401 for each modified video 401a - 401d, the type of any new content and the license for it (e.g., but not limited to, only child - appropriate language, or only content without copyright), etc. In operation 515, owner 410 may upload the original video 401 and associate it with one of the modification profiles, or select one or more restrictions, or both.

[0050] Next, content delivery service 455 can start streaming the uploaded original video 401 to viewer 430, and then in operation 520, can begin collecting statistical values and demographic profiles regarding viewer behavior. Also, in operation 520, content delivery service 455 can receive comments and reactions from viewer 430 and then host them. In operation 525, video creation service 460 can begin analyzing the statistical values collected by content delivery service 455, as well as the content of the submitted comments and reactions. For example, without limitation, if the statistical values indicate that a viewer looped a certain portion of video 401 three times between time points 2:23 and 3:12, video creation service 460 may flag that segment as being problematic. Similarly, exemplary comments that may indicate that a segment of original video 401 needs correction can include statements such as "The panel demonstration is too fast to see the details," "The video doesn't mention the legroom inside the car," and "How can this function with the latest version of the GNU / Linux operating system?"

[0051] In some embodiments, feedback engine 468 of video creation service 460 may generate insights by clustering comments processed via natural language processing (NLP) based on the meaning of the comments. NLP can also, without limitation, extract keywords such as "speed up," "accelerate," "faster," etc., and use them to cluster into actions indicating acceleration of segments of the original video. Another example is clustering sentences containing keywords such as "information is missing," "not clear," and "not understandable" to indicate segments of the video where some aspects need clarification, which can be converted, for example, into an action to add a more detailed explanation of that segment of the original video.

[0052] In some embodiments, the feedback engine 468 of the video creation service 460 may also generate insights by applying sentiment analysis to comments and reactions. For example, if a plurality of comments or reactions or both use extremely negative language, the system may also weight more heavily the relevant segments for modification. Examples of negative comments that are determined to modify such segments include "There is no way to see the feet of the car", "The use of the camera when presenting the feet is terrible", and "It is really stressful to see the feet in this video". Sentiment analysis may also utilize the timestamps associated with comments to generate directions and trends for the sentiment (e.g., comments are generally favorable before some event and generally not favorable after that event).

[0053] In operation 530, the video segment analysis engine 462 of the video creation service 460 may select one or more video segments for modification according to the insights. In some embodiments, the video segments that may receive the benefit of the modification can be identified by, for example, matching the insights obtained from operation 525 with the content of the automatically generated transcript of video 401. Some embodiments may also perform object identification analysis on video 401 and compare the resulting list of objects with the insights. Some embodiments may also utilize the timestamps associated with statistical values, or the timestamps submitted in comments and / or reactions, or both, to identify segments for modification.

[0054] In operation 540, the content modifier 464 of the video creation service 460 may determine how to modify the identified segments of the original video 401. This operation may include comparing the insights from operation 525 with a menu of potential operations, such as stretching the segment to play it slower, shortening the segment to play it faster, adding new frames to new content, adding an overlay to new content, adding a text bubble to new content, and adding content from external sources 465 such as other websites, clip art, digital models, blogs, etc. This operation may also include generating multiple potential modifications and then making a selection from among the potential modifications using the preferences and restrictions received in operation 510.

[0055] For each operation involving new content, the content modifier 464 of the video creation service 460 may first create a revised script in response to the comments (i.e., modify the generated transcript), and then use GAN technology to create a simulation of the voice and appearance of the performer / narrator 420. Some embodiments may also supplement the visual material with supplementary text material such as live captioning or text overlay or both.

[0056] In some embodiments, using GAN to add additional material may include: synthesizing new frames from existing footage of the performer / narrator and a newly created script with new statements, and synthesizing audio similar to the voice or appearance or both of the video performer / narrator. When creating new visual effects such as creating and adding new stylized text material, GAN can learn the styling of the visual effects used in the original video and then be used to style the new overlay so that it matches the original style.

[0057] Next, in operation 550, the content selection module 466 of the video creation service 460 may improve part of the video modification considering the constraints or optimizations specified in operation 510. For example, some embodiments may operate to prioritize the modification of videos that add information related to trending topics, and this modification is in response to concerns about the maximum number of comments or in response to concerns of the most respected and influential viewers 430. However, adding content to address all of these three cases may make the resulting modified videos 401a - 401d too long. In this case, in one embodiment, the video creation service 460 can present all or a subset of the possible combinations of new versions of the original video to the video owner 410, so that the video owner can select the updated version to be transferred to the content delivery service 455, or the video owner can specify a set of priorities for the modification operations. In this case, a ranked list of the possible updated versions is generated based on this priority, and the first version that meets the constraints of the video owner can be selected.

[0058] In some cases, some insights may lead to competing modifications. For example, some viewers 430 may desire more details, while other viewers may not desire as much detail. In such cases, the content selection module 466 of the video creation service 460 may, in operation 570, select which modifications should actually be inserted based on the priorities specified in operation 510. In some embodiments, the content selection module 466 of the video creation service 460 may respond by creating multiple different modified versions of the videos 401a - 401d targeted at each group of viewers 430 in operation 570, and then upload all of the versions to the content delivery service 455. In some embodiments, the video creation service 460 may additionally or alternatively automatically generate its own comments and reactions to the original video 401 and decide to issue them to the comment or message board of the content delivery service 455, or both, in operation 570 to address any thread concerns (e.g., in case of competition, the comments may be calculated as being more appropriate as a text response, such as a lower - priority modification or a response to a comment requesting a URL, or both).

[0059] In operation 580, the content modifier 464 of the video creation service 460 may process the selected modifications to create one or more additional modified versions of the videos 401a - 401d. Also, in operation 590, the video creation service 460 may generate metadata and insert the metadata into the modified videos 401a - 401d to identify which parts of the modified videos 401a - 401d are new, what changes were made, when the changes were made, etc.

[0060] The video creation service 460 may optionally seek the necessary license and / or the approval of the original owner 410 of the video 401 for the modification in operation 592. In operation 594, the video creation service 460 may automatically upload the modified versions of videos 401a - 401d and start collecting statistical values, comments, and reactions for the modified versions 401a - 401d. In operation 596, the video creation service 460 may compare the collected statistical values (such as the number of views, viewing duration, etc.) of the modified versions 401a - 401d, as well as the collected comments and reactions, with the statistical values, comments, and reactions for the original version of the video 401. If the statistical values, comments, and sentiment do not improve the video, the video creation service 460 may roll back the changes and notify the original owner. Alternatively, some embodiments may conduct A / B tests on the various versions of videos 401, 401a - 401d and select the version with better statistical values, comments, and reactions.

[0061] Computer program product Although the present invention has been described in detail with reference to its specific examples, the present invention may also be embodied in other specific forms without departing from its essential idea or nature. For example, the present invention may be a system, method, or computer program product or a combination thereof in any possible integration of technical detail levels. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to execute aspects of the present invention. The computer-readable program instructions may be stored and executed on a single computer, or may be divided among different computers for storage and execution at the same location or different locations.

[0062] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing, but is not limited thereto. A non-exhaustive listing of more specific examples of computer-readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or raised structures in a groove on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium shall not be construed to be a transitory signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted through an electrical wire.

[0063] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a single computing / processing device or to an external computer or an external storage device via a network, such as, for example, the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. Each computing / processing device's network adapter card or network interface receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within the respective single computing / processing device.

[0064] Computer-readable program instructions for carrying out the operation of the present invention may be written in any combination of one or more programming languages, including assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or object-oriented programming languages such as Smalltalk(R), C++, and procedural programming languages or similar programming languages such as the "C" programming language, in either source code or object code. The computer-readable program instructions may be executed entirely on the customer's computer, partly as a stand-alone software package on the customer's computer, partly on the customer's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the customer's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may utilize the state information of the computer-readable program instructions to execute the computer-readable program instructions to customize the electronic circuit in order to implement aspects of the present invention.

[0065] These computer-readable program instructions, when executed via a processor of a computer or other programmable data processing apparatus, may cause the means for implementing the functions / acts specified in one or more blocks of a flowchart, a block diagram, or both to be created, thereby making a machine, such as a computer or other programmable data processing apparatus. These computer-readable program instructions may also be stored in a computer-readable storage medium that includes instructions for implementing the functions / acts of one or more blocks of a flowchart, a block diagram, or both, such that the computer-readable storage medium causes a computer, a programmable data processing apparatus, or other device or combination thereof to function in a particular manner.

[0066] The computer-readable program instructions may also be loaded onto a computer, other programmable apparatus, or other device to create a computer-implemented process such that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / acts specified in one or more blocks of a flowchart, a block diagram, or both by causing a series of operational steps to be performed on the computer, other programmable apparatus, or other device.

[0067] General Introduction Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions. Moreover, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block of the flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or combinations of dedicated hardware and computer instructions.

[0068] Any specific program nomenclature used in this description is for convenience only, and thus the present invention should not be limited to using only any particular application identified by, or implied by, or both, such nomenclature. Thus, for example, a routine executed to implement an embodiment of the present invention may be referred to by a "program," "application," "server," or other meaningful name, whether implemented as part of an operating system or a particular application. In fact, other alternative hardware or software or both environments may be used without departing from the scope of the present invention.

[0069] Accordingly, the embodiments described herein are considered to be illustrative in all respects and not restrictive, and it is desirable that the appended claims for determining the scope of the present invention be referred to.

Claims

1. A method for automatically generating an extension for audiovisual (AV) content, comprising: receiving, by a network interface, data regarding consumer interaction with the original AV content, the data including comments about the original AV content; by a processing unit, analyzing the comments of the data regarding consumer interaction to generate consumer insights about the original AV content; automatically associating the consumer insights with segments of the original AV content; automatically generating content for the segments according to the consumer insights; injecting the generated content into the original AV content to create modified AV content; automatically transmitting, by the network interface, the modified AV content. A method as described above.

2. The method according to claim 1, further comprising collecting the data regarding the consumer interaction with the original AV content.

3. The analyzing includes: performing natural language processing on the comments to determine meaning; using the determined meaning to cluster the comments. The method according to claim 1.

4. The analyzing includes performing sentiment analysis of the comments to determine sentiment and sentiment direction. The method according to claim 1.

5. The automatically generating the content includes: generating a transcript of the original AV content; comparing the consumer insights with the transcript. The method according to claim 1.

6. The automatically generating the content further includes: modifying the transcript according to the consumer insights; generating a simulated performance using the modified transcript. The method according to claim 5.

7. The automatically generating the content further includes generating the simulated performance using an adversarial generation network, and the simulated performance resembles the voice and appearance of a performer. The method according to claim 6.

8. generating supplementary text materials using the consumer insights; overlaying the supplementary materials on segments of the content; The method according to claim 5, further comprising:

9. analyzing the data about the consumer interaction to generate a plurality of consumer insights about the original AV content; detecting a conflict between at least two of the plurality of insights; selecting a priority insight from among the at least two conflicting insights, wherein the automatically generated content corresponds to the priority insight; automatically generating comments according to the lower priority insight; issuing the generated comments as a response to comments about the original AV content; The method according to claim 5, further comprising:

10. selecting external content according to the consumer insights; adding the external content to the original AV content; The method according to claim 5, further comprising:

11. generating metadata about the modified AV content; attaching the metadata to the modified AV content; The method according to claim 1, further comprising:

12. the consumer interaction includes comments about the content, data about the consumer use of the content; The method according to claim 1.

13. receiving data about the consumer interaction with the modified AV content; comparing the data about the consumer interaction with the modified AV content with the data about the consumer interaction with the original AV content; removing the modified AV content or the original AV content based on the comparison; The method according to claim 1, further comprising:

14. Receiving a profile from an owner of the original AV content, the profile specifying at least one preference and at least one restriction, said receiving; Automatically generating content for the segment by the processing unit according to the at least one preference and the at least one restriction; The method according to claim 1, further comprising.

15. Automatically generating a plurality of potential contents for the segment according to the consumer insights; Estimating an additional length of time associated with the plurality of potential contents; Selecting from among the plurality of potential contents using the at least one preference and the at least one restriction; The method according to claim 14, further comprising.

16. The method according to claim 1, wherein the data regarding the consumer interaction with the original AV content includes trend search terms for the original AV content.

17. The method according to claim 1, further comprising automatically modifying the segment by the processing unit according to the consumer insights.

18. A content manager, A content creation server communicably coupled to a plurality of content consumer devices, the server comprising a processor coupled to a memory, the processor and the memory being configured to: Receive data regarding consumer interaction with original audiovisual (AV) content, the data including comments about the original audiovisual (AV) content, said receiving; Analyze the comments of the data regarding consumer interaction to generate consumer insights about the original audiovisual (AV) content; Automatically associate the consumer insights with segments of the original audiovisual (AV) content; Automatically generate content for the segments according to the consumer insights; Inject the generated content into the original audiovisual (AV) content to create modified AV content; Automatically transmit the modified AV content; The content creation server configured to perform; A content manager comprising.

19. A computer program that causes a computer to execute the method according to any one of claims 1 to 17.

Images