Method for automatically generating enhancement to AV content, content manager, and program

Abstract

To provide a method, and the like, for automatically generating enhancements to audiovisual (AV) content.SOLUTION: A method for automatically generating enhancements to audio-visual (AV) content, a content manager, and a computer program product are provided. The method may comprise: by a processing unit, analyzing data about consumer interactions to generate consumer insights about the original AV content; automatically associating the consumer insights with a segment of original AV content; automatically generating content for the segment responsive to the consumer insights; and injecting the generated content into the original AV content to create modified AV content. The method may further comprise, by a network interface, receiving data about consumer interactions with original AV content and automatically transmitting the modified AV content.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to creating and managing audiovisual (AV) content, and more particularly to automatically expanding AV content in response to viewer interaction with the AV content.

Background Art

[0002] The development of the EDVAC system in 1948 is often cited as the beginning of the computer age. Since then, computer systems have evolved into extremely complex devices. Today's computer systems typically include a combination of sophisticated hardware and software components, application programs, operating systems, processors, buses, memories, input / output devices, and the like. As performance has increased with advancements in semiconductor processing and computer architecture, more evolved computer software has evolved to take advantage of the higher performance of these functions, leading to the current computer systems that are much more powerful than just a few years ago.

[0003] A variety of content creators have come to utilize these improved functions via content streaming and podcasting. Content streaming generally refers to the transmission of audio or video or both files, typically over the Internet, from a server to a client, and is widely used to view video clips and movies on various media players such as televisions, computers, tablets, and smartphones. Typically, in content streaming, the viewer / end-user can start playing the content before the entire file has been transmitted. In contrast, podcasting also generally refers to the transmission of audio or video or both files, typically over the Internet, from a server to a client. However, in podcasting, the transmission generally occurs before the viewer starts playing the content.

[0004] Streaming and podcasting have proven themselves to be excellent platforms for, among other things, advertisements, product reviews, problem troubleshooting (e.g., information technology, home maintenance, industrial-related issues). SUMMARY OF THE INVENTION

[0005] According to an embodiment of the present disclosure, a method for automatically generating an extension for audiovisual (AV) content. One embodiment may include analyzing, by a processing unit, data about consumer interaction to generate consumer insights about the original AV content, automatically associating the consumer insights with segments of the original AV content, automatically generating content for the segments according to the consumer insights, and injecting the generated content into the original AV content to create modified AV content. The embodiment may further include receiving, by a network interface, data about consumer interaction with the original AV content and automatically transmitting the modified AV content.

[0006] According to an embodiment of the present disclosure, a content manager. One embodiment includes a content creation server communicatively coupled to a plurality of content consumer devices, the server including a processor coupled to a memory. The processor and the memory are configured to receive data about consumer interaction with original audiovisual (AV) content, analyze the data about consumer interaction to generate consumer insights about the original AV content, automatically associate the consumer insights with segments of the original AV content, automatically generate content for the segments according to the consumer insights, inject the generated content into the original AV content to create modified AV content, and automatically transmit the modified AV content.

[0007] According to an embodiment of the present disclosure, a computer program product for automatically generating an extension for audiovisual (AV) content. The computer program product in one embodiment may include a computer-readable storage medium having program instructions executable by a processor embodied therein. The program instructions may cause the processor to receive data about consumer interaction with original AV content, analyze the data about consumer interaction to generate consumer insights about the original AV content, automatically associate the consumer insights with segments of the original AV content, automatically generate content for the segments according to the consumer insights, inject the generated content into the original AV content to create modified AV content, and automatically transmit the modified AV content.

[0008] The above summary is not intended to describe every illustrative embodiment or every implementation of the present disclosure.

[0009] The drawings included in this application are incorporated herein and form a part thereof. These drawings illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure. The drawings merely illustrate specific embodiments and do not limit the present disclosure.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Best Mode for Carrying Out the Invention

[0011] The present invention is subject to various modifications and alternative forms, specific examples of which are shown by way of illustration in the drawings and will be described in detail. However, it should be understood that there is no intention to limit the present invention to the specific embodiments described. On the contrary, it is intended to cover all modifications, equivalents, and alternative forms falling within the spirit and scope of the present invention.

[0012] Aspects of the present disclosure relate to creating and managing audio-visual (AV) content, and more particularly to automatically extending AV content in response to consumer interactions with the AV content, such as comments and viewing behavior. The present disclosure is not necessarily limited to such applications, but various aspects of the present disclosure can be understood through discussion of various examples using this context.

[0013] Some embodiments of the present disclosure can provide an automated system for generating new versions of original content, such as videos or podcasts. This system can convert data about consumer interactions with content, such as general public comments and viewing / listening behaviors (e.g., at which parts of a video a viewer starts interacting or at which parts a user stops interacting), and external data such as trend search terms, into insights; use the insights to automatically generate content, or to modify content, or both; input the generated, or modified, or both content to create a revised version of the content that meets the owner's constraints and goals (e.g., new video duration, external factors affecting video viewing time meet very demanding revisions); and minimize competing requests through comment creation and addition in the thread of requested content changes.

[0014] In this way, some embodiments can reduce the time and resources required to create various versions of AV content tailored to different viewer profiles, to correct errors in original content, to supplement original content, to update original content, to maintain viewer engagement with the content, or a combination thereof. Specifically, some embodiments can enable the automatic correction of minor but relevant details that may be missing or incorrect in the original version of the AV content, or the automatic addition of information in response to popular topics (e.g., release of a new version of a smartphone), or both. Correction of these errors and deficiencies may lead to further viewing of the AV content, an increase in the time spent on the AV content, and thus a greater opportunity for branding and investment by the owner of the AV content.

[0015] In an exemplary operation, a content creator / owner can create an original version of a video and then upload the video to a streaming service. Next, the streaming service both distributes the video to end users and provides a feedback mechanism such as a public bulletin board or a comment bulletin board. When uploading the original video to the streaming service, the content owner may specify that they desire, or permit, or both, an automatically generated extension for the video. Some content owners may also specify one or more restrictions for such an extension, such as "I want up to 4 versions of this video, and each version should be less than 130% of the duration of the original video."

[0016] Next, the content delivery service 455 (see FIG. 4, described in more detail below) can begin delivering the original video to the end user, and the end user can submit feedback to the streaming service in the form of text comments on the bulletin board / comment board. Examples of user comments can include "It would be perfect if it could be viewed at 1.25x speed", "It's difficult to view this URL in the browser at 12:54", and "Does this work on the newly released iPhone?". The streaming service can also enable users to react to comments submitted by others, such as by indicating that others "like (highly evaluate)" or "dislike (lowly evaluate)" the comment, or by submitting reaction comments to the original comment, or both (collectively user "reactions"). These reactions can be threaded using the base comment of the reaction so that, regardless of when and by whom two comments are submitted, a reaction stating "Since this is set to the variable PYTHON, this person is probably using 3.7" appears near the base comment "What version of Python was this tested with?".

[0017] The video creation service 460 may analyze the submitted comments and the submitted reactions to detect problems in improving the original video, or areas therefor, or both. This may include using natural language processing (NLP) techniques, including sentiment analysis, to analyze and understand the meaning, tone, and direction of the comments and reactions. In some embodiments, this may include resolving conflicts between the submitted comments and reactions, such as preferring more recent comments, or comments by those with a higher online rating, or both. The video creation service 460 may then associate the resulting insights with specific segments of the original video. This may also include using NLP techniques, such as generating a time-indexed transcript of the original video. This may also include using object identification techniques to classify the content of the original video.

[0018] Next, in some embodiments, an optimal fix can be selected from a menu of potential actions in response to one or more of the comments or reactions or both. In some embodiments, this menu of potential actions may include, but is not limited to, increasing the broadcast speed of a portion of the video (e.g., from 1x speed to 1.25x speed), decreasing the broadcast speed of a portion of the video (e.g., from 1x speed to 0.8x speed), searching for and inserting supplementary external material, creating and inserting supplementary original material, and editing a portion of the original material. In some embodiments, this may include generating multiple candidates for addition and then making a selection from among the candidates using the preferences and limitations provided by the content owner.

[0019] In some embodiments, the supplementary material may include overlaying text material on an existing video segment, such as adding a URL where a particular product can be obtained, or a URL where further information can be found, or a URL with some additional explanatory text. In some embodiments, the supplementary material may include using deep learning and adversarial generative network (GAN) techniques to generate a realistic simulation of the voice or face or both of the original video's content presenter / actor. In this way, for example, if the content or reaction indicates that the original content contains an error, it is possible to create a new version of the video using the correct facts, and in the new version, this fact appears as if the content presenter is actually stating the correct facts. Some embodiments may automatically issue the updated version, and some embodiments may request that the owner of the original video approve the change.

[0020] In some embodiments, the video creation service 460 may generate multiple different versions of the edit or the supplementary material or both, creating multiple different revised versions of the original video, where each revised version targets a different viewer profile. Continuing with the above example, the video creation service 460 may create a first version of the video targeting an information technology (IT) specialization, where an overlay is added and "Python version 3.7" is used, and a second version of the video reveals that it targets retail users of a particular brand of smartphone, and in this second version, the GAN creates a new video segment announcing the compatibility and optimal settings for the latest version of the smartphone.

[0021] After issuing the updated video to the content delivery service 455, in some embodiments, the video production service 460 may then conduct A / B tests on various versions (e.g., original version vs. corrected version, new version A vs. new version B, etc.) to determine whether there are additional areas for improvement and also to determine which version has a higher score for viewer engagement. If the new version has a better score, the content delivery service 455 can then license the resulting production to the content owner. In some embodiments, the video production service 460 may also pay viewers who submitted comments and reactions to the original or corrected content used to create the new version of the video, as well as to external sources for any content used. In yet other embodiments, the content delivery service 455 may create a marketplace for video links (cross-reference), where different versions of the original content can be presented to different users based on their viewer profiles.

[0022] Cloud computing Figure 1 illustrates a cloud environment consistent with some embodiments. While this disclosure includes a detailed description of cloud computing, it should be understood that the implementations of the teachings presented herein are not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0023] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or service provider interaction. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0024] The characteristics are as follows: · On-demand self-service: Cloud consumers can unilaterally provision computing capabilities such as server time and network storage automatically as needed, without requiring human interaction with the service provider. · Broad network access: The capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs). · Resource pooling: The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to demand. There is a sense of location independence in that consumers generally have no control or knowledge of the exact location of the provided resources, although it may be possible to specify location at a higher level of abstraction (e.g., country, state, or data center). · Rapid elasticity: The capabilities are provisioned rapidly and elastically, and in some cases, automatically, quickly scaled out, and rapidly released and quickly scaled in. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be purchased at any time in any quantity. · Service measurement: The cloud system automatically controls and optimizes resource usage by leveraging measurement capabilities at several levels of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active customer accounts). Resource usage is monitored, controlled, and reported, providing transparency to both the provider and consumer of the services being utilized.

[0025] The service model is as follows: · Software as a Service (SaaS): The functionality provided to the consumer is to use the provider's applications running on the cloud infrastructure. The applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functionality, except for limited customer-specific application configurations on an exception basis. · Platform as a Service (PaaS): The functionality provided to the consumer is to deploy consumer-created, or acquired, applications created using programming languages and tools supported by the provider onto the cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but has control over the deployed applications and, in some cases, the applications hosting the environment configuration. · Infrastructure as a Service (IaaS): The functions provided to consumers are to provision other basic computing resources such as processing, storage, networks, and any software that consumers can deploy and run, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and in some cases limited control over select networking components (e.g., host firewalls).

[0026] The deployment models are as follows: · Private cloud: The cloud infrastructure is operated solely for a particular organization. It is managed by that organization or a third party and can exist on-premises or off-premises. · Community cloud: The cloud infrastructure is shared by several organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, and compliance considerations). It is managed by an organization or a third party and can exist on-premises or off-premises. · Public cloud: The cloud infrastructure is made available to the general public or a large industry group and is owned by an organization that sells cloud services. · Hybrid cloud: The cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain unique entities but are joined by standardized or proprietary technologies (e.g., cloud bursting for load balancing between clouds) that enable data and application portability.

[0027] The cloud computing environment is service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. The core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0028] Next, referring to FIG. 1, an exemplary cloud computing environment 50 is depicted. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 that can communicate with local computing devices used by cloud consumers, such as, for example, a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automotive computer system 54N or a combination thereof. The nodes 10 can communicate with each other. In one or more networks, such as the private, community, public, or hybrid clouds or combinations thereof as described above herein, these can be grouped physically or virtually (not shown). Thereby, the cloud computing environment 50 can provide an infrastructure, platform, software, or a combination thereof as a service such that cloud consumers do not need to maintain resources on local computing devices. It will be understood that the types of computing devices 54A - N shown in FIG. 1 are merely exemplary, and that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of computerized device on or via any type of network (e.g., using a web browser) or both.

[0029] Referring to FIG. 2, a set of functional abstraction layers provided by the cloud computing environment 50 (FIG. 1) is shown. It should be understood in advance that the components, layers, and functions shown in FIG. 2 are merely illustrative and the embodiments of the present invention are not limited thereto. As depicted, the following layers and corresponding functions are provided.

[0030] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: mainframe 61, RISC (Reduced Instruction Set Computer) architecture-based server 62, server 63, blade server 64, storage device 65, and network and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0031] The virtualization layer 70 provides an abstraction layer that provides the following examples of virtual entities: virtual server 71, virtual storage 72, virtual network 73 including virtual private network, virtual applications and operating systems 74, and virtual client 75.

[0032] In one example, the management layer 80 can provide the functions described below. Resource provisioning 81 provides for the dynamic procurement of computing resources and other resources utilized to perform tasks within a cloud computing environment. Metering and pricing 82 provides for cost tracking when resources are utilized within a cloud computing environment, and for charging or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides for the verification of identification information about cloud consumers and tasks, and for the protection of data and other resources. The customer portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides for the allocation and management of cloud computing resources such that the required service levels are met. Service level agreement (SLA) planning and enforcement 85 provides for the pre-negotiation of cloud computing resources for which future demands are expected according to the SLA, and for the procurement of cloud computing resources.

[0033] The workload layer 90 provides examples of functionality for which a cloud computing environment is utilized. Examples of workloads and functions resulting from this layer include: mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analytics processing 94, transaction processing 95, and content manager 96.

[0034] Data processing system FIG. 3 illustrates an embodiment of a data processing system (DPS) 300 suitable for use as a cloud computing node 10 within a cloud computing environment 50 consistent with several embodiments. In some embodiments, the DPS 300 may be implemented as a processor incorporated into a personal computer, a server computer, a portable computer such as a laptop or notebook computer, a PDA (personal digital assistant), a tablet computer, or a smartphone, an automobile, an aircraft, a teleconferencing system, an appliance, or any other suitable type of electronic device. Moreover, there may be components other than those shown in FIG. 3 or additional components with respect to those shown in FIG. 3, and the number, type, and configuration of such components may vary. Moreover, FIG. 3 depicts only representative and major components of the DPS 300, and individual components may be much more complex than represented in FIG. 3.

[0035] The data processing system 300 of FIG. 3 may include a plurality of central processing units 310a - 310d (collectively referred to herein as processor 310 or CPU 310), and the plurality of central processing units are connected by a system bus 322 to a memory 312, a mass storage interface 314, a terminal / display interface 316, a network interface 318, and an input / output (I / O) interface 320. The mass storage interface 314 of the present embodiment connects the system bus 322 to one or more mass storage devices such as a direct access storage device 340, a universal serial bus (USB) storage device 341, or a read / write optical disk drive 342. Through the network interface 318, the DPS 300a can communicate with other DPS 300b on the communication medium 306. The memory 312 may also include an operating system 324, a plurality of application programs 326, and program data 328.

[0036] The embodiment of the DPS300 in FIG. 3 may be a general-purpose computing device. Thus, the processor 310 may be any device capable of executing program instructions stored in the memory 312, and it may itself be constructed from one or more microprocessors or integrated circuits or both. In this embodiment, the DPS300 includes a plurality of processors or processing cores or both, as is common in large-scale, more high-performance computer systems. However, in other embodiments, the DPS300 may include a single-processor system, or a single processor designed to emulate a multi-processor system, or both. Furthermore, the processor 310 may be implemented using a plurality of heterogeneous DPS300s, in which system, the main processor exists on a single chip with secondary processors. As another illustrative example, the processor 310 may be a symmetric multi-processor system including a plurality of processors of the same type.

[0037] When the DPS300 is powered on, the associated processor 310 may first execute program instructions that configure an operating system 324 that manages the physical and logical resources of the DPS300. These resources may include the memory 312, the mass storage interface 314, the terminal / display interface 316, the network interface 318, and the system bus 322. Similar to the processor 310, some embodiments of the DPS300 may utilize a plurality of system interfaces 314, 316, 318, 320, and buses 322, thereby being able to include its own separate, fully-programmed microprocessors.

[0038] Instructions for an operating system, application, or program, or a combination thereof (collectively referred to as "program code," "computer-usable program code," or "computer-readable program code") may initially be disposed in a mass storage device 340, 341, 342 that communicates with the processor 310 through the system bus 322. The program code in various embodiments can be embodied in various physical or tangible computer-readable media such as the system memory 312 or the mass storage devices 340, 341, 342. In the illustrative example of FIG. 3, the instructions may be stored in a functional form of permanent storage on the direct access storage device 340. These instructions may then be loaded into the memory 312 for execution by the processor 310. However, the program code may also be selectively disposed in a functional form on a removable computer-readable medium 342 and may be loaded into or transferred to the DPS 300 for execution by the processor 310.

[0039] The system bus 322 may be any device that facilitates communication between the processor 310, the memory 312, and the interfaces 314, 316, 318, 320. Moreover, the system bus 322 of this embodiment is a relatively simple single bus structure that provides a direct communication path between the system bus 322 and other bus structures, including, but not limited to, point-to-point links in a hierarchical, star, or web configuration, multiple hierarchical buses, parallel and redundant paths, and is consistent with the present disclosure.

[0040] Memory 312 and mass storage 340, 341, 342 can operate in cooperation to store operating system 324, application program 326, and program data 328. In an exemplary embodiment, memory 312 is a random access semiconductor device capable of storing data and programs. Although FIG. 3 conceptually depicts the device as a single, monolithic entity, memory 312 may, in some embodiments, have a more complex arrangement, such as a hierarchical structure of caches and other memory devices. For example, memory 312 can exist at multiple levels of cache, and these caches can be further divided by function such that one cache holds instructions while another holds non-instruction data used by one or more processors. Memory 312 can also be distributed and associated with different processors 310 or sets of processors 310, as is known in any of various so-called non-uniform memory access (NUMA) computer architectures. Moreover, some embodiments may utilize a virtual addressing mechanism that enables the DPS 300 to behave as if it had access to a large, single memory entity instead of access to multiple, smaller memory entities such as memory 312 and mass storage devices 340, 341, 342.

[0041] The operating system 324, application program 326, and program data 328 are illustrated as being included within the memory 312, but some or all of these may be physically located on different computer systems and, for example, in some embodiments may be accessed remotely via the communication medium 306. Thus, the operating system 324, application program 326, and program data 328 are illustrated as being included within the memory 312, but these elements need not all necessarily be simultaneously included in exactly the same physical device, and may even be present in the virtual memory of other DPSs 300.

[0042] System interfaces 314, 316, 318, 320 support communication with various storage and I / O devices. The mass storage interface 314 may support attachment of one or more mass storages 340, 341, 342, which typically are storage devices such as rotating magnetic disk drives, solid state storage devices (SSDs) that use integrated circuit assemblies as memory for persistent data storage, typically using flash memory, or a combination of both. However, the mass storage devices 340, 341, 342 may also include other devices, such as an array of disk drives (commonly referred to as a RAID array) configured to appear as a single large storage device to the host, or a hard disk drive, tape (e.g., mini DV), writable compact disk (e.g., CD-R and CD-RW), digital versatile disk (e.g., DVD, DVD-R, DVD+R, DVD+RW, DVD-RAM), holographic storage system, blue laser disk, archival storage media such as the IBM(R) Millipede device, or a combination thereof, etc.

[0043] The terminal / display interface 316 may be used to directly connect one or more display units, such as the monitor 380, to the DPS 300. These display units 380 may be non-intelligent (i.e., dumb) terminals, such as LED monitors, or may be fully programmable workstations used to enable IT administrators and customers to communicate with the DPS 300. However, note that while the display interface 316 is provided to support communication with one or more display units 380, the DPS 300 does not necessarily require a display unit 380, as all necessary interaction with customers and other processes may occur via the network interface 318.

[0044] The communication medium 306 may be any suitable network or combination of networks and can support any suitable protocol suitable for the communication of data and / or code to / from the plurality of DPS300s. Thus, the network interface 318 may be any device that facilitates such communication regardless of whether the network connection is made using modern analog or digital or both technologies or via some future networking mechanism. Suitable communication media 306 include, but are not limited to, the "InfiniBand" or IEEE (Institute of Electrical and Electronics Engineers) 802.3x "Ethernet(R)" specifications; cellular transmission networks; wireless networks implementing one of IEEE802.11x, IEEE802.16, General Packet Radio Service ("GPRS"), FRS (Family Radio Service), or the Bluetooth specification; networks implemented using one or more of Ultra-Wideband ("UWB") technology as described in FCC02-48. Those skilled in the art will understand that many different networks and transport protocols can be used to implement the communication medium 306. The Transmission Control Protocol / Internet Protocol ("TCP / IP") suite includes suitable network and transport protocols.

[0045] System Architecture Figure 4 depicts a system 400 architecture consistent with several embodiments. The system 400 embodiment of FIG. 4 may include an owner / creator 410 of the original video 401, a person 420 captured in the original video 401, and a plurality of viewers 430a - 430d (collectively viewers 430) of the original video 401. Each of the owner 410 and the viewers 430 may have a DPS 411, 431a - 431d (e.g., a laptop, smartphone, tablet, etc., each of which may be the above-described DPS 300 in some embodiments) with a network interface that can access one or more common computer networks (e.g., the Internet, a proprietary social network platform, etc.). In some embodiments, the system 400 may also include a cloud computing infrastructure 450 (such as the above-described cloud computing environment 50) that hosts a content manager 496, and the content manager 496 may include a content delivery service 455 and a video creation service 460.

[0046] During operation, the owner 410 of this embodiment may use their DPS 411 to upload the original version of the video 401 to the content delivery service 455. The content delivery service 455 may then store the original version of the video 401 in its computing resources 450 and may stream the original version of the video 401 to the DPS 431 associated with the viewers 430 upon request. In some embodiments, a group of viewers 430 may receive the stream substantially simultaneously (e.g., a live stream). In other embodiments, each of the viewers 430 may receive the stream at different times (e.g., an on-demand stream, a podcast, etc.).

[0047] In this embodiment, viewer 430 can consume the original video 401 from the content delivery service 455. Some of the viewers 430 may respond by posting comments or reactions or both to the content delivery service 455, thereby making these comments or reactions or both available to other viewers 430. Additionally, the content delivery service 455 may collect and use statistical values about the activities of the viewers 430, such as how long they watched the video 401, which segment they started watching from, which segment they stopped watching at, which segments the viewers repeated watching (i.e., looped), whether the viewers 430 paused the video 401 at any segment, whether they searched for explanatory material from an external source, etc. In some embodiments, the video production service 460 may automatically analyze the collected statistical values, posted comments, or posted reactions or a combination thereof using natural language processing (NLP) techniques to generate insights about problems or potential areas for supplementation / improvement associated with the original video 401 or both. The video production service 460 may then calculate one or more preferred changes to the video 401 from a menu of potential actions and then automatically generate one or more modified videos 401a - 401d that include such changes. Some embodiments may further utilize conditions specified by the owner 410, such as limitations on the amount of computing power used when calculating the preferred changes, the total time of the newly created videos, the percentage increase in time of the new videos, the type of additional content (e.g., only child - appropriate language usage, only content without copyright), etc.

[0048] In some embodiments, the video creation service 460 further includes a related video segment analysis engine 462 for matching the calculated insights with corresponding segments of the video 401, a content modifier 464 for generating new content (e.g., modifying existing content, creating new content, creating new simulation content, or importing existing content from an external data source 465, or a combination thereof) and inserting the content into the video 401, a content selection module 466 for determining, according to constraints, which modifications to make, which modifications to include, or both, and a feedback engine 468 that can analyze viewer statistics, viewer comments, and viewer reactions to the video 401 using NLP.

[0049] Content Expansion Figures 5A-5B (collectively Figure 5) illustrate one way to expand content 500, consistent with several embodiments. In operation 510, owner 410 of video 401 can define one or more video creation profiles. Each profile may include one or more preferences, such as the limit on the maximum number of modified videos to be created, the desirability weight of one or more user profiles for which the modified videos are to be created, the limit on how frequently the modified videos are to be created, the limit on how much time the modified version should represent, what types of third-party content may be used, what types of corrective operations are permitted, etc. Each profile may also include one or more restrictions, such as the magnitude of computing power to be used, the total playback time of any modified videos 401a-401d, the percentage increase in playback time compared to the original video 401 for each modified video 401a-401d, the type of any new content and the license therefor (e.g., but not limited to, only child-appropriate language or only content without copyright), etc. In operation 515, owner 410 may upload the original video 401 and associate it with one of the modification profiles, or select one or more restrictions, or both.

[0050] Next, the content delivery service 455 can start streaming the uploaded original video 401 to the viewer 430, and then in operation 520, can begin to collect statistical values and demographic profiles about the viewer behavior. Also, in operation 520, the content delivery service 455 can receive comments and reactions from the viewer 430 and then host them. In operation 525, the video creation service 460 can begin to analyze the statistical values collected by the content delivery service 455, as well as the content of the submitted comments and reactions. For example, without limitation, if the statistical values indicate that the viewer looped a certain part of the video 401 three times between time points 2:23 and 3:12, the video creation service 460 may flag that segment as being problematic. Similarly, exemplary comments that may indicate that a segment of the original video 401 needs correction can include statements such as "The panel demonstration is too fast to see the details," "The video does not mention the legroom inside the car," and "How can this function with the latest version of the GNU / Linux operating system?"

[0051] In some embodiments, the feedback engine 468 of the video creation service 460 may generate insights by clustering comments processed via natural language processing (NLP) based on the meaning of the comments. NLP can also, without limitation, extract keywords such as "speed up," "accelerate," "faster," etc., and use them to cluster into actions indicating acceleration of segments of the original video. Another example is to cluster sentences containing keywords such as "information is missing," "not clear," and "not understandable" to indicate segments of the video that need clarification in some aspects, which can be converted, for example, into an action to add a more detailed explanation of that segment of the original video.

[0052] In some embodiments, the feedback engine 468 of the video creation service 460 may also generate insights by applying sentiment analysis to comments and reactions. For example, if a plurality of comments or reactions or both use extremely negative words, the system may also weight the relevant segments more heavily for modification. Examples of negative comments with a greater weight that determine to modify such segments include "There is no way to see the feet of the car," "The use of the camera when presenting the feet is terrible," and "It is really stressful to see the feet in this video." Sentiment analysis may also utilize the timestamps associated with the comments to generate the direction and trend for the sentiment (e.g., the comments are generally favorable before some event and generally not favorable after that event).

[0053] In operation 530, the video segment analysis engine 462 of the video creation service 460 may select one or more video segments for modification according to the insights. In some embodiments, the video segments that may receive the benefit of the modification can be identified by, for example, matching the insights obtained from operation 525 with the content of the automatically generated transcript of the video 401. Some embodiments may also perform object identification analysis on the video 401 and compare the resulting list of objects with the insights. Some embodiments may also utilize the timestamps associated with the statistical values, or the timestamps submitted in the comments and / or reactions, or both, to identify the segments for modification.

[0054] In operation 540, the content modifier 464 of the video creation service 460 may determine how to modify the identified segments of the original video 401. This operation may include comparing the insights from operation 525 with a menu of potential operations, such as stretching the segment to play it slower, shortening the segment to play it faster, adding new frames to new content, adding overlays to new content, adding text bubbles to new content, and adding content from external sources 465 such as other websites, clip art, digital models, blogs, etc. This operation may also include generating a plurality of potential modifications and then making a selection from among the potential modifications using the preferences and restrictions received in operation 510.

[0055] For each operation involving new content, the content modifier 464 of the video creation service 460 may first create a revision script in response to the comments (i.e., modify the generated transcript), and then use GAN technology to create a simulation of the voice and appearance of the performer / narrator 420. Some embodiments may also supplement the audiovisual material with supplementary text material such as live captioning or text overlays or both.

[0056] In some embodiments, using GAN to add additional material may include: synthesizing new frames from existing footage of the performer / narrator and a newly created script with new statements, and synthesizing audio similar to the voice or appearance or both of the video performer / narrator. When creating new visual effects, such as creating and adding new stylized text material, GAN can learn the styling of the visual effects used in the original video and then be used to style the new overlay so that it matches the original style.

[0057] Next, in operation 550, the content selection module 466 of the video creation service 460 may improve part of the video modification considering the constraints or optimizations specified in operation 510. For example, some embodiments may operate to prioritize the modification of the video by adding information related to trending topics, where this modification is in response to concerns about the maximum number of comments or in response to concerns of the most respected and influential viewers 430. However, adding content to address all three of these cases may make the resulting modified videos 401a - 401d too long. In this case, in one embodiment, the video creation service 460 can present to the video owner 410 all or a subset of the possible combinations of new versions of the original video, so that the video owner can select the updated version to be transferred to the content delivery service 455, or the video owner can specify a set of priorities for the modification operations. In this case, a ranked list of the possible updated versions is generated based on this priority, and the first version that meets the constraints of the video owner can be selected.

[0058] In some cases, some insights may lead to competing modifications. For example, some viewers 430 may desire more details, while other viewers may not desire as much detail. In such cases, the content selection module 466 of the video creation service 460 may, in operation 570, select which modifications should actually be inserted based on the priorities specified in operation 510. In some embodiments, the content selection module 466 of the video creation service 460 may respond by creating multiple different modified versions of videos 401a - 401d targeted at each group of viewers 430 in operation 570, and then upload all of the versions to the content delivery service 455. In some embodiments, the video creation service 460 may additionally or alternatively automatically generate its own comments and reactions to the original video 401 and decide to issue them to the comment or message board, or both, of the content delivery service 455 to address any thread concerns in operation 570 (e.g., in case of competition, comments may be calculated as being more appropriate as a text response to lower - priority modifications, or comments requesting a URL, or both).

[0059] In operation 580, the content modifier 464 of the video creation service 460 may process the selected modifications to create one or more additional modified versions of videos 401a - 401d. Also, in operation 590, the video creation service 460 may generate metadata and insert the metadata into the modified videos 401a - 401d to identify which parts of the modified videos 401a - 401d are new, what changes were made, when the changes were made, etc.

[0060] The video creation service 460 may optionally seek the necessary license and / or approval from the original owner 410 of the video 401 for the modification in operation 592. In operation 594, the video creation service 460 may automatically upload the modified versions of the videos 401a - 401d and start collecting statistical values, comments, and reactions for the modified versions 401a - 401d. In operation 596, the video creation service 460 may compare the collected statistical values (such as the number of views, viewing duration, etc.) of the modified versions 401a - 401d, as well as the collected comments and reactions, with the statistical values, comments, and reactions for the original version of the video 401. If the statistical values, comments, and sentiment do not improve the video, the video creation service 460 may roll back the changes and notify the original owner. Alternatively, some embodiments may conduct A / B tests on the various versions of the videos 401, 401a - 401d and select the version with better statistical values, comments, and reactions.

[0061] Computer program product Although the present invention has been described in detail with reference to its specific examples, the present invention may also be embodied in other specific forms without departing from its essential idea or nature. For example, the present invention may be a system, method, or computer program product or a combination thereof in the integration at any possible technical detail level. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to execute aspects of the present invention. The computer-readable program instructions may be stored and executed on a single computer, or may be divided among different computers for storage and execution at the same location or different locations.

[0062] A computer-readable storage medium can be a tangible device that holds and stores instructions for use by an instruction execution device. The computer-readable storage medium can be, for example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing, but is not limited thereto. A non-exhaustive listing of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in a groove with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed to be a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through an optical fiber cable), or an electrical signal transmitted through an electrical wire.

[0063] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a separate computing / processing device or to an external computer or an external storage device via a network, such as, for example, the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. Each computing / processing device's network adapter card or network interface receives the computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within the separate computing / processing device.

[0064] Computer-readable program instructions for performing the operations of the present invention may be in any combination of one or more programming languages, including assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuits, or object-oriented programming languages such as Smalltalk(R), C++, and procedural programming languages or similar programming languages such as the "C" programming language, and may be in either source code or object code. The computer-readable program instructions may be executed entirely on the customer's computer, partly on the customer's computer as a stand-alone software package, partly on the customer's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the customer's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may utilize the state information of the computer-readable program instructions to execute the computer-readable program instructions to customize the electronic circuit in order to implement aspects of the present invention.

[0065] These computer-readable program instructions, when executed via a processor of a computer or other programmable data processing apparatus, may create means for implementing the functions / acts specified in one or more blocks of a flowchart, a block diagram, or both, thereby causing a machine, such as a computer or other programmable data processing apparatus, to function in a particular manner. These computer-readable program instructions may also be stored in a computer-readable storage medium that includes instructions for implementing the functions / acts of the one or more blocks of a flowchart, a block diagram, or both, such that when the instructions are executed by a computer, a programmable data processing apparatus, or other device, the computer-readable storage medium causes the device to function in a particular manner.

[0066] These computer-readable program instructions may also be loaded onto a computer, other programmable apparatus, or other device to create a computer-implemented process such that the instructions, when executed on the computer, other programmable apparatus, or other device, implement the functions / acts specified in one or more blocks of a flowchart, a block diagram, or both by causing a series of operational steps to be performed on the computer, other programmable apparatus, or other device.

[0067] General Introduction Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions. Moreover, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block of the flowchart or block diagram can represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams or flowchart illustrations, and combinations of blocks in the block diagrams or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified functions or acts, or combinations of dedicated hardware and computer instructions.

[0068] Any specific program nomenclature used in this description is for convenience only, and thus the present invention should not be limited to using only any specific application identified by, or implied by, or both, such nomenclature. Thus, for example, routines executed to implement embodiments of the present invention may be referred to by a "program", "application", "server", or other meaningful name, whether implemented as part of an operating system or a specific application. In fact, other alternative hardware or software or both environments may be used without departing from the scope of the present invention.

[0069] Accordingly, the embodiments described herein are considered to be illustrative in all respects and not restrictive, and it is desirable that the appended claims for determining the scope of the present invention be referred to.

Claims

1. 1. A method for automatically generating enhancements to audiovisual (AV) content, comprising: receiving, via a network interface, data about consumer interaction with the original AV content; The processing unit analyzing the data about consumer interactions to generate consumer insights about the original AV content; automatically associating said consumer insights with segments of said original AV content; automatically generating content for the segment in response to the consumer insights; generating modified AV content by inputting the generated content into the original AV content; automatically transmitting the modified AV content via the network interface; A method comprising:

2. The method of claim 1 , further comprising: collecting the data about the consumer interactions with the original AV content.

3. The method of claim 1 , wherein the data includes comments about the original AV content, and further comprising analyzing the comments to generate the consumer insights.

4. The analyzing step comprises: performing natural language processing on the comments to determine meaning; clustering the comments using the determined meanings; and The method of claim 3 , comprising:

5. The method of claim 3 , wherein the analyzing comprises performing a sentiment analysis of the comments to determine sentiment and sentiment direction.

6. said automatically generating said content further comprises: generating a transcript of the original AV content; comparing said consumer insights with said transcript; The method of claim 1 , comprising:

7. said automatically generating said content further comprises: modifying the transcript in response to the consumer insights; and generating a simulated performance using the modified transcript; The method of claim 6 further comprising:

8. 8. The method of claim 7, wherein the automatically generating the content further comprises generating the simulated performance using a generative adversarial network, the simulated performance resembling a voice and appearance of a performer.

9. generating supplemental text material using said consumer insights; and overlaying supplemental material onto said segments of content; The method of claim 6 further comprising:

10. analyzing the data about consumer interactions to generate a plurality of consumer insights about the original AV content; detecting a conflict between at least two of the plurality of insights; selecting a preferred insight from among the at least two competing insights, wherein the automatically generated content is responsive to the preferred insight; Automatically generate comments according to the lower priority insight; and publishing the generated comments as responses to comments on the original AV content; The method of claim 6 further comprising:

11. selecting external content in response to the consumer insights; adding the external content to the original AV content; The method of claim 6 further comprising:

12. generating metadata about the modified AV content; adding said metadata to said modified AV content; The method of claim 1 further comprising:

13. The consumer interaction, Comments on said content; and Data about consumers' use of said content; The method of claim 1 , comprising:

14. receiving data about the consumer interaction with the modified AV content; comparing the data about the consumer interaction with the modified AV content with the data about the consumer interaction with the original AV content; Removing the modified AV content or the original AV content based on the comparison; and The method of claim 1 further comprising:

15. receiving a profile from an owner of the original AV content, the profile specifying at least one preference and at least one restriction; automatically generating, by the processing unit, content for the segment in response to the at least one preference and the at least one restriction; The method of claim 1 further comprising:

16. automatically generating a plurality of potential content for the segment in response to the consumer insights; estimating additional lengths of time associated with the plurality of potential pieces of content; and selecting from among the plurality of potential content using the at least one preference and the at least one restriction; The method of claim 15 further comprising:

17. The method of claim 1 , wherein the data about consumer interactions with the original AV content includes trending search terms related to the original AV content.

18. The method of claim 1 , further comprising automatically revising, by the processing unit, the segments in response to the consumer insights.

19. A content manager, 1. A content creation server communicatively coupled to a plurality of content consumer devices, the server comprising: a processor coupled to a memory, the processor and the memory comprising: receiving data about consumer interaction with original viewing (AV) content; analyzing the data about consumer interactions to generate consumer insights about the original AV content; automatically associating said consumer insights with segments of said original AV content; automatically generating content for the segment in response to the consumer insights; generating modified AV content by inputting the generated content into the original AV content; automatically transmitting the modified AV content; the content creation server configured to A content manager comprising:

20. 1. A computer program product for automatically generating enhancements to audiovisual (AV) content, the computer program product including a computer-readable storage medium having program instructions embodied therein, the program instructions executable by a processor to cause the processor to receiving data about consumer interaction with the original audiovisual content; analyzing the data about consumer interactions to generate consumer insights about the original AV content; automatically associating said consumer insights with segments of said original AV content; automatically generating content for the segment in response to the consumer insights; generating modified AV content by inputting the generated content into the original AV content; automatically transmitting the modified AV content; A computer program product that causes

Images