Methods for automatically generating enhancements for AV content, content managers, programs

Abstract

To provide a method, and the like, for automatically generating enhancements to audiovisual (AV) content.SOLUTION: A method for automatically generating enhancements to audio-visual (AV) content, a content manager, and a computer program product are provided. The method may comprise: by a processing unit, analyzing data about consumer interactions to generate consumer insights about the original AV content; automatically associating the consumer insights with a segment of original AV content; automatically generating content for the segment responsive to the consumer insights; and injecting the generated content into the original AV content to create modified AV content. The method may further comprise, by a network interface, receiving data about consumer interactions with original AV content and automatically transmitting the modified AV content.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to creating and managing audiovisual (AV) content, and more particularly, to automatically expanding AV content in response to viewer interaction with the AV content.

Background Art

[0002] The development of the EDVAC system in 1948 is often cited as the beginning of the computer age. Since then, computer systems have evolved into extremely complex devices. Today's computer systems typically include a combination of sophisticated hardware and software components, application programs, operating systems, processors, buses, memories, input / output devices, and the like. As performance has increased with advancements in semiconductor processing and computer architecture, more evolved computer software has evolved to utilize the higher performance of these functions, leading to current computer systems that are much more powerful than just a few years ago.

[0003] A wide variety of content creators have begun to utilize these improved features through content streaming and podcasting. Content streaming generally refers to the transmission of audio, video, or both files from a server to a client, typically over the internet, and is widely used to view video clips and movies on various media players such as televisions, computers, tablets, and smartphones. Typically, with content streaming, the viewer / end user can begin playing the content before the entire file has been transmitted. In contrast, podcasting also generally refers to the transmission of audio, video, or both files from a server to a client, also typically over the internet. However, with podcasting, the transmission generally occurs before the viewer begins playing the content.

[0004] Streaming and podcasting have proven to be excellent platforms in themselves for advertising, product reviews, and troubleshooting problems (e.g., information technology, home maintenance, and industry-related issues). [Overview of the project]

[0005] Embodiments of the present disclosure provide a method for automatically generating enhancements to video content (AV) content. One embodiment may include, by a processing unit, analyzing data about consumer interactions to generate consumer insights about original AV content, automatically associating the consumer insights with segments of the original AV content, automatically generating content for the segments according to the consumer insights, and creating modified AV content by inserting the generated content into the original AV content. The embodiment may further include, by a network interface, receiving data about consumer interactions with the original AV content and automatically transmitting the modified AV content.

[0006] According to embodiments of the present disclosure, a content manager is provided. One embodiment includes a content creation server communicatively coupled to a plurality of content consumer devices, the server including a memory-coupled processor. The processor and memory may be configured to receive data about consumer interactions with original viewing (AV) content, analyze the data about consumer interactions to generate consumer insights about the original AV content, automatically associate the consumer insights with segments of the original AV content, automatically generate content for the segments according to the consumer insights, feed the generated content into the original AV content to create modified AV content, and automatically transmit the modified AV content.

[0007] Embodiments of the present disclosure provide a computer program product for automatically generating enhancements to video content (AV). The computer program product in one embodiment may include a computer-readable storage medium in which program instructions executable by a processor are embodied. The program instructions may cause the processor to receive data about consumer interaction with original AV content, analyze the data about consumer interaction to generate consumer insights about the original AV content, automatically associate the consumer insights with segments of the original AV content, automatically generate content for the segments according to the consumer insights, input the generated content into the original AV content to create modified AV content, and automatically transmit the modified AV content.

[0008] The above summary is not intended to describe any of the exemplary embodiments or implementations of this disclosure.

[0009] The drawings included in this application are incorporated herein and form part thereof. These drawings illustrate embodiments of the disclosure and, together with the description, serve to illustrate the principles of the disclosure. The drawings are merely illustrative of specific embodiments and do not limit the disclosure. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows a cloud computing environment consistent with several embodiments. [Figure 2] This figure shows an abstract model layer consistent with several embodiments. [Figure 3] This figure shows a data processing system consistent with several embodiments. [Figure 4] This figure shows a system architecture consistent with several embodiments. [Figure 5A] This figure shows a method for expanding content, consistent with several embodiments. [Figure 5B] This figure shows a method for expanding content, consistent with several embodiments. [Modes for carrying out the invention]

[0011] The present invention follows various modifications and alternative forms, specific examples of which are shown in the drawings and described in detail. However, it should be understood that the invention is not intended to be limited to the specific embodiments described. On the contrary, it is intended to cover all modifications, equivalents, and alternative forms that fall within the spirit and scope of the invention.

[0012] A part of this disclosure relates to creating and managing AV content, and a more detailed part relates to automatically expanding AV content in response to consumer interaction with the AV content, such as comments and viewing behavior. While this disclosure is not necessarily limited to such uses, various parts of this disclosure can be understood through discussions of various examples of its use in this context.

[0013] Some embodiments of this disclosure can provide an automated system for generating new versions of original content, such as a video or podcast. This system may include: converting data about consumer interaction with the content, such as public comments and viewing / listening behavior (e.g., at what point in the video viewers begin to interact or at what point in the video users stop interacting), as well as external data such as trend search terms, into insights; using the insights to automatically generate content, modify content, or both; feeding generated, modified, or both content into a modified version of the content that meets the owner's constraints and goals (e.g., new video length, external factors affecting video viewing time, meeting highly demanding modifications); and minimizing conflicting requests through the creation and addition of comments in the thread for the requested content modification.

[0014] In this method, several embodiments can reduce the time and resources required to create diverse versions of AV content tailored to various viewer profiles, to correct errors in the original content, to supplement the original content, to update the original content, or to maintain viewer engagement with the content, or a combination thereof. Specifically, some embodiments can enable the automatic correction of minor but relevant details that may be missing or incorrect in the original version of AV content, or the automatic addition of information in response to trending topics (e.g., the release of a new version of a smartphone), or both. Correcting these errors and deficiencies can lead to increased viewing and time spent on AV content, and ultimately, significant branding and investment opportunities for AV content owners.

[0015] In an exemplary scenario, a content creator / owner might create an original version of a video and then upload it to a streaming service. The streaming service then both delivers the video to end users and provides feedback mechanisms, such as a public forum or comment board. When uploading the original video to the streaming service, the content owner may specify that they want, or allow, automatically generated extensions to that video, or both. Some content owners may also specify one or more restrictions on such extensions, such as "I want no more than four versions of this video, and each version should be less than 130% of the original video's duration."

[0016] Next, the content distribution service 455 (described in more detail below, see Figure 4) can begin distributing the original video to end users, who can submit feedback to the streaming service in the form of text comments on a message board / comment board. Examples of user comments include, "It would be perfect if I could watch it at 1.25x speed," "It's difficult to view this URL in a browser at 12:54," and "Does this work on the newly released iPhone?" The streaming service can also allow users to react to comments submitted by others by indicating that others "like" or "dislike" the comment, or by submitting reaction comments to the original comment, or both (collectively referred to as user "reactions"). These reactions may be threaded using the base comment of the reactions, so that the reaction stating "This is what's set in the variable PYTHON, so this person is probably using 3.7" appears near the base comment "Which version of Python was this tested with?", regardless of when and by whom the two comments were submitted.

[0017] The video creation service 460 may analyze submitted comments and reactions to detect problems, or areas for improvement, in the original video, or both. This may include analyzing and understanding the meaning, tone, and direction of comments and reactions using natural language processing (NLP) techniques, including sentiment analysis. In some embodiments, this may include resolving conflicts between submitted comments and between reactions, such as prioritizing more recent comments, comments from people with higher online ratings, or both. The video creation service 460 may then associate the insights obtained with specific segments of the original video. This, again, may include using NLP techniques, such as generating a time-indexed transcript of the original video. This may also include using object recognition techniques to classify the content of the original video.

[0018] Next, in some embodiments, the optimal correction can be selected from a menu of potential actions in response to one or more comments or reactions, or both. In some embodiments, this menu of potential actions may include, but are not limited to, increasing the broadcast speed of a portion of the video (e.g., from 1x to 1.25x), decreasing the broadcast speed of a portion of the video (e.g., from 1x to 0.8x), searching for and inserting supplemental external material, creating and inserting supplemental original material, and editing portions of the original material. In some embodiments, this may include generating multiple candidates for addition, and then making a selection from the candidates using preferences and constraints given by the content owner.

[0019] In some embodiments, supplementary material may include overlaying text material onto existing video segments, such as adding URLs where a particular product can be obtained, or URLs where further information can be found, or URLs with some additional descriptive text. In some embodiments, supplementary material may include generating realistic simulations of the voice and / or face of the content presenter / actor in the original video by using deep learning and generative adversarial network (GAN) techniques. In this way, for example, if content or reactions indicate that the original content contains an error, it is possible to create a new version of the video with the correct facts, in which the facts appear as if the content presenter is actually speaking the correct facts. In some embodiments, the updated version may be published automatically, and in some embodiments, the owner of the original video may be required to acknowledge the changes.

[0020] In some embodiments, the video creation service 460 may produce multiple different modified versions of the original video by generating multiple different versions of edits or supplementary materials or both, in which case each modified version targets a different audience profile. Continuing the example above, the video creation service 460 may create a first version of the video targeting information technology (IT) professionals, in which an overlay is added and "Python version 3.7" is used, and a second version of the video clearly targets retail users of a specific brand of smartphone, in which a GAN creates a new video segment announcing compatibility and optimal settings for the latest version of the smartphone.

[0021] After issuing the updated video to the content delivery service 455, in some embodiments, the video creation service 460 may then conduct A / B testing on various versions (e.g., original version vs. revised version, new version A vs. new version B, etc.) to determine whether there are additional areas for improvement and also to determine which version has a higher viewer engagement score. If the new version has a better score, the content delivery service 455 can then license the resulting production to the content owner. In some embodiments, the video creation service 460 may also pay viewers who submitted comments and reactions to the original or corrected content used to create the new version of the video, as well as to external sources for any content used. In still other embodiments, the content delivery service 455 may create a marketplace for video links (cross-reference), where different versions of the original content can be presented to different users based on their viewer profiles.

[0022] Cloud computing FIG. 1 illustrates a cloud environment consistent with some embodiments. While this disclosure includes a detailed description of cloud computing, it should be understood that the implementations of the teachings set forth herein are not limited to a cloud computing environment. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0023] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0024] The characteristics are as follows: · On-demand self-service: Cloud consumers can unilaterally provision computing capabilities such as server time and network storage automatically as needed, without requiring human interaction with the service provider. · Broad network access: The capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs). · Resource pooling: The provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and re-assigned according to demand. There is a sense of location independence in that consumers generally do not have control or information about the exact location of the provided resources, although location can sometimes be specified at a higher level of abstraction (e.g., country, state, or data center). · Rapid elasticity: The capabilities are provisioned rapidly and elastically, and in some cases, automatically, quickly scaled out, and rapidly released and quickly scaled in. For consumers, the capabilities available for provisioning often appear to be unlimited and can be purchased at any time and in any quantity. • Service Measurement: Cloud systems automatically control and optimize resource usage by leveraging measurement capabilities at several levels of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active customer accounts). Resource usage is monitored, controlled, and reported, providing transparency to both service providers and consumers of the services being used.

[0025] The service model is as follows: Software as a Service (SaaS): The functionality provided to consumers is the use of the provider's applications running on cloud infrastructure. The applications are accessible from various client devices through thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application functions, with the exception of limited customer-specific application configurations. Platform as a Service (PaaS): The functionality offered to consumers is the deployment of consumer-created or acquired applications, written using programming languages ​​and tools supported by the provider, onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and, in some cases, the applications hosting the environment configuration. Infrastructure as a Service (IaaS): The functionality provided to consumers is the provisioning of processing, storage, networking, and other basic computing resources that consumers can deploy and run any software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have limited control over the operating system, storage, deployed applications, and possibly selected networking components (e.g., host firewalls).

[0026] The deployment model is as follows: • Private Cloud: Cloud infrastructure is operated exclusively for a specific organization. It is managed by that organization or a third party and can reside on-premises or off-premises. • Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community that has shared issues (e.g., mission, security requirements, policies, and compliance concerns). It can be managed by an organization or a third party and can reside on-premises or off-premises. • Public Cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by the organization that sells cloud services. • Hybrid Cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain unique entities, but are joined by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load balancing across clouds).

[0027] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure, which includes a network of interconnected nodes.

[0028] Referring now to Figure 1, an exemplary cloud computing environment 50 is depicted. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 that can communicate with local computing devices used by cloud consumers, such as personal digital assistants (PDAs) or mobile phones 54A, desktop computers 54B, laptop computers 54C, or automotive computer systems 54N or a combination thereof. The nodes 10 can communicate with each other. In one or more networks, such as private, community, public, or hybrid clouds or a combination thereof as described herein, they can be grouped physically or virtually (not shown). This allows the cloud computing environment 50 to provide infrastructure, platforms, or software or a combination thereof as a service, without requiring cloud consumers to maintain resources on their local computing devices. The types of computing devices 54A-N shown in Figure 1 are intended to be illustrative only, and it should be understood that the computing node 10 and the cloud computing environment 50 can communicate with any type of computerized device over any type of network or network addressable connection (e.g., using a web browser) or both.

[0029] Referring to Figure 2, a set of functional abstraction layers provided by the cloud computing environment 50 (Figure 1) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 2 are intended to be illustrative only, and embodiments of the present invention are not limited thereto. The following layers and corresponding functions are provided, as described:

[0030] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include: a mainframe 61, RISC (Reduced Instruction Set Computer) architecture-based servers 62, 63, blade servers 64, storage devices 65, and networks and network components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0031] The virtualization layer 70 provides an abstraction layer that provides the following examples of virtual entities: virtual servers 71, virtual storage 72, virtual networks 73 including virtual private networks, virtual applications and operating systems 74, and virtual clients 75.

[0032] In one example, the management layer 80 may provide the following functions: Resource provisioning 81 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 82 provides cost tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides verification of identities for cloud consumers and tasks, as well as protection for data and other resources. The customer portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources to ensure that required service levels are met. Service level agreement (SLA) planning and execution 85 provides pre-arrangements for cloud computing resources whose future demands are anticipated in accordance with SLAs, and procurement of cloud computing resources.

[0033] Workload Layer 90 provides examples of the functionality that cloud computing environments utilize. Examples of workloads and functions derived from this layer include: mapping and navigation 91, software development and lifecycle management 92, virtual classroom education delivery 93, data analytics processing 94, transaction processing 95, and content manager 96.

[0034] Data Processing System Figure 3 illustrates an embodiment of a data processing system (DPS) 300 suitable for use as a cloud computing node 10 within a cloud computing environment 50, consistent across several embodiments. In some embodiments, the DPS 300 may be implemented as a processor, smart device, or any other suitable type of electronic device, embedded in a personal computer, server computer, portable computer such as a laptop or notebook computer, PDA (personal digital assistant), tablet computer, or large device such as a smartphone, automobile, aircraft, teleconferencing system, or appliance. Furthermore, there may be components other than those shown in Figure 3, or additional components beyond those shown in Figure 3, and the number, type, and configuration of such components may vary. Moreover, Figure 3 only depicts representative and major components of the DPS 300, and individual components may be far more complex than those represented in Figure 3.

[0035] The data processing system 300 in Figure 3 may include a plurality of central processing units 310a to 310d (collectively referred to herein as processors 310 or CPUs 310), which are connected by a system bus 322 to memory 312, a mass storage interface 314, a terminal / display interface 316, a network interface 318, and an input / output ("I / O") interface 320. The mass storage interface 314 in this embodiment connects the system bus 322 to one or more mass storage devices, such as a direct access storage device 340, a Universal Serial Bus ("USB") storage device 341, or a read / write optical disk drive 342. The network interface 318 allows DPS 300a to communicate with other DPS 300b over a communication medium 306. The memory 312 may also include an operating system 324, a plurality of application programs 326, and program data 328.

[0036] The embodiment of DPS300 in Figure 3 may be a general-purpose computing device. Therefore, the processor 310 may be any device capable of executing program instructions stored in memory 312, and may itself be constructed from one or more microprocessors or integrated circuits, or both. In this embodiment, DPS300 includes multiple processors or processing cores, or both, as is common in large, high-performance computer systems. However, in other embodiments, DPS300 may include a single-processor system, or a single processor designed to emulate a multiprocessor system, or both. Furthermore, the processor 310 may be implemented using multiple heterogeneous DPS300s, in which the primary processor resides on a single chip accompanied by secondary processors. As another illustrative example, the processor 310 may be a symmetrical multiprocessor system including multiple processors of the same type.

[0037] When the DPS300 starts up, the associated processor 310 may first execute program instructions that constitute the operating system 324, which manages the physical and logical resources of the DPS300. These resources may include memory 312, a mass storage interface 314, a terminal / display interface 316, a network interface 318, and a system bus 322. Similar to processor 310, some embodiments of the DPS300 may utilize multiple system interfaces 314, 316, 318, 320, and bus 322, thereby allowing each to include its own separate, fully programmed microprocessor.

[0038] Instructions for an operating system, application, or program, or a combination thereof (collectively referred to as “program code,” “computer-readable program code,” or “computer-readable program code”) may initially be located in mass storage devices 340, 341, 342 that communicate with the processor 310 via the system bus 322. In various embodiments, the program code can be embodied in various physical or tangible computer-readable media, such as system memory 312 or mass storage devices 340, 341, 342. In the illustrative example of Figure 3, the instructions may be stored in a functional form of persistent memory on the direct-access storage device 340. These instructions may then be loaded into memory 312 for execution by the processor 310. However, the program code may also be located in a functional form on a selectively removable computer-readable medium 342, which may be loaded into or transferred to the DPS 300 for execution by the processor 310.

[0039] The system bus 322 may be any device that facilitates communication between the processor 310, memory 312, and interfaces 314, 316, 318, and 320. Furthermore, the system bus 322 in this embodiment is a relatively simple, single bus structure that provides a direct communication path between the system bus 322 and other bus structures, but may include, but is not limited to, hierarchical structures, point-to-point links in star or web configurations, multiple hierarchical buses, parallel and redundant paths, etc., and is consistent with this disclosure.

[0040] The memory 312 and the mass storages 340, 341, and 342 can work together to store the operating system 324, application programs 326, and program data 328. In the exemplary embodiment, the memory 312 is a random-access semiconductor device capable of storing data and programs. Although Figure 3 conceptually depicts the device as a single, monolithic entity, the memory 312 may have more complex configurations, such as a hierarchical structure of caches and other memory devices, in some embodiments. For example, the memory 312 may reside in multiple levels of caches, and these caches may be further divided by function, such that one cache holds instructions while another holds non-instruction data used by one or more processors. The memory 312 may be further distributed and associated with different processors 310 or sets of processors 310, as is known in any of the various so-called non-uniform memory access (NUMA) computer architectures. Furthermore, some embodiments may utilize a virtual addressing mechanism that allows the DPS300 to behave as if it had access to a single, large storage entity instead of accessing multiple smaller storage entities such as memory 312 and mass storage devices 340, 341, and 342.

[0041] Although the operating system 324, application program 326, and program data 328 are illustrated as being contained within memory 312, some or all of them may be physically located on different computer systems and, for example in some embodiments, may be accessed remotely via a communication medium 306. Therefore, although the operating system 324, application program 326, and program data 328 are illustrated as being contained within memory 312, these elements do not necessarily all need to be contained simultaneously on the exact same physical device, and may even reside in the virtual memory of another DPS 300.

[0042] System interfaces 314, 316, 318, and 320 support communication with a variety of storage and I / O devices. The mass storage interface 314 may support the attachment of one or more mass storage devices 340, 341, and 342, which are typically rotating magnetic disk drive storage devices, solid-state storage devices (SSDs) that use integrated circuit assemblies as memory for persistently storing data, usually using flash memory, or a combination of both. However, the high-capacity storage devices 340, 341, and 342 may also include other devices, such as arrays of disk drives (commonly called RAID arrays) configured to appear to the host as a single large storage device, or archival storage media such as hard disk drives, tapes (e.g., MiniDV), writable compact discs (e.g., CD-R and CD-RW), digital versatile discs (e.g., DVD, DVD-R, DVD+R, DVD+RW, DVD-RAM), holographic storage systems, blue laser discs, IBM(R) Millipede devices, or combinations thereof.

[0043] The terminal / display interface 316 may be used to directly connect one or more display units, such as monitors 380, to the DPS300. These display units 380 may be non-intelligent (i.e., dumb) terminals such as LED monitors, or they may be fully programmable workstations used to enable IT administrators and customers to communicate with the DPS300. However, it should be noted that while the display interface 316 is provided to support communication with one or more display units 380, the DPS300 does not necessarily require display units 380, as all necessary interactions with customers and other processes can occur via the network interface 318.

[0044] The communication medium 306 may be any suitable network or combination of networks and can support any suitable protocol suitable for data or code or both communication to and from multiple DPS 300s. Thus, the network interface 318 may be any device that facilitates such communication, regardless of whether the network connection is made using modern analog or digital or both technologies, or via some future networking mechanism. Suitable communication mediums 306 include, but are not limited to, the “InfiniBand” or IEEE (Institute of Electrical and Electronics Engineers) 802.3x “Ethernet(R)” specification; cellular transmission networks; wireless networks implementing one of the following: IEEE 802.11x, IEEE 802.16, General-Purpose Packet Radio Service ("GPRS"), FRS (Family Radio Service), or Bluetooth specification; and networks implemented using one or more of the following: ultra-wideband ("UWB") technologies as described in FCC 02-48. Those skilled in the art will understand that many different network and transport protocols can be used to implement the communication medium 306. The Transmission Control Protocol / Internet Protocol ("TCP / IP") suite includes suitable network and transport protocols.

[0045] System Architecture Figure 4 depicts a system 400 architecture consistent with several embodiments. The system 400 embodiment in Figure 4 may include an owner / creator 410 of the original video 401, a person 420 captured in the original video 401, and several viewers 430a-430d (collectively referred to as viewers 430) of the original video 401. Each of the owner 410 and viewers 430 may have DPS 411, 431a-431d (e.g., laptops, smartphones, tablets, etc., each of which may be a DPS 300 as described above in some embodiments) with a network interface that allows access to one or more common computer networks (e.g., the internet, a proprietary social networking platform, etc.). In some embodiments, system 400 may also include a cloud computing infrastructure 450 (e.g., a cloud computing environment 50 as described above) that hosts a content manager 496, the content manager 496 may include a content distribution service 455 and a video creation service 460.

[0046] During operation, the owner 410 of this embodiment may use their DPS 411 to upload the original version of the video 401 to the content distribution service 455. The content distribution service 455 may then store the original version of the video 401 in its computing resource 450 and stream the original version of the video 401 to a DPS 431 associated with a viewer 430 upon request. In some embodiments, a group of viewers 430 may receive the stream substantially simultaneously (e.g., a live stream). In other embodiments, each of the viewers 430 may receive the stream at different times (e.g., an on-demand stream, a podcast, etc.).

[0047] In this embodiment, viewers 430 can consume the original video 401 from the content distribution service 455. Some viewers 430 may respond by posting comments or reactions or both to the content distribution service 455, thereby making these comments or reactions or both available to other viewers 430. In addition, the content distribution service 455 may collect and use statistics about the viewers 430's activity, such as how long they watched the video 401, at which segment they started watching, at which segment they stopped watching, which segments of the video 401 viewers watched repeatedly (i.e., looped), whether viewers 430 paused the video 401 at any segment, and whether they searched for explanatory material from external sources. In some embodiments, the video creation service 460 may automatically analyze collected statistics, posted comments, or posted reactions, or a combination thereof, using natural language processing (NLP) techniques to generate insights into problems or areas for potential supplementation / improvement, or both, associated with the original video 401. The video creation service 460 may then calculate one or more desirable changes to the video 401 from a menu of potential actions, and then automatically generate one or more modified videos 401a-401d containing such changes. In some embodiments, when calculating the desirable changes, further conditions specified by the owner 410 may be utilized, such as limitations on the amount of computing power used, the total time of the newly created videos, the percentage increase in the time of the new videos, and the type of added content (e.g., only child-friendly language, only copyright-free content).

[0048] In some embodiments, the video creation service 460 may further include a relevant video segment analysis engine 462 for matching calculated insights with corresponding segments of video 401; a content modifier 464 for generating new content (e.g., modifying existing content, creating new content, creating new simulation content, or importing existing content from an external data source 465, or a combination thereof) and inserting that content into video 401; a content selection module 466 for determining, according to constraints, which modifications to make, which modifications to include, or both; and a feedback engine 468 that can use NLP to analyze viewer statistics, viewer comments, and viewer reactions to video 401.

[0049] Content expansion Figures 5A to 5B (collectively Figure 5) illustrate one method of extending content 500, consistent with several embodiments. In operation 510, the owner 410 of video 401 can define one or more video creation profiles. Each profile may include one or more preferences, such as a limit on the maximum number of modified videos to be created, a preference weight for one or more user profiles on which the modified video is created, a limit on how often modified videos are created, a limit on how long the modified version should be displayed, what types of third-party content may be used, and what types of modification operations are permitted. Each profile may also include one or more restrictions, such as a limit on the amount of computing power used, the total playback time of any modified videos 401a to 401d, the percentage increase in playback time of each modified video 401a to 401d compared to the original video 401, and any types of new content and licenses therefor (e.g., only child-friendly language, or only copyright-free content, but not limited to these). In operation 515, the owner 410 may upload the original video 401 and associate it with one of the modification profiles, or select one or more restrictions, or both.

[0050] Next, the content distribution service 455 may begin streaming the uploaded original video 401 to the viewer 430, and then in operation 520, may begin collecting statistics and demographic profiles about viewer behavior. Also in operation 520, the content distribution service 455 may receive and then host comments and reactions from the viewer 430. In operation 525, the video creation service 460 may begin analyzing the statistics collected by the content distribution service 455, as well as the content of the submitted comments and reactions. For example, but not limited to, if statistics show that a viewer looped a certain portion of video 401 three times between time points 2:23 and 3:12, the video creation service 460 may flag that segment as problematic. Similarly, exemplary comments that may indicate that a segment of the original video 401 needs correction may include statements such as, "The panel demonstration is too fast to see the details," "The video doesn't mention the legroom in the car," and "How can this work on the latest version of the GNU / Linux operating system?"

[0051] In some embodiments, the feedback engine 468 of the video creation service 460 may generate insights by clustering comments processed via natural language processing (NLP) based on the meaning of the comments. NLP can also be used, but is not limited to, to extract keywords such as “speed up,” “accelerate,” and “faster,” and cluster them to actions indicating acceleration in a segment of the original video. Another example is clustering sentences containing keywords such as “missing information,” “not clear,” and “not understandable,” to indicate segments of the video where some aspects require clarification, which can be translated, for example, into actions that add a more detailed explanation of that segment in the original video.

[0052] In some embodiments, the feedback engine 468 of the video creation service 460 may also generate insights by applying sentiment analysis to comments and reactions. For example, if multiple comments or reactions, or both, use extremely negative language, the system may also weight the relevant segment more heavily for correction. Examples of negative comments with a heavier weight that would lead to a decision to correct such a segment include "There's no way to see the car's feet," "The use of the camera when showing the feet is terrible," and "It's really stressful to see the feet in this video." Sentiment analysis may also utilize timestamps associated with comments to generate direction and trends in sentiment (for example, comments are generally favorable before a certain event and generally unfavorable after that event).

[0053] In operation 530, the video segment analysis engine 462 of the video creation service 460 may select one or more video segments for modification based on the insights. In some embodiments, video segments that may benefit from modification can be identified, for example, by matching the insights obtained from operation 525 with the content of the automatically generated transcript of video 401. In some embodiments, object identification analysis may also be performed on video 401 and the resulting list of objects may be compared with the insights. In some embodiments, segments for modification may also be identified using timestamps associated with statistics, or timestamps submitted in comments and / or reactions, or both.

[0054] In action 540, the content modifier 464 of the video creation service 460 may determine how to modify the identified segment of the original video 401. This action may include comparing insights from action 525 with a menu of potential actions and may include stretching the segment to play slowly, shortening the segment to play quickly, adding new frames to the new content, adding an overlay to the new content, adding a text bubble to the new content, and adding content from external sources 465 such as other websites, clip art, digital models, or blogs. This action may also include generating multiple potential modifications and then selecting from among the potential modifications using the preferences and limitations received in action 510.

[0055] For each action involving new content, the content modifier 464 of the video creation service 460 may first create a revised script in response to comments (i.e., modify the generated transcript), and then use GAN technology to create a simulation of the voice and appearance of the performer / narrator 420. Some embodiments may also supplement the viewing material with supplemental text material, such as live captioning or text overlays or both.

[0056] In some embodiments, using GANs to add additional material may include: synthesizing new frames and newly created scripts with new statements from existing footage of a performer / narrator, as well as synthesizing audio that resembles the voice and / or appearance of the video performer / narrator. When creating new visual effects, such as creating and adding new styled text material, the GAN can learn the styling of the visual effects used in the original video and then use that to style the new overlay so that the overlay matches the original style.

[0057] Next, in operation 550, the content selection module 466 of the video creation service 460 may refine some of the video modifications to take into account the constraints or optimizations specified in operation 510. For example, some embodiments may operate to prioritize modifications to the video that add information related to trending topics, such that the modifications address the concerns of the largest number of comments or the concerns of the most respected and influential audience 430. However, adding content to address all three of these cases may make the resulting modified videos 401a-401d too long. In this case, in one embodiment, the video creation service 460 may present the video owner 410 with all or a subset of possible combinations of new versions of the original video, so that the video owner can select which updated version should be forwarded to the content distribution service 455, or the video owner can specify a set of priorities for the modification operations. In this case, a ranked list of possible updated versions is generated based on this priority, and a first version that satisfies the video owner's constraints may be selected.

[0058] In some cases, several insights may lead to conflicting modifications; for example, some viewers 430 may want more detail, while others may not. In such cases, the content selection module 466 of the video creation service 460 may, in operation 570, select which modifications should actually be inserted based on the priority specified in operation 510. In some embodiments, the content selection module 466 of the video creation service 460 may, in operation 570, respond by creating multiple different modified versions of videos 401a-401d targeting each group of viewers 430, and then upload all versions to the content distribution service 455. In some embodiments, the video creation service 460 may additionally or alternatively decide to automatically generate its own comments and reactions to the original video 401 and publish them to the comment / message or both bulletin board of the content distribution service 455 in operation 570 to address any thread concerns (for example, in the case of a conflict, the comment may respond to a lower-priority modification, or a modification where a text response is calculated to be more appropriate than the other, such as a response to a comment requesting a URL, or both).

[0059] In operation 580, the content modifier 464 of the video creation service 460 may process the selected modifications to create one or more additional modified versions of videos 401a to 401d. In operation 590, the video creation service 460 may also generate metadata and insert it into the modified videos 401a to 401d to identify which parts of the modified videos 401a to 401d are new, what changes were made, when the changes were made, etc.

[0060] The video creation service 460 may optionally seek the necessary licenses or authorization from the original owner 410 of video 401, or both, for the modifications in operation 592. In operation 594, the video creation service 460 may automatically upload the modified versions of videos 401a-401d and begin collecting statistics, comments, and reactions for the modified versions 401a-401d. In operation 596, the video creation service 460 may compare the collected statistics (e.g., views, viewing duration, etc.) and collected comments and reactions for the modified versions 401a-401d with the statistics, comments, and reactions for the original version of video 401. If the statistics, comments, and sentiment do not improve the video, the video creation service 460 may roll back the changes and notify the original owner, etc. Alternatively, some embodiments may conduct A / B testing on various versions of videos 401, 401a-401d and select the version with better statistics, comments, and reactions.

[0061] Computer program products While the present invention has been described in detail with reference to specific examples thereof, the present invention may also be embodied in other specific forms without departing from its essential idea or nature. For example, the present invention may be a system, method, or computer program product or combination thereof in an integration of any possible level of technical detail. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to perform an aspect of the present invention. The computer-readable program instructions may be stored and executed on a single computer, or they may be divided between different computers for storage and execution, in the same or different locations.

[0062] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by instruction-executing devices. Computer-readable storage media may, but are not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. A non-exhaustive enumeration of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random-access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punched cards or raised structures in a groove on which instructions are recorded, and any suitable combination thereof. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmitting media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0063] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to an individual computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. The network adapter card or network interface of each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage on a computer-readable storage medium within the individual computing / processing device.

[0064] The computer-readable program instructions for performing the operation of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or object-oriented programming languages ​​such as Smalltalk(R) and C++, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the customer's computer, some as a standalone software package on the customer's computer, some on the customer's computer and some on a remote computer, or all on a remote computer or on a server. In the latter scenario, the remote computer may be connected to the customer's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can be personalized by executing computer-readable program instructions by utilizing state information of computer-readable program instructions in order to carry out aspects of the present invention.

[0065] These computer-readable program instructions may be provided to a computer or other programmable data processing processor to create a machine, so that instructions executed via the processor of the computer or other programmable data processing device may create means for implementing functions / operations specified in one or more blocks of a flowchart or block diagram or both. These computer-readable program instructions may also be stored on a computer-readable storage medium on which the instructions are stored can be instructed to function in a particular way to a computer, a programmable data processing device, or other device or combination thereof, so that the manufacturing article contains instructions for implementing modes of functions / operations specified in one or more blocks of a flowchart or block diagram or both.

[0066] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device to create a computer implementation process that implements a function / action specified in one or more blocks of a flowchart or block diagram, or both, causing the computer, other programmable device, or other device to perform a series of operable steps.

[0067] General Overview Aspects of the present invention have been described herein with reference to flowcharts or block diagrams, or both, of methods, apparatuses (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or both, and combinations of blocks in a flowchart or block diagram, or both, are implemented by computer-readable program instructions. Furthermore, the flowcharts and block diagrams in the drawings illustrate the architecture, function, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram can represent a module, segment, or portion of instructions, containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in a block may occur in a different order than shown in the drawings. For example, two consecutively shown blocks may actually be executed substantially simultaneously, or blocks may sometimes be executed in reverse order depending on the function they involve. It should also be noted that each block in a block diagram or flowchart, or both, and any combination of blocks in a block diagram or flowchart, or both, are implemented by a dedicated hardware-based system that performs a specified function or action, or executes a combination of dedicated hardware and computer instructions.

[0068] Any specific program naming conventions used in this description are for convenience only, and therefore the present invention should not be limited to using only any particular application identified, implied, or both by such naming conventions. For example, routines performed to implement embodiments of the present invention, whether implemented as part of an operating system or a particular application, could also be referred to as “programs,” “applications,” “servers,” or other meaningful names. In fact, other alternative hardware and / or software environments may be used without departing from the scope of the present invention.

[0069] Therefore, the embodiments described herein are considered in all respects to be illustrative and non-limiting, and it is desirable that the appended claims be referenced for determining the scope of the invention.

Claims

1. A method for automatically expanding viewing (AV) content, In an electronic device having one or more processors and memory, Providing AV content to multiple viewers, Receiving consumer dialogue data associated with the AV content, which includes statistics on the viewing behavior of the multiple viewers; Based on the aforementioned consumer dialogue data, select one or more segments of the AV content, To generate automatically expanded AV content, the system automatically expands one or more segments of the AV content based on the statistics of viewing behavior, The network interface automatically transmits the automatically expanded AV content, Methods that include...

2. The method according to claim 1, wherein automatically expanding the one or more segments includes prioritizing the modification of the one or more segments.

3. The consumer dialogue data further includes comments associated with the AV content, Selecting one or more segments of the AV content includes selecting one or more segments based on the comments associated with the AV content. The method according to claim 1.

4. Selecting one or more segments based on the comments associated with the AV content, To generate a set of insights based on the aforementioned comments, Matching a first insight from the set of insights, which corresponds to at least one of a first problem with the AV content, a first supplement to the AV content, and a first improvement to the AV content, with a first content of the transcript of the AV content, Matching a second insight from the set of insights different from the first insight, which corresponds to at least one of a second problem with the AV content, a second supplement to the AV content, and a second improvement to the AV content, with a second content different from the first content of the AV content transcript, including, The method according to claim 3.

5. The method according to claim 4, wherein generating the set of insights includes clustering the comments based on the meaning of the comments.

6. The method according to claim 1, wherein selecting one or more segments of the AV content includes weighting a first segment more heavily than a second segment.

7. The method according to claim 1, wherein the statistics of the viewing behavior of the plurality of viewers include viewing time, start segment, stop segment, repeat viewing count, or any combination thereof.

8. The method according to claim 1, wherein providing the AV content to multiple viewers includes implementing conditions in the AV content.

9. The method according to claim 1, wherein automatically expanding one or more segments of the AV content includes generating a revision script for the AV content.

10. The method according to claim 1, wherein receiving consumer interaction data associated with the AV content includes collecting a demographic profile.

11. A computer program that causes a computer to perform the method according to any one of claims 1 to 10.

12. A memory for storing the computer program described in claim 11, One or more processors capable of executing the computer program stored in the memory, An electronic device equipped with the following features.

Images