Method, recording medium, computer program and system for training a neural network

Abstract

To provide a control system.SOLUTION: A neural network comprises an input layer, one or more hidden layers, and an output layer. The input layer is divided into a situation context input sub-layer, (in some embodiments) a background context input sub-layer, and an environmental input sub-layer. The output layer includes selecting or sequencing output sub-layer and an environmental output sub-layer. Each of the layers (including the sub-layers) includes a plurality of neurons, and each of the plurality of neurons includes an activation value. Situation context, environmental information, and background context may be input to the neural network. The neural network generates outputs that are used to dynamically select and sequence selected storylines that are used to modify a story based on viewer sentiment, environment or context, or a combination thereof.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to artificial intelligence and artificial reality. More particularly, the present invention relates to dynamically modifying and / or adapting one or more media (video and / or audio) streams based on a user or viewer's context and / or environment to provide an enhanced viewer experience and / or to more effectively achieve the objectives of the media provider. [Background technology]

[0002] The fields of artificial intelligence and consumer electronics are converging in the film and entertainment industry to bring video and audio content in multiple venues and through multiple paths to users. Movies, concerts and events are experienced not only in venues (such as theaters, concert halls and stadiums) and on television, but also as media on computers and iPhones, where content can be consumed on demand, continuously or intermittently, and at various times, locations and user situations. Typically, however, these media are available only in fixed sequences of events, images, sounds or speech, or combinations thereof, with fixed content. Such media tend to be static, fixed and incapable of responding to changes in the environment in which the media is presented and / or the context in which the user or viewer consumes the media. In many cases, once content is created, there is no way to change the content based on the sentiment, context or environment of the user or viewer, for example, the composition (demand), number, mood, emotion, background of the user or viewer, or parameters of the environment in which the media is presented, or a combination of these.

[0003] There is a need to be able to modify media content and sequencing in a manner that is adaptable to the sentiment, context and environment of a user or viewer. Summary of the Invention

[0004] Viewed from a first aspect, the present invention provides a control system for adapting a media stream, the control system comprising: a neural network having an input layer, one or more hidden layers and an output layer, the input layer having a situational context input sublayer and an environmental input sublayer, the output layer having a selection or ordering output sublayer and an environmental output sublayer, each of the layers having a plurality of neurons, each of the plurality of neurons having an activation value; one or more situational context modules each having one or more context inputs from one or more sensors monitoring a viewer and one or more sentiment outputs connected to the situational context input sublayer; one or more environmental information modules having one or more environmental sensor inputs and an environmental output connected to the environmental input sublayer, one or more selection modules connected to the selection or ordering output sublayer; and one or more ordering modules connected to the selection or ordering output sublayer, the selection module operable to select one or more selected storylines and the ordering module operable to sequence the selected storylines into a story to be played.

[0005] Viewed from a further aspect, the present invention provides a method for training a neural network, the method comprising: for a plurality of sentiment activity values ​​and sentiment neurons, respectively, inputting the sentiment activity values ​​to sentiment neurons in a situation context input sublayer, the situation context input sublayer being part of an input layer of the neural network, the sentiment activity values ​​forming a sentiment input pattern; and for a plurality of environment activity values ​​and environment neurons, respectively, inputting the environment activity values ​​to environment neurons in an environment input sublayer, the environment input sublayer being part of an input layer of the neural network. The method includes inputting environmental activation values, which form an environmental input pattern; propagating the sentiment input pattern and the environmental input pattern through a neural network; determining how much to change one or more weights and one or more biases by minimizing a loss function applied to an output layer of the neural network, the output layer having a selection or ordering output sublayer and an environmental output sublayer, each having output activation values; backpropagating to change the weights and biases; and repeating the previous two steps until the output activation values ​​reach a desired result, terminating training.

[0006] Viewed from a further aspect, the present invention provides a control implementation method for controlling a control system as defined in any one of claims 1 to 14, the method comprising the steps of: monitoring a viewer and one or more sentiment outputs connected to a situation context input sublayer; selecting one or more selected storylines; and sequencing the selected storylines into a story to be played.

[0007] Viewed from a further aspect, the present invention provides a computer program product for managing a system, the computer program product including a computer readable storage medium readable by a processing circuit and storing executable instructions for causing the processing circuit to perform a method of the present invention.

[0008] Viewed from a further aspect, the present invention provides a computer program stored on a computer readable medium and readable into an internal memory of a digital computer, the computer program comprising software code portions for performing the method of the present invention when the program is executed on a computer.

[0009] Viewed from a further aspect, the present invention provides a storyline control system comprising: a neural network having an input layer, one or more hidden layers, and an output layer, the input layer having a situational context input sublayer, a background context input sublayer, and an environmental input sublayer, the output layer having a selection or ordering output sublayer and an environmental output sublayer, each of the layers having a plurality of neurons, each of the plurality of neurons having an activation value; one or more situational context modules each having one or more context inputs from one or more sensors monitoring a viewer and one or more sentiment outputs connected to the situational context input sublayer; one or more environmental information modules having one or more environmental sensor inputs and an environmental output connected to the environmental input sublayer; one or more selection modules connected to the selection or ordering output sublayer; and one or more ordering modules connected to the selection or ordering output sublayer, wherein the selection module selects one or more selected storylines and the ordering module orders the selected storylines into a story to be played.

[0010] The present invention is a storyline control system that includes methods for training and running the system.

[0011] A neural network is used in the system. In some embodiments, the neural network is a convolutional neural network (CNN). The neural network has an input layer, one or more hidden layers, and an output layer. The input layer is divided into a situation context input sublayer, a background context input sublayer (in some embodiments), and an environmental input sublayer. The output layer has a selection or ordering output sublayer and an environmental output sublayer. Each of the layers (including the sublayers) has a plurality of neurons, each of the plurality of neurons having an activation value.

[0012] The one or more situational context modules each have one or more context inputs from one or more sensors monitoring the audience and one or more sentiment outputs connected to the situational context input sublayer.

[0013] The one or more environmental information modules have one or more environmental sensor inputs for monitoring the viewer environment and an environmental output connected to the environmental input sublayer.

[0014] In some embodiments, the one or more background context modules have one or more background inputs containing viewer characteristics and a background output connected to the environment input sublayer.

[0015] One or more selection modules are coupled to the selection or ordering output sublayer to control the selection of one or more selected storylines.

[0016] One or more sequencing modules may be connected to the selection or sequencing output sublayer to sequence the selected storylines, e.g., by a storyline control system, causing the selected storyline to change the story before the story is played or while the story is being played.

[0017] The present invention will now be described, by way of example only, with reference to preferred embodiments as illustrated in the following drawings. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a block diagram illustrating one architecture of the present invention. [Diagram 2] 1 illustrates one or more users in an environment. [Diagram 3] Shows a story with one or more main storylines and one or more alternative storylines, including start points, alternative start points, branching points, end points and alternative end points. [Figure 4] FIG. 2 is a system architecture diagram of a neural network embodiment of the present invention. [Diagram 5] 4 is a flow chart of a training process used to train one embodiment of the present invention. [Figure 6] 3 is an operational process flow diagram illustrating steps of one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] As used herein, the term "audience" refers to one or more users. The terms "user" and "audience" are used interchangeably without loss of generality. By way of non-limiting example, audience context may include user sentiment, composition (e.g., user demographics - age, education, status, social and political viewpoints), profile (e.g., likes, dislikes), feelings, emotions, location, number, time of day, environmental response, media response, or emotional state, or a combination thereof. In some embodiments, two types of user / audience context are used: background context and situational context.

[0020] Media refers to any sensory output of a system that imparts one or more physical sensations to a user. Non-limiting examples of media can include video, audio, speech, changes in the environment of a room or space (e.g., air currents, temperature, humidity, etc.), smells, lighting, special effects, etc.

[0021] The environment or user / viewer environment is the totality of some or all of the sensory output at any given time being output by the system.

[0022] Experience or viewer / user experience is the perceptual experience felt by a user / viewer at any given time caused by the environment in which the media / story is being played.

[0023] One or more media contain a story, and each story has one or more "storylines", e.g., subcomponents or segments. A storyline has a start point and an end point. Typically, the storylines are played in sequence, e.g., output on one or more output devices, to create the one or more stories and associated environments that are part of the stories. One or more second storylines of the same or different media or both may start or end based on the start / end or at a cue in the first storyline, or both.

[0024] For example, a first video storyline plays a sequence of images of "Little Red Riding Hood" walking up to the door of "Grandma's" house. When "Little Red Riding Hood" opens the door (a cue in the first video story), a first audio storyline begins playing and a door opening sound is heard. After the door opens, the first video and first audio storyline end, for example, at an end point. At the end point of the first video and first audio storyline, a second video storyline begins (at its start point) and a sequence of images is played showing "Little Red Riding Hood" walking into the house. In the middle of the second video storyline, "Little Red Riding Hood" meets the "Big Bad Wolf". This cue point in the second video storyline waits for the start of a second audio segment of the omnibus music (at its start point) and for the first control system storyline to turn down the lights and turn up the audio volume.

[0025] The content, selection and sequencing of these storylines drive one or more system outputs, creating a dynamic user / viewer environment while the story is playing. If the content, selection and environment change over time, the user / viewer experience changes over time.

[0026] A "story" is a selection of one or more storylines that are played in a storyline sequence. Each storyline has one or more storyline contents. Each storyline is played or output in one or more media types, e.g., video, audio, environmental changes (temperature, airflow), smell, lighting, etc. The storyline selection, storyline content, and storyline sequencing all affect the user / viewer's environment and experience. Thus, changing any of the storyline selection, content, or sequencing, or combinations thereof, during a story, changes the user / viewer's environment and experience. Changes in the environment and experience change the user context, which is input into the system and may then change the storyline selection, content, or sequencing, or combinations thereof.

[0027] The present invention is a system and method that enables storyline selection and sequencing in a story, in real time, while the story is being delivered and / or played to a user / viewer. Storyline changes can be based on situational context (user / viewer experience), environmental or background context, or a combination of these.

[0028] In some embodiments, the system monitors or inputs the user / viewer context and determines how to modify the selection and / or sequencing of one or more storylines to change the story or environment or both and thus the viewer experience. In some embodiments, the modifications are made based on some criteria, such as the media provider's objectives for creating an experience, communicating a message, or gauging viewer / user response, or a combination thereof.

[0029] In some embodiments, two types of user / viewer context are used: background context and situational context. User / viewer background context is the context that exists prior to the playing of the story. Non-limiting examples of background context include user composition (e.g., demographics - age, education, socio-economic status, income level, social and political viewpoints), user profile (likes, dislikes), time of day, number of users in the audience, weather, location of the audience. Non-limiting examples of situational context include one or more user sentiments in the audience, such as feelings, emotions, reactions to the environment (verbal descriptions of the environment / story, facial expression reactions, etc.).

[0030] The system can collect background context information about one or more users in the audience. This information can be collected from social media, user input, monitoring users' use of the system, image recognition, etc. In some embodiments, background context information including non-user information, e.g., weather, is input. Background context information can be obtained from calendars, system clocks, network searches, etc., and information from user surveys, for example, can be used.

[0031] In addition, the system monitors and inputs situational context. In some embodiments, situational context is collected from a facial recognition system used to identify facial expressions and / or the user's mood. In some embodiments, the system analyzes voice input for word use, meaning, expression, volume, etc. to determine emotional state. In some embodiments, the system can measure, record, infer, or predict, or a combination of, viewer responses (e.g., through facial recognition and analysis and voice analysis, etc.) to changes in environmental input, media type, and / or story and storyline media content, along with changes in storyline selection and sequencing.

[0032] While presenting storyline sequences in a story, the system monitors environmental inputs such as sound, sound levels, light levels, light frequencies, smells, humidity, temperature, airflow, etc.

[0033] In a preferred embodiment, a neural network is used. During a training phase, training data including situational context information, background context information, or environmental information, or a combination thereof, is input to the neural network. The neural network is trained using known backpropagation techniques as described below.

[0034] During training, inputs are fed into the input layer and the output of the output layer is compared to the desired outcome. For example, the difference is used in minimizing a loss function to determine how to change the internal details of the neural network. The inputs are fed back again and the new outputs are compared to the desired outputs. The process is repeated until the system is trained.

[0035] During the operation phase, the output of the neural network controls the selection and sequencing of storyline and environmental outputs to generate one or more stories that are played as media. The resulting stories are played to a viewer (or a model of a viewer) to create an environment.

[0036] During operation, the viewer's experience is monitored and the selection and / or ordering of storylines (including environmental storyline outputs) is altered as determined by the trained neural network. To alter the story, the selection controller alters the selection of storylines and the sequence controller alters the sequence of stories to generate an altered story. The altered story may have storylines added, removed or modified or any combination thereof compared to the previous storyline.

[0037] During the operation phase, the trained system becomes increasingly capable of selecting and sequencing storylines and environmental outputs to generate one or more viewer experiences more predictably. The operation system can create stories, monitor the viewer experience, and dynamically change the selection and sequencing of one or more storylines (including environmental outputs) based on the immediate viewer experience to create different environments and outcomes for the viewer with predictable viewer reactions or responses. These changes may be made for future stories or may occur "on the fly" while a story is playing.

[0038] It should be noted that in some embodiments, the operating system is trained to provide output based on a viewer experience that incorporates dynamic background context, situational context or environment, or a combination thereof.

[0039] In accordance with the present invention, artificial intelligence can be used to increase a viewer's engagement with media (e.g., movies) by adapting its storyline in order to make the viewer's experience more exciting, thrilling, and impactful.

[0040] In accordance with the present invention, the number of views of a media presentation (e.g., a movie) may be increased because the viewing experience becomes more intense as the media presentation changes depending on the background and situational context by adapting the storyline, resulting in an improvement in the viewer experience as measured by achieving the media provider's goals, which may include selling more tickets, having more network views, or reaching a more diverse audience, or a combination of these.

[0041] The present invention can monitor and record viewer experience with one or more storylines or stories, for example by measuring situational context, background context or environment, or a combination thereof, and storing this information, for example on external storage, for later analysis and use.

[0042] In some embodiments, a convolutional neural network (CNN) is used to modify storyline selection or storyline ordering or both, or environmental control, or a combination of these.

[0043] Reference is now made to the drawings, and in particular to FIG. 1, which is a block diagram of one architecture 100 of the present invention.

[0044] In one embodiment, the control system 160 is connected 110C to one or more networks, cloud environments, remote storage or application servers 110, or combinations thereof. The network connection 110C may be any known connection, with a standard interface 110C to any standard network 110, including the Internet, an intranet, a wide area network (WAN), a local area network (LAN), the cloud, or a radio frequency connection, such as Wi-fi, or combinations thereof.

[0045] The network connection 110C is connected to components of the control system 160 by any known system bus 101 via a communication interface 104. Among these, media data can be communicated to and from the system 160 via the communication interface 104.

[0046] One or more computer processors, such as a central processing unit (CPU) 103, a co-processor 103, or a graphical processing unit (GPU), or a combination thereof, are connected to the system bus 101.

[0047] One or more input / output (I / O) controllers 105 are connected to the system bus 101. The I / O controllers 105 are connected to one or more I / O connections 115. The I / O connections or buses 115 connect the I / O devices by hardwiring or wirelessly, for example, radio frequency or optical connections. Examples of I / O devices include external storage 111 (e.g., portable data storage such as CD / DVD devices, external hard drives, cloud storage, etc.), one or more input sensors 112, one or more displays 113, for example, graphical user interfaces (GUIs), and one or more other controllers 114, for example, environmental controllers. Media content or storyline information or both may be communicated over the network connections 110C or the I / O connections 115 or a combination thereof, for example, to the external storage 111, the displays / GUIs 113, and other (e.g., environmental) outputs 114.

[0048] One or more memories 102 are connected to the system bus 101. Accessibly stored in the memory 102 are system software 106, such as one or more operating systems (OpSys), operational storage 107, such as a cache memory, and one or more application software modules 108.

[0049] Other applications may be resident in memory 102 for execution on system 160. One or more functions of these applications may also be executed externally to system 160, with results provided to system 160 for execution. The situational context module 120 analyzes inputs that affect the situational context while a storyline or story is being played. The background context module 130 analyzes background context information. The environment module 140 analyzes the effect of the environment while a storyline or story is being played. The selection module 150 selects a storyline for playing. The sequencing module 165 selects the sequence in which the selected storyline is played. The output of the selection module 150 and the sequencing module 165 may be executed before playing a story or dynamically while playing a story, based on the analysis of the situational context 120, background context 130, environment information 140, or system configuration, or a combination thereof. The environment module 170 coordinates the environment output with the selected and sequenced storyline and programs, and controls the environment output via the I / O controller 105.

[0050] 2 is a diagram of one or more users / viewers 250 in a user environment 200. The environment 200 includes a space with one or more inputs and one or more outputs. The inputs include sensors 112, such as image capture devices (e.g., cameras 205), audio capture devices 210 (microphones 210), and a graphical user interface (GUI) 113. The inputs also include environmental inputs, such as a temperature detector 230, a humidity detector (not shown) 112, and a position detector 280. The outputs include the GUI 113, a television, a display, a home entertainment system 255, and audio output devices (e.g., speakers) 215. Several environmental outputs (e.g., 225) are controlled by an environmental controller 105, e.g., via a bus 115, to control environmental parameters including lighting 225, temperature, humidity, and airflow.

[0051] Auxiliary devices such as mobile phone 275, data assistants, monitors (e.g., "fitbit" and heart rate monitors), blood pressure monitors, respiration detectors, motion detectors, and computers provide input or output or both to a user, typically 250. Inputs such as microphone 210, temperature sensor 230, mobile phone 275, and camera 205 and outputs such as speaker 215, mobile phone 275, lights 225, and the like may be located anywhere in environment 200, including toys 245, etc. The inputs and outputs may be hardwired or wirelessly connected, for example, using bus 115.

[0052] An audience (typically 250) includes one or more users 250. Users 250 have different characteristics. For example, one user 250 may be a parent 250P and another user 250 may be a youth 250C. An audience 250 can be diverse. Non-limiting examples of an audience 250 include a family watching TV, a group watching a movie, a group playing video games, a crowd at an entertainment venue, viewers of a commercial, students in a classroom, a group at a theme park, a movie theater audience, and people attending an athletic or political event.

[0053] The control system 160 may be located anywhere there is a connection to I / O devices (112, 113, 114, etc.). For example, the control system 160 may be located within an entertainment system 255. In some embodiments, the control system 160 is connected (wired or wirelessly) to the network / cloud 110 via the communication interface 104 110C.

[0054] The sensors 112 capture signals (e.g., images and audio) from the user 250 via bus connections 115 to the respective I / O controllers 105, among other things. The signals are processed and the content of the received signals is analyzed by one or more of the situation context module 120, the background context module 130 and the environment module 140. In some embodiments, information is also received from one or more of the network / cloud 110, user input via the interface 113, sensors, or input from auxiliary devices such as the mobile phone 275 and storage 111, or a combination thereof.

[0055] Using these inputs, the situational context module 120, background context module 130 and environment module 140 determine the situational context 260, background context or environmental parameters (e.g., temperature, volume, lighting levels, humidity, etc.) in the environment 200, or a combination thereof.

[0056] Non-limiting examples are provided.

[0057] The stereo camera 205 provides signals to the situational context module 120 for the situational context module 120 to perform body pose recognition and determine one or more reactions 260 of the user 250, such as surprise, laughter, sadness, excitement, etc.

[0058] The camera 205 provides a signal to the situational context module 120, which performs facial recognition and determines a reaction 260 of the user 250 based on facial expressions.

[0059] The infrared camera 205 provides signals to the situational context module 120 which determines the body and facial temperatures representative of the reaction 260 of the user 250 .

[0060] The thermometer 230 sensor 112 provides a signal to the environmental module 140 to measure the ambient temperature, which is an environmental parameter of the environment 200 .

[0061] The microphone 210 provides a signal to the environment module 140 to determine the ambient sound level within the environment 200 .

[0062] The microphone 210 provides signals to the situational context module 120, which performs speech and sound (e.g., crying, laughing) recognition. Natural language processing (NLP) may be used to determine one or more user 250 responses 260. In some embodiments, the situational context module 120 performs NLP to detect key phrases that refer to the story being played. The NLP can detect phases of speech by the user 250 that indicate the user's 250 expectations and / or sentiment regarding the story's progression and / or ending.

[0063] Using image data received from the sensor 112, the situational context module 120 can perform age recognition, which may provide an indication of the reaction 260 of the user 250.

[0064] The situational context module 120 can use the motion data from the motion sensor 112 (eg, rapid movement, no movement, gait, etc.) to determine a user response 260.

[0065] The background context module 130 uses information received from the network 110, user input 113 (e.g., user surveys) or stored memory 111, or a combination thereof, to form a user profile for one or more users 250. Information may be collected from descriptions and activity on social media accounts, information about the user's groups and friends on social media, the user's search activity, web sites visited, merchandise information sought, items purchased, etc. Similar information may be accessed from activity on auxiliary devices such as a mobile phone 275. Of course, appropriate authorization and access privileges must be obtained to access this information.

[0066] The output of the situational context module 120 includes a set of sentiments or user reactions or both, each having a reaction value / score representing the level of sentiment / reaction 260 for one or more of the users. This represents the sentiment status or sentiment pattern of each of the users 250. In some embodiments, this sentiment pattern / status is input to a situational context sublayer in a neural network as described in FIG. 4. In some embodiments, the users 250 are first classified according to their sentiment status. In some embodiments, an aggregator aggregates the sentiment / reactions 260 of multiple users 250 and determines an aggregated set of sentiments / reactions 260 and corresponding values ​​that represent the sentiment status of one or more groups (250P, 250C) in the audience 250 or the entire audience 250 or both. The sentiment status may change while the story is playing.

[0067] The output of the background context module 130 is a user profile of one or more of the users 250. Each user profile has multiple user characteristics with associated characteristic values / scores representing the level of each of the user characteristics for a given user 250. In some embodiments, the user profiles are grouped by similarity. In some embodiments, a profile aggregator aggregates the profiles of multiple users 250 and determines an aggregate profile with respective profile values / scores representing the profiles of one or more groups (250P, 250C) of the audience 250 or the entire audience 250 or both. The profiles may change while the story is playing, although in some embodiments the changes in the profiles are expected to be less dynamic. In some embodiments, the profile of the characteristics is input to the background context sublayer 454 as a pattern of activity values ​​410, as shown in FIG. 4.

[0068] The output of the environmental module 140 is an environmental profile of the environment 200. The environmental profile has multiple environmental parameters, each with an associated parameter value / score that represents the level that each parameter has in the environment 200 at a given time. The environmental profile changes while the story is being played. The control system 160 can also modify the environment 200 and thus the environmental profile by changing the selection and sequencing of storylines. In some embodiments, the environmental profile of the environmental parameters is input into the environmental sublayer 456 as a pattern of activity values ​​410, as shown in FIG. 4.

[0069] FIG. 3 shows a story 300 having one or more main storylines ( 304 , 394 ) and one or more alternative storylines 383 , including a start point 301 , an alternative start point 302 , a branching point 380 , an end point 309 and an alternative end point 310 .

[0070] Figure 3 is an illustration of a story 300 having one or more storylines (303, 323, 333, 343, 363, 393, typically 383) of a main storyline or story that runs from a start point 301 directly to an end point 309. There is a storyline 383 that defines a first story and an alternative storyline 383. The alternative story develops from the first / original story by adding, inserting or deleting storylines 383 or a combination thereof in the first / original story.

[0071] A story 300, for example, has a start point 301 and an end point 309. Alternate start points 302 and alternative end points 310 may exist.

[0072] In addition, there are branching points, which are points in the story 300 where the story 300 can be altered by inserting, deleting, or adding storylines 383. In some embodiments, alternative storylines 383 begin and end, for example at branching points 340 and 345, and alter the content of the original story without changing the continuity of the story.

[0073] A branching point usually coincides with a storyline start point, e.g., 320, 330, 340, 360, 370, typically 380, or a storyline end point 315, 325, 335, 365, typically 385. By coinciding a storyline start point 380 with a branching point and a storyline end point 385 with a branching point, the story 300 may be altered by playing the inserted storyline (with its start and end points coinciding with the branching point) in place of the previous storyline in the sequence.

[0074] For example, story 300 is originally sequenced to start at 301 and play storyline 304 between points 301 and branching point 315. The system can modify story 300 by starting at starting point 302, playing storyline 303 instead of storyline 304, and ending storyline 303 at branching point 315, which is common to the ending points of both storylines 303 and 304. By substituting storyline 304 for storyline 303, control system 160 has changed the selection (selecting 304 instead of 303) and ordering (playing 304 first and not playing 303) of storyline 383 in story 300.

[0075] As a further example, assume an original story 300 begins at 301 and continues by playing a single storyline between starting point 301 and directly at 309. This original story can be altered in many ways by selecting different storylines in a different order. The story can begin at 301 and continue to branch point 320, where storyline 323 is selected and played to branch point 325. Alternatively, the original story can begin at 302 and continue to branch point 330, where system 160 selects storyline 333 and returns to the original storyline at branch point 335. Again, the system can play a storyline beginning at 301, where system 160 can select storyline 343 to begin at branch point 340 and return to the original storyline at branch point 345. Storylines 393 and 394 provide alternative ending points (either 309 or 310) in story 300, depending on which story ending point the system selects and plays at branching point 370. Note that within an alternative storyline (e.g., 363), there may be branching points 367 that can start or end another storyline 305, for example at a cue point.

[0076] "Playing" a storyline 383 enables one or more outputs to provide media to a viewer 250, as described above, that corresponds to the selected storyline 383, sequenced and played.

[0077] FIG. 4 is a system architecture diagram of one embodiment of a neural network 400 of the present invention.

[0078] Neural network 400 includes a number of neurons, typically 405. Each of the neurons 405 stores a value referred to as an activation value 410. For example, neuron 405 holds an activation value of "3." For clarity, most of the neurons and activation values ​​are not numbered in FIG.

[0079] Neural network 400 includes multiple layers, e.g., 420, 422, 424, 425, typically 426. There is a first or input layer 420 and a final or output layer 426. Between the input layer 420 and the output layer 426 are one or more hidden layers, e.g., 422, 424. Each of the layers 425 has a number of neurons 405. In some embodiments, the number of layers 425 and the number of neurons 405 in each layer are determined empirically by experimentation.

[0080] In some embodiments, every neuron in the previous layer is connected to each of the neurons in the next layer by an edge 415. For example, a typical neuron 406 in the next (hidden) layer 422 is individually connected to each neuron 405 in the input layer 420 by an edge 415. In some embodiments, one or more of the edges 415 has an associated weight W 418. In a similar manner 430, each neuron 406 in the next layer, e.g., 422, is connected to all of the neurons 405 in the previous layer, e.g., 420, by an edge 415. The same kind of connections 415 are made between each neuron in the second hidden layer 424 and each neuron in the first hidden layer 422, as well as between each neuron 495 in the output layer 426 and all of the neurons in the second hidden layer 424. These connections 430 are not shown in FIG. 4 for clarity.

[0081] In some embodiments, the activity value 410 in each neuron 406 is determined by a weighted sum of the activity values ​​410 of each connected neuron 405 in the previous layer. Each activity value 410 is weighted by the weight (W, 418) of the edge 415 that connects the neuron 406 to each of the corresponding neurons 405 in the previous layer, e.g., 420.

[0082] Thus, the pattern of activation values ​​410 in a previous layer, e.g., 420, together with the weights (w, 418) of each edge 415, determine the pattern of activation values ​​406 in the next layer, e.g., 422. In a similar manner, a weighted sum of the set of activation values ​​406 in the previous layer, e.g., 422, determines the pattern of activation values ​​of each neuron, typically 405, and thus the pattern of activation values ​​of neurons in the next layer, e.g., 424. This process continues until a pattern of activation values ​​emerges, represented by the activation values ​​in each of the neurons 495, typically 490, in the output layer 426. Thus, given a pattern of activation values ​​405 in the input layer 420, the structure of the neural network 400, the weights (w, 418) and biases b (described below) determine the pattern of activation values ​​output, which are the activation values ​​of each of the neurons 495 in the output layer 326. As the set of activation values ​​in the input layer 420 changes, so does the set of activation values ​​in the output layer 426. The set of varying activation values ​​in the hidden layers (422, 424) is a level of abstraction that may or may not have physical meaning.

[0083] In some embodiments, the input layer 420 is subdivided into two or more sublayers, such as 452, 454, and typically 456.

[0084] The input sublayer 452 is a situational context sublayer 452 that receives the activation values ​​410 from the output of the situational context module 120. The pattern of activation values ​​at the input layer 452 represents the status of the sentiment / reaction 260 of the viewer 250. For example, neurons 405 in the situational context sublayer 452 represent the viewer's reactions such as happiness, excitement, and anxiety.

[0085] The input sublayer 454 is the background context sublayer 453, which receives activity values ​​410 from the output of the background context module 130. The activity values ​​410 of the neurons 405 in the background context sublayer 454 represent the values / scores of the traits in the profile of the user / viewer 250. In some embodiments, the pattern of activity values ​​410 in the background context sublayer 454 represents the background context status of the traits at the time of the user / viewer's 250 profile.

[0086] The input sublayer 456 is an environmental sublayer 456 that receives activity values ​​410 from the output of the environmental module 140. The activity values ​​410 of the neurons 405 in the environmental sublayer 456 are values / scores that represent environmental parameters in an environmental profile. In some embodiments, the pattern of activity values ​​410 in the environmental sublayer 456 represents the environmental profile status as it changes over time while the story is being played.

[0087] In some embodiments, output layer 426 is subdivided into two or more sublayers, such as 482 and 484 , typically 480 .

[0088] Output sublayer 482 has neurons 495 that determine which storylines are selected and when each of the selected storylines begins and ends, for example, at branching points (380, 385).

[0089] The output sublayer 484 has neurons 495 that determine how the control system 160 controls the I / O devices, and in particular, the environmental outputs to change the environment 200 .

[0090] One mathematical representation of the transition from one layer to the next in neural network 400 is as follows:

[0091]

number

[0092] Here, a n 1 is the activation value 410 of the n-th neuron 406 in the next level, here level 1; n,k is the weight (w, 418) of the edge 415 between the k-th neuron 405 of the current level, here level 0, and the n-th neuron 406 of the next level, here level 1; b n is the bias value for the weighted sum of the nth neuron 406 at the next level. In some embodiments, the bias value can be thought of as a threshold for turning on a neuron.

[0093] The term σ is a scaling factor, which can be, for example, a sigmoid function or a rectified linear unit, e.g., ReLU(a)=max(0,a).

[0094] The neural network is trained by finding values ​​for all the weights (w, 418) and biases b. In some embodiments, the well-known backpropagation method is used to find the weight and bias values.

[0095] In some embodiments, to begin training, the weights and biases are set to either random values ​​or some initial set of values. The output of the output layer 426, i.e., the activation pattern, is compared to the desired outcome. Comparison of the actual output to the desired outcome via a loss function (e.g., the square root of the sum of squared differences) measures how close the output is to the desired output for a given input. The loss function is minimized, e.g., by gradient descent, via an iterative process to determine how much and in what direction to change the weights and biases to move them closer to the desired values. The weights and biases are modified, i.e., backpropagated, and another iteration is performed. Multiple iterations are performed until the output layer produces an activation pattern that is close to the desired outcome for a given activation pattern imposed on the input layer.

[0096] In an alternative embodiment, neural network 400 is a convolutional neural network (CNN). In a CNN, one or more of the hidden layers are convolutional layers where convolution is performed with one or more filters to detect, emphasize, or de-emphasize patterns within the layer. For example, there are different filter types for detecting sub-shapes in images or sub-patterns in speech or both. In a preferred embodiment, filter 470 is a matrix of values ​​that is convolved over the input to the layer to generate new patterns of the input to the layer.

[0097] FIG. 5 is a flow chart of a training process 500 used to train one embodiment of the present invention.

[0098] The training process 500 begins with an untrained neural network 400. Initial activation patterns from the situation context module 120, background context module 130 and environment module 140 are input to the input layer 420. Alternatively, the inputs may be simulated.

[0099] For example, the output of the situational context module 120 is input (505) to the situational context sublayer 452 as a pattern of activity values ​​410 representing the status of the sentiment / reaction 260 of the viewer 250. The output of the background context module 130 is input to the background context sublayer 454 as a pattern of activity values ​​410 representing the status of the background context status at a point in time, i.e., the status of a characteristic user / viewer 250 profile. The output of the environment module 140 is input (505) to the environment sublayer 456 as a pattern of activity values ​​410 representing the status of the environmental parameters in the environment profile.

[0100] In step 510, the weights 418 and bias b are initially set.

[0101] In step 520 , the activation values ​​410 are propagated to the output layer 426 .

[0102] In step 530, the output of sublayer 482 is compared to the desired selection and ordering of storylines. Additionally, the output of sublayer 484 is compared to the desired configuration of environmental outputs. A loss function is minimized to determine the magnitude and direction of change required for a new set of weights 418 and biases b.

[0103] In step 540, the new weights 418 and biases b are backpropagated in known manner and a new set of outputs (482, 484) is received.

[0104] After re-entering the input activation values, a check is made at step 550. If the outputs of sublayers 482 and 484 are within the tolerances of the desired selection and ordering and environmental outputs, then process 500 ends. If not, control returns to minimizing the loss function at step 530 and the process is repeated again.

[0105] FIG. 6 is a flow chart of an operational process 600 illustrating the operational steps of one embodiment of the present invention.

[0106] In step 605, the output of the situational context module 120 is input (605) to the situational context sublayer 452 as a pattern of activity values ​​410 representing the status of the sentiment / reaction 260 of the viewer 250. The output of the background context module 130 is input (605) to the background context sublayer 454 as a pattern of activity values ​​410 representing the status of the background context status at a point in time, i.e., the status of the profile of the characteristic user / viewer 250. The output of the environment module 140 is input (605) to the environment sublayer 456 as a pattern of activity values ​​410 representing the status of the environmental parameters in the environment profile.

[0107] In step 610, the output of sublayer 484 is provided to environmental controller 170. The output of environmental controller 170 controls environmental output 114 via I / O controller and bus 115 to modify the environment (620).

[0108] In step 610, the output of sublayer 482 is provided to selection module 150 and ordering module 165. Selection module 150 determines which storyline 283 has been selected for playback from the pattern of activity values ​​in sublayer 482. Sequencing module 165 determines the ordering for playing the selected storyline.

[0109] In one embodiment, in step 630, the selection module 150 examines the list of available storylines. If a storyline is not selected, it continues examining the list until it encounters and selects all storylines specified by the pattern of activity values ​​in sublayer 482. In an alternative embodiment, the pattern of activity values ​​in sublayer 482 directly identifies which storyline is selected.

[0110] The ordering module 165 determines the sequence in which the selected stories are played from the pattern of activity values ​​in the sublayer 482. The pattern of activity values ​​in the sublayer 482 also determines the branching points at which the selected storylines start and end. Step 640 monitors the playing of the stories or the sequence in which the stories are played. In step 640, the ordering module 165 determines whether a branching point has been reached and whether the ordering module 165 needs to modify (e.g., add, delete, insert) one of the selected storylines at this branching point. If not, the ordering module 165 waits until the next branching point is reached and the determination is made again.

[0111] If a storyline from the selected storyline needs to be played at the branching point, then the storyline changes in step 650. The new storyline is played and ends at the appropriate branching point (e.g., as indicated by the pattern of activity values ​​in sublayer 482) and the original storyline is not played at this point in the sequence.

[0112] Various embodiments are identified in the following numbered clauses: (Appendix 1) 1. A control system for adapting a media stream, comprising: a neural network having an input layer, one or more hidden layers, and an output layer, the input layer having a situation context input sublayer and an environmental input sublayer, the output layer having a selection or ordering output sublayer and an environmental output sublayer, each of the layers having a plurality of neurons, each of the plurality of neurons having an activation value; one or more situational context modules, each having one or more context inputs from one or more sensors monitoring an audience, and one or more sentiment outputs connected to the situational context input sublayer; one or more environmental information modules having one or more environmental sensor inputs and an environmental output connected to said environmental input sublayer; one or more selection modules connected to the selection or ordering output sublayer; one or more ordering modules connected to said selection or ordering output sublayer; wherein the selection module is operable to select one or more selected storylines and the sequencing module is operable to sequence the selected storylines into a story to be played. (Appendix 2) 2. The control system of claim 1, wherein the context input includes one or more of a facial image, an infrared image, an audio input, an audio volume level detector, text, spoken words, a mobile phone input, heart rate, blood pressure, and respiratory rate. (Appendix 3) The situation context module comprises: Face recognition, location recognition, natural language processing (NLP), key phrase recognition and voice recognition 3. The control system of claim 1 or 2, operable to perform one or more of the functions of (Appendix 4) The sentiment output may be: Feelings, emotions, laughter, sadness, anticipation, fear, excitement, reactions to the environment, reactions to one or more storylines The control system according to any one of appendixes 1 to 3, wherein the viewer sentiment is represented by one or more of the following: (Appendix 5) The environmental sensor input is Audio capture devices, microphones, video capture devices, cameras, infrared cameras, network connections, weather inputs, location sensors, mobile phones, thermometers, humidity detectors, airflow detectors, light intensity detectors, light detectors and motion sensors. 5. The control system according to any one of claims 1 to 4, comprising one or more of the following: (Appendix 6) 6. The control system of claim 1, wherein the environmental outputs include one or more of a volume control, a lighting control, a temperature control, a humidity control, a heating system control, a cooling system control, and an airflow control. (Appendix 7) The control system of any of claims 1 to 6, wherein the selection module is operable to select one or more of the selected storylines based on a dynamic pattern of the sentiment output and a dynamic pattern of the environmental output processed by the neural network. (Appendix 8) The control system of any of claims 1 to 7, wherein the ordering module is operable to order one or more of the selected storylines based on dynamic patterns of the sentiment output and dynamic patterns of the environmental output processed by the neural network. (Appendix 9) 9. The control system of claim 1, wherein the neural network is a convolutional neural network (CNN). (Appendix 10) A control system according to any one of appendices 1 to 9, wherein one or more of the selected storylines begins and ends at a branching point. (Appendix 11) 11. The control system of any one of claims 1 to 10, wherein the neural network further includes a background context input sublayer. (Appendix 12) The activity values ​​of one or more neurons in the background context input sublayer are Audience demographics, age, education level, socio-economic status, income level, social and political viewpoints, user profile, likes, dislikes, time of day, number of viewers, weather, location of viewers 12. The control system of claim 11, comprising one or more of the following expressions: (Appendix 13) One or more pieces of user context information Among the sources of social media posts, social media usage, mobile usage, audience research, search history, calendar, system clock, image analysis and location information 13. The control system of claim 11 or 12, originating from one or more of the following sources: (Appendix 14) A control system as described in any of Appendix 11 to 13, wherein the selection module is operable to select one or more of the selected storylines, and the ordering module orders one or more of the selected storylines based on dynamic patterns of sentiment output, background context and environmental output processed by the neural network. (Appendix 15) 1. A method of training a neural network, comprising: For a plurality of sentiment activity values ​​and sentiment neurons, inputting the sentiment activity values ​​to sentiment neurons in a situation context input sublayer, the situation context input sublayer being part of an input layer of the neural network, the sentiment activity values ​​forming a sentiment input pattern; inputting, for a plurality of environmental activity values ​​and environmental neurons, the environmental activity values ​​into environmental neurons in an environmental input sublayer, the environmental input sublayer being part of the input layer of the neural network, the environmental activity values ​​forming an environmental input pattern; propagating the sentiment input patterns and the environmental input patterns through the neural network; determining how to change one or more weights and one or more biases by minimizing a loss function applied to an output layer of the neural network, the output layer having a selection or ordering output sublayer and an environment output sublayer, each having output activation values; backpropagating to vary the weights and biases; repeating the previous two steps until the output activation value reaches a desired result, and then terminating the training; A method comprising: (Appendix 16) The method includes, after the training is completed, selecting one or more selected storylines; inserting the selected storyline within an initial story, where a start point of the selected storyline begins at a start branch point of the initial story and an end point of the selected storyline ends at an end branch point of the initial story; 16. The method of claim 15, comprising: (Appendix 17) 17. The method of claim 16, wherein the environmental output layer has a dynamic pattern of output activity values ​​that control one or more environmental outputs associated with the selected storyline. (Appendix 18) 18. The method of claim 16 or 17, wherein the steps of selecting the selected storylines and ordering the selected storylines are determined by a dynamic sentiment input pattern. (Appendix 19) A control implementation method for controlling a control system according to any one of claims 1 to 14, comprising: monitoring an audience and one or more sentiment outputs connected to a situational context input sublayer; selecting one or more selected storylines; sequencing the selected storylines into a story to be played; A method comprising: (Appendix 20) 1. A computer program product for managing a system, comprising: A computer-readable storage medium readable by a processing circuit and storing executable instructions for causing said processing circuit to perform the method according to any one of claims 15 to 19. (c) computer program products, (Appendix 21) A computer program stored on a computer readable medium and readable into the internal memory of a digital computer, the program comprising software code portions for performing the method according to any one of claims 15 to 19 when the program is run on a computer. As noted above, in embodiments of the present invention, the context input may include one or more of face images, infrared images, audio input, audio volume level detector, text, spoken words, mobile phone input, heart rate, blood pressure, and respiration rate. Additionally, the activity values ​​of one or more neurons of the background context input sublayer may include representations of one or more of audience demographics, age, education level, socio-economic status, income level, social and political viewpoints, user profile, likes, dislikes, time of day, number of audience users, weather, and viewer location. Additionally, the one or more user background context information may originate from one or more sources of social media posts, social media usage, mobile phone usage, audience surveys, search history, calendar, system clock, image analysis, and location information sources. Of course, appropriate authorization and access privileges must be obtained to access the information. The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications, or technical improvements to the technology found in the market, or to allow other skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A method of training a neural network, comprising the steps of: inputting, for a plurality of sentiment activity values ​​and sentiment neurons, the sentiment activity values ​​into sentiment neurons in a situation context input sublayer, the situation context input sublayer being part of an input layer of the neural network, the sentiment activity values ​​forming a sentiment input pattern; inputting, for a plurality of environmental activation values ​​and environmental neurons, the environmental activation values ​​to environmental neurons in an environmental input sublayer, the environmental input sublayer being part of the input layer of the neural network, the environmental activation values ​​forming an environmental input pattern; propagating the sentiment input patterns and the environmental input patterns through the neural network; determining how to change one or more weights and one or more biases by minimizing a loss function applied to an output layer of the neural network, the output layer having a selection or ordering output sublayer and an environment output sublayer, each having an output activation value; backpropagating to vary the weights and biases; repeating the previous two steps until the output activation value reaches a desired result, and then terminating the training; A method comprising:

2. The method includes, after the training is completed, selecting one or more selected storylines; inserting the selected storyline within an initial story, where a start point of the selected storyline begins at a start branch point of the initial story and an end point of the selected storyline ends at an end branch point of the initial story; The method of claim 1 , comprising:

3. The method of claim 2 , wherein the environmental output sublayer comprises a dynamic pattern of output activity values ​​that control one or more environmental outputs associated with the selected storyline.

4. The method of claim 3, wherein the environmental outputs include one or more of volume control, lighting control, temperature control, humidity control, heating system control, cooling system control and airflow control.

5. A method according to any one of claims 2 to 4, wherein one or more of the selected storylines begins and ends at a branching point.

6. The method of any one of claims 2 to 5, wherein the steps of selecting the selected storylines and ordering the selected storylines are determined by a dynamic sentiment input pattern.

7. A method according to any one of claims 1 to 6, wherein the neural network is a convolutional neural network (CNN).

8. The method of claim 1, wherein the neural network further includes a background context input sublayer.

9. The activity values ​​of one or more neurons of the background context input sublayer are Audience demographics, age, education level, socio-economic status, income level, social and political viewpoints, user profile, likes, dislikes, time of day, number of viewers, weather, location of viewers The method of claim 8 , comprising one or more representations of:

10. A recording medium for storing a computer program for managing a system, A recording medium readable by a processing circuit and storing executable instructions for performing the method according to any one of claims 1 to 9 by said processing circuit.

11. A computer program stored on a computer readable medium and readable into an internal memory of a digital computer, the program comprising software code portions for performing the method according to any one of claims 1 to 9 when said program is executed on a computer.

12. A system for training a neural network, comprising: one or more processors; one or more memories coupled to said one or more processors; wherein the one or more processors perform a method by executing program instructions stored in the one or more memories, the method comprising: inputting, for a plurality of sentiment activity values ​​and sentiment neurons, the sentiment activity values ​​into sentiment neurons in a situation context input sublayer, the situation context input sublayer being part of an input layer of the neural network, the sentiment activity values ​​forming a sentiment input pattern; inputting, for a plurality of environmental activation values ​​and environmental neurons, the environmental activation values ​​to environmental neurons in an environmental input sublayer, the environmental input sublayer being part of the input layer of the neural network, the environmental activation values ​​forming an environmental input pattern; propagating the sentiment input patterns and the environmental input patterns through the neural network; determining how to change one or more weights and one or more biases by minimizing a loss function applied to an output layer of the neural network, the output layer having a selection or ordering output sublayer and an environment output sublayer, each having an output activation value; backpropagating to vary the weights and biases; repeating the previous two steps until the output activation value reaches a desired result, and then terminating the training; Including, the system.

Images