Artificial intelligence-based system and method for creating digital arts and stories

Abstract

The present invention discloses an artificial-intelligence based system and method for creating digital arts and stories. The system comprises a computing device in communication with a database via a network. The computing device is configured to create one or more interlinked panels and one or more buttons. Each panel comprises one or more choices. The computing device is configured to enable a user to navigate through the panels and buttons, and select one or more choices. The computing device is configured to enable the user to create a digital breadcrumb path to track every user selection and contextual change through a series of panels and buttons. The computing device is configured to send the digital breadcrumb path to an artificial intelligence (AI) model, and enable the AI model to create at least one of digital arts and stories based on the digital breadcrumb path.

Description

ARTIFICIAL INTELLIGENCE-BASED SYSTEM AND METHOD FOR CREATING DIGITAL ARTS AND STORIESCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Provisional Application Serial No. 63 / 585,243, entitled “ARTIFICIAL INTELLIGENCE-BASED SYSTEM AND METHOD FOR CREATING DIGITAL ARTS AND STORIES” filed September 26, 2023, the contents of which are incorporated herein by reference in their entirety.TECHNICAL FIELD

[0002] The present disclosure relates generally to a field of digital content creation using artificial intelligence, and more particularly, to an artificial-intelligence-based system and method for creating digital arts and stories.BACKGROUND

[0003] Artworks and stories serve as delightful mediums for sharing experiences, thoughts, and narratives with friends and family. Almost everyone has engaged in this form of storytelling, which spans from personal anecdotes to family and cultural histories. In the modern era, these artworks and stories can be shared digitally. Digital forms of artworks and interactive stories offer numerous advantages over traditional methods. The conventional approach to creating artworks and stories necessitates a separate infrastructure setup and various accessories.

[0004] Moreover, most digital artworks are crafted by individuals with computer skills using specialized software. These digital narratives are often created on systems that rely on graphical interfaces dependent on icons, menus, and some form of pointingor cursor control device. Such systems are typically non-portable and complex to operate, making them challenging to share. They also require substantial training, rendering them unsuitable for widespread consumer use. Furthermore, both conventional methods and systems are not ideal for children and individuals with disabilities.

[0005] Consequently, there is a need for an artificial-intelligence-based system and method for creating digital arts and stories. This system and method should be user- friendly for all age groups and accessible to users with disabilities, thereby enabling everyone to create and enjoy digital arts and stories.SUMMARY OF THE INVENTION

[0006] Broadly, the present disclosure is directed to the creation of digital stories and artworks. Systems and methods of this disclosure are directed towards content creation, including digital picture and story generation and / or digital picture and story editing, including collaborative content creation using an artificial-intelligence-based system.

[0007] In one embodiment, a user accesses initial content, such as a digital story or picture, using a computing device hosting a digital story application. The initial content includes one or more pieces of content, where each piece includes one or more of textual, visual, and / or audio aspects. Examples of these aspects include, without limitation, text, drawings, graphs, graphics, images, artwork, charts, photographs, animations, videos, streaming video, audio, streaming music, musical tracks, musical scales, dialogue, and sound effects. The digital story application can enable a user to advance one piece of content at a time (e.g., page by page) by making selections. Resulting digital stories can be shared and accessed by viewers different (e g., remote) from the user (or users) who generated the digital story. A “user” as defined herein refers to any type of user who may interact with the disclosed system and methods at any level or role. In one embodiment, a user may act in a maker role (designer) to create one or more interlinked panels and one or more buttons. In one embodiment, the system allows the maker and / or user to be part of the panel (ecosystem) design process by including configuration notes, which are an initial set of prompts or instructions created, added, and / or modified by a user or maker of the system. In one embodiment, the configuration notes act as a preamble set of breadcrumbs to assist in setting the stage or scenario for the content to be created. For example, the configuration notes may include one or more details such as a keyword indicating the mood / feeling of the story or other content. In one embodiment, a user may act in a maker role (designer or builder) to design the panels and buttons as well as the flow of control and actions triggered by user actions or contextual changes. This allows a maker to create an interactive "ecosystem," which offers the user a self-contained experience where the user cancreate artworks and stories.

[0008] In one or more embodiments, one or more additional users (e.g., co-author or collaborators) may interact with the disclosed system and methods. In one embodiment, a user may act as a collaborator, which may be one or more person or other connected system and methods that shares in the effort to create an ecosystem for the user or one or more person or other system and methods that participates with the user in interacting with the ecosystem or shares in the experience of the resulting artworks or stories.

[0009] The present invention discloses an artificial-intelligence-based system and method for creating digital arts and stories. The digital story creation system comprises a user device associated with a user, a database configured to store information related to the user, and a computing device in communication with the database and the user device via a network. The computing device comprises at least one memory storing a set of program modules and at least one processor. Further, many of the functions of the system may utilize artificial intelligence and / or machine learning (AI / ML). AI / ML may be within the device or connected to the device over a network.

[0010] As one other example, the systems of the current invention may be combined with newer categories of Al such as what is known as generative Al or artificial general intelligence. Generative Al describes algorithms that can be used to create new content, including audio, code, images, text, simulations and videos. Examples of generative Al include programs such as ChatGPT by OpenAI (Generative Pre-trained Transformer), a chatbot that can generate an answer to almost any question it is asked, and DALL-E, a tool for Al-generated art. For example, such generative Al systems might be used to compose an audible sentence or picture to convey a story to the user.

[0011] As described, ChatGPT is an Al-powered chatbot that uses natural language processing and machine learning algorithms to understand and respond to user inputs.It is part of the GPT family of models, which are known for their ability to generate human-like language and understand the context and nuances of human conversation. ChatGPT can be integrated with various messaging platforms and can be used for a wide range of applications. It can answer questions, provide information, and engage in free-form conversation with users. Unlike rule-based chatbots, ChatGPT is capable of learning from previous interactions, improving its responses over time. It is also able to generate unique and creative responses, making it more effective in interacting with humans. To use ChatGPT, a user simply types in a question or input and the Al model responds with a natural language response. The responses are generated in real-time and are tailored to the specific context and needs of the user.

[0012] In an example for the present invention, to generate a response, a user inputs a prompt, and an Al-model uses its transformer model to generate a new sentence or phrase that is relevant to the input. An Al-model, e.g., ChatGPT, can also generate longer texts such as paragraphs or even entire articles or books. Some AIs, such as OpenAI’s DALL-E, can also generate computer-generated pictures or music. Such responses include the information as discussed to produce a comprehensive output, e.g., a story, response, or summary.

[0013] The computing device of the present invention is configured to create one or more interlinked panels and one or more buttons. Each panel comprises one or more choices. The computing device is configured to enable a user to navigate through the panels and buttons, and select one or more choices. The computing device is configured to enable the user to create a digital breadcrumb path to track every user selection and contextual change through a series of panels and buttons. The computing device is configured to send the digital breadcrumb path to an artificial intelligence (Al) model, and enable the Al model to create at least one of digital arts and stories based on the digital breadcrumb path. In one embodiment, the stories comprise interactive stories. The computing device is configured to display a panel comprising illustrations.

[0014] The computing device is further configured to automatically redirect the user to the story after a pause or interruption of the process by the user. The computing device is further configured to enable the maker to create or modify one or more interlinked panels and one or more buttons, which are used by the user to traverse the story-creating ecosystem. The computing device is further configured to read the created story for the user via a text-to-speech module. The computing device is further configured to enable the user to save the created digital arts and stories.

[0015] In one embodiment, the computing device is configured to enter events, either in real-time or subsequently, through input data. For example, a user may input or trigger an entry that the user has eaten a meal or completed a task. The input device is configured to receive data input of at least one of a context input data, a user input data, and a third-party input data. In one example, context input data includes a time of day, for example, 3 :00 pm which is related to the time of a child coming home from school. In another example, the context input data is entered for activities that occur during the day, e.g., meals and other activities. In another example, the context input data is from one or more sensors for physiological parameters, for example, heart rate, blood pressure, oxygen levels, glucose level, insulin levels. In yet another example, the context input data includes occurrences, for example, ringing of doorbell, person entering a room. The user input data includes input data from the user. In one embodiment, an input device is configured to detect volitional user input of the user. For example, the input device is configured to detect movement of a finger or extremity, which triggers a switch signal. The third-party input data includes input data from people other than the user, for example, doctors, assistants, caretakers, or other third parties, including through one or more input devices. The third-party input data can be by controller, sensors such as RFID, voice or facial recognition sensors, audio or visual input devices, e.g., cameras, microphones, or other user interface. Other sensors may be utilized in the system, such as a proximity sensor, photodetector, a Hall-effect sensor, a radio frequency identifier (RFID) sensor, a biomedical sensor (such as electromyography, a moisture sensor, a fluid sensor, a temperature sensor, anelectrodermal activity sensor, a chemical presence sensor, a biological presence sensor, sound sensor, vibration sensor, and a pH level sensor), and a force sensor that may sense a mechanical force such as a pressure sensor or a flex sensor. Biopotential sensors may include but are not limited to electromyography (EMG) sensors, ECG sensors, respiration, galvanic skin response (GSR), or others. Other types of sensors may also be incorporated into the devices described herein.

[0016] In one embodiment, a method for creating digital arts and stories is disclosed. The method is executed by the system of the present invention. The system comprises the user device associated with the user, the database configured to store information related to the user, and the computing device in communication with the database and the user device via the network. The computing device comprises at least one memory storing a set of program modules and at least one processor. The computing device comprises an artificial intelligence (Al) model. Such artificial intelligence (Al) model may be networked with the system and may be cloud-based.

[0017] At one step, the computing device is configured to create one or more interlinked panels and one or more buttons. Each panel comprises one or more choices. At another step, the computing device is configured to enable a user to navigate through the panels and buttons, and select one or more choices. At yet another step, the computing device is configured to enable the user to create a digital breadcrumb path to track every user selection and contextual change through a series of panels and buttons. At yet another step, the computing device is configured to send the digital breadcrumb path to an artificial intelligence (Al) model. At yet another step, the computing device is configured to enable the Al model to create at least one of digital arts and stories based on the digital breadcrumb path. In one embodiment, the stories comprise interactive stories. At yet another step, the computing device is configured to display a story and illustrations based on the digital breadcrumb path (selections) made by the user.

[0018] At yet another step, the computing device is further configured to automatically redirect the user to the story after a pause or interruption of the process by the user. At yet another step, the computing device is further configured to enable the user to create one or more interlinked panels and one or more buttons. At yet another step, the computing device is further configured to read the created story for the user via a text-to-speech module. At yet another step, the computing device is further configured to enable the user to save the created digital arts and stories.BRIEF DESCRIPTION OF THE DRAWINGS

[0018] FIG. 1 exemplarily illustrates an environment of an artificial intelligencebased system for creating digital arts and stories, according to an embodiment of the present invention.

[0019] FIG. 2 exemplarily illustrates a flowchart of a method for creating digital arts and stories, according to an embodiment of the present invention.

[0020] FIG. 3 exemplarily illustrates a screenshot of a user interface providing one or more options to create a desired type of digital arts and stories, according to an embodiment of the present invention.

[0021] FIG. 4 exemplarily illustrates a screenshot of a user interface for creating stories, according to an embodiment of the present invention.

[0022] FIG. 5 exemplarily illustrates a screenshot of a user interface that enables a user acting in a maker role to create or modify story-creating parameters through rules and configuration notes according to an embodiment of the present invention.

[0023] FIG. 6 exemplarily illustrates a screenshot of a user interface that enables the user to select one or more storytelling items, e.g., tools, to eventually become part of the digital breadcrumb path for the story, according to an embodiment of the present invention.

[0024] FIG. 7 exemplarily illustrates a screenshot of a user interface to enable a user acting in a maker role to create or modify breadcrumbs that are associated with particular panel buttons, according to an embodiment of the present invention.

[0025] FIG. 8 exemplarily illustrates a screenshot of a user interface displaying a calmenvironment, according to an embodiment of the present invention.

[0026] FIG. 9 exemplarily illustrates a screenshot of a user interface to create context and actions, according to an embodiment of the present invention.

[0027] FIG. 10 exemplarily illustrates screenshots of series of panels and buttons selected by the user to create a digital art, according to an embodiment of the present invention.

[0028] FIG. 11 exemplarily illustrates a screenshot of a user interface displaying the digital instruction set or sets created from the accumulated breadcrumbs based on the user’s actions, context, and configuration notes, according to another embodiment of the present invention.

[0029] FIG. 12 exemplarily illustrates a screenshot of an Al-created story from the accumulated breadcrumbs (from FIG. 11), based on the digital instruction set, according to another embodiment of the present invention.DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0030] Example embodiments of the disclosure now will be described more fully hereinafter with reference to the accompanying drawings, in which example embodiments are shown. The concepts discussed herein may, however, be embodied in many different forms and should not be construed as limited to the example embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope to those of ordinary skill in the art. Like numbers refer to like elements but not necessarily the same or identical elements throughout. It is noted that various connections between elements are discussed in the following description. These connections are general and, unless specified otherwise, may be direct or indirect, wired or wireless, and the specification is not intended to be limiting in this respect.

[0031] The inventors have contemplated that the subject matter of the present invention might also be embodied, in other ways, to include different steps or a different combination of steps similar to the ones described in this document, in conjunction with present and future technological advances. Similar symbols used in different illustrative figures identify similar components unless contextually stated otherwise. The terms herein, “steps,” “block,” and “flow” are used below, to explain different elements of the method employed, and should not be interpreted as implying any particular order among different steps unless any specific order is explicitly described for the embodiments of this invention.

[0032] The descriptors “module” and “unit” for components used in the description below are assigned or mixed in consideration of ease in writing the specification and do not have distinctive meanings or roles by themselves. In the following description, detailed descriptions of well-known functions or constructions will be omitted since they would obscure the invention in unnecessary detail. Additionally, the accompanying drawings are used to help easily understand embodiments disclosedherein but the technical idea of the present disclosure is not limited thereto. It should be understood that variations, equivalents, or substitutes contained in the concept and technical scope of the present disclosure are also included. It will be understood that the terms “first” and “second” are used herein to describe various components, but these components should not be limited by these terms. These terms are used only to distinguish one component from other components.

[0033] FIG. 1 exemplarily illustrates an environment 100 of an artificial-intelligence- based system for creating digital arts and stories, according to an embodiment of the present invention. The system can be used for education, training, children’s playtime, assistive technology (AT), and augmentative and alternative communication (AAC), among other realms, to create unique stories and artwork based on the user’s actions. A user is a person who generates original content and / or edits previously created content. As examples, user can be an original author, a fan of the subject and / or author, a caregiver, a parent, a sibling, etc. The system creates digital breadcrumbs to track each user selection and contextual change through a series of panels and buttons. These breadcrumbs are then sent to an artificial intelligence (Al) system to create the work of art and interactive stories. The system comprises a computing device 102 in communication with a database 104 via a network 106. The system further comprises a user device 108 in communication with the computing device 102 via the network 106. The computing device 102 further comprises an artificial intelligence (Al) model, which may be cloud-based.

[0034] In one embodiment, the computing device 102 is a server. The server can be any suitable server(s) for storing information, data, programs, and / or any other suitable content. For example, the server is at least one of a general or special purpose computer. The server operates as a single computer, which could be a hardware and / or software server, a workstation, a desktop, a laptop, a tablet, a mobile phone, a mainframe, a supercomputer, a server farm, and so forth. Although the server is illustrated as a single device, the functions performed by the server can be performed using any suitablenumber of computing devices 102.

[0035] The user device 108, for example, includes, but is not limited to, a general computing device, a tablet computing device, a smartphone, a wearable computing device (such as an augmented reality device or watch), a desktop computer, a laptop computer, a mobile phone, a personal digital assistant, a tablet computer, and / or any other suitable type of computer. The user device 108 is associated with a user. In one example, the database 104 resides in the server. In another example, the database 104 resides separately from the server. Regardless of its location, the database 104 comprises a memory to store and organize data for use by the server or the computing devices 102. In one or more embodiments, the database 104 comprises information related to the user.

[0036] The network 106 generally represents one or more interconnected networks, over which the computing device 102, the user device 108, and the database 104 can communicate with each other. The network 106 may be any type of wired or wireless network, for example, the Internet, cellular, an intranet, a wide area network (WAN), a local area network (LAN), a metropolitan area network (MAN), Bluetooth, Bluetooth Low Energy (BLE), enterprise private network (EPN), worldwide interoperability for microwave access (WiMax), and a virtual private network (VPN). A person skilled in the art will recognize that the network 106 may also be a combination of more than one type of network. For example, the network 106 may be a combination of a LAN and the Internet. Moreover, the network 106 may be implemented as a wired network, a wireless network, or a combination thereof.

[0037] The computing device 102 comprises at least one processor and at least one memory. The memory comprises a set of program modules executed by the processor. In another embodiment, the set of program modules is provided on different computing devices 102. The computing device 102 is configured to create one or more interlinked panels and one or more buttons. The one or more interlinked panels and one or morebuttons are configured to enable the user to navigate through the panels and buttons and to create a digital breadcrumb path to track each selection and contextual change. In one embodiment, the digital breadcrumb path includes the initial user selections and contextual changes and is used in conjunction with the configuration notes to create the synthesized instruction set or sets (query or prompt) to send to an artificial intelligence (Al) model.

[0038] In one embodiment, the digital breadcrumb path includes contextual parameters. Contextual parameters as used herein means parameters relating to the environment, surroundings, and location of the individual, including, but not limited to, date and time, ambient temperature and weather, global positioning, and the like. For example, traditionally, context includes information about a location and its surrounding environment, and the identity of things named in a text such as people, places, books, and so on.

[0039] The configuration notes are created according to an initial set of prompts created, added, and / or modified by a user or maker of the system. In one embodiment, the configuration notes may include one or more user profile parameters, which may include at least one prompt about the user including gender, age, geographic location, language of a character, education level, interests, likes, dislikes, scenarios, etc. Further, the configuration notes may include one or more details of objects, places, plants and animals, weather and environmental factors, events, timelines, etc.

[0040] In one embodiment, the configuration notes may include one or more attribute keyword. An attribute keyword is a keyword that can indicate the attribute of an element to be included in the story or story content, and may include a keyword indicating the mood / feeling of the story or story content, a keyword indicating the type of the medium / purpose, and the like. For example, attribute keywords may include keywords representing the type of media / purpose, and keywords that indicate mood / feeling may include funny, scary, important, light, stylish, fashionable, trendy,interesting, cute, funny, neat, and the like.

[0041] In one embodiment, the configuration notes include one or more attributes of the user or of other characters or elements to be integrated into the created story or story content. In one embodiment, the configuration notes include one or more character models. The term “character model” refers to a set of characteristics that define a character or element included in the story. For example, the characteristics may designate an age, gender / sex, race, ethnicity, nationality and other basic personality traits for an individual character. As one example, a first character model may be designated to correspond to a narrator of a story, where the narrator represents a person or character who is telling a story, filling in background information and / or bridging gaps between dialogue. As further examples, secondary characteristic models may be designated in different types of stories, books, textual content and the like. As one example, a character may be designated as a family member, friend, teacher, medical personnel, coworker, staff person and the like. As another example, personal preferences of the user may be incorporated, e.g., favorite animals, colors, memories, places, objects, etc.

[0042] In one embodiment, the configuration notes may include two or more versions that utilize different character models for a common character in the story based on an age or other demographic characteristic of the target customer / user. For example, a story may be converted into an adult-type story format where the characters are adults and the story is provided to customers / users who are adults. The same story may be converted into a child-type format where the characters are children and the story is provided to customers / users who are children.

[0043] In one embodiment, the configuration notes may indicate a particular character model to be well-suited for certain types of literary forms, such as autobiographies, biographies, comedies, dramas, essays, fantasies, historical fiction, historical nonfiction, novels, poetry, science fiction, short stories and the like. Optionally, thecharacter model may indicate that a particular character model should not be used with certain types of literary forms.

[0044] In one embodiment, the configuration notes may include one or more preferences, for example, based on intent (e.g., instructional, humorous), style of delivery (e.g., realistic versus make-believe), methods of delivery (text only, with black-and-white video), genre (drama, science fiction), and so on. The content to be created may also be marked with preferences according to identified objects, locations, individuals, words, languages, ethnic contexts, actions, music, sound effects, environment, and so on. Content may be further indicated with attributes according to cognitive and emotional reactions of an individual including surprise, amusement, fear, horror, anger, rage, disgust, annoyance, contempt, sadness, joy, confusion, interest, boredom, calmness, anxiety, anticipation, envy, sexual desire, love, friendship, and so on.

[0045] Optionally, the configuration notes may indicate certain narrative traits or attributes may be afforded different weights, with a weighted combination (e.g., weighted sum) utilized to select the character model. For example, it may be determined that gender and background are important factors, while nationality and / or race are unimportant factors. The weighted combination of the narrative factors may be utilized to identify the most suitable character model.

[0046] The computing device 102 is configured to send synthesized instructions (prompt or query) to an artificial intelligence (Al) model. In one embodiment, the synthesized instructions or query represents digital instructions or set of instructions that are sent to an artificial intelligence (Al) model. In one embodiment, the computing device 102 is configured to use the initial user selections and contextual changes to create the configuration notes, in order to send initial synthesized instructions or queries to an artificial intelligence (Al) model. In one embodiment, the configuration notes are crafted from the initial user selections and contextual changes according to the rulesentered into the system by a maker user. The maker creates one or more interlinked panels and one or more buttons according to the rules set forth in the configuration notes.

[0047] In another embodiment, the computing device 102 is designed to enable a user to act in a maker role (designer) to create one or more interlinked panels and one or more buttons. The user and designer may be the same person or may be different (such as a parent, caregiver, editor, co-worker, etc.), which allows the maker / user to be part of the process of creating interlinked panels and buttons. In one embodiment, the system allows the user to be part of the panel (ecosystem) design process, expressing pleasure or displeasure with the artwork chosen to represent buttons, and so on. The system is designed to enable an artificial intelligence (Al) model to create digital content based on the instruction set (query or prompt), crafted from the breadcrumb path, and return the created content back to the user.

[0048] The returned digital content from the artificial intelligence (Al) model includes one or more pieces of content, where each piece includes one or more of textual, visual, and / or audio aspects. Examples of these aspects include, without limitation, text, drawings, graphs, graphics, images, artwork, charts, photographs, animations, videos, streaming video, audio, streaming music, musical tracks, musical scales, dialogue, and sound effects. The digital story application can enable a user to advance one piece of content at a time (e.g., page by page), or the digital story application can coordinate presentation of each piece of content (e.g., page) in the initial configuration notes (Fig. 5). Digital story presentation in a viewer can include coordination of presentation of the textual, visual, audio and / or other aspects of each piece of content contained from the initial user selections and contextual changes. Resulting digital stories can be shared and accessed by viewers different (e.g., remotely) from the user (or users) who generated the digital story. In one or more embodiments, one or more remote users (collaborators) can take in the content using a digital story application or viewer and view, add to, or modify the content in a collaborative format.

[0049] In one embodiment, the content generated and / or edited in example environment 100 includes various types of written and / or pictorial communication. For instance, content can include literature that can be expressed in visual form. In various embodiments, content edited and generated by digital story environment 100 includes, for example and without limitation, graphic novels, books such as children's books and young adult books, comic books, plays, theater, video content, musical scores, musical content, medical content, and greeting cards. For discussion purposes only, content edited and / or generated by digital story environment 100 will be referred to herein as “a digital story” or “digital stories.” The use of “a digital story” or “digital stories” is not limited to a particular medium, but rather encompasses various forms of content, including music composition and performance, picture artwork, video and film generation, theater, medicine, and education curriculum and teaching.

[0050] In one embodiment, the story includes an interactive story. The computing device 102 is configured to allow the user to save the art and interactive story created by the user. Furthermore, the computing device 102 is equipped to read the created story for the user via a text-to-speech module and to enable the user to save the created digital arts and stories. In one embodiment, the computing device 102 is configured with a digital story application, which accesses a digital story by communicating with a server over network 106, where server is in communication with digital story database 104. In some implementations, the digital story application interfaces with another application such that digital story generation, editing, and / or viewing occurs within that other application. In one embodiment, the digital story application is a plug-in or addin for another application. In other instances, the digital story application is a standalone application or executable application file. In one embodiment, the digital story application can be stored at a third-party service such as the Apple App Store or Google Store and downloaded onto device 102. In another embodiment, the computing device 102 is designed to enable a user to act in a maker role (designer) to create one or more interlinked panels and one or more buttons. The user and maker may be the same personor one or more other persons (such as a parent, caregiver, editor, co-worker, etc.) may act as makers. It allows the maker / user to be part of the process of creating interlinked panels and buttons. Thus, the system allows the user to be part of the panel (ecosystem) design process, expressing pleasure or displeasure with the artwork chosen to represent buttons, and so on.

[0051] In one embodiment, a method to create interlinking panels and buttons is disclosed. The computing device 102 is designed to allow the user to traverse through a series of panels via user-initiated activations (e.g., button presses) and other contexts (e.g., a pre-set timeout of a particular panel) to a terminus panel. The terminus panel triggers the artificial intelligence model to create a story based on the sequence of choices and context switches. In one embodiment, the user is a panel maker. The computing device 102 is configured to enable the user to create panels and populate fields in each panel with "breadcrumbs," text that is added to a log of events that will form the basis of a call to the artificial intelligence (Al) model. Once the maker associates the breadcrumbs with actions, those breadcrumbs are collected as the user interacts with the panel set. At the terminus panel, the collection of text (breadcrumbs) is sent as a prompt to the artificial intelligence model.

[0052] In another embodiment, the computing device 102 is configured to enable the user acting in the role of designer (maker) to directly access the artificial intelligence model to directly generate graphics for the panels and buttons. In one embodiment, the computing device 102 is configured to display a panel comprising illustrations to help users relax if the user is tense or other “non-standard” user activations are recorded. For example, if a user of assistive technology (AT) or augmentative and alternative communication (AAC) has unintended switches as the result of fasciculation or an inability to relax after making the original switch, a panel may trigger a timeout that redirects the normal story flow. In one embodiment, the computing device 102 is configured to provide a panel that represents a quiet place of rest to help users relax. The computing device 102 is configured to reinforce good assistive technology (AT)switching techniques and help users to relax. Once the user is relaxed, the computing device 102 is configured to automatically take the user back to the main branch of the story.

[0053] FIG. 2 exemplarily illustrates a flowchart of a method 200 for creating digital arts and stories, according to an embodiment of the present invention. The method 200 is executed utilizing system 100 of the present invention. The system comprises the computing device 102 in communication with the database 104 via the network 106. The system further comprises the user device 108 in communication with the computing device 102 via the network 106. The computing device 102 further comprises the artificial intelligence (Al) model.

[0054] At step 202, the computing device 102 is configured to create one or more interlinked panels and one or more buttons. Each panel comprises one or more choices. At step 204, the computing device 102 is configured to enable a user to navigate through the panels and buttons, and select one or more choices. At step 206, the computing device 102 is configured to enable the user to create a digital breadcrumb path to track each user selection and contextual change through a series of panels and buttons. At step 208, the computing device 102 is configured to send the digital breadcrumb path to an artificial intelligence (Al) model. In another embodiment, the digital breadcrumb path includes all of the initial user selections and contextual changes, which are synthesized into the initial instructions or query to send to an artificial intelligence (Al) model based on the instructions of the configuration notes. At step 210, the computing device 102 is configured to enable the Al model to create at least one of digital arts and stories based on the digital breadcrumb path. In one embodiment, the stories comprise interactive stories.

[0055] In one embodiment, the computing device 102 is further configured to automatically redirect the user to the story after a pause or interruption by the user. The performance of the user (selections, actions, timeliness, etc.) can be reflected in thestoryline, such as by altering the next plot points. In one example, a lack of timely response may trigger a timeout of the story action. For example, the story could trigger a response of “You seem tired. Let’s take a nap!” The computing device 102 is further configured to enable the user to create one or more interlinked panels and one or more buttons. The computing device 102 is further configured to read the created story for the user via a text-to-speech module. The computing device 102 is further configured to enable the user to save the created digital arts and stories.

[0056] In one embodiment, the computing device 102 is configured with a contextual module, which enables contextual information functionalities. In an example embodiment, the contextual information may be certain information or data automatically provided by a system service, such as a location from a global positioning system service; a date from a calendar service; or a time from a clock service.

[0057] Contextual information can be used for a wide variety of applications and can generally be characterized as information that is relevant to an understanding of something. For example, traditionally, context includes information about a location and its surrounding environment, the identity of things named in a text such as people, places, books, and so on. However, other types of information promise to be of as much or even greater benefit for next-generation communication services. Using context information, computing device 102 may utilize an artificial intelligence (Al) model to search and provide content that is relevant to the context information. The application may receive contextual information when the user of the device is interacting with one or more specified applications, and retrieve content from the application being accessed when such application meets a set of criteria or predetermined context.

[0058] In one or more embodiments, the contextual application may be configured to monitor all applications running on an assistive technology system. When a user interaction occurs for any application running an assistive technology system, a contextual application may be notified. The contextual application may then determinewhether the application with which the interaction occurs is identified in a defined configuration. In some other example aspects, the contextual application may be configured to monitor only specific applications such as, for example, a Contacts application, for interaction by a user of an assistive technology system .

[0059] In one or more embodiments, the contextual application may be configured to monitor or detect applications for a particular event occurring in the assistive technology system. An event may be any event performed as a result of user interaction with an assistive technology system, or any event performed as a result of sensors such as RFID, GPS, facial recognition, voice recognition, time / date, etc. In some example embodiments, events may refer to events triggered by device sensors or system services, such as, for example, location services, accessibility helper services, or date and time services.

[0060] User interactions may refer to an event within the assistive technology system wherein the user accesses certain applications such that a window of a contacts list application is displayed in the user interface of the assistive technology system. Once context information is set and / or retrieved, the context information may be utilized by one or more contextual applications to retrieve content related to the context information from one or more applications. In one example embodiment, contextual applications may run in the background while another application is being accessed such that when a user accesses and interacts with an application, the contextual application may retrieve the context information without user intervention and / or unobtrusively from an application and utilize the retrieved context information to search for content from any application. In one or more embodiments, the present invention provides systems and methods for implementing an interaction model with the user in order to interact with data of any data source in a natural way. In some examples, the system receives natural language input from the user and processes the input using multiple technologies such as keyword mapping, fuzzy logic, contextsensitivity and historical search data.

[0061] In another embodiment, the computing device 102 is configured with a sound effect module, which enables sound effect recording and / or sound effect replacement functionalities. The sound effect module can include a data store of various types of sound effects usable by a user generating or editing a digital story. The sound effect module can display all of the original sound effects on a timeline and enable manipulation of those sound effects, specifically the timing and / or type of sound effect. The sound effect module also enables replacement of sound effect tracks on the timeline. The sound effect module can also enable recording of new sounds, for example, by interfacing with one or more microphone devices. Additionally, sound effect module can access and / or import media files from external locations to supplement available sound effect options. The sound effect module can also access and store recording and / or sound effects as part of the digital story database 104. The sound effect module can be accessed via the digital story application. The digital story application can access the network 106 and the digital story database 104 to access additional sound effects. Sound effects that are recorded using the sound effect module such as new sounds can be stored in the digital story database 104, another location on network 106, or an external location. By way of example, the sound effect module can be connected to a third-party service such as Soundsnap.com, a sound effect library. Sounds effects can be added to the sound effect module and then incorporated into the digital story.

[0062] In another embodiment, the computing device 102 is configured with a soundtrack module that enables management of digital music tracks. Example soundtrack management operations provided by soundtrack module include adding, moving, replacing, and / or deleting digital music files within a soundtrack timeline. The soundtrack module can interface with one or more external music services to access and / or play parts or all of various digital audio tracks. Example third party music streaming applications include, but are not limited to, Spotify, iTunes, Napster, Amazon, and Tidal. In some instances, the soundtrack module enables recording froma phone, computing device, tablet, or other musical recording device, such that the user can record their own musical track and use it during playback of the digital story.

[0063] FIG. 3 exemplarily illustrates a screenshot 300 of a user interface providing one or more options to create a desired type of digital arts and stories, according to an embodiment of the present invention. The user interface provides options to create a painting, create at least two types of themed cartoons, and create an adventure story.

[0064] FIG. 4 exemplarily illustrates a screenshot 400 of a user interface for creating stories, according to an embodiment of the present invention. The panels include a start panel. The Start Panel is used to reset the breadcrumbs log. The panel maker or the user is enabled to create a series of panels, link them according to user-initiated actions, and other contexts, and add breadcrumb text that describes the meaning of each action or context switch.

[0065] FIG. 5 exemplarily illustrates a screenshot 500 of a maker interface for creating a story using a description, according to an embodiment of the present invention. The maker interface enables the maker to input configuration notes that are used as the starting breadcrumbs, which can be used in creating the story. The artificial intelligence model is configured to create the story corresponding to the description in the digital breadcrumb path. The digital breadcrumb path includes all of the initial user selections and contextual changes, which are synthesized into the instructions or queries to send to an artificial intelligence (Al) model based on the particular instructions of the configuration notes.

[0066] FIG. 6 exemplarily illustrates a screenshot 600 of a user interface that enables the user to select one or more tools, according to an embodiment of the present invention. The selectable tools represent choices or options for the user to direct the story in a particular direction. In one embodiment, the user interface enables a user to choose one or more pathways or decisions for an adventure or an interactive story.

[0067] FIG. 7 exemplarily illustrates a screenshot 700 of a designer user (maker) interface that enables a maker to create button actions, according to an embodiment of the present invention. The meaning and actions of various keystrokes or inputs, which are selectable by the user, are added to the breadcrumb fields associated with the buttons, such that a specific button press (in this case, A through D) will insert that breadcrumb into the accumulated breadcrumb path being dynamically built. The maker can prioritize actions in the configuration notes such that when sent to an artificial intelligence (Al) model, certain aspects are prioritized in the content creation.

[0068] FIG. 8 exemplarily illustrates a screenshot 800 of a user interface displaying a calm environment, according to an embodiment of the present invention. The computing device 102 is configured to enable other “non-standard” user activations to create a change in the storyline. In this case, the user is taken to a panel to rest (provide no activations) for a pre-set period. The breadcrumb in this event is “Time to relax. Quiet, so he doesn't hear us.” After the timeout, the user can be sent to any part of the story (any panel) the maker chooses.

[0069] FIG. 9 exemplarily illustrates a screenshot 900 of a user interface for creating contextual prompts and actions, according to an embodiment of the present invention. The user interface allows a maker to configure various contextual prompts and actions to provide instructional data to set the context of the story or content. This would include cases where there is a timeout (no activation was made within a preset period) or when a “non-standard switch activation” was received.

[0070] FIG. 10 exemplarily illustrates a screenshot 1000 of a series of panels selected by the user to create digital art, according to an embodiment of the present invention. The digital art could be created by creating an interlinking set of panels and buttons to allow users to choose picture styles, locations, and content. At step 1, a first panel displays choices to create the digital art and stories. The choices include, but are notlimited to, painting, cartoons, and stories. The user selects paintings from the displayed choices. At step 2, the panel displays different types of paintings including, but not limited to, impressionist, oil, cubist, and watercolor paintings. The user chooses watercolor painting. At step 3, the panel displays locations for the setting for creating the painting. The user selects a beach environment setting. At step 4, the user selects an object, a boat. At step 5, the user selects another object, pelicans. At step 6, a blank easel is displayed. Furthermore, step 6 indicates that the options selected by the user are sent to the artificial intelligence model. At step 7, the computing device 102 is configured to create and display an artwork based on the user’s selections.

[0071] In another example, the computing device 102 is configured to provide a T- Rex-based story, and the goal is for the user to outrun the T-Rex by making timely switch activations. Each activation invokes a new panel with a more distant T-Rex. If an activation is not made before a pre-set timeout, the previous panel is invoked, bringing the T-Rex closer. The size of the T-rex is represented by numbers for explanatory purposes. However, the computing device 102 is configured to graphically display the T-rex of different sizes to the user. The higher the number, the bigger the T- rex appears (looks closer to the user) as rendered by the computing device 102.

[0072] If a panel graphic of the T-rex is of size 4, the breadcrumb-timely activation is “The roars are loud. The T-Rex is still close. Don't lose strength. Keep running.” and the breadcrumb-timeout is “Oh no, he's catching up, and he's really close now!”. If a panel graphic of the T-rex is of size 3, the breadcrumb-timely activation is “Good, we've put some distance between us and the T-Rex.” and the breadcrumb-timeout is “The T- Rex is getting closer. We must run faster!”

[0073] If a panel graphic of the T-rex is of size 2, the breadcrumb-timely activation is “Yay, we have found our friend, the Brontosaurus! The treasure and escaping the T-Rex are only one step away!” and the breadcrumb-timeout is “We have a lead, but he has caught up a little just now.” If a panel graphic of the T-rex is of size 1, the breadcrumb-timely activation is “We did it! We escaped the T-Rex and found the treasure!” and the breadcrumb-timeout is “Oh, we were doing so well, but we are getting tired.”

[0074] In another example, the system enables the user to create a story, and the method of creating the story is disclosed as follows. The system enables the user to navigate through the panels and buttons. The actions performed by the user through the panels and buttons, and the corresponding breadcrumbs of the actions, are disclosed as follows. For example, the user initially starts the “T-rex Island" adventure, and the corresponding breadcrumb includes “Please create a 500-word, fun story featuring a nine-year-old named James and his Australian Shepherd named Gracie based on the following sequence of events.” At another step, the user chooses a smoke bomb from possible options: rope, pickaxe, smoke bomb, and wagon. The breadcrumb corresponding to this action includes “They brought a smoke bomb with them for the adventure”.

[0075] At yet another step, the user makes a normal activation (a button press). The breadcrumb corresponding to this action includes “The T-Rex is close, so close. We need to start running away.” At yet another step, the user makes a normal activation. The breadcrumb corresponding to this action includes “The roars are loud. T-Rex is still close. Don't lose strength. Keep running.” At yet another step, the user makes an out-of-the-ordinary activation, rerouting the flow of the story to a period of rest. The breadcrumb corresponding to this action includes “Time to relax. Quiet, so he doesn't hear us ” At yet another step, the user must rest without activations for a pre-set period. For this step, no breadcrumb is applicable. At yet another step, the user makes a normal activation. The breadcrumb corresponding to this action includes “Good, we've put some distance between us and the T-Rex.” At yet another step, the user doesn't make an activation before a pre-set timeout. The breadcrumb corresponding to this action includes “We have a lead, but he has caught up a little just now.” In this event, the story flow is routed to the previous panel.

[0076] At yet another step, the user makes a normal activation. The breadcrumb corresponding to this action includes “Good, we've put some distance between us and the T-Rex.” At yet another step, the user makes a normal activation. The breadcrumb corresponding to this action includes “Yay, we have found our friend, the Brontosaurus. The treasure and escaping the T-Rex are only one step away!”. At yet another step, the user makes a normal activation. The breadcrumb corresponding to this action includes “We did it! We escaped the T-Rex and found the treasure!”. At this transition, the computing device 102 indicates that this is the adventure's terminus. The instruction set includes the accumulated breadcrumbs, which are transmitted to the Al model for the creation of a story. FIG. 11 exemplarily illustrates a screenshot 1100 of a user interface displaying the accumulated breadcrumbs, according to another embodiment of the present invention. Based on the accumulated breadcrumbs, the computing device 102 receives and displays the Al-created story. FIG. 12 exemplarily illustrates a screenshot 1200 of a user interface displaying the Al-created story. The quality of the story produced by the systems and methods of the present invention is dependent on the quality of the breadcrumbs that form the instructions or prompts for the Al model to generate the story. The computing device 102 is configured to enable the user to save the stories for later use or to immediately read them out loud to the user through a text- to-speech function.

[0077] Advantageously, the system could be used for education, training, children's playtime, assistive technology (AT), and augmentative and alternative communication (AAC), among other uses, to create unique stories and artwork based on the user's actions / selections onscreen (such as while playing an interactive game). In the case of assistive technology (AT), the system offers users with significant motor deficits, such as those challenged by the sometimes-uncontrollable movements triggered by cerebral palsy (CP), a safe and productive platform to create art and stories using their preferred access method.

[0078] In some embodiments, the digital story generation system includes one ormore software modules programmed with computer readable electronic instructions to perform one or more operations associated with a generating a digital story. The digital story generation system can operate a system for receiving, storing, editing, collaborating, mixing, displaying, modifying, publishing, and sending digital fdes. In some embodiments, the digital story generation system includes a collaboration system that operates a system for a collaborator or more than one collaborator to edit a digital story. The system can provide an interface for accessing digital fdes, editing the fdes, combining fdes, and publishing fdes.

[0079] In some embodiments, the system allows for publishing. Once the user opts to make the content available for publication, the system can publish the content. Publishing can be made to (a) a select group of viewers (for example, friends, colleagues, caregivers, relatives, etc.) who have access rights to see content, (b) to the public via a digital story application, (c) print-on-demand services, or (d) via other means that can show the content.

[0080] In another embodiment, advertisements may be displayed in connection with the digital stories. Ads may be display ads that are placed in close proximity to the digital stories. Other ads may be placed within the digital stories. Advertisers may bid on priority to be placed with digital stories. Advertisers may use geographic or other demographics to place ads. The digital story application can also be linked to a third- party advertising platform, such as Google AdWords.

[0081] Although the features, functions, components, and parts have been described herein in accordance with the teachings of the present disclosure, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all embodiments of the teachings of the disclosure that fairly fall within the scope of permissible equivalents.

[0082] Many modifications and other implementations of the disclosure set forthherein will be apparent having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the disclosure is not to be limited to the specific implementations disclosed and that modifications and other implementations are intended to be included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

CLAIMSWhat is claimed is:

1. An artificial-intelligence-based system for creating digital content, comprising: a user device associated with a user; a database configured to store information related to the user, and a computing device in communication with the database and the user device via a network, wherein the computing device comprises at least one memory storing a set of program modules and at least one processor, wherein the computing device comprises an artificial intelligence (Al) model, wherein the computing device is configured to: create one or more interlinked panels and one or more buttons, wherein each panel comprises one or more choices; enable a user to navigate through the panels and buttons, and select one or more choices; create a digital breadcrumb path to track every user selection and contextual change through a series of panels and buttons; send the digital breadcrumb path to the artificial intelligence (Al) model; and enable the Al model to create at least one of digital content based on the digital breadcrumb path.

2. The system of claim 1 , wherein the digital content includes at least one of digital arts and stories.

3. The system of claim 2, wherein the stories include interactive stories.

4. The system of claim 1, wherein the computing device is configured to display alternate panel illustrations upon input of non-standard user activations.

5. The system of claim 1, wherein the computing device is configured to automatically redirect the user to the content creation after a pause or interruption of the process.

6. The system of claim 1, wherein the computing device is configured to enable the user to create one or more interlinked panels and one or more buttons.

7. The system of claim 1, wherein the computing device is configured to read a created story for the user via a text-to-speech module.

8. The system of claim 1, wherein the computing device is configured to enable the user to include configuration notes as a preamble set of breadcrumbs to assist in setting the stage or scenario for the content to be created.

9. The system of claim 1, wherein the computing device is configured to enable triggering of the flow of control and actions by user actions or contextual input.

10. The system of claim 1, wherein the computing device is configured to enable the user to save the created digital content.

11. A method for creating digital content, comprising: providing a user device associated with a user, a database configured to store information related to the user, and a computing device in communication with the database and the user device via a network, wherein the computing device comprises at least one memory storing a set of program modules and at least one processor, wherein the computing device is incorporated with an artificial intelligence model; creating, at the computing device, one or more interlinked panels and one or more buttons, wherein each panel comprises one or more choices; enabling, at the computing device, a user to navigate through the panels andbuttons, and select one or more choices; creating, at the computing device, a digital breadcrumb path to track every user selection and contextual change through a series of panels and buttons; sending the digital breadcrumb path to an artificial intelligence (Al) model, and enabling, at the computing device, the Al model to create digital content based on the digital breadcrumb path.

12. The method of claim 11, wherein the digital content includes at least one of digital arts and stories.

13. The method of claim 11, wherein the stories include interactive stories.

14. The method of claim 11, further comprises a step of: displaying, at the user device, a change in the panel illustrations upon input of non-standard user activations.

15. The method of claim 11, further comprises a step of: automatically redirecting, at the computing device, the user to the content creation after a pause or interruption by the user.

16. The method of claim 11, further comprises a step of: enabling, at the computing device, the user to create one or more interlinked panels and one or more buttons.

17. The method of claim 11, further comprises a step of: enabling, at the computing device, the user to include configuration notes as a preamble set of breadcrumbs to assist in setting the stage or scenario for the content to be created.

18. The method of claim 11, further comprises a step of: enabling, at the computing device, the triggering of the flow of control and actions by user actions or contextual input.

19. The method of claim 11, further comprises a step of reading, via the computing device, the created digital content for the user via a text-to-speech module.

20. The method of claim 11, further comprises a step of: enabling, at the computing device, the user to save the created digital content.

Images