System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting a synopsis by a user; means for automatically generating text and illustrations from the synopsis using a generative AI model; means for combining the automatically generated text and illustrations to create a picture book layout; and means for transmitting the picture book layout for printing and mailing.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Curious children need many picture books, but it is difficult to gather a large number of picture books using conventional methods. In addition, commercially available picture books often do not fully meet the interests and preferences of individual children. Furthermore, it is difficult to include the child or a specific character in the content of the picture book, and there is no easy way to create an original, personalized picture book. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for a user to input a synopsis, a means for automatically generating text and illustrations from the synopsis using a generative AI model, a means for combining the automatically generated text and illustrations to create a picture book layout, and a means for transmitting the picture book layout data for printing and mailing. Furthermore, by further including a means for having the user and specific characters appear in the text and illustrations, and a means for creating a series of generated picture books to create multiple picture books, it becomes possible to easily create a large number of personalized original picture books.

[0006] "User" refers to an individual who uses the system to input a picture book synopsis and create an original picture book.

[0007] A "plot" is a brief summary of the story of a picture book that the user provides to the system, and is the information that serves as the basis for the generation AI to generate text and illustrations.

[0008] A "generative AI model" refers to an artificial intelligence program that automatically generates text and illustrations from an input synopsis.

[0009] "Sentences" refers to a set of strings of characters and sentences that are automatically generated from the plot by a generative AI model and are used as the story of a picture book.

[0010] "Illustrations" refer to images and paintings generated by generative AI models and used as visual elements in picture books.

[0011] "Layout" refers to the combination of generated text and illustrations arranged in a printable format.

[0012] "Means for sending for printing and mailing" refers to the means for sending the layout data of the completed picture book to a printing service and carrying out the procedure for mailing the picture book to the user's address.

[0013] "Characters" refers to people, animals, fictional beings, etc. that appear in picture books, and can include the user themselves or specific characters.

[0014] "Serialization" refers to the process of creating multiple related picture books based on the worldview and characters of a generated picture book. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] To solve the problem that the invention aims to solve, the present invention provides a system that allows users to easily create picture books using generative AI. This system is executed through the following processing steps. Each processing step is mainly executed by a terminal, a server, and a user.

[0037] Getting User Input

[0038] The terminal provides the user with an interface for inputting the storyline of the picture book. The user inputs the storyline and character information into this interface. The terminal acquires this input information and sends it to the server.

[0039] Generative AI generates text and illustrations

[0040] The server receives the synopsis from the user and uses a generation AI to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI generates a story for each page based on the synopsis, and the illustration generation AI then creates appropriate illustrations based on the generated text.

[0041] Picture book layout generation

[0042] The server combines the generated text and illustrations and creates a printable layout, including a title page, body pages, and other decorative elements. If desired, the user can include specific characters (e.g., the child himself).

[0043] Printing and mailing arrangements

[0044] The server sends the layout data of the completed picture book to the printing service, which then carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[0045] Get feedback

[0046] The device asks the user for feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts in a feedback form. This feedback is sent to the server and used to improve the system and the generation AI.

[0047] Specific examples

[0048] For example, if a user inputs a story about an animal's adventure, the server can receive the story and use the AI ​​to generate specific sentences and illustrations like the one below.

[0049] 1. Sentence Generation: "One day, Peter the rabbit went out in search of a hidden treasure deep in the forest."

[0050] 2. Illustration generation: A picture of Peter walking through the forest with a backpack on.

[0051] The generated text and illustrations are arranged on the title page and body pages using layout generation, and then sent to a printing service. The book is then mailed to the user's address, and the user receives their original picture book. If the user wishes to create another picture book, they can also create a sequel as a series.

[0052] In this way, the present invention realizes a system that allows users to easily create original picture books and provide entertainment for children.

[0053] The processing flow will be explained below.

[0054] Step 1: User enters synopsis

[0055] Subject: User

[0056] Specific behavior:

[0057] The user inputs the story synopsis through the system interface, which is a text input form where the user writes a brief summary of the story and information about the characters.

[0058] Subject: Device

[0059] Specific behavior:

[0060] The terminal collects the synopsis entered by the user and transmits it to the server.

[0061] Step 2: Generating text and illustrations using generative AI

[0062] Subject: Server

[0063] Specific behavior:

[0064] The server receives the synopsis sent by the user, then uses a text generation AI model to automatically generate text for each page of the picture book based on the synopsis, and then uses an illustration generation AI model to automatically generate illustrations that match the generated text.

[0065] Examples:

[0066] For example, if the plot is "Animals on an adventure searching for hidden treasure in the forest," the text generation AI will generate the sentence "Peter the rabbit went searching for hidden treasure deep in the forest." Next, the illustration generation AI will generate an illustration depicting a scene of Peter walking through the forest.

[0067] Step 3: Creating a picture book layout

[0068] Subject: Server

[0069] Specific behavior:

[0070] The server combines the generated text and illustrations to create a printable picture book layout, including a title page, body pages, and other decorative elements, and can even include specific characters (such as the child) if the user so desires.

[0071] Examples:

[0072] The generated text, "Peter the Rabbit Searches for Treasure," is then integrated with the illustrations and placed appropriately on each page of the picture book. Backgrounds and headings are also added to the pages.

[0073] Step 4: Arrange for printing and mailing

[0074] Subject: Server

[0075] Specific behavior:

[0076] The server sends the completed picture book layout data to the printing service and carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[0077] Examples:

[0078] The completed layout is sent to a printing company, which prints the picture book and delivers it to an address specified by the user.

[0079] Step 5:

[0080] Get feedback

[0081] Subject: Device

[0082] Specific behavior:

[0083] The device sends the user a notification requesting feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts in a feedback form.

[0084] Subject: User

[0085] Specific behavior:

[0086] Users receive a notification, fill out a feedback form with their thoughts and opinions about the picture book, and send it to the system via their terminal.

[0087] Subject: Server

[0088] Specific behavior:

[0089] The server receives feedback sent by users and analyzes it to improve the quality of the system and the generated AI.

[0090] These steps allow users to easily create and enjoy original picture books, and the system is continually improved through feedback.

[0091] Example 1

[0092] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0093] The traditional picture book creation process is time-consuming and laborious, requiring specialized knowledge, making it difficult for ordinary users to easily create original picture books. Another issue is that it is difficult to improve the system based on user feedback.

[0094] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0095] In this invention, the server includes means for a user to input a synopsis, means for automatically generating text and illustrations from the synopsis using a generative AI model, means for creating a picture book layout by combining the automatically generated text and illustrations, means for transmitting the picture book layout data for printing and mailing, and means for obtaining feedback from the user regarding the generated picture book. This enables even general users to easily create original picture books, and further enables continuous improvement of the system based on the obtained feedback.

[0096] "User" refers to an individual or group that uses the picture book creation system to create a picture book based on their own storyline.

[0097] A "synopsis" is a short story description entered by the user as an overview of the picture book's contents.

[0098] A "generative AI model" is an artificial intelligence model that automatically generates text and illustrations from a synopsis entered by the user.

[0099] "Text generation AI" is the part of the generative AI model that generates detailed text based on the synopsis entered by the user.

[0100] "Illustration generation AI" is the part of the generative AI model that depicts appropriate images based on the generated text.

[0101] "Layout" refers to the design that combines the generated text and illustrations and arranges them in a printable format.

[0102] A "printing service" is a company or system that provides a service of printing actual picture books based on layout data sent from a server and mailing them to users.

[0103] "Feedback" refers to the impressions and suggestions for improvement provided by users about the picture books they receive, and is data for future system improvements.

[0104] The "system" refers to the entire set of components that execute a series of processes, including user input, generation of text and illustrations using a generative AI model, creation of layout, arrangement of printing and mailing, and obtaining feedback.

[0105] "Character" means a person, animal, or fictional being that appears in the synopsis entered by the user and in the generated text and illustrations.

[0106] "Serialization" is the process of creating multiple picture books in succession as sequels to a previous picture book.

[0107] MODE FOR CARRYING OUT THE INVENTION

[0108] The present invention relates to a system that allows users to easily create original picture books. The system is implemented through the following specific steps. Each step is mainly performed by a terminal, a server, and a user.

[0109] 1. Getting User Input

[0110] The device provides the user with an interface for entering information about the storyline and characters in the book. This interface includes input fields, drop-down menus, character selection options, etc. The user enters the appropriate information into these fields and clicks the "Submit" button. The device then packages the entered information as JSON-formatted data and sends it to the server via a secure HTTP request.

[0111] 2. Generating text and illustrations using generative AI

[0112] The server processes the synopsis information sent from the device and passes it to the generative AI model. The generative AI model includes a text generation AI and an illustration generation AI. The text generation AI generates a detailed story based on the synopsis, and the illustration generation AI draws appropriate illustrations based on the content of the generated text. At this time, a prompt sentence is input into the generative AI model to generate the necessary data.

[0113] Specific examples

[0114] The server generates the following prompt: "The characters are Peter the rabbit, Phil the fox, and Oliver the owl. Describe their adventure in search of a hidden treasure deep in the forest. The scene begins with Peter putting on his backpack and setting off."

[0115] This prompt is fed into the AI ​​model, which generates the following text and illustration:

[0116] 1. Sentence Generation: "One day, Peter the rabbit went out in search of a hidden treasure deep in the forest."

[0117] 2. Illustration generation: A picture of Peter walking through the forest with a backpack on.

[0118] 3. Picture book layout generation

[0119] The server combines the generated text and illustrations and lays them out in a printable format, taking into account design elements such as pagination, font settings, and illustration positioning. A PDF file is generated using layout editing software (e.g., Adobe InDesign) or a custom layout engine.

[0120] 4. Printing and mailing arrangements

[0121] The server sends the completed picture book layout data to a partner printing service. The printing service prints the picture book based on the received PDF file and arranges for it to be mailed to the user's specified address. Information such as the tracking number is returned to the server and notified to the user.

[0122] 5. Get feedback

[0123] After the picture book arrives, the device displays a notification requesting feedback. The user enters their thoughts and suggestions for improvement in the feedback form on the device. The entered feedback is sent back to the server and used to improve the system in the future.

[0124] Specific actions

[0125] After the user finishes reading the picture book, the device displays a notification saying, "Please tell us what you thought of the picture book." The user fills in the feedback form, saying, "The illustrations were great, but the story was a little short." After submitting the form, the feedback is saved on the server.

[0126] In this way, the present invention allows users to easily create and receive original picture books. Furthermore, the feedback function allows for continuous improvement of the system. The entire system is linked by users, terminals, servers, and printing services, enabling smooth picture book creation and distribution.

[0127] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0128] Step 1: Getting User Input

[0129] The device provides the user with an interface for inputting the storyline and character information of the picture book. The user fills in the storyline and character information in the input fields on this interface and clicks the send button.

[0130] Input: Synopsis text, character information (e.g. "A story about the adventures of animals" or "Peter the rabbit, Phil the fox, Oliver the owl")

[0131] Output: JSON format data (e.g., { "storybook": { "title": "Animal Adventures", "summary": "Animal Adventures", "characters": ["Peter the Rabbit", "Phil the Fox", "Oliver the Owl"]}}")

[0132] Step 2: Generating text and illustrations using generative AI

[0133] The server receives the plot information sent from the device and passes it to the generative AI model, which includes a text generation AI and an illustration generation AI, and uses these to generate the story and illustrations.

[0134] Input: Synopsis data in JSON format

[0135] Data processing: Input plot and character information into the generative AI model as prompts (e.g., "The characters are Peter the rabbit, Phil the fox, and Oliver the owl. Describe their adventure as they search for a hidden treasure deep in the forest. The story begins with Peter putting on his backpack and setting off.").

[0136] Output: Generated text and illustration (e.g., text "One day, Peter the rabbit went out to look for hidden treasure deep in the forest.", illustration "Peter is walking through the forest with his backpack on.")

[0137] Step 3: Creating a picture book layout

[0138] The server combines the generated text and illustrations and lays them out in a printable format, generating a PDF file while taking into account design elements such as pagination, font settings, and illustration positioning.

[0139] Input: Generated text and illustrations

[0140] Data processing: Create layouts using layout editing software (e.g. Adobe InDesign) or custom layout engines, and arrange pages, fonts, and illustrations.

[0141] Output: A printable PDF file (e.g. "storybook.pdf")

[0142] Step 4: Arrange for printing and mailing

[0143] The server sends the completed picture book layout data to a partner printing service, which prints the picture book based on the PDF file and mails it to the address specified by the user.

[0144] Input: Printable PDF file, user address information

[0145] Data processing: Send the PDF file to a printing service and provide shipping details

[0146] Output: Shipped notification and tracking information (e.g., "Shipped notification, tracking number 123456789")

[0147] Step 5: Getting feedback

[0148] After receiving the picture book, the device displays a notification requesting feedback. The user enters their thoughts and suggestions for improvement in the feedback form and submits it. The feedback information is stored on the server and used to improve the system.

[0149] Input: User impressions and suggestions for improvement entered in the feedback form (e.g., "The illustrations were great, but the story was a little short.")

[0150] Data calculation: Feedback information is sent to the server and stored in the database

[0151] Output: Saved feedback information (e.g., "{ "feedback": { "user_id": "001", "comments": "The illustrations were great, but the story was a little short"}}")

[0152] Through these steps, users can easily create and receive original picture books, and the feedback function allows for continuous improvement of the system.

[0153] (Application example 1)

[0154] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0155] Conventional picture book production systems lack the means for users to easily create original picture books and distribute them digitally, making it difficult to reach many users and distribute them instantly.

[0156] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0157] In this invention, the server includes means for a user to input a synopsis, means for automatically generating text and illustrations from the synopsis using a generative AI model, means for combining the automatically generated text and illustrations to create a picture book layout, means for saving the generated picture book in a digital format and sharing it with other users, and means for distributing it as an e-book in cooperation with an external content distribution service. This allows users to easily create original picture books and instantly distribute them in digital format to many users.

[0158] "User" refers to the person who enters the storyline and character information of the picture book.

[0159] A "generative AI model" refers to an artificial intelligence model used to automatically generate text and illustrations from a synopsis.

[0160] "Sentence" refers to text automatically generated by a generative AI model.

[0161] "Illustrations" refer to pictures or images automatically generated by generative AI models.

[0162] "Picture book layout" refers to the page composition that combines and arranges generated text and illustrations.

[0163] "Digital format" refers to a data format that can be stored and displayed electronically.

[0164] "External content distribution service" refers to a third-party online platform for publishing and selling the generated picture books as e-books.

[0165] "Sharing" refers to the act of sharing the generated picture book with other users.

[0166] This invention is a system that allows users to easily create original picture books and store and distribute them in digital form. A specific embodiment for realizing this system will be described below.

[0167] Getting User Input

[0168] The user inputs the plot and character information using their own device (e.g., a smartphone). The device provides an interface for the user to perform input operations. The device then transmits the input information to the server.

[0169] Generating text and illustrations using a generative AI model

[0170] The server uses a generative AI model to automatically generate text and illustrations based on the synopsis and character information received from the user. Specifically, the server uses the "generative AI model" to generate text and illustrations by inputting the following prompts:

[0171] Example prompt sentence:

[0172] Generate a picture book story with the following synopsis: A brave little cat has an adventure in a big city. Characters: Timmy the Cat

[0173] Picture book layout generation

[0174] The generated text and illustrations are integrated by the server to create a picture book layout, which is then stored in digital format.

[0175] Save and share digitally

[0176] The server stores the layout of the created picture book in a digital format and provides a means for sharing it with other users, allowing the user to share the created picture book with other users.

[0177] Collaboration with content distribution services

[0178] Furthermore, the server also provides a means to link with external content distribution services and distribute the created picture books as e-books, allowing users to instantly and widely distribute the picture books they have created.

[0179] System Configuration

[0180] Hardware: Smartphone (user input), Server (data processing and storage)

[0181] Software: Generative AI model (generating text and illustrations), interface app (user input)

[0182] Data processing and calculation:

[0183] The synopsis and character information entered by the user is obtained as text data.

[0184] Using a generative AI model, text and illustrations are generated based on input data.

[0185] Consolidate generated data and store and distribute it digitally.

[0186] Specific examples of program processing

[0187] For example, if a user inputs the synopsis "A brave little cat goes on an adventure in a big city," the generative AI model will generate the sentence "One day, Timmy, a brave little cat, heads to a big city and starts his adventure," along with an illustration of the cat on an adventure. The generated sentences and illustrations are then integrated into the layout of the picture book by the server and saved in digital format.

[0188] In this way, the present invention realizes a system that enables users to easily create original picture books and instantly distribute them in digital form.

[0189] ---

[0190] This explanation corresponds to the "Form for carrying out the invention."

[0191] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0192] Step 1:

[0193] The user uses a smartphone or PC to input the storyline and character information into the interface. The information entered by the user is saved as text data on the device. This text data is then sent to the server.

[0194] Step 2:

[0195] The server analyzes the received text data. Specifically, it creates a prompt sentence for the generative AI model based on the synopsis and character information entered by the user. The prompt sentence is input into the generative AI model, and the picture book text is automatically generated. The generated text is saved as text data within the server.

[0196] Step 3:

[0197] The server creates a prompt for generating illustrations based on the generated text. The prompt is input into an illustration generation AI model, which automatically generates illustrations for the picture book. The generated illustrations are saved as image data on the server.

[0198] Step 4:

[0199] The server integrates the generated text and illustrations and automatically creates a picture book layout. This layout arranges the text and illustrations on specific pages, forming a single picture book as a whole. The generated layout data is saved in digital format.

[0200] Step 5:

[0201] The server provides a means for users to share the digital data of the created picture book with other users. This means allows users to share the picture book they created with other users. This data is also linked to external content distribution services and distributed as an e-book.

[0202] Step 6:

[0203] The server also collects feedback from users. After a user finishes reading a picture book, they enter their impressions and opinions into a feedback form and send it to the server. This feedback data is used to improve the generative AI model and the overall quality of the system.

[0204] Through the above steps, the present invention allows users to easily create original picture books and share and distribute them in digital format.

[0205] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0206] This invention combines a system that allows users to easily create picture books using generative AI with an emotion engine that recognizes the user's emotions, thereby enabling the creation of more personalized picture books. This system is executed through the following processing steps. Each processing step is mainly executed by the terminal, server, and user.

[0207] Getting User Input

[0208] The terminal provides the user with an interface for inputting the storyline of the picture book. The user inputs the storyline and character information into this interface. The terminal acquires this input information and sends it to the server.

[0209] Optimization by Emotion Engine

[0210] Before receiving the synopsis sent by the user, the server uses an emotion engine to recognize the user's emotions. This allows the server to check whether the input synopsis and character settings are appropriate for the user's emotions. If necessary, the emotion engine suggests optimizing the synopsis and settings based on the user's emotions.

[0211] Generative AI generates text and illustrations

[0212] The server receives a synopsis optimized based on the user's emotions and uses a generation AI to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI generates a story for each page based on the synopsis, and the illustration generation AI then creates appropriate illustrations based on the generated text. The emotion engine then rechecks the generated text and illustrations and makes fine adjustments as necessary.

[0213] Picture book layout generation

[0214] The server combines the generated text and illustrations and creates a printable layout, including a title page, body pages, and other decorative elements. Additionally, the layout can include specific characters (e.g., the child himself) if desired.

[0215] Printing and mailing arrangements

[0216] The server sends the layout data of the completed picture book to the printing service, which then carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[0217] Get feedback

[0218] The device sends the user a notification requesting feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts in a feedback form. The feedback also includes the user's emotions, which the emotion engine analyzes and reflects in the next generation.

[0219] Specific examples

[0220] For example, suppose a user inputs "a story about animals' adventures" as the synopsis. The server first checks the user's emotions using the emotion engine and determines whether the input synopsis is appropriate for the user's current emotions. If the server determines that the user is feeling a little depressed, the emotion engine will suggest, "Why not change it to a fun adventure story?"

[0221] Based on the optimized synopsis, the generative AI model generates specific sentences and illustrations like the following:

[0222] 1. Sentence Generation: "One day, Peter the rabbit went out in search of a hidden treasure deep in the forest."

[0223] 2. Illustration generation: A picture of Peter walking through the forest with a backpack on.

[0224] The generated text and illustrations are arranged on the title page and body pages using layout generation, and then sent to a printing service. The book is then mailed to the user's address, and the user receives their original picture book. If the user wishes to create another picture book, they can also create a sequel as a series.

[0225] In this way, by combining emotion engines, we can create a system that provides even more personalized, original picture books that best suit the user's emotions.

[0226] The processing flow will be explained below.

[0227] Step 1: User enters synopsis

[0228] Subject: User

[0229] Specific behavior:

[0230] The user inputs the story synopsis through the system interface, which is a text input form where the user writes a brief summary of the story and information about the characters.

[0231] Subject: Device

[0232] Specific behavior:

[0233] The terminal collects the synopsis entered by the user and transmits it to the server.

[0234] Step 2: Optimizing with an Emotional Engine

[0235] Subject: Server

[0236] Specific behavior:

[0237] Before receiving the synopsis sent by the user, the server activates the emotion engine and collects data to recognize the user's emotions. If the device has a built-in camera or microphone, the emotion engine analyzes facial expressions and voice tones to check whether the synopsis entered by the user is appropriate for the user's emotions.

[0238] Examples:

[0239] For example, if a user inputs "forest adventure," the emotion engine will read emotions such as "fun" or "exciting" from the user's facial expressions and tone, and determine whether the synopsis is appropriate based on those emotions. If necessary, the emotion engine will make optimization suggestions such as "How about adding more specific adventure elements?"

[0240] Step 3: Generative AI generates text and illustrations

[0241] Subject: Server

[0242] Specific behavior:

[0243] The server receives the synopsis optimized based on the user's emotions and uses a generation AI to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI generates a story for each page based on the synopsis, and the illustration generation AI creates illustrations based on the generated text.

[0244] Specific behavior:

[0245] Based on the plot, the AI ​​generates the sentence "Peter the rabbit went in search of hidden treasure deep in the forest," and then the AI ​​generates an illustration of Peter carrying a backpack and walking through the forest.

[0246] Subject: Emotion Engine

[0247] Specific behavior:

[0248] The emotion engine rechecks the generated text and illustrations and fine-tunes them to better suit the user's emotions. For example, if the user is expressing a happy emotion, the engine will add humor to the text.

[0249] Step 4: Creating a layout for your picture book

[0250] Subject: Server

[0251] Specific behavior:

[0252] The server combines the generated text and illustrations to create a printable layout, including a title page, body pages, and other decorative elements, and can include specific characters (e.g., the child himself) if the user so desires.

[0253] Specific behavior:

[0254] The completed text and illustrations are arranged on each page, and decorations such as backgrounds and headings are added. If a specific character appears, an illustration of that character is also added.

[0255] Step 5: Arrange for printing and mailing

[0256] Subject: Server

[0257] Specific behavior:

[0258] The server sends the completed picture book layout data to the printing service and carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[0259] Specific behavior:

[0260] The printing service prints the layout data and delivers the picture book to the address specified by the user.

[0261] Step 6: Getting feedback

[0262] Subject: Device

[0263] Specific behavior:

[0264] The device sends the user a notification requesting feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts and opinions in a feedback form. This feedback includes emotions, which the system analyzes and reflects in the next generation.

[0265] Subject: User

[0266] Specific behavior:

[0267] Users receive a notification, fill out a feedback form with their thoughts and opinions about the picture book, and send it to the system via their terminal.

[0268] Subject: Server

[0269] Specific behavior:

[0270] The server analyzes the received feedback and uses it to generate the next picture book. The emotion engine also analyzes this feedback to understand changes in the user's emotions.

[0271] These steps allow users to easily create original picture books with emotionally appropriate content, and the system is continually improved through feedback.

[0272] Example 2

[0273] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0274] Conventional picture book creation systems automatically generate text and illustrations based on a synopsis entered by the user, but because they do not take into account the user's emotions, it is difficult to create a personalized picture book that meets the user's expectations. In addition, there is no way to check the text and illustrations generated based on the user's emotions and make fine adjustments as necessary.

[0275] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0276] In this invention, the server includes means for a user to input a synopsis, means for automatically generating text and illustrations from the synopsis using a generative AI model, means for combining the automatically generated text and illustrations to create a picture book layout, means for transmitting the picture book layout data for printing and mailing, means for recognizing the user's emotions and optimizing the synopsis and settings based on the user's emotions, and means for checking the generated text and illustrations based on the user's emotions and fine-tuning them as necessary, thereby enabling the creation of a personalized picture book optimized based on the user's emotions.

[0277] A "user" is an entity that uses the system to input the storyline and characters of a picture book.

[0278] A "generative AI model" is an artificial intelligence algorithm that automatically generates text and illustrations from an input synopsis.

[0279] A "synopsis" is a short sentence that describes the outline of the story of a picture book.

[0280] "Sentence" refers to text that is automatically generated by the generative AI model based on the plot.

[0281] "Illustrations" are images automatically generated by a generative AI model based on text generation.

[0282] "Layout" is a format for combining automatically generated text and illustrations to arrange them into a picture book.

[0283] "Printing and mailing" is the process of producing a physical picture book based on the completed picture book layout data and delivering it to a specified address.

[0284] The "emotion engine" is an algorithm that recognizes the user's emotions from the synopsis and character information entered by the user and suggests optimizations.

[0285] "Optimization" means adjusting the input plot and settings based on the user's emotions.

[0286] "Fine-tuning" means rechecking the generated text and illustrations and correcting the content if necessary.

[0287] This invention is a system that allows users to easily create personalized picture books by combining a generative AI model and an emotion engine. This system generates optimized text and illustrations that take emotions into consideration based on the plot and character information entered by the user, and then creates a picture book layout that combines them. A detailed explanation of the system is provided below.

[0288] This system is mainly run by a server, a terminal, and a user. The general flow of the system is as follows: user input, optimization by an emotion engine, generation of text and illustrations by generative AI, generation of picture book layout, printing and mailing, and obtaining feedback.

[0289] The device provides the user with an interface for inputting the storyline and character information of the picture book. The user inputs the storyline and character information into input fields displayed on the device, and the data is sent from the device to the server. This data can be sent in an appropriate data format such as JSON.

[0290] After receiving user input data from the device, the server launches the emotion engine and analyzes the data. The emotion engine recognizes the user's emotions using natural language processing libraries (e.g., NLTK, BERT, etc.). If the input content does not match the user's emotions, the emotion engine makes optimization suggestions to the server and generates optimization candidates. The server presents them to the user, and if the user approves, the optimized synopsis proceeds to the next process.

[0291] The server then receives the optimized synopsis and uses a generative AI model (e.g., GPT-3, DALL-E, etc.) to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI first creates a story for each page based on the synopsis, and then the illustration generation AI creates appropriate illustrations based on the generated text. The generated text and illustrations are then checked again by the emotion engine and fine-tuned as necessary.

[0292] The server uses the generated text and illustrations to create a layout for the book. This is done using a layout engine (e.g., LaTeX or an automated script in InDesign). The layout includes a title page, body pages, and other decorative elements. Additionally, the user can configure the layout to include specific characters of their choice.

[0293] The completed picture book layout data is converted to PDF format and sent to the printing service API. The printing service then prints the picture book based on that data and delivers it to the address specified by the user. At this time, the user is notified of a tracking number so that they can track the delivery status.

[0294] After receiving the picture book, the device sends a notification to the user requesting feedback. The user enters their impressions of the book into a feedback form, and the information is sent to the server. The feedback also includes the user's emotions, which the emotion engine analyzes and reflects in the next generation process.

[0295] Specific examples

[0296] For example, suppose a user inputs a story about animals' adventures as a synopsis. The server first checks the user's emotions using the emotion engine to determine whether the input synopsis is appropriate for the user's current emotions. If the server determines that the user is feeling a little depressed, the emotion engine will suggest changing it to a fun adventure story. Based on the optimized synopsis, the generative AI model will generate specific sentences and illustrations like the one below.

[0297] Sentence Generation: "One day, Peter the rabbit went out in search of hidden treasure deep in the forest."

[0298] Illustration generation: A picture of Peter walking through the forest with a backpack on.

[0299] The generated text and illustrations are arranged on the title page and body pages using layout generation, and then sent to a printing service. The book is then mailed to the user's address, and the user receives their original picture book. If the user wishes to create another picture book, they can also create a sequel as a series.

[0300] Prompt Sentence Examples

[0301] "Please provide a synopsis of a children's animal adventure story. Please also include information about the characters."

[0302] The present invention allows users to easily create and enjoy original picture books that are personalized to their own emotions.

[0303] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0304] Step 1:

[0305] The device provides the user with an interface for inputting the storyline of the picture book. The user uses this interface to input the storyline and character information. The input data is sent to the server in JSON format by the device.

[0306] Input: User-entered plot and character information

[0307] Output: Data sent to the server in JSON format

[0308] Specifically, the user enters "Synopsis: Animals go on a great adventure in the forest" and the characters "Peter the rabbit, Reina the fox, and Beatrice the bear" into the interface. The device converts this information into JSON data "{'summary': 'Animals go on a great adventure in the forest', 'characters': ['Peter', 'Reina', 'Beatrice']}" and sends it to the server.

[0309] Step 2:

[0310] The server analyzes the user input data received from the device. The server then activates an emotion engine to analyze this data and recognize the user's emotions. The emotion engine uses a natural language processing library to analyze the input text and output the results. If necessary, it generates optimization suggestions, which the server presents to the user.

[0311] Input: User-entered data sent from the terminal

[0312] Output: Emotion recognition results from the emotion engine and optimization suggestions as needed

[0313] Specifically, the server receives the data "{'summary': 'Animals have a great adventure in the forest', 'characters': ['Peter', 'Reina', 'Beatrice']}" and, if the server uses the emotion engine to recognize that the user's emotion is "depressed," it generates a suggestion such as "Why not change it to a more cheerful adventure story?"

[0314] Step 3:

[0315] The server presents the emotion engine's suggestions to the user, and the user approves or modifies the suggestions, resulting in optimized data that is then sent back to the server. Based on the optimized synopsis received by the server, the generative AI model is used to automatically generate the text and illustrations for the picture book. First, the text generation AI generates a story for each page, and then the illustration generation AI generates illustrations based on the story.

[0316] Input: User-approved or revised optimized synopsis

[0317] Output: Text and illustrations generated by a generative AI model

[0318] Specifically, the user approves the optimization proposal, and based on the optimized plot summary, "Animals have a fun adventure," the text generation AI generates text such as "One day, Peter the rabbit went out to search for hidden treasure deep in the forest." The illustration generation AI then uses that text to generate an illustration of Peter walking through the forest with a backpack on.

[0319] Step 4:

[0320] The server combines the generated text and illustrations to create a picture book layout. It uses a layout engine to automatically generate a layout including a title page, body pages, and decorative elements. The server then converts the completed layout into PDF format and sends it to the print service API.

[0321] Input: Text and illustrations generated by a generative AI model

[0322] Output: Layout data in PDF format sent to the print service API

[0323] Specifically, the server generates a title page called "Peter and the Forest Treasure" and a layout with text and illustrations on each page, and sends it in PDF format to the "Printing Service API."

[0324] Step 5:

[0325] The printing service receives the layout data in PDF format, prints the picture book, and delivers it to the address specified by the user. The user is then notified of the tracking number so that they can track the delivery status.

[0326] Input: PDF layout data sent from the server

[0327] Output: Printed book and tracking number notification to user

[0328] Specifically, the printing service prints the picture book based on the PDF and ships it to the user's address. The user can then check the delivery status using the tracking number provided.

[0329] Step 6:

[0330] The device sends a notification to the user requesting feedback on the picture book. After receiving and reading the book, the user enters their thoughts in a feedback form, and the information is sent to the server. The server analyzes the feedback using an emotion engine and reflects it in the next generation process.

[0331] Input: User feedback and emotional information

[0332] Output: The results of the emotion engine analysis and improvements to be reflected in the next generation process

[0333] Specifically, the device sends a notification to the user saying, "How did you like the picture book?", and the user enters feedback such as, "My child really enjoyed it. I'd like to add more animal species next time." The server analyzes this information and reflects it in the next generation process.

[0334] (Application example 2)

[0335] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0336] Conventional picture book generation systems lacked the ability to personalize the plot and character information entered by the user, making it particularly difficult to generate optimal content that reflected the user's emotions.Furthermore, there was no mechanism for incorporating feedback from the generated picture book into the next generation.

[0337] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a user to input a summary, a means for automatically generating text and images from the summary using a generative AI model, a means for creating a reading material layout by combining the automatically generated text and images, and a means for recognizing the user's emotions using an emotion engine and optimizing the summary and character settings. This makes it possible to generate personalized reading material that is optimized for the user's emotions.

[0338] A "user" is an entity that uses the system to generate reading material, for example, someone who inputs information via a smart device.

[0339] The "summary" includes a brief summary of the story and information about the characters entered by the user.

[0340] A "generative AI model" is an artificial intelligence program that automatically generates text and images based on a summary entered by the user.

[0341] The "emotion engine" is a program that recognizes the user's emotions and optimizes the story outline and character settings based on those emotions.

[0342] "Sentence" refers to the text portion of the story generated by the generative AI model.

[0343] "Imagery" refers to visual illustrations or pictures related to the narrative generated by the generative AI model.

[0344] "Reading material layout" refers to creating a page layout for a reading material by combining automatically generated text and images.

[0345] "Printing" refers to the process of producing a reading material in a physical form based on the layout data of the generated reading material.

[0346] "Mail" means delivery of printed materials to the address specified by the user.

[0347] "Personalization" refers to providing content that is optimized to the user's emotions and preferences.

[0348] An embodiment of the present invention provides a system that allows users to easily create and view personalized reading material through a picture book streaming app using a smartphone or head-mounted display (HMD).

[0349] System Overview:

[0350] Hardware configuration:

[0351] Smartphone

[0352] Head-mounted display (HMD)

[0353] server

[0354] Software configuration:

[0355] Emotion Engine

[0356] Generative AI Models

[0357] Web frameworks (e.g., Flask, Django)

[0358] REST API

[0359] Process flow:

[0360] 1. Getting user input:

[0361] Using an application on a device (smartphone or HMD), the user inputs a summary of the story, for example, "One day, Peter the rabbit went out to search for a hidden treasure deep in the forest."

[0362] Additionally, the user can input their current emotions, choosing from options such as "happy," "sad," or "angry."

[0363] 2. Emotional engine optimization:

[0364] The server uses an emotion engine to recognize the user's emotion, and then checks whether the summary entered by the user is appropriate for that emotion.

[0365] If necessary, the sentiment engine suggests optimizing the summary based on the user's sentiment.

[0366] 3. Generative AI model for generating text and images:

[0367] The server receives the summary optimized by the emotion engine, and the generative AI model generates narrative text and images based on the summary.

[0368] For example, a text generation AI generates the sentence, "One day, Peter the rabbit went out to search for hidden treasure deep in the forest," and an image generation AI generates an illustration depicting that scene.

[0369] 4. Generating reading material arrangements:

[0370] The server combines the generated text and images to create the page layout of the reading material, including the title page and body pages.

[0371] 5. Getting feedback:

[0372] The system asks users for feedback on the generated reading material, including their feelings, which the emotion engine analyzes and reflects in the next generation.

[0373] Example prompt sentence:

[0374] "Generate a fun adventure story about Peter the Rabbit's search for hidden treasure."

[0375] This invention is a system that combines an emotion engine and a generative AI model to provide highly personalized reading material to users. By linking smart devices with a server, content can be generated based on the user's emotions, and further optimized content can be provided through feedback.

[0376] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0377] Step 1:

[0378] The device provides an interface for the user to input a story summary and their current emotions. Using this interface, the user can input a summary such as "One day, Peter the rabbit went out to search for hidden treasure deep in the forest" and the emotion "fun." This input information is sent to the server. The input is the user's story summary and emotions, and the output is the user's input data sent to the server.

[0379] Step 2:

[0380] The server sends the received user summary and emotion to the emotion engine. The emotion engine checks the suitability of the input summary based on the user's emotion and optimizes it if necessary. For example, if the user inputs "fun," the emotion engine suggests a title and character settings that increase the fun factor. In this step, the input is the user's summary and emotion, and the output is an optimized summary suggestion.

[0381] Step 3:

[0382] The server receives the summary optimized by the emotion engine and sends it to the generative AI model. The generative AI model generates story text and images based on this optimized summary. For example, the text generation part generates the sentence "One day, Peter the rabbit went out to search for hidden treasure deep in the forest," and the image generation part generates an illustration depicting that scene. The input is the optimized summary, and the output is the generated text and images.

[0383] Step 4:

[0384] The server combines the generated text and images to automatically create page layouts for the reading material, including title and body pages. For example, the title page displays the title "The Adventures of Peter Rabbit," and the body pages combine the generated text with corresponding illustrations. The input is the generated text and images, and the output is the page layout.

[0385] Step 5:

[0386] The server provides the layout data of the completed book to the user through the streaming app. If the user wishes, it can also send it to a printing service and mail the physical book to the user's address. In this step, the input is the layout data of the book, and the output is the provision of the book in streaming format and arrangement of a printing service.

[0387] Step 6:

[0388] The device sends a notification to the user requesting feedback on the generated reading material. After reading the reading material, the user enters their impressions in a feedback form. This feedback is sent to the server and reflected in the next generation. The input is the user's feedback, and the output is the feedback data analyzed by the emotion engine.

[0389] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0390] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0391] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0392] [Second embodiment]

[0393] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0394] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0395] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0396] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0397] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0398] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0399] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0400] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0401] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0402] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0403] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0404] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0405] To solve the problem that the invention aims to solve, the present invention provides a system that allows users to easily create picture books using generative AI. This system is executed through the following processing steps. Each processing step is mainly executed by a terminal, a server, and a user.

[0406] Getting User Input

[0407] The terminal provides the user with an interface for inputting the storyline of the picture book. The user inputs the storyline and character information into this interface. The terminal acquires this input information and sends it to the server.

[0408] Generative AI generates text and illustrations

[0409] The server receives the synopsis from the user and uses a generation AI to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI generates a story for each page based on the synopsis, and the illustration generation AI then creates appropriate illustrations based on the generated text.

[0410] Picture book layout generation

[0411] The server combines the generated text and illustrations and creates a printable layout, including a title page, body pages, and other decorative elements. If desired, the user can include specific characters (e.g., the child himself).

[0412] Printing and mailing arrangements

[0413] The server sends the layout data of the completed picture book to the printing service, which then carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[0414] Get feedback

[0415] The device asks the user for feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts in a feedback form. This feedback is sent to the server and used to improve the system and the generation AI.

[0416] Specific examples

[0417] For example, if a user inputs a story about an animal's adventure, the server can receive the story and use the AI ​​to generate specific sentences and illustrations like the one below.

[0418] 1. Sentence Generation: "One day, Peter the rabbit went out in search of a hidden treasure deep in the forest."

[0419] 2. Illustration generation: A picture of Peter walking through the forest with a backpack on.

[0420] The generated text and illustrations are arranged on the title page and body pages using layout generation, and then sent to a printing service. The book is then mailed to the user's address, and the user receives their original picture book. If the user wishes to create another picture book, they can also create a sequel as a series.

[0421] In this way, the present invention realizes a system that allows users to easily create original picture books and provide entertainment for children.

[0422] The processing flow will be explained below.

[0423] Step 1: User enters synopsis

[0424] Subject: User

[0425] Specific behavior:

[0426] The user inputs the story synopsis through the system interface, which is a text input form where the user writes a brief summary of the story and information about the characters.

[0427] Subject: Device

[0428] Specific behavior:

[0429] The terminal collects the synopsis entered by the user and transmits it to the server.

[0430] Step 2: Generating text and illustrations using generative AI

[0431] Subject: Server

[0432] Specific behavior:

[0433] The server receives the synopsis sent by the user, then uses a text generation AI model to automatically generate text for each page of the picture book based on the synopsis, and then uses an illustration generation AI model to automatically generate illustrations that match the generated text.

[0434] Examples:

[0435] For example, if the plot is "Animals on an adventure searching for hidden treasure in the forest," the text generation AI will generate the sentence "Peter the rabbit went searching for hidden treasure deep in the forest." Next, the illustration generation AI will generate an illustration depicting a scene of Peter walking through the forest.

[0436] Step 3: Creating a picture book layout

[0437] Subject: Server

[0438] Specific behavior:

[0439] The server combines the generated text and illustrations to create a printable picture book layout, including a title page, body pages, and other decorative elements, and can even include specific characters (such as the child) if the user so desires.

[0440] Examples:

[0441] The generated text, "Peter the Rabbit Searches for Treasure," is then integrated with the illustrations and placed appropriately on each page of the picture book. Backgrounds and headings are also added to the pages.

[0442] Step 4: Arrange for printing and mailing

[0443] Subject: Server

[0444] Specific behavior:

[0445] The server sends the completed picture book layout data to the printing service and carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[0446] Examples:

[0447] The completed layout is sent to a printing company, which prints the picture book and delivers it to an address specified by the user.

[0448] Step 5:

[0449] Get feedback

[0450] Subject: Device

[0451] Specific behavior:

[0452] The device sends the user a notification requesting feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts in a feedback form.

[0453] Subject: User

[0454] Specific behavior:

[0455] Users receive a notification, fill out a feedback form with their thoughts and opinions about the picture book, and send it to the system via their terminal.

[0456] Subject: Server

[0457] Specific behavior:

[0458] The server receives feedback sent by users and analyzes it to improve the quality of the system and the generated AI.

[0459] These steps allow users to easily create and enjoy original picture books, and the system is continually improved through feedback.

[0460] Example 1

[0461] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0462] The traditional picture book creation process is time-consuming and laborious, requiring specialized knowledge, making it difficult for ordinary users to easily create original picture books. Another issue is that it is difficult to improve the system based on user feedback.

[0463] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0464] In this invention, the server includes means for a user to input a synopsis, means for automatically generating text and illustrations from the synopsis using a generative AI model, means for creating a picture book layout by combining the automatically generated text and illustrations, means for transmitting the picture book layout data for printing and mailing, and means for obtaining feedback from the user regarding the generated picture book. This enables even general users to easily create original picture books, and further enables continuous improvement of the system based on the obtained feedback.

[0465] "User" refers to an individual or group that uses the picture book creation system to create a picture book based on their own storyline.

[0466] A "synopsis" is a short story description entered by the user as an overview of the picture book's contents.

[0467] A "generative AI model" is an artificial intelligence model that automatically generates text and illustrations from a synopsis entered by the user.

[0468] "Text generation AI" is the part of the generative AI model that generates detailed text based on the synopsis entered by the user.

[0469] "Illustration generation AI" is the part of the generative AI model that depicts appropriate images based on the generated text.

[0470] "Layout" refers to the design that combines the generated text and illustrations and arranges them in a printable format.

[0471] A "printing service" is a company or system that provides a service of printing actual picture books based on layout data sent from a server and mailing them to users.

[0472] "Feedback" refers to the impressions and suggestions for improvement provided by users about the picture books they receive, and is data for future system improvements.

[0473] The "system" refers to the entire set of components that execute a series of processes, including user input, generation of text and illustrations using a generative AI model, creation of layout, arrangement of printing and mailing, and obtaining feedback.

[0474] "Character" means a person, animal, or fictional being that appears in the synopsis entered by the user and in the generated text and illustrations.

[0475] "Serialization" is the process of creating multiple picture books in succession as sequels to a previous picture book.

[0476] MODE FOR CARRYING OUT THE INVENTION

[0477] The present invention relates to a system that allows users to easily create original picture books. The system is implemented through the following specific steps. Each step is mainly performed by a terminal, a server, and a user.

[0478] 1. Getting User Input

[0479] The device provides the user with an interface for entering information about the storyline and characters in the book. This interface includes input fields, drop-down menus, character selection options, etc. The user enters the appropriate information into these fields and clicks the "Submit" button. The device then packages the entered information as JSON-formatted data and sends it to the server via a secure HTTP request.

[0480] 2. Generating text and illustrations using generative AI

[0481] The server processes the synopsis information sent from the device and passes it to the generative AI model. The generative AI model includes a text generation AI and an illustration generation AI. The text generation AI generates a detailed story based on the synopsis, and the illustration generation AI draws appropriate illustrations based on the content of the generated text. At this time, a prompt sentence is input into the generative AI model to generate the necessary data.

[0482] Specific examples

[0483] The server generates the following prompt: "The characters are Peter the rabbit, Phil the fox, and Oliver the owl. Describe their adventure in search of a hidden treasure deep in the forest. The scene begins with Peter putting on his backpack and setting off."

[0484] This prompt is fed into the AI ​​model, which generates the following text and illustration:

[0485] 1. Sentence Generation: "One day, Peter the rabbit went out in search of a hidden treasure deep in the forest."

[0486] 2. Illustration generation: A picture of Peter walking through the forest with a backpack on.

[0487] 3. Picture book layout generation

[0488] The server combines the generated text and illustrations and lays them out in a printable format, taking into account design elements such as pagination, font settings, and illustration positioning. A PDF file is generated using layout editing software (e.g., Adobe InDesign) or a custom layout engine.

[0489] 4. Printing and mailing arrangements

[0490] The server sends the completed picture book layout data to a partner printing service. The printing service prints the picture book based on the received PDF file and arranges for it to be mailed to the user's specified address. Information such as the tracking number is returned to the server and notified to the user.

[0491] 5. Get feedback

[0492] After the picture book arrives, the device displays a notification requesting feedback. The user enters their thoughts and suggestions for improvement in the feedback form on the device. The entered feedback is sent back to the server and used to improve the system in the future.

[0493] Specific actions

[0494] After the user finishes reading the picture book, the device displays a notification saying, "Please tell us what you thought of the picture book." The user fills in the feedback form, saying, "The illustrations were great, but the story was a little short." After submitting the form, the feedback is saved on the server.

[0495] In this way, the present invention allows users to easily create and receive original picture books. Furthermore, the feedback function allows for continuous improvement of the system. The entire system is linked by users, terminals, servers, and printing services, enabling smooth picture book creation and distribution.

[0496] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0497] Step 1: Getting User Input

[0498] The device provides the user with an interface for inputting the storyline and character information of the picture book. The user fills in the storyline and character information in the input fields on this interface and clicks the send button.

[0499] Input: Synopsis text, character information (e.g. "A story about the adventures of animals" or "Peter the rabbit, Phil the fox, Oliver the owl")

[0500] Output: JSON format data (e.g., { "storybook": { "title": "Animal Adventures", "summary": "Animal Adventures", "characters": ["Peter the Rabbit", "Phil the Fox", "Oliver the Owl"]}}")

[0501] Step 2: Generating text and illustrations using generative AI

[0502] The server receives the plot information sent from the device and passes it to the generative AI model, which includes a text generation AI and an illustration generation AI, and uses these to generate the story and illustrations.

[0503] Input: Synopsis data in JSON format

[0504] Data processing: Input plot and character information into the generative AI model as prompts (e.g., "The characters are Peter the rabbit, Phil the fox, and Oliver the owl. Describe their adventure as they search for a hidden treasure deep in the forest. The story begins with Peter putting on his backpack and setting off.").

[0505] Output: Generated text and illustration (e.g., text "One day, Peter the rabbit went out to look for hidden treasure deep in the forest.", illustration "Peter is walking through the forest with his backpack on.")

[0506] Step 3: Creating a picture book layout

[0507] The server combines the generated text and illustrations and lays them out in a printable format, generating a PDF file while taking into account design elements such as pagination, font settings, and illustration positioning.

[0508] Input: Generated text and illustrations

[0509] Data processing: Create layouts using layout editing software (e.g. Adobe InDesign) or custom layout engines, and arrange pages, fonts, and illustrations.

[0510] Output: A printable PDF file (e.g. "storybook.pdf")

[0511] Step 4: Arrange for printing and mailing

[0512] The server sends the completed picture book layout data to a partner printing service, which prints the picture book based on the PDF file and mails it to the address specified by the user.

[0513] Input: Printable PDF file, user address information

[0514] Data processing: Send the PDF file to a printing service and provide shipping details

[0515] Output: Shipped notification and tracking information (e.g., "Shipped notification, tracking number 123456789")

[0516] Step 5: Getting feedback

[0517] After receiving the picture book, the device displays a notification requesting feedback. The user enters their thoughts and suggestions for improvement in the feedback form and submits it. The feedback information is stored on the server and used to improve the system.

[0518] Input: User impressions and suggestions for improvement entered in the feedback form (e.g., "The illustrations were great, but the story was a little short.")

[0519] Data calculation: Feedback information is sent to the server and stored in the database

[0520] Output: Saved feedback information (e.g., "{ "feedback": { "user_id": "001", "comments": "The illustrations were great, but the story was a little short"}}")

[0521] Through these steps, users can easily create and receive original picture books, and the feedback function allows for continuous improvement of the system.

[0522] (Application example 1)

[0523] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0524] Conventional picture book production systems lack the means for users to easily create original picture books and distribute them digitally, making it difficult to reach many users and distribute them instantly.

[0525] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0526] In this invention, the server includes means for a user to input a synopsis, means for automatically generating text and illustrations from the synopsis using a generative AI model, means for combining the automatically generated text and illustrations to create a picture book layout, means for saving the generated picture book in a digital format and sharing it with other users, and means for distributing it as an e-book in cooperation with an external content distribution service. This allows users to easily create original picture books and instantly distribute them in digital format to many users.

[0527] "User" refers to the person who enters the storyline and character information of the picture book.

[0528] A "generative AI model" refers to an artificial intelligence model used to automatically generate text and illustrations from a synopsis.

[0529] "Sentence" refers to text automatically generated by a generative AI model.

[0530] "Illustrations" refer to pictures or images automatically generated by generative AI models.

[0531] "Picture book layout" refers to the page composition that combines and arranges generated text and illustrations.

[0532] "Digital format" refers to a data format that can be stored and displayed electronically.

[0533] "External content distribution service" refers to a third-party online platform for publishing and selling the generated picture books as e-books.

[0534] "Sharing" refers to the act of sharing the generated picture book with other users.

[0535] This invention is a system that allows users to easily create original picture books and store and distribute them in digital form. A specific embodiment for realizing this system will be described below.

[0536] Getting User Input

[0537] The user inputs the plot and character information using their own device (e.g., a smartphone). The device provides an interface for the user to perform input operations. The device then transmits the input information to the server.

[0538] Generating text and illustrations using a generative AI model

[0539] The server uses a generative AI model to automatically generate text and illustrations based on the synopsis and character information received from the user. Specifically, the server uses the "generative AI model" to generate text and illustrations by inputting the following prompts:

[0540] Example prompt sentence:

[0541] Generate a picture book story with the following synopsis: A brave little cat has an adventure in a big city. Characters: Timmy the Cat

[0542] Picture book layout generation

[0543] The generated text and illustrations are integrated by the server to create a picture book layout, which is then stored in digital format.

[0544] Save and share digitally

[0545] The server stores the layout of the created picture book in a digital format and provides a means for sharing it with other users, allowing the user to share the created picture book with other users.

[0546] Collaboration with content distribution services

[0547] Furthermore, the server also provides a means to link with external content distribution services and distribute the created picture books as e-books, allowing users to instantly and widely distribute the picture books they have created.

[0548] System Configuration

[0549] Hardware: Smartphone (user input), Server (data processing and storage)

[0550] Software: Generative AI model (generating text and illustrations), interface app (user input)

[0551] Data processing and calculation:

[0552] The synopsis and character information entered by the user is obtained as text data.

[0553] Using a generative AI model, text and illustrations are generated based on input data.

[0554] Consolidate generated data and store and distribute it digitally.

[0555] Specific examples of program processing

[0556] For example, if a user inputs the synopsis "A brave little cat goes on an adventure in a big city," the generative AI model will generate the sentence "One day, Timmy, a brave little cat, heads to a big city and starts his adventure," along with an illustration of the cat on an adventure. The generated sentences and illustrations are then integrated into the layout of the picture book by the server and saved in digital format.

[0557] In this way, the present invention realizes a system that enables users to easily create original picture books and instantly distribute them in digital form.

[0558] ---

[0559] This explanation corresponds to the "Form for carrying out the invention."

[0560] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0561] Step 1:

[0562] The user uses a smartphone or PC to input the storyline and character information into the interface. The information entered by the user is saved as text data on the device. This text data is then sent to the server.

[0563] Step 2:

[0564] The server analyzes the received text data. Specifically, it creates a prompt sentence for the generative AI model based on the synopsis and character information entered by the user. The prompt sentence is input into the generative AI model, and the picture book text is automatically generated. The generated text is saved as text data within the server.

[0565] Step 3:

[0566] The server creates a prompt for generating illustrations based on the generated text. The prompt is input into an illustration generation AI model, which automatically generates illustrations for the picture book. The generated illustrations are saved as image data on the server.

[0567] Step 4:

[0568] The server integrates the generated text and illustrations and automatically creates a picture book layout. This layout arranges the text and illustrations on specific pages, forming a single picture book as a whole. The generated layout data is saved in digital format.

[0569] Step 5:

[0570] The server provides a means for users to share the digital data of the created picture book with other users. This means allows users to share the picture book they created with other users. This data is also linked to external content distribution services and distributed as an e-book.

[0571] Step 6:

[0572] The server also collects feedback from users. After a user finishes reading a picture book, they enter their impressions and opinions into a feedback form and send it to the server. This feedback data is used to improve the generative AI model and the overall quality of the system.

[0573] Through the above steps, the present invention allows users to easily create original picture books and share and distribute them in digital format.

[0574] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0575] This invention combines a system that allows users to easily create picture books using generative AI with an emotion engine that recognizes the user's emotions, thereby enabling the creation of more personalized picture books. This system is executed through the following processing steps. Each processing step is mainly executed by the terminal, server, and user.

[0576] Getting User Input

[0577] The terminal provides the user with an interface for inputting the storyline of the picture book. The user inputs the storyline and character information into this interface. The terminal acquires this input information and sends it to the server.

[0578] Optimization by Emotion Engine

[0579] Before receiving the synopsis sent by the user, the server uses an emotion engine to recognize the user's emotions. This allows the server to check whether the input synopsis and character settings are appropriate for the user's emotions. If necessary, the emotion engine suggests optimizing the synopsis and settings based on the user's emotions.

[0580] Generative AI generates text and illustrations

[0581] The server receives a synopsis optimized based on the user's emotions and uses a generation AI to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI generates a story for each page based on the synopsis, and the illustration generation AI then creates appropriate illustrations based on the generated text. The emotion engine then rechecks the generated text and illustrations and makes fine adjustments as necessary.

[0582] Picture book layout generation

[0583] The server combines the generated text and illustrations and creates a printable layout, including a title page, body pages, and other decorative elements. Additionally, the layout can include specific characters (e.g., the child himself) if desired.

[0584] Printing and mailing arrangements

[0585] The server sends the layout data of the completed picture book to the printing service, which then carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[0586] Get feedback

[0587] The device sends the user a notification requesting feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts in a feedback form. The feedback also includes the user's emotions, which the emotion engine analyzes and reflects in the next generation.

[0588] Specific examples

[0589] For example, suppose a user inputs "a story about animals' adventures" as the synopsis. The server first checks the user's emotions using the emotion engine and determines whether the input synopsis is appropriate for the user's current emotions. If the server determines that the user is feeling a little depressed, the emotion engine will suggest, "Why not change it to a fun adventure story?"

[0590] Based on the optimized synopsis, the generative AI model generates specific sentences and illustrations like the following:

[0591] 1. Sentence Generation: "One day, Peter the rabbit went out in search of a hidden treasure deep in the forest."

[0592] 2. Illustration generation: A picture of Peter walking through the forest with a backpack on.

[0593] The generated text and illustrations are arranged on the title page and body pages using layout generation, and then sent to a printing service. The book is then mailed to the user's address, and the user receives their original picture book. If the user wishes to create another picture book, they can also create a sequel as a series.

[0594] In this way, by combining emotion engines, we can create a system that provides even more personalized, original picture books that best suit the user's emotions.

[0595] The processing flow will be explained below.

[0596] Step 1: User enters synopsis

[0597] Subject: User

[0598] Specific behavior:

[0599] The user inputs the story synopsis through the system interface, which is a text input form where the user writes a brief summary of the story and information about the characters.

[0600] Subject: Device

[0601] Specific behavior:

[0602] The terminal collects the synopsis entered by the user and transmits it to the server.

[0603] Step 2: Optimizing with an Emotional Engine

[0604] Subject: Server

[0605] Specific behavior:

[0606] Before receiving the synopsis sent by the user, the server activates the emotion engine and collects data to recognize the user's emotions. If the device has a built-in camera or microphone, the emotion engine analyzes facial expressions and voice tones to check whether the synopsis entered by the user is appropriate for the user's emotions.

[0607] Examples:

[0608] For example, if a user inputs "forest adventure," the emotion engine will read emotions such as "fun" or "exciting" from the user's facial expressions and tone, and determine whether the synopsis is appropriate based on those emotions. If necessary, the emotion engine will make optimization suggestions such as "How about adding more specific adventure elements?"

[0609] Step 3: Generative AI generates text and illustrations

[0610] Subject: Server

[0611] Specific behavior:

[0612] The server receives the synopsis optimized based on the user's emotions and uses a generation AI to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI generates a story for each page based on the synopsis, and the illustration generation AI creates illustrations based on the generated text.

[0613] Specific behavior:

[0614] Based on the plot, the AI ​​generates the sentence "Peter the rabbit went in search of hidden treasure deep in the forest," and then the AI ​​generates an illustration of Peter carrying a backpack and walking through the forest.

[0615] Subject: Emotion Engine

[0616] Specific behavior:

[0617] The emotion engine rechecks the generated text and illustrations and fine-tunes them to better suit the user's emotions. For example, if the user is expressing a happy emotion, the engine will add humor to the text.

[0618] Step 4: Creating a layout for your picture book

[0619] Subject: Server

[0620] Specific behavior:

[0621] The server combines the generated text and illustrations to create a printable layout, including a title page, body pages, and other decorative elements, and can include specific characters (e.g., the child himself) if the user so desires.

[0622] Specific behavior:

[0623] The completed text and illustrations are arranged on each page, and decorations such as backgrounds and headings are added. If a specific character appears, an illustration of that character is also added.

[0624] Step 5: Arrange for printing and mailing

[0625] Subject: Server

[0626] Specific behavior:

[0627] The server sends the completed picture book layout data to the printing service and carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[0628] Specific behavior:

[0629] The printing service prints the layout data and delivers the picture book to the address specified by the user.

[0630] Step 6: Getting feedback

[0631] Subject: Device

[0632] Specific behavior:

[0633] The device sends the user a notification requesting feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts and opinions in a feedback form. This feedback includes emotions, which the system analyzes and reflects in the next generation.

[0634] Subject: User

[0635] Specific behavior:

[0636] Users receive a notification, fill out a feedback form with their thoughts and opinions about the picture book, and send it to the system via their terminal.

[0637] Subject: Server

[0638] Specific behavior:

[0639] The server analyzes the received feedback and uses it to generate the next picture book. The emotion engine also analyzes this feedback to understand changes in the user's emotions.

[0640] These steps allow users to easily create original picture books with emotionally appropriate content, and the system is continually improved through feedback.

[0641] Example 2

[0642] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0643] Conventional picture book creation systems automatically generate text and illustrations based on a synopsis entered by the user, but because they do not take into account the user's emotions, it is difficult to create a personalized picture book that meets the user's expectations. In addition, there is no way to check the text and illustrations generated based on the user's emotions and make fine adjustments as necessary.

[0644] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0645] In this invention, the server includes means for a user to input a synopsis, means for automatically generating text and illustrations from the synopsis using a generative AI model, means for combining the automatically generated text and illustrations to create a picture book layout, means for transmitting the picture book layout data for printing and mailing, means for recognizing the user's emotions and optimizing the synopsis and settings based on the user's emotions, and means for checking the generated text and illustrations based on the user's emotions and fine-tuning them as necessary, thereby enabling the creation of a personalized picture book optimized based on the user's emotions.

[0646] A "user" is an entity that uses the system to input the storyline and characters of a picture book.

[0647] A "generative AI model" is an artificial intelligence algorithm that automatically generates text and illustrations from an input synopsis.

[0648] A "synopsis" is a short sentence that describes the outline of the story of a picture book.

[0649] "Sentence" refers to text that is automatically generated by the generative AI model based on the plot.

[0650] "Illustrations" are images automatically generated by a generative AI model based on text generation.

[0651] "Layout" is a format for combining automatically generated text and illustrations to arrange them into a picture book.

[0652] "Printing and mailing" is the process of producing a physical picture book based on the completed picture book layout data and delivering it to a specified address.

[0653] The "emotion engine" is an algorithm that recognizes the user's emotions from the synopsis and character information entered by the user and suggests optimizations.

[0654] "Optimization" means adjusting the input plot and settings based on the user's emotions.

[0655] "Fine-tuning" means rechecking the generated text and illustrations and correcting the content if necessary.

[0656] This invention is a system that allows users to easily create personalized picture books by combining a generative AI model and an emotion engine. This system generates optimized text and illustrations that take emotions into consideration based on the plot and character information entered by the user, and then creates a picture book layout that combines them. A detailed explanation of the system is provided below.

[0657] This system is mainly run by a server, a terminal, and a user. The general flow of the system is as follows: user input, optimization by an emotion engine, generation of text and illustrations by generative AI, generation of picture book layout, printing and mailing, and obtaining feedback.

[0658] The device provides the user with an interface for inputting the storyline and character information of the picture book. The user inputs the storyline and character information into input fields displayed on the device, and the data is sent from the device to the server. This data can be sent in an appropriate data format such as JSON.

[0659] After receiving user input data from the device, the server launches the emotion engine and analyzes the data. The emotion engine recognizes the user's emotions using natural language processing libraries (e.g., NLTK, BERT, etc.). If the input content does not match the user's emotions, the emotion engine makes optimization suggestions to the server and generates optimization candidates. The server presents them to the user, and if the user approves, the optimized synopsis proceeds to the next process.

[0660] The server then receives the optimized synopsis and uses a generative AI model (e.g., GPT-3, DALL-E, etc.) to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI first creates a story for each page based on the synopsis, and then the illustration generation AI creates appropriate illustrations based on the generated text. The generated text and illustrations are then checked again by the emotion engine and fine-tuned as necessary.

[0661] The server uses the generated text and illustrations to create a layout for the book. This is done using a layout engine (e.g., LaTeX or an automated script in InDesign). The layout includes a title page, body pages, and other decorative elements. Additionally, the user can configure the layout to include specific characters of their choice.

[0662] The completed picture book layout data is converted to PDF format and sent to the printing service API. The printing service then prints the picture book based on that data and delivers it to the address specified by the user. At this time, the user is notified of a tracking number so that they can track the delivery status.

[0663] After receiving the picture book, the device sends a notification to the user requesting feedback. The user enters their impressions of the book into a feedback form, and the information is sent to the server. The feedback also includes the user's emotions, which the emotion engine analyzes and reflects in the next generation process.

[0664] Specific examples

[0665] For example, suppose a user inputs a story about animals' adventures as a synopsis. The server first checks the user's emotions using the emotion engine to determine whether the input synopsis is appropriate for the user's current emotions. If the server determines that the user is feeling a little depressed, the emotion engine will suggest changing it to a fun adventure story. Based on the optimized synopsis, the generative AI model will generate specific sentences and illustrations like the one below.

[0666] Sentence Generation: "One day, Peter the rabbit went out in search of hidden treasure deep in the forest."

[0667] Illustration generation: A picture of Peter walking through the forest with a backpack on.

[0668] The generated text and illustrations are arranged on the title page and body pages using layout generation, and then sent to a printing service. The book is then mailed to the user's address, and the user receives their original picture book. If the user wishes to create another picture book, they can also create a sequel as a series.

[0669] Prompt Sentence Examples

[0670] "Please provide a synopsis of a children's animal adventure story. Please also include information about the characters."

[0671] The present invention allows users to easily create and enjoy original picture books that are personalized to their own emotions.

[0672] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0673] Step 1:

[0674] The device provides the user with an interface for inputting the storyline of the picture book. The user uses this interface to input the storyline and character information. The input data is sent to the server in JSON format by the device.

[0675] Input: User-entered plot and character information

[0676] Output: Data sent to the server in JSON format

[0677] Specifically, the user enters "Synopsis: Animals go on a great adventure in the forest" and the characters "Peter the rabbit, Reina the fox, and Beatrice the bear" into the interface. The device converts this information into JSON data "{'summary': 'Animals go on a great adventure in the forest', 'characters': ['Peter', 'Reina', 'Beatrice']}" and sends it to the server.

[0678] Step 2:

[0679] The server analyzes the user input data received from the device. The server then activates an emotion engine to analyze this data and recognize the user's emotions. The emotion engine uses a natural language processing library to analyze the input text and output the results. If necessary, it generates optimization suggestions, which the server presents to the user.

[0680] Input: User-entered data sent from the terminal

[0681] Output: Emotion recognition results from the emotion engine and optimization suggestions as needed

[0682] Specifically, the server receives the data "{'summary': 'Animals have a great adventure in the forest', 'characters': ['Peter', 'Reina', 'Beatrice']}" and, if the server uses the emotion engine to recognize that the user's emotion is "depressed," it generates a suggestion such as "Why not change it to a more cheerful adventure story?"

[0683] Step 3:

[0684] The server presents the emotion engine's suggestions to the user, and the user approves or modifies the suggestions, resulting in optimized data that is then sent back to the server. Based on the optimized synopsis received by the server, the generative AI model is used to automatically generate the text and illustrations for the picture book. First, the text generation AI generates a story for each page, and then the illustration generation AI generates illustrations based on the story.

[0685] Input: User-approved or revised optimized synopsis

[0686] Output: Text and illustrations generated by a generative AI model

[0687] Specifically, the user approves the optimization proposal, and based on the optimized plot summary, "Animals have a fun adventure," the text generation AI generates text such as "One day, Peter the rabbit went out to search for hidden treasure deep in the forest." The illustration generation AI then uses that text to generate an illustration of Peter walking through the forest with a backpack on.

[0688] Step 4:

[0689] The server combines the generated text and illustrations to create a picture book layout. It uses a layout engine to automatically generate a layout including a title page, body pages, and decorative elements. The server then converts the completed layout into PDF format and sends it to the print service API.

[0690] Input: Text and illustrations generated by a generative AI model

[0691] Output: Layout data in PDF format sent to the print service API

[0692] Specifically, the server generates a title page called "Peter and the Forest Treasure" and a layout with text and illustrations on each page, and sends it in PDF format to the "Printing Service API."

[0693] Step 5:

[0694] The printing service receives the layout data in PDF format, prints the picture book, and delivers it to the address specified by the user. The user is then notified of the tracking number so that they can track the delivery status.

[0695] Input: PDF layout data sent from the server

[0696] Output: Printed book and tracking number notification to user

[0697] Specifically, the printing service prints the picture book based on the PDF and ships it to the user's address. The user can then check the delivery status using the tracking number provided.

[0698] Step 6:

[0699] The device sends a notification to the user requesting feedback on the picture book. After receiving and reading the book, the user enters their thoughts in a feedback form, and the information is sent to the server. The server analyzes the feedback using an emotion engine and reflects it in the next generation process.

[0700] Input: User feedback and emotional information

[0701] Output: The results of the emotion engine analysis and improvements to be reflected in the next generation process

[0702] Specifically, the device sends a notification to the user saying, "How did you like the picture book?", and the user enters feedback such as, "My child really enjoyed it. I'd like to add more animal species next time." The server analyzes this information and reflects it in the next generation process.

[0703] (Application example 2)

[0704] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0705] Conventional picture book generation systems lacked the ability to personalize the plot and character information entered by the user, making it particularly difficult to generate optimal content that reflected the user's emotions.Furthermore, there was no mechanism for incorporating feedback from the generated picture book into the next generation.

[0706] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a user to input a summary, a means for automatically generating text and images from the summary using a generative AI model, a means for creating a reading material layout by combining the automatically generated text and images, and a means for recognizing the user's emotions using an emotion engine and optimizing the summary and character settings. This makes it possible to generate personalized reading material that is optimized for the user's emotions.

[0707] A "user" is an entity that uses the system to generate reading material, for example, someone who inputs information via a smart device.

[0708] The "summary" includes a brief summary of the story and information about the characters entered by the user.

[0709] A "generative AI model" is an artificial intelligence program that automatically generates text and images based on a summary entered by the user.

[0710] The "emotion engine" is a program that recognizes the user's emotions and optimizes the story outline and character settings based on those emotions.

[0711] "Sentence" refers to the text portion of the story generated by the generative AI model.

[0712] "Imagery" refers to visual illustrations or pictures related to the narrative generated by the generative AI model.

[0713] "Reading material layout" refers to creating a page layout for a reading material by combining automatically generated text and images.

[0714] "Printing" refers to the process of producing a reading material in a physical form based on the layout data of the generated reading material.

[0715] "Mail" means delivery of printed materials to the address specified by the user.

[0716] "Personalization" refers to providing content that is optimized to the user's emotions and preferences.

[0717] An embodiment of the present invention provides a system that allows users to easily create and view personalized reading material through a picture book streaming app using a smartphone or head-mounted display (HMD).

[0718] System Overview:

[0719] Hardware configuration:

[0720] Smartphone

[0721] Head-mounted display (HMD)

[0722] server

[0723] Software configuration:

[0724] Emotion Engine

[0725] Generative AI Models

[0726] Web frameworks (e.g., Flask, Django)

[0727] REST API

[0728] Process flow:

[0729] 1. Getting user input:

[0730] Using an application on a device (smartphone or HMD), the user inputs a summary of the story, for example, "One day, Peter the rabbit went out to search for a hidden treasure deep in the forest."

[0731] Additionally, the user can input their current emotions, choosing from options such as "happy," "sad," or "angry."

[0732] 2. Emotional engine optimization:

[0733] The server uses an emotion engine to recognize the user's emotion, and then checks whether the summary entered by the user is appropriate for that emotion.

[0734] If necessary, the sentiment engine suggests optimizing the summary based on the user's sentiment.

[0735] 3. Generative AI model for generating text and images:

[0736] The server receives the summary optimized by the emotion engine, and the generative AI model generates narrative text and images based on the summary.

[0737] For example, a text generation AI generates the sentence, "One day, Peter the rabbit went out to search for hidden treasure deep in the forest," and an image generation AI generates an illustration depicting that scene.

[0738] 4. Generating reading material arrangements:

[0739] The server combines the generated text and images to create the page layout of the reading material, including the title page and body pages.

[0740] 5. Getting feedback:

[0741] The system asks users for feedback on the generated reading material, including their feelings, which the emotion engine analyzes and reflects in the next generation.

[0742] Example prompt sentence:

[0743] "Generate a fun adventure story about Peter the Rabbit's search for hidden treasure."

[0744] This invention is a system that combines an emotion engine and a generative AI model to provide highly personalized reading material to users. By linking smart devices with a server, content can be generated based on the user's emotions, and further optimized content can be provided through feedback.

[0745] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0746] Step 1:

[0747] The device provides an interface for the user to input a story summary and their current emotions. Using this interface, the user can input a summary such as "One day, Peter the rabbit went out to search for hidden treasure deep in the forest" and the emotion "fun." This input information is sent to the server. The input is the user's story summary and emotions, and the output is the user's input data sent to the server.

[0748] Step 2:

[0749] The server sends the received user summary and emotion to the emotion engine. The emotion engine checks the suitability of the input summary based on the user's emotion and optimizes it if necessary. For example, if the user inputs "fun," the emotion engine suggests a title and character settings that increase the fun factor. In this step, the input is the user's summary and emotion, and the output is an optimized summary suggestion.

[0750] Step 3:

[0751] The server receives the summary optimized by the emotion engine and sends it to the generative AI model. The generative AI model generates story text and images based on this optimized summary. For example, the text generation part generates the sentence "One day, Peter the rabbit went out to search for hidden treasure deep in the forest," and the image generation part generates an illustration depicting that scene. The input is the optimized summary, and the output is the generated text and images.

[0752] Step 4:

[0753] The server combines the generated text and images to automatically create page layouts for the reading material, including title and body pages. For example, the title page displays the title "The Adventures of Peter Rabbit," and the body pages combine the generated text with corresponding illustrations. The input is the generated text and images, and the output is the page layout.

[0754] Step 5:

[0755] The server provides the layout data of the completed book to the user through the streaming app. If the user wishes, it can also send it to a printing service and mail the physical book to the user's address. In this step, the input is the layout data of the book, and the output is the provision of the book in streaming format and arrangement of a printing service.

[0756] Step 6:

[0757] The device sends a notification to the user requesting feedback on the generated reading material. After reading the reading material, the user enters their impressions in a feedback form. This feedback is sent to the server and reflected in the next generation. The input is the user's feedback, and the output is the feedback data analyzed by the emotion engine.

[0758] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0759] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0760] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0761] [Third embodiment]

[0762] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0763] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0764] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0765] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0766] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0767] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0768] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0769] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0770] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0771] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0772] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0773] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0774] To solve the problem that the invention aims to solve, the present invention provides a system that allows users to easily create picture books using generative AI. This system is executed through the following processing steps. Each processing step is mainly executed by a terminal, a server, and a user.

[0775] Getting User Input

[0776] The terminal provides the user with an interface for inputting the storyline of the picture book. The user inputs the storyline and character information into this interface. The terminal acquires this input information and sends it to the server.

[0777] Generative AI generates text and illustrations

[0778] The server receives the synopsis from the user and uses a generation AI to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI generates a story for each page based on the synopsis, and the illustration generation AI then creates appropriate illustrations based on the generated text.

[0779] Picture book layout generation

[0780] The server combines the generated text and illustrations and creates a printable layout, including a title page, body pages, and other decorative elements. If desired, the user can include specific characters (e.g., the child himself).

[0781] Printing and mailing arrangements

[0782] The server sends the layout data of the completed picture book to the printing service, which then carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[0783] Get feedback

[0784] The device asks the user for feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts in a feedback form. This feedback is sent to the server and used to improve the system and the generation AI.

[0785] Specific examples

[0786] For example, if a user inputs a story about an animal's adventure, the server can receive the story and use the AI ​​to generate specific sentences and illustrations like the one below.

[0787] 1. Sentence Generation: "One day, Peter the rabbit went out in search of a hidden treasure deep in the forest."

[0788] 2. Illustration generation: A picture of Peter walking through the forest with a backpack on.

[0789] The generated text and illustrations are arranged on the title page and body pages using layout generation, and then sent to a printing service. The book is then mailed to the user's address, and the user receives their original picture book. If the user wishes to create another picture book, they can also create a sequel as a series.

[0790] In this way, the present invention realizes a system that allows users to easily create original picture books and provide entertainment for children.

[0791] The processing flow will be explained below.

[0792] Step 1: User enters synopsis

[0793] Subject: User

[0794] Specific behavior:

[0795] The user inputs the story synopsis through the system interface, which is a text input form where the user writes a brief summary of the story and information about the characters.

[0796] Subject: Device

[0797] Specific behavior:

[0798] The terminal collects the synopsis entered by the user and transmits it to the server.

[0799] Step 2: Generating text and illustrations using generative AI

[0800] Subject: Server

[0801] Specific behavior:

[0802] The server receives the synopsis sent by the user, then uses a text generation AI model to automatically generate text for each page of the picture book based on the synopsis, and then uses an illustration generation AI model to automatically generate illustrations that match the generated text.

[0803] Examples:

[0804] For example, if the plot is "Animals on an adventure searching for hidden treasure in the forest," the text generation AI will generate the sentence "Peter the rabbit went searching for hidden treasure deep in the forest." Next, the illustration generation AI will generate an illustration depicting a scene of Peter walking through the forest.

[0805] Step 3: Creating a picture book layout

[0806] Subject: Server

[0807] Specific behavior:

[0808] The server combines the generated text and illustrations to create a printable picture book layout, including a title page, body pages, and other decorative elements, and can even include specific characters (such as the child) if the user so desires.

[0809] Examples:

[0810] The generated text, "Peter the Rabbit Searches for Treasure," is then integrated with the illustrations and placed appropriately on each page of the picture book. Backgrounds and headings are also added to the pages.

[0811] Step 4: Arrange for printing and mailing

[0812] Subject: Server

[0813] Specific behavior:

[0814] The server sends the completed picture book layout data to the printing service and carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[0815] Examples:

[0816] The completed layout is sent to a printing company, which prints the picture book and delivers it to an address specified by the user.

[0817] Step 5:

[0818] Get feedback

[0819] Subject: Device

[0820] Specific behavior:

[0821] The device sends the user a notification requesting feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts in a feedback form.

[0822] Subject: User

[0823] Specific behavior:

[0824] Users receive a notification, fill out a feedback form with their thoughts and opinions about the picture book, and send it to the system via their terminal.

[0825] Subject: Server

[0826] Specific behavior:

[0827] The server receives feedback sent by users and analyzes it to improve the quality of the system and the generated AI.

[0828] These steps allow users to easily create and enjoy original picture books, and the system is continually improved through feedback.

[0829] Example 1

[0830] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0831] The traditional picture book creation process is time-consuming and laborious, requiring specialized knowledge, making it difficult for ordinary users to easily create original picture books. Another issue is that it is difficult to improve the system based on user feedback.

[0832] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0833] In this invention, the server includes means for a user to input a synopsis, means for automatically generating text and illustrations from the synopsis using a generative AI model, means for creating a picture book layout by combining the automatically generated text and illustrations, means for transmitting the picture book layout data for printing and mailing, and means for obtaining feedback from the user regarding the generated picture book. This enables even general users to easily create original picture books, and further enables continuous improvement of the system based on the obtained feedback.

[0834] "User" refers to an individual or group that uses the picture book creation system to create a picture book based on their own storyline.

[0835] A "synopsis" is a short story description entered by the user as an overview of the picture book's contents.

[0836] A "generative AI model" is an artificial intelligence model that automatically generates text and illustrations from a synopsis entered by the user.

[0837] "Text generation AI" is the part of the generative AI model that generates detailed text based on the synopsis entered by the user.

[0838] "Illustration generation AI" is the part of the generative AI model that depicts appropriate images based on the generated text.

[0839] "Layout" refers to the design that combines the generated text and illustrations and arranges them in a printable format.

[0840] A "printing service" is a company or system that provides a service of printing actual picture books based on layout data sent from a server and mailing them to users.

[0841] "Feedback" refers to the impressions and suggestions for improvement provided by users about the picture books they receive, and is data for future system improvements.

[0842] The "system" refers to the entire set of components that execute a series of processes, including user input, generation of text and illustrations using a generative AI model, creation of layout, arrangement of printing and mailing, and obtaining feedback.

[0843] "Character" means a person, animal, or fictional being that appears in the synopsis entered by the user and in the generated text and illustrations.

[0844] "Serialization" is the process of creating multiple picture books in succession as sequels to a previous picture book.

[0845] MODE FOR CARRYING OUT THE INVENTION

[0846] The present invention relates to a system that allows users to easily create original picture books. The system is implemented through the following specific steps. Each step is mainly performed by a terminal, a server, and a user.

[0847] 1. Getting User Input

[0848] The device provides the user with an interface for entering information about the storyline and characters in the book. This interface includes input fields, drop-down menus, character selection options, etc. The user enters the appropriate information into these fields and clicks the "Submit" button. The device then packages the entered information as JSON-formatted data and sends it to the server via a secure HTTP request.

[0849] 2. Generating text and illustrations using generative AI

[0850] The server processes the synopsis information sent from the device and passes it to the generative AI model. The generative AI model includes a text generation AI and an illustration generation AI. The text generation AI generates a detailed story based on the synopsis, and the illustration generation AI draws appropriate illustrations based on the content of the generated text. At this time, a prompt sentence is input into the generative AI model to generate the necessary data.

[0851] Specific examples

[0852] The server generates the following prompt: "The characters are Peter the rabbit, Phil the fox, and Oliver the owl. Describe their adventure in search of a hidden treasure deep in the forest. The scene begins with Peter putting on his backpack and setting off."

[0853] This prompt is fed into the AI ​​model, which generates the following text and illustration:

[0854] 1. Sentence Generation: "One day, Peter the rabbit went out in search of a hidden treasure deep in the forest."

[0855] 2. Illustration generation: A picture of Peter walking through the forest with a backpack on.

[0856] 3. Picture book layout generation

[0857] The server combines the generated text and illustrations and lays them out in a printable format, taking into account design elements such as pagination, font settings, and illustration positioning. A PDF file is generated using layout editing software (e.g., Adobe InDesign) or a custom layout engine.

[0858] 4. Printing and mailing arrangements

[0859] The server sends the completed picture book layout data to a partner printing service. The printing service prints the picture book based on the received PDF file and arranges for it to be mailed to the user's specified address. Information such as the tracking number is returned to the server and notified to the user.

[0860] 5. Get feedback

[0861] After the picture book arrives, the device displays a notification requesting feedback. The user enters their thoughts and suggestions for improvement in the feedback form on the device. The entered feedback is sent back to the server and used to improve the system in the future.

[0862] Specific actions

[0863] After the user finishes reading the picture book, the device displays a notification saying, "Please tell us what you thought of the picture book." The user fills in the feedback form, saying, "The illustrations were great, but the story was a little short." After submitting the form, the feedback is saved on the server.

[0864] In this way, the present invention allows users to easily create and receive original picture books. Furthermore, the feedback function allows for continuous improvement of the system. The entire system is linked by users, terminals, servers, and printing services, enabling smooth picture book creation and distribution.

[0865] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0866] Step 1: Getting User Input

[0867] The device provides the user with an interface for inputting the storyline and character information of the picture book. The user fills in the storyline and character information in the input fields on this interface and clicks the send button.

[0868] Input: Synopsis text, character information (e.g. "A story about the adventures of animals" or "Peter the rabbit, Phil the fox, Oliver the owl")

[0869] Output: JSON format data (e.g., { "storybook": { "title": "Animal Adventures", "summary": "Animal Adventures", "characters": ["Peter the Rabbit", "Phil the Fox", "Oliver the Owl"]}}")

[0870] Step 2: Generating text and illustrations using generative AI

[0871] The server receives the plot information sent from the device and passes it to the generative AI model, which includes a text generation AI and an illustration generation AI, and uses these to generate the story and illustrations.

[0872] Input: Synopsis data in JSON format

[0873] Data processing: Input plot and character information into the generative AI model as prompts (e.g., "The characters are Peter the rabbit, Phil the fox, and Oliver the owl. Describe their adventure as they search for a hidden treasure deep in the forest. The story begins with Peter putting on his backpack and setting off.").

[0874] Output: Generated text and illustration (e.g., text "One day, Peter the rabbit went out to look for hidden treasure deep in the forest.", illustration "Peter is walking through the forest with his backpack on.")

[0875] Step 3: Creating a picture book layout

[0876] The server combines the generated text and illustrations and lays them out in a printable format, generating a PDF file while taking into account design elements such as pagination, font settings, and illustration positioning.

[0877] Input: Generated text and illustrations

[0878] Data processing: Create layouts using layout editing software (e.g. Adobe InDesign) or custom layout engines, and arrange pages, fonts, and illustrations.

[0879] Output: A printable PDF file (e.g. "storybook.pdf")

[0880] Step 4: Arrange for printing and mailing

[0881] The server sends the completed picture book layout data to a partner printing service, which prints the picture book based on the PDF file and mails it to the address specified by the user.

[0882] Input: Printable PDF file, user address information

[0883] Data processing: Send the PDF file to a printing service and provide shipping details

[0884] Output: Shipped notification and tracking information (e.g., "Shipped notification, tracking number 123456789")

[0885] Step 5: Getting feedback

[0886] After receiving the picture book, the device displays a notification requesting feedback. The user enters their thoughts and suggestions for improvement in the feedback form and submits it. The feedback information is stored on the server and used to improve the system.

[0887] Input: User impressions and suggestions for improvement entered in the feedback form (e.g., "The illustrations were great, but the story was a little short.")

[0888] Data calculation: Feedback information is sent to the server and stored in the database

[0889] Output: Saved feedback information (e.g., "{ "feedback": { "user_id": "001", "comments": "The illustrations were great, but the story was a little short"}}")

[0890] Through these steps, users can easily create and receive original picture books, and the feedback function allows for continuous improvement of the system.

[0891] (Application example 1)

[0892] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0893] Conventional picture book production systems lack the means for users to easily create original picture books and distribute them digitally, making it difficult to reach many users and distribute them instantly.

[0894] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0895] In this invention, the server includes means for a user to input a synopsis, means for automatically generating text and illustrations from the synopsis using a generative AI model, means for combining the automatically generated text and illustrations to create a picture book layout, means for saving the generated picture book in a digital format and sharing it with other users, and means for distributing it as an e-book in cooperation with an external content distribution service. This allows users to easily create original picture books and instantly distribute them in digital format to many users.

[0896] "User" refers to the person who enters the storyline and character information of the picture book.

[0897] A "generative AI model" refers to an artificial intelligence model used to automatically generate text and illustrations from a synopsis.

[0898] "Sentence" refers to text automatically generated by a generative AI model.

[0899] "Illustrations" refer to pictures or images automatically generated by generative AI models.

[0900] "Picture book layout" refers to the page composition that combines and arranges generated text and illustrations.

[0901] "Digital format" refers to a data format that can be stored and displayed electronically.

[0902] "External content distribution service" refers to a third-party online platform for publishing and selling the generated picture books as e-books.

[0903] "Sharing" refers to the act of sharing the generated picture book with other users.

[0904] This invention is a system that allows users to easily create original picture books and store and distribute them in digital form. A specific embodiment for realizing this system will be described below.

[0905] Getting User Input

[0906] The user inputs the plot and character information using their own device (e.g., a smartphone). The device provides an interface for the user to perform input operations. The device then transmits the input information to the server.

[0907] Generating text and illustrations using a generative AI model

[0908] The server uses a generative AI model to automatically generate text and illustrations based on the synopsis and character information received from the user. Specifically, the server uses the "generative AI model" to generate text and illustrations by inputting the following prompts:

[0909] Example prompt sentence:

[0910] Generate a picture book story with the following synopsis: A brave little cat has an adventure in a big city. Characters: Timmy the Cat

[0911] Picture book layout generation

[0912] The generated text and illustrations are integrated by the server to create a picture book layout, which is then stored in digital format.

[0913] Save and share digitally

[0914] The server stores the layout of the created picture book in a digital format and provides a means for sharing it with other users, allowing the user to share the created picture book with other users.

[0915] Collaboration with content distribution services

[0916] Furthermore, the server also provides a means to link with external content distribution services and distribute the created picture books as e-books, allowing users to instantly and widely distribute the picture books they have created.

[0917] System Configuration

[0918] Hardware: Smartphone (user input), Server (data processing and storage)

[0919] Software: Generative AI model (generating text and illustrations), interface app (user input)

[0920] Data processing and calculation:

[0921] The synopsis and character information entered by the user is obtained as text data.

[0922] Using a generative AI model, text and illustrations are generated based on input data.

[0923] Consolidate generated data and store and distribute it digitally.

[0924] Specific examples of program processing

[0925] For example, if a user inputs the synopsis "A brave little cat goes on an adventure in a big city," the generative AI model will generate the sentence "One day, Timmy, a brave little cat, heads to a big city and starts his adventure," along with an illustration of the cat on an adventure. The generated sentences and illustrations are then integrated into the layout of the picture book by the server and saved in digital format.

[0926] In this way, the present invention realizes a system that enables users to easily create original picture books and instantly distribute them in digital form.

[0927] ---

[0928] This explanation corresponds to the "Form for carrying out the invention."

[0929] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0930] Step 1:

[0931] The user uses a smartphone or PC to input the storyline and character information into the interface. The information entered by the user is saved as text data on the device. This text data is then sent to the server.

[0932] Step 2:

[0933] The server analyzes the received text data. Specifically, it creates a prompt sentence for the generative AI model based on the synopsis and character information entered by the user. The prompt sentence is input into the generative AI model, and the picture book text is automatically generated. The generated text is saved as text data within the server.

[0934] Step 3:

[0935] The server creates a prompt for generating illustrations based on the generated text. The prompt is input into an illustration generation AI model, which automatically generates illustrations for the picture book. The generated illustrations are saved as image data on the server.

[0936] Step 4:

[0937] The server integrates the generated text and illustrations and automatically creates a picture book layout. This layout arranges the text and illustrations on specific pages, forming a single picture book as a whole. The generated layout data is saved in digital format.

[0938] Step 5:

[0939] The server provides a means for users to share the digital data of the created picture book with other users. This means allows users to share the picture book they created with other users. This data is also linked to external content distribution services and distributed as an e-book.

[0940] Step 6:

[0941] The server also collects feedback from users. After a user finishes reading a picture book, they enter their impressions and opinions into a feedback form and send it to the server. This feedback data is used to improve the generative AI model and the overall quality of the system.

[0942] Through the above steps, the present invention allows users to easily create original picture books and share and distribute them in digital format.

[0943] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0944] This invention combines a system that allows users to easily create picture books using generative AI with an emotion engine that recognizes the user's emotions, thereby enabling the creation of more personalized picture books. This system is executed through the following processing steps. Each processing step is mainly executed by the terminal, server, and user.

[0945] Getting User Input

[0946] The terminal provides the user with an interface for inputting the storyline of the picture book. The user inputs the storyline and character information into this interface. The terminal acquires this input information and sends it to the server.

[0947] Optimization by Emotion Engine

[0948] Before receiving the synopsis sent by the user, the server uses an emotion engine to recognize the user's emotions. This allows the server to check whether the input synopsis and character settings are appropriate for the user's emotions. If necessary, the emotion engine suggests optimizing the synopsis and settings based on the user's emotions.

[0949] Generative AI generates text and illustrations

[0950] The server receives a synopsis optimized based on the user's emotions and uses a generation AI to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI generates a story for each page based on the synopsis, and the illustration generation AI then creates appropriate illustrations based on the generated text. The emotion engine then rechecks the generated text and illustrations and makes fine adjustments as necessary.

[0951] Picture book layout generation

[0952] The server combines the generated text and illustrations and creates a printable layout, including a title page, body pages, and other decorative elements. Additionally, the layout can include specific characters (e.g., the child himself) if desired.

[0953] Printing and mailing arrangements

[0954] The server sends the layout data of the completed picture book to the printing service, which then carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[0955] Get feedback

[0956] The device sends the user a notification requesting feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts in a feedback form. The feedback also includes the user's emotions, which the emotion engine analyzes and reflects in the next generation.

[0957] Specific examples

[0958] For example, suppose a user inputs "a story about animals' adventures" as the synopsis. The server first checks the user's emotions using the emotion engine and determines whether the input synopsis is appropriate for the user's current emotions. If the server determines that the user is feeling a little depressed, the emotion engine will suggest, "Why not change it to a fun adventure story?"

[0959] Based on the optimized synopsis, the generative AI model generates specific sentences and illustrations like the following:

[0960] 1. Sentence Generation: "One day, Peter the rabbit went out in search of a hidden treasure deep in the forest."

[0961] 2. Illustration generation: A picture of Peter walking through the forest with a backpack on.

[0962] The generated text and illustrations are arranged on the title page and body pages using layout generation, and then sent to a printing service. The book is then mailed to the user's address, and the user receives their original picture book. If the user wishes to create another picture book, they can also create a sequel as a series.

[0963] In this way, by combining emotion engines, we can create a system that provides even more personalized, original picture books that best suit the user's emotions.

[0964] The processing flow will be explained below.

[0965] Step 1: User enters synopsis

[0966] Subject: User

[0967] Specific behavior:

[0968] The user inputs the story synopsis through the system interface, which is a text input form where the user writes a brief summary of the story and information about the characters.

[0969] Subject: Device

[0970] Specific behavior:

[0971] The terminal collects the synopsis entered by the user and transmits it to the server.

[0972] Step 2: Optimizing with an Emotional Engine

[0973] Subject: Server

[0974] Specific behavior:

[0975] Before receiving the synopsis sent by the user, the server activates the emotion engine and collects data to recognize the user's emotions. If the device has a built-in camera or microphone, the emotion engine analyzes facial expressions and voice tones to check whether the synopsis entered by the user is appropriate for the user's emotions.

[0976] Examples:

[0977] For example, if a user inputs "forest adventure," the emotion engine will read emotions such as "fun" or "exciting" from the user's facial expressions and tone, and determine whether the synopsis is appropriate based on those emotions. If necessary, the emotion engine will make optimization suggestions such as "How about adding more specific adventure elements?"

[0978] Step 3: Generative AI generates text and illustrations

[0979] Subject: Server

[0980] Specific behavior:

[0981] The server receives the synopsis optimized based on the user's emotions and uses a generation AI to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI generates a story for each page based on the synopsis, and the illustration generation AI creates illustrations based on the generated text.

[0982] Specific behavior:

[0983] Based on the plot, the AI ​​generates the sentence "Peter the rabbit went in search of hidden treasure deep in the forest," and then the AI ​​generates an illustration of Peter carrying a backpack and walking through the forest.

[0984] Subject: Emotion Engine

[0985] Specific behavior:

[0986] The emotion engine rechecks the generated text and illustrations and fine-tunes them to better suit the user's emotions. For example, if the user is expressing a happy emotion, the engine will add humor to the text.

[0987] Step 4: Creating a layout for your picture book

[0988] Subject: Server

[0989] Specific behavior:

[0990] The server combines the generated text and illustrations to create a printable layout, including a title page, body pages, and other decorative elements, and can include specific characters (e.g., the child himself) if the user so desires.

[0991] Specific behavior:

[0992] The completed text and illustrations are arranged on each page, and decorations such as backgrounds and headings are added. If a specific character appears, an illustration of that character is also added.

[0993] Step 5: Arrange for printing and mailing

[0994] Subject: Server

[0995] Specific behavior:

[0996] The server sends the completed picture book layout data to the printing service and carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[0997] Specific behavior:

[0998] The printing service prints the layout data and delivers the picture book to the address specified by the user.

[0999] Step 6: Getting feedback

[1000] Subject: Device

[1001] Specific behavior:

[1002] The device sends the user a notification requesting feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts and opinions in a feedback form. This feedback includes emotions, which the system analyzes and reflects in the next generation.

[1003] Subject: User

[1004] Specific behavior:

[1005] Users receive a notification, fill out a feedback form with their thoughts and opinions about the picture book, and send it to the system via their terminal.

[1006] Subject: Server

[1007] Specific behavior:

[1008] The server analyzes the received feedback and uses it to generate the next picture book. The emotion engine also analyzes this feedback to understand changes in the user's emotions.

[1009] These steps allow users to easily create original picture books with emotionally appropriate content, and the system is continually improved through feedback.

[1010] Example 2

[1011] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1012] Conventional picture book creation systems automatically generate text and illustrations based on a synopsis entered by the user, but because they do not take into account the user's emotions, it is difficult to create a personalized picture book that meets the user's expectations. In addition, there is no way to check the text and illustrations generated based on the user's emotions and make fine adjustments as necessary.

[1013] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1014] In this invention, the server includes means for a user to input a synopsis, means for automatically generating text and illustrations from the synopsis using a generative AI model, means for combining the automatically generated text and illustrations to create a picture book layout, means for transmitting the picture book layout data for printing and mailing, means for recognizing the user's emotions and optimizing the synopsis and settings based on the user's emotions, and means for checking the generated text and illustrations based on the user's emotions and fine-tuning them as necessary, thereby enabling the creation of a personalized picture book optimized based on the user's emotions.

[1015] A "user" is an entity that uses the system to input the storyline and characters of a picture book.

[1016] A "generative AI model" is an artificial intelligence algorithm that automatically generates text and illustrations from an input synopsis.

[1017] A "synopsis" is a short sentence that describes the outline of the story of a picture book.

[1018] "Sentence" refers to text that is automatically generated by the generative AI model based on the plot.

[1019] "Illustrations" are images automatically generated by a generative AI model based on text generation.

[1020] "Layout" is a format for combining automatically generated text and illustrations to arrange them into a picture book.

[1021] "Printing and mailing" is the process of producing a physical picture book based on the completed picture book layout data and delivering it to a specified address.

[1022] The "emotion engine" is an algorithm that recognizes the user's emotions from the synopsis and character information entered by the user and suggests optimizations.

[1023] "Optimization" means adjusting the input plot and settings based on the user's emotions.

[1024] "Fine-tuning" means rechecking the generated text and illustrations and correcting the content if necessary.

[1025] This invention is a system that allows users to easily create personalized picture books by combining a generative AI model and an emotion engine. This system generates optimized text and illustrations that take emotions into consideration based on the plot and character information entered by the user, and then creates a picture book layout that combines them. A detailed explanation of the system is provided below.

[1026] This system is mainly run by a server, a terminal, and a user. The general flow of the system is as follows: user input, optimization by an emotion engine, generation of text and illustrations by generative AI, generation of picture book layout, printing and mailing, and obtaining feedback.

[1027] The device provides the user with an interface for inputting the storyline and character information of the picture book. The user inputs the storyline and character information into input fields displayed on the device, and the data is sent from the device to the server. This data can be sent in an appropriate data format such as JSON.

[1028] After receiving user input data from the device, the server launches the emotion engine and analyzes the data. The emotion engine recognizes the user's emotions using natural language processing libraries (e.g., NLTK, BERT, etc.). If the input content does not match the user's emotions, the emotion engine makes optimization suggestions to the server and generates optimization candidates. The server presents them to the user, and if the user approves, the optimized synopsis proceeds to the next process.

[1029] The server then receives the optimized synopsis and uses a generative AI model (e.g., GPT-3, DALL-E, etc.) to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI first creates a story for each page based on the synopsis, and then the illustration generation AI creates appropriate illustrations based on the generated text. The generated text and illustrations are then checked again by the emotion engine and fine-tuned as necessary.

[1030] The server uses the generated text and illustrations to create a layout for the book. This is done using a layout engine (e.g., LaTeX or an automated script in InDesign). The layout includes a title page, body pages, and other decorative elements. Additionally, the user can configure the layout to include specific characters of their choice.

[1031] The completed picture book layout data is converted to PDF format and sent to the printing service API. The printing service then prints the picture book based on that data and delivers it to the address specified by the user. At this time, the user is notified of a tracking number so that they can track the delivery status.

[1032] After receiving the picture book, the device sends a notification to the user requesting feedback. The user enters their impressions of the book into a feedback form, and the information is sent to the server. The feedback also includes the user's emotions, which the emotion engine analyzes and reflects in the next generation process.

[1033] Specific examples

[1034] For example, suppose a user inputs a story about animals' adventures as a synopsis. The server first checks the user's emotions using the emotion engine to determine whether the input synopsis is appropriate for the user's current emotions. If the server determines that the user is feeling a little depressed, the emotion engine will suggest changing it to a fun adventure story. Based on the optimized synopsis, the generative AI model will generate specific sentences and illustrations like the one below.

[1035] Sentence Generation: "One day, Peter the rabbit went out in search of hidden treasure deep in the forest."

[1036] Illustration generation: A picture of Peter walking through the forest with a backpack on.

[1037] The generated text and illustrations are arranged on the title page and body pages using layout generation, and then sent to a printing service. The book is then mailed to the user's address, and the user receives their original picture book. If the user wishes to create another picture book, they can also create a sequel as a series.

[1038] Prompt Sentence Examples

[1039] "Please provide a synopsis of a children's animal adventure story. Please also include information about the characters."

[1040] The present invention allows users to easily create and enjoy original picture books that are personalized to their own emotions.

[1041] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1042] Step 1:

[1043] The device provides the user with an interface for inputting the storyline of the picture book. The user uses this interface to input the storyline and character information. The input data is sent to the server in JSON format by the device.

[1044] Input: User-entered plot and character information

[1045] Output: Data sent to the server in JSON format

[1046] Specifically, the user enters "Synopsis: Animals go on a great adventure in the forest" and the characters "Peter the rabbit, Reina the fox, and Beatrice the bear" into the interface. The device converts this information into JSON data "{'summary': 'Animals go on a great adventure in the forest', 'characters': ['Peter', 'Reina', 'Beatrice']}" and sends it to the server.

[1047] Step 2:

[1048] The server analyzes the user input data received from the device. The server then activates an emotion engine to analyze this data and recognize the user's emotions. The emotion engine uses a natural language processing library to analyze the input text and output the results. If necessary, it generates optimization suggestions, which the server presents to the user.

[1049] Input: User-entered data sent from the terminal

[1050] Output: Emotion recognition results from the emotion engine and optimization suggestions as needed

[1051] Specifically, the server receives the data "{'summary': 'Animals have a great adventure in the forest', 'characters': ['Peter', 'Reina', 'Beatrice']}" and, if the server uses the emotion engine to recognize that the user's emotion is "depressed," it generates a suggestion such as "Why not change it to a more cheerful adventure story?"

[1052] Step 3:

[1053] The server presents the emotion engine's suggestions to the user, and the user approves or modifies the suggestions, resulting in optimized data that is then sent back to the server. Based on the optimized synopsis received by the server, the generative AI model is used to automatically generate the text and illustrations for the picture book. First, the text generation AI generates a story for each page, and then the illustration generation AI generates illustrations based on the story.

[1054] Input: User-approved or revised optimized synopsis

[1055] Output: Text and illustrations generated by a generative AI model

[1056] Specifically, the user approves the optimization proposal, and based on the optimized plot summary, "Animals have a fun adventure," the text generation AI generates text such as "One day, Peter the rabbit went out to search for hidden treasure deep in the forest." The illustration generation AI then uses that text to generate an illustration of Peter walking through the forest with a backpack on.

[1057] Step 4:

[1058] The server combines the generated text and illustrations to create a picture book layout. It uses a layout engine to automatically generate a layout including a title page, body pages, and decorative elements. The server then converts the completed layout into PDF format and sends it to the print service API.

[1059] Input: Text and illustrations generated by a generative AI model

[1060] Output: Layout data in PDF format sent to the print service API

[1061] Specifically, the server generates a title page called "Peter and the Forest Treasure" and a layout with text and illustrations on each page, and sends it in PDF format to the "Printing Service API."

[1062] Step 5:

[1063] The printing service receives the layout data in PDF format, prints the picture book, and delivers it to the address specified by the user. The user is then notified of the tracking number so that they can track the delivery status.

[1064] Input: PDF layout data sent from the server

[1065] Output: Printed book and tracking number notification to user

[1066] Specifically, the printing service prints the picture book based on the PDF and ships it to the user's address. The user can then check the delivery status using the tracking number provided.

[1067] Step 6:

[1068] The device sends a notification to the user requesting feedback on the picture book. After receiving and reading the book, the user enters their thoughts in a feedback form, and the information is sent to the server. The server analyzes the feedback using an emotion engine and reflects it in the next generation process.

[1069] Input: User feedback and emotional information

[1070] Output: The results of the emotion engine analysis and improvements to be reflected in the next generation process

[1071] Specifically, the device sends a notification to the user saying, "How did you like the picture book?", and the user enters feedback such as, "My child really enjoyed it. I'd like to add more animal species next time." The server analyzes this information and reflects it in the next generation process.

[1072] (Application example 2)

[1073] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1074] Conventional picture book generation systems lacked the ability to personalize the plot and character information entered by the user, making it particularly difficult to generate optimal content that reflected the user's emotions.Furthermore, there was no mechanism for incorporating feedback from the generated picture book into the next generation.

[1075] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a user to input a summary, a means for automatically generating text and images from the summary using a generative AI model, a means for creating a reading material layout by combining the automatically generated text and images, and a means for recognizing the user's emotions using an emotion engine and optimizing the summary and character settings. This makes it possible to generate personalized reading material that is optimized for the user's emotions.

[1076] A "user" is an entity that uses the system to generate reading material, for example, someone who inputs information via a smart device.

[1077] The "summary" includes a brief summary of the story and information about the characters entered by the user.

[1078] A "generative AI model" is an artificial intelligence program that automatically generates text and images based on a summary entered by the user.

[1079] The "emotion engine" is a program that recognizes the user's emotions and optimizes the story outline and character settings based on those emotions.

[1080] "Sentence" refers to the text portion of the story generated by the generative AI model.

[1081] "Imagery" refers to visual illustrations or pictures related to the narrative generated by the generative AI model.

[1082] "Reading material layout" refers to creating a page layout for a reading material by combining automatically generated text and images.

[1083] "Printing" refers to the process of producing a reading material in a physical form based on the layout data of the generated reading material.

[1084] "Mail" means delivery of printed materials to the address specified by the user.

[1085] "Personalization" refers to providing content that is optimized to the user's emotions and preferences.

[1086] An embodiment of the present invention provides a system that allows users to easily create and view personalized reading material through a picture book streaming app using a smartphone or head-mounted display (HMD).

[1087] System Overview:

[1088] Hardware configuration:

[1089] Smartphone

[1090] Head-mounted display (HMD)

[1091] server

[1092] Software configuration:

[1093] Emotion Engine

[1094] Generative AI Models

[1095] Web frameworks (e.g., Flask, Django)

[1096] REST API

[1097] Process flow:

[1098] 1. Getting user input:

[1099] Using an application on a device (smartphone or HMD), the user inputs a summary of the story, for example, "One day, Peter the rabbit went out to search for a hidden treasure deep in the forest."

[1100] Additionally, the user can input their current emotions, choosing from options such as "happy," "sad," or "angry."

[1101] 2. Emotional engine optimization:

[1102] The server uses an emotion engine to recognize the user's emotion, and then checks whether the summary entered by the user is appropriate for that emotion.

[1103] If necessary, the sentiment engine suggests optimizing the summary based on the user's sentiment.

[1104] 3. Generative AI model for generating text and images:

[1105] The server receives the summary optimized by the emotion engine, and the generative AI model generates narrative text and images based on the summary.

[1106] For example, a text generation AI generates the sentence, "One day, Peter the rabbit went out to search for hidden treasure deep in the forest," and an image generation AI generates an illustration depicting that scene.

[1107] 4. Generating reading material arrangements:

[1108] The server combines the generated text and images to create the page layout of the reading material, including the title page and body pages.

[1109] 5. Getting feedback:

[1110] The system asks users for feedback on the generated reading material, including their feelings, which the emotion engine analyzes and reflects in the next generation.

[1111] Example prompt sentence:

[1112] "Generate a fun adventure story about Peter the Rabbit's search for hidden treasure."

[1113] This invention is a system that combines an emotion engine and a generative AI model to provide highly personalized reading material to users. By linking smart devices with a server, content can be generated based on the user's emotions, and further optimized content can be provided through feedback.

[1114] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1115] Step 1:

[1116] The device provides an interface for the user to input a story summary and their current emotions. Using this interface, the user can input a summary such as "One day, Peter the rabbit went out to search for hidden treasure deep in the forest" and the emotion "fun." This input information is sent to the server. The input is the user's story summary and emotions, and the output is the user's input data sent to the server.

[1117] Step 2:

[1118] The server sends the received user summary and emotion to the emotion engine. The emotion engine checks the suitability of the input summary based on the user's emotion and optimizes it if necessary. For example, if the user inputs "fun," the emotion engine suggests a title and character settings that increase the fun factor. In this step, the input is the user's summary and emotion, and the output is an optimized summary suggestion.

[1119] Step 3:

[1120] The server receives the summary optimized by the emotion engine and sends it to the generative AI model. The generative AI model generates story text and images based on this optimized summary. For example, the text generation part generates the sentence "One day, Peter the rabbit went out to search for hidden treasure deep in the forest," and the image generation part generates an illustration depicting that scene. The input is the optimized summary, and the output is the generated text and images.

[1121] Step 4:

[1122] The server combines the generated text and images to automatically create page layouts for the reading material, including title and body pages. For example, the title page displays the title "The Adventures of Peter Rabbit," and the body pages combine the generated text with corresponding illustrations. The input is the generated text and images, and the output is the page layout.

[1123] Step 5:

[1124] The server provides the layout data of the completed book to the user through the streaming app. If the user wishes, it can also send it to a printing service and mail the physical book to the user's address. In this step, the input is the layout data of the book, and the output is the provision of the book in streaming format and arrangement of a printing service.

[1125] Step 6:

[1126] The device sends a notification to the user requesting feedback on the generated reading material. After reading the reading material, the user enters their impressions in a feedback form. This feedback is sent to the server and reflected in the next generation. The input is the user's feedback, and the output is the feedback data analyzed by the emotion engine.

[1127] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1128] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1129] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1130] [Fourth embodiment]

[1131] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1132] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1133] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1134] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1135] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1136] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1137] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1138] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1139] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1140] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1141] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1142] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1143] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1144] To solve the problem that the invention aims to solve, the present invention provides a system that allows users to easily create picture books using generative AI. This system is executed through the following processing steps. Each processing step is mainly executed by a terminal, a server, and a user.

[1145] Getting User Input

[1146] The terminal provides the user with an interface for inputting the storyline of the picture book. The user inputs the storyline and character information into this interface. The terminal acquires this input information and sends it to the server.

[1147] Generative AI generates text and illustrations

[1148] The server receives the synopsis from the user and uses a generation AI to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI generates a story for each page based on the synopsis, and the illustration generation AI then creates appropriate illustrations based on the generated text.

[1149] Picture book layout generation

[1150] The server combines the generated text and illustrations and creates a printable layout, including a title page, body pages, and other decorative elements. If desired, the user can include specific characters (e.g., the child himself).

[1151] Printing and mailing arrangements

[1152] The server sends the layout data of the completed picture book to the printing service, which then carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[1153] Get feedback

[1154] The device asks the user for feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts in a feedback form. This feedback is sent to the server and used to improve the system and the generation AI.

[1155] Specific examples

[1156] For example, if a user inputs a story about an animal's adventure, the server can receive the story and use the AI ​​to generate specific sentences and illustrations like the one below.

[1157] 1. Sentence Generation: "One day, Peter the rabbit went out in search of a hidden treasure deep in the forest."

[1158] 2. Illustration generation: A picture of Peter walking through the forest with a backpack on.

[1159] The generated text and illustrations are arranged on the title page and body pages using layout generation, and then sent to a printing service. The book is then mailed to the user's address, and the user receives their original picture book. If the user wishes to create another picture book, they can also create a sequel as a series.

[1160] In this way, the present invention realizes a system that allows users to easily create original picture books and provide entertainment for children.

[1161] The processing flow will be explained below.

[1162] Step 1: User enters synopsis

[1163] Subject: User

[1164] Specific behavior:

[1165] The user inputs the story synopsis through the system interface, which is a text input form where the user writes a brief summary of the story and information about the characters.

[1166] Subject: Device

[1167] Specific behavior:

[1168] The terminal collects the synopsis entered by the user and transmits it to the server.

[1169] Step 2: Generating text and illustrations using generative AI

[1170] Subject: Server

[1171] Specific behavior:

[1172] The server receives the synopsis sent by the user, then uses a text generation AI model to automatically generate text for each page of the picture book based on the synopsis, and then uses an illustration generation AI model to automatically generate illustrations that match the generated text.

[1173] Examples:

[1174] For example, if the plot is "Animals on an adventure searching for hidden treasure in the forest," the text generation AI will generate the sentence "Peter the rabbit went searching for hidden treasure deep in the forest." Next, the illustration generation AI will generate an illustration depicting a scene of Peter walking through the forest.

[1175] Step 3: Creating a picture book layout

[1176] Subject: Server

[1177] Specific behavior:

[1178] The server combines the generated text and illustrations to create a printable picture book layout, including a title page, body pages, and other decorative elements, and can even include specific characters (such as the child) if the user so desires.

[1179] Examples:

[1180] The generated text, "Peter the Rabbit Searches for Treasure," is then integrated with the illustrations and placed appropriately on each page of the picture book. Backgrounds and headings are also added to the pages.

[1181] Step 4: Arrange for printing and mailing

[1182] Subject: Server

[1183] Specific behavior:

[1184] The server sends the completed picture book layout data to the printing service and carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[1185] Examples:

[1186] The completed layout is sent to a printing company, which prints the picture book and delivers it to an address specified by the user.

[1187] Step 5:

[1188] Get feedback

[1189] Subject: Device

[1190] Specific behavior:

[1191] The device sends the user a notification requesting feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts in a feedback form.

[1192] Subject: User

[1193] Specific behavior:

[1194] Users receive a notification, fill out a feedback form with their thoughts and opinions about the picture book, and send it to the system via their terminal.

[1195] Subject: Server

[1196] Specific behavior:

[1197] The server receives feedback sent by users and analyzes it to improve the quality of the system and the generated AI.

[1198] These steps allow users to easily create and enjoy original picture books, and the system is continually improved through feedback.

[1199] Example 1

[1200] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1201] The traditional picture book creation process is time-consuming and laborious, requiring specialized knowledge, making it difficult for ordinary users to easily create original picture books. Another issue is that it is difficult to improve the system based on user feedback.

[1202] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1203] In this invention, the server includes means for a user to input a synopsis, means for automatically generating text and illustrations from the synopsis using a generative AI model, means for creating a picture book layout by combining the automatically generated text and illustrations, means for transmitting the picture book layout data for printing and mailing, and means for obtaining feedback from the user regarding the generated picture book. This enables even general users to easily create original picture books, and further enables continuous improvement of the system based on the obtained feedback.

[1204] "User" refers to an individual or group that uses the picture book creation system to create a picture book based on their own storyline.

[1205] A "synopsis" is a short story description entered by the user as an overview of the picture book's contents.

[1206] A "generative AI model" is an artificial intelligence model that automatically generates text and illustrations from a synopsis entered by the user.

[1207] "Text generation AI" is the part of the generative AI model that generates detailed text based on the synopsis entered by the user.

[1208] "Illustration generation AI" is the part of the generative AI model that depicts appropriate images based on the generated text.

[1209] "Layout" refers to the design that combines the generated text and illustrations and arranges them in a printable format.

[1210] A "printing service" is a company or system that provides a service of printing actual picture books based on layout data sent from a server and mailing them to users.

[1211] "Feedback" refers to the impressions and suggestions for improvement provided by users about the picture books they receive, and is data for future system improvements.

[1212] The "system" refers to the entire set of components that execute a series of processes, including user input, generation of text and illustrations using a generative AI model, creation of layout, arrangement of printing and mailing, and obtaining feedback.

[1213] "Character" means a person, animal, or fictional being that appears in the synopsis entered by the user and in the generated text and illustrations.

[1214] "Serialization" is the process of creating multiple picture books in succession as sequels to a previous picture book.

[1215] MODE FOR CARRYING OUT THE INVENTION

[1216] The present invention relates to a system that allows users to easily create original picture books. The system is implemented through the following specific steps. Each step is mainly performed by a terminal, a server, and a user.

[1217] 1. Getting User Input

[1218] The device provides the user with an interface for entering information about the storyline and characters in the book. This interface includes input fields, drop-down menus, character selection options, etc. The user enters the appropriate information into these fields and clicks the "Submit" button. The device then packages the entered information as JSON-formatted data and sends it to the server via a secure HTTP request.

[1219] 2. Generating text and illustrations using generative AI

[1220] The server processes the synopsis information sent from the device and passes it to the generative AI model. The generative AI model includes a text generation AI and an illustration generation AI. The text generation AI generates a detailed story based on the synopsis, and the illustration generation AI draws appropriate illustrations based on the content of the generated text. At this time, a prompt sentence is input into the generative AI model to generate the necessary data.

[1221] Specific examples

[1222] The server generates the following prompt: "The characters are Peter the rabbit, Phil the fox, and Oliver the owl. Describe their adventure in search of a hidden treasure deep in the forest. The scene begins with Peter putting on his backpack and setting off."

[1223] This prompt is fed into the AI ​​model, which generates the following text and illustration:

[1224] 1. Sentence Generation: "One day, Peter the rabbit went out in search of a hidden treasure deep in the forest."

[1225] 2. Illustration generation: A picture of Peter walking through the forest with a backpack on.

[1226] 3. Picture book layout generation

[1227] The server combines the generated text and illustrations and lays them out in a printable format, taking into account design elements such as pagination, font settings, and illustration positioning. A PDF file is generated using layout editing software (e.g., Adobe InDesign) or a custom layout engine.

[1228] 4. Printing and mailing arrangements

[1229] The server sends the completed picture book layout data to a partner printing service. The printing service prints the picture book based on the received PDF file and arranges for it to be mailed to the user's specified address. Information such as the tracking number is returned to the server and notified to the user.

[1230] 5. Get feedback

[1231] After the picture book arrives, the device displays a notification requesting feedback. The user enters their thoughts and suggestions for improvement in the feedback form on the device. The entered feedback is sent back to the server and used to improve the system in the future.

[1232] Specific actions

[1233] After the user finishes reading the picture book, the device displays a notification saying, "Please tell us what you thought of the picture book." The user fills in the feedback form, saying, "The illustrations were great, but the story was a little short." After submitting the form, the feedback is saved on the server.

[1234] In this way, the present invention allows users to easily create and receive original picture books. Furthermore, the feedback function allows for continuous improvement of the system. The entire system is linked by users, terminals, servers, and printing services, enabling smooth picture book creation and distribution.

[1235] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1236] Step 1: Getting User Input

[1237] The device provides the user with an interface for inputting the storyline and character information of the picture book. The user fills in the storyline and character information in the input fields on this interface and clicks the send button.

[1238] Input: Synopsis text, character information (e.g. "A story about the adventures of animals" or "Peter the rabbit, Phil the fox, Oliver the owl")

[1239] Output: JSON format data (e.g., { "storybook": { "title": "Animal Adventures", "summary": "Animal Adventures", "characters": ["Peter the Rabbit", "Phil the Fox", "Oliver the Owl"]}}")

[1240] Step 2: Generating text and illustrations using generative AI

[1241] The server receives the plot information sent from the device and passes it to the generative AI model, which includes a text generation AI and an illustration generation AI, and uses these to generate the story and illustrations.

[1242] Input: Synopsis data in JSON format

[1243] Data processing: Input plot and character information into the generative AI model as prompts (e.g., "The characters are Peter the rabbit, Phil the fox, and Oliver the owl. Describe their adventure as they search for a hidden treasure deep in the forest. The story begins with Peter putting on his backpack and setting off.").

[1244] Output: Generated text and illustration (e.g., text "One day, Peter the rabbit went out to look for hidden treasure deep in the forest.", illustration "Peter is walking through the forest with his backpack on.")

[1245] Step 3: Creating a picture book layout

[1246] The server combines the generated text and illustrations and lays them out in a printable format, generating a PDF file while taking into account design elements such as pagination, font settings, and illustration positioning.

[1247] Input: Generated text and illustrations

[1248] Data processing: Create layouts using layout editing software (e.g. Adobe InDesign) or custom layout engines, and arrange pages, fonts, and illustrations.

[1249] Output: A printable PDF file (e.g. "storybook.pdf")

[1250] Step 4: Arrange for printing and mailing

[1251] The server sends the completed picture book layout data to a partner printing service, which prints the picture book based on the PDF file and mails it to the address specified by the user.

[1252] Input: Printable PDF file, user address information

[1253] Data processing: Send the PDF file to a printing service and provide shipping details

[1254] Output: Shipped notification and tracking information (e.g., "Shipped notification, tracking number 123456789")

[1255] Step 5: Getting feedback

[1256] After receiving the picture book, the device displays a notification requesting feedback. The user enters their thoughts and suggestions for improvement in the feedback form and submits it. The feedback information is stored on the server and used to improve the system.

[1257] Input: User impressions and suggestions for improvement entered in the feedback form (e.g., "The illustrations were great, but the story was a little short.")

[1258] Data calculation: Feedback information is sent to the server and stored in the database

[1259] Output: Saved feedback information (e.g., "{ "feedback": { "user_id": "001", "comments": "The illustrations were great, but the story was a little short"}}")

[1260] Through these steps, users can easily create and receive original picture books, and the feedback function allows for continuous improvement of the system.

[1261] (Application example 1)

[1262] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1263] Conventional picture book production systems lack the means for users to easily create original picture books and distribute them digitally, making it difficult to reach many users and distribute them instantly.

[1264] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1265] In this invention, the server includes means for a user to input a synopsis, means for automatically generating text and illustrations from the synopsis using a generative AI model, means for combining the automatically generated text and illustrations to create a picture book layout, means for saving the generated picture book in a digital format and sharing it with other users, and means for distributing it as an e-book in cooperation with an external content distribution service. This allows users to easily create original picture books and instantly distribute them in digital format to many users.

[1266] "User" refers to the person who enters the storyline and character information of the picture book.

[1267] A "generative AI model" refers to an artificial intelligence model used to automatically generate text and illustrations from a synopsis.

[1268] "Sentence" refers to text automatically generated by a generative AI model.

[1269] "Illustrations" refer to pictures or images automatically generated by generative AI models.

[1270] "Picture book layout" refers to the page composition that combines and arranges generated text and illustrations.

[1271] "Digital format" refers to a data format that can be stored and displayed electronically.

[1272] "External content distribution service" refers to a third-party online platform for publishing and selling the generated picture books as e-books.

[1273] "Sharing" refers to the act of sharing the generated picture book with other users.

[1274] This invention is a system that allows users to easily create original picture books and store and distribute them in digital form. A specific embodiment for realizing this system will be described below.

[1275] Getting User Input

[1276] The user inputs the plot and character information using their own device (e.g., a smartphone). The device provides an interface for the user to perform input operations. The device then transmits the input information to the server.

[1277] Generating text and illustrations using a generative AI model

[1278] The server uses a generative AI model to automatically generate text and illustrations based on the synopsis and character information received from the user. Specifically, the server uses the "generative AI model" to generate text and illustrations by inputting the following prompts:

[1279] Example prompt sentence:

[1280] Generate a picture book story with the following synopsis: A brave little cat has an adventure in a big city. Characters: Timmy the Cat

[1281] Picture book layout generation

[1282] The generated text and illustrations are integrated by the server to create a picture book layout, which is then stored in digital format.

[1283] Save and share digitally

[1284] The server stores the layout of the created picture book in a digital format and provides a means for sharing it with other users, allowing the user to share the created picture book with other users.

[1285] Collaboration with content distribution services

[1286] Furthermore, the server also provides a means to link with external content distribution services and distribute the created picture books as e-books, allowing users to instantly and widely distribute the picture books they have created.

[1287] System Configuration

[1288] Hardware: Smartphone (user input), Server (data processing and storage)

[1289] Software: Generative AI model (generating text and illustrations), interface app (user input)

[1290] Data processing and calculation:

[1291] The synopsis and character information entered by the user is obtained as text data.

[1292] Using a generative AI model, text and illustrations are generated based on input data.

[1293] Consolidate generated data and store and distribute it digitally.

[1294] Specific examples of program processing

[1295] For example, if a user inputs the synopsis "A brave little cat goes on an adventure in a big city," the generative AI model will generate the sentence "One day, Timmy, a brave little cat, heads to a big city and starts his adventure," along with an illustration of the cat on an adventure. The generated sentences and illustrations are then integrated into the layout of the picture book by the server and saved in digital format.

[1296] In this way, the present invention realizes a system that enables users to easily create original picture books and instantly distribute them in digital form.

[1297] ---

[1298] This explanation corresponds to the "Form for carrying out the invention."

[1299] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1300] Step 1:

[1301] The user uses a smartphone or PC to input the storyline and character information into the interface. The information entered by the user is saved as text data on the device. This text data is then sent to the server.

[1302] Step 2:

[1303] The server analyzes the received text data. Specifically, it creates a prompt sentence for the generative AI model based on the synopsis and character information entered by the user. The prompt sentence is input into the generative AI model, and the picture book text is automatically generated. The generated text is saved as text data within the server.

[1304] Step 3:

[1305] The server creates a prompt for generating illustrations based on the generated text. The prompt is input into an illustration generation AI model, which automatically generates illustrations for the picture book. The generated illustrations are saved as image data on the server.

[1306] Step 4:

[1307] The server integrates the generated text and illustrations and automatically creates a picture book layout. This layout arranges the text and illustrations on specific pages, forming a single picture book as a whole. The generated layout data is saved in digital format.

[1308] Step 5:

[1309] The server provides a means for users to share the digital data of the created picture book with other users. This means allows users to share the picture book they created with other users. This data is also linked to external content distribution services and distributed as an e-book.

[1310] Step 6:

[1311] The server also collects feedback from users. After a user finishes reading a picture book, they enter their impressions and opinions into a feedback form and send it to the server. This feedback data is used to improve the generative AI model and the overall quality of the system.

[1312] Through the above steps, the present invention allows users to easily create original picture books and share and distribute them in digital format.

[1313] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1314] This invention combines a system that allows users to easily create picture books using generative AI with an emotion engine that recognizes the user's emotions, thereby enabling the creation of more personalized picture books. This system is executed through the following processing steps. Each processing step is mainly executed by the terminal, server, and user.

[1315] Getting User Input

[1316] The terminal provides the user with an interface for inputting the storyline of the picture book. The user inputs the storyline and character information into this interface. The terminal acquires this input information and sends it to the server.

[1317] Optimization by Emotion Engine

[1318] Before receiving the synopsis sent by the user, the server uses an emotion engine to recognize the user's emotions. This allows the server to check whether the input synopsis and character settings are appropriate for the user's emotions. If necessary, the emotion engine suggests optimizing the synopsis and settings based on the user's emotions.

[1319] Generative AI generates text and illustrations

[1320] The server receives a synopsis optimized based on the user's emotions and uses a generation AI to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI generates a story for each page based on the synopsis, and the illustration generation AI then creates appropriate illustrations based on the generated text. The emotion engine then rechecks the generated text and illustrations and makes fine adjustments as necessary.

[1321] Picture book layout generation

[1322] The server combines the generated text and illustrations and creates a printable layout, including a title page, body pages, and other decorative elements. Additionally, the layout can include specific characters (e.g., the child himself) if desired.

[1323] Printing and mailing arrangements

[1324] The server sends the layout data of the completed picture book to the printing service, which then carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[1325] Get feedback

[1326] The device sends the user a notification requesting feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts in a feedback form. The feedback also includes the user's emotions, which the emotion engine analyzes and reflects in the next generation.

[1327] Specific examples

[1328] For example, suppose a user inputs "a story about animals' adventures" as the synopsis. The server first checks the user's emotions using the emotion engine and determines whether the input synopsis is appropriate for the user's current emotions. If the server determines that the user is feeling a little depressed, the emotion engine will suggest, "Why not change it to a fun adventure story?"

[1329] Based on the optimized synopsis, the generative AI model generates specific sentences and illustrations like the following:

[1330] 1. Sentence Generation: "One day, Peter the rabbit went out in search of a hidden treasure deep in the forest."

[1331] 2. Illustration generation: A picture of Peter walking through the forest with a backpack on.

[1332] The generated text and illustrations are arranged on the title page and body pages using layout generation, and then sent to a printing service. The book is then mailed to the user's address, and the user receives their original picture book. If the user wishes to create another picture book, they can also create a sequel as a series.

[1333] In this way, by combining emotion engines, we can create a system that provides even more personalized, original picture books that best suit the user's emotions.

[1334] The processing flow will be explained below.

[1335] Step 1: User enters synopsis

[1336] Subject: User

[1337] Specific behavior:

[1338] The user inputs the story synopsis through the system interface, which is a text input form where the user writes a brief summary of the story and information about the characters.

[1339] Subject: Device

[1340] Specific behavior:

[1341] The terminal collects the synopsis entered by the user and transmits it to the server.

[1342] Step 2: Optimizing with an Emotional Engine

[1343] Subject: Server

[1344] Specific behavior:

[1345] Before receiving the synopsis sent by the user, the server activates the emotion engine and collects data to recognize the user's emotions. If the device has a built-in camera or microphone, the emotion engine analyzes facial expressions and voice tones to check whether the synopsis entered by the user is appropriate for the user's emotions.

[1346] Examples:

[1347] For example, if a user inputs "forest adventure," the emotion engine will read emotions such as "fun" or "exciting" from the user's facial expressions and tone, and determine whether the synopsis is appropriate based on those emotions. If necessary, the emotion engine will make optimization suggestions such as "How about adding more specific adventure elements?"

[1348] Step 3: Generative AI generates text and illustrations

[1349] Subject: Server

[1350] Specific behavior:

[1351] The server receives the synopsis optimized based on the user's emotions and uses a generation AI to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI generates a story for each page based on the synopsis, and the illustration generation AI creates illustrations based on the generated text.

[1352] Specific behavior:

[1353] Based on the plot, the AI ​​generates the sentence "Peter the rabbit went in search of hidden treasure deep in the forest," and then the AI ​​generates an illustration of Peter carrying a backpack and walking through the forest.

[1354] Subject: Emotion Engine

[1355] Specific behavior:

[1356] The emotion engine rechecks the generated text and illustrations and fine-tunes them to better suit the user's emotions. For example, if the user is expressing a happy emotion, the engine will add humor to the text.

[1357] Step 4: Creating a layout for your picture book

[1358] Subject: Server

[1359] Specific behavior:

[1360] The server combines the generated text and illustrations to create a printable layout, including a title page, body pages, and other decorative elements, and can include specific characters (e.g., the child himself) if the user so desires.

[1361] Specific behavior:

[1362] The completed text and illustrations are arranged on each page, and decorations such as backgrounds and headings are added. If a specific character appears, an illustration of that character is also added.

[1363] Step 5: Arrange for printing and mailing

[1364] Subject: Server

[1365] Specific behavior:

[1366] The server sends the completed picture book layout data to the printing service and carries out the procedure to mail the picture book to the user's address. The printing service prints the picture book based on the received layout data and mails it to the specified address.

[1367] Specific behavior:

[1368] The printing service prints the layout data and delivers the picture book to the address specified by the user.

[1369] Step 6: Getting feedback

[1370] Subject: Device

[1371] Specific behavior:

[1372] The device sends the user a notification requesting feedback on the completed picture book. After receiving and reading the book, the user enters their thoughts and opinions in a feedback form. This feedback includes emotions, which the system analyzes and reflects in the next generation.

[1373] Subject: User

[1374] Specific behavior:

[1375] Users receive a notification, fill out a feedback form with their thoughts and opinions about the picture book, and send it to the system via their terminal.

[1376] Subject: Server

[1377] Specific behavior:

[1378] The server analyzes the received feedback and uses it to generate the next picture book. The emotion engine also analyzes this feedback to understand changes in the user's emotions.

[1379] These steps allow users to easily create original picture books with emotionally appropriate content, and the system is continually improved through feedback.

[1380] Example 2

[1381] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1382] Conventional picture book creation systems automatically generate text and illustrations based on a synopsis entered by the user, but because they do not take into account the user's emotions, it is difficult to create a personalized picture book that meets the user's expectations. In addition, there is no way to check the text and illustrations generated based on the user's emotions and make fine adjustments as necessary.

[1383] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1384] In this invention, the server includes means for a user to input a synopsis, means for automatically generating text and illustrations from the synopsis using a generative AI model, means for combining the automatically generated text and illustrations to create a picture book layout, means for transmitting the picture book layout data for printing and mailing, means for recognizing the user's emotions and optimizing the synopsis and settings based on the user's emotions, and means for checking the generated text and illustrations based on the user's emotions and fine-tuning them as necessary, thereby enabling the creation of a personalized picture book optimized based on the user's emotions.

[1385] A "user" is an entity that uses the system to input the storyline and characters of a picture book.

[1386] A "generative AI model" is an artificial intelligence algorithm that automatically generates text and illustrations from an input synopsis.

[1387] A "synopsis" is a short sentence that describes the outline of the story of a picture book.

[1388] "Sentence" refers to text that is automatically generated by the generative AI model based on the plot.

[1389] "Illustrations" are images automatically generated by a generative AI model based on text generation.

[1390] "Layout" is a format for combining automatically generated text and illustrations to arrange them into a picture book.

[1391] "Printing and mailing" is the process of producing a physical picture book based on the completed picture book layout data and delivering it to a specified address.

[1392] The "emotion engine" is an algorithm that recognizes the user's emotions from the synopsis and character information entered by the user and suggests optimizations.

[1393] "Optimization" means adjusting the input plot and settings based on the user's emotions.

[1394] "Fine-tuning" means rechecking the generated text and illustrations and correcting the content if necessary.

[1395] This invention is a system that allows users to easily create personalized picture books by combining a generative AI model and an emotion engine. This system generates optimized text and illustrations that take emotions into consideration based on the plot and character information entered by the user, and then creates a picture book layout that combines them. A detailed explanation of the system is provided below.

[1396] This system is mainly run by a server, a terminal, and a user. The general flow of the system is as follows: user input, optimization by an emotion engine, generation of text and illustrations by generative AI, generation of picture book layout, printing and mailing, and obtaining feedback.

[1397] The device provides the user with an interface for inputting the storyline and character information of the picture book. The user inputs the storyline and character information into input fields displayed on the device, and the data is sent from the device to the server. This data can be sent in an appropriate data format such as JSON.

[1398] After receiving user input data from the device, the server launches the emotion engine and analyzes the data. The emotion engine recognizes the user's emotions using natural language processing libraries (e.g., NLTK, BERT, etc.). If the input content does not match the user's emotions, the emotion engine makes optimization suggestions to the server and generates optimization candidates. The server presents them to the user, and if the user approves, the optimized synopsis proceeds to the next process.

[1399] The server then receives the optimized synopsis and uses a generative AI model (e.g., GPT-3, DALL-E, etc.) to automatically generate the text and illustrations for the picture book. Specifically, the text generation AI first creates a story for each page based on the synopsis, and then the illustration generation AI creates appropriate illustrations based on the generated text. The generated text and illustrations are then checked again by the emotion engine and fine-tuned as necessary.

[1400] The server uses the generated text and illustrations to create a layout for the book. This is done using a layout engine (e.g., LaTeX or an automated script in InDesign). The layout includes a title page, body pages, and other decorative elements. Additionally, the user can configure the layout to include specific characters of their choice.

[1401] The completed picture book layout data is converted to PDF format and sent to the printing service API. The printing service then prints the picture book based on that data and delivers it to the address specified by the user. At this time, the user is notified of a tracking number so that they can track the delivery status.

[1402] After receiving the picture book, the device sends a notification to the user requesting feedback. The user enters their impressions of the book into a feedback form, and the information is sent to the server. The feedback also includes the user's emotions, which the emotion engine analyzes and reflects in the next generation process.

[1403] Specific examples

[1404] For example, suppose a user inputs a story about animals' adventures as a synopsis. The server first checks the user's emotions using the emotion engine to determine whether the input synopsis is appropriate for the user's current emotions. If the server determines that the user is feeling a little depressed, the emotion engine will suggest changing it to a fun adventure story. Based on the optimized synopsis, the generative AI model will generate specific sentences and illustrations like the one below.

[1405] Sentence Generation: "One day, Peter the rabbit went out in search of hidden treasure deep in the forest."

[1406] Illustration generation: A picture of Peter walking through the forest with a backpack on.

[1407] The generated text and illustrations are arranged on the title page and body pages using layout generation, and then sent to a printing service. The book is then mailed to the user's address, and the user receives their original picture book. If the user wishes to create another picture book, they can also create a sequel as a series.

[1408] Prompt Sentence Examples

[1409] "Please provide a synopsis of a children's animal adventure story. Please also include information about the characters."

[1410] The present invention allows users to easily create and enjoy original picture books that are personalized to their own emotions.

[1411] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1412] Step 1:

[1413] The device provides the user with an interface for inputting the storyline of the picture book. The user uses this interface to input the storyline and character information. The input data is sent to the server in JSON format by the device.

[1414] Input: User-entered plot and character information

[1415] Output: Data sent to the server in JSON format

[1416] Specifically, the user enters "Synopsis: Animals go on a great adventure in the forest" and the characters "Peter the rabbit, Reina the fox, and Beatrice the bear" into the interface. The device converts this information into JSON data "{'summary': 'Animals go on a great adventure in the forest', 'characters': ['Peter', 'Reina', 'Beatrice']}" and sends it to the server.

[1417] Step 2:

[1418] The server analyzes the user input data received from the device. The server then activates an emotion engine to analyze this data and recognize the user's emotions. The emotion engine uses a natural language processing library to analyze the input text and output the results. If necessary, it generates optimization suggestions, which the server presents to the user.

[1419] Input: User-entered data sent from the terminal

[1420] Output: Emotion recognition results from the emotion engine and optimization suggestions as needed

[1421] Specifically, the server receives the data "{'summary': 'Animals have a great adventure in the forest', 'characters': ['Peter', 'Reina', 'Beatrice']}" and, if the server uses the emotion engine to recognize that the user's emotion is "depressed," it generates a suggestion such as "Why not change it to a more cheerful adventure story?"

[1422] Step 3:

[1423] The server presents the emotion engine's suggestions to the user, and the user approves or modifies the suggestions, resulting in optimized data that is then sent back to the server. Based on the optimized synopsis received by the server, the generative AI model is used to automatically generate the text and illustrations for the picture book. First, the text generation AI generates a story for each page, and then the illustration generation AI generates illustrations based on the story.

[1424] Input: User-approved or revised optimized synopsis

[1425] Output: Text and illustrations generated by a generative AI model

[1426] Specifically, the user approves the optimization proposal, and based on the optimized plot summary, "Animals have a fun adventure," the text generation AI generates text such as "One day, Peter the rabbit went out to search for hidden treasure deep in the forest." The illustration generation AI then uses that text to generate an illustration of Peter walking through the forest with a backpack on.

[1427] Step 4:

[1428] The server combines the generated text and illustrations to create a picture book layout. It uses a layout engine to automatically generate a layout including a title page, body pages, and decorative elements. The server then converts the completed layout into PDF format and sends it to the print service API.

[1429] Input: Text and illustrations generated by a generative AI model

[1430] Output: Layout data in PDF format sent to the print service API

[1431] Specifically, the server generates a title page called "Peter and the Forest Treasure" and a layout with text and illustrations on each page, and sends it in PDF format to the "Printing Service API."

[1432] Step 5:

[1433] The printing service receives the layout data in PDF format, prints the picture book, and delivers it to the address specified by the user. The user is then notified of the tracking number so that they can track the delivery status.

[1434] Input: PDF layout data sent from the server

[1435] Output: Printed book and tracking number notification to user

[1436] Specifically, the printing service prints the picture book based on the PDF and ships it to the user's address. The user can then check the delivery status using the tracking number provided.

[1437] Step 6:

[1438] The device sends a notification to the user requesting feedback on the picture book. After receiving and reading the book, the user enters their thoughts in a feedback form, and the information is sent to the server. The server analyzes the feedback using an emotion engine and reflects it in the next generation process.

[1439] Input: User feedback and emotional information

[1440] Output: The results of the emotion engine analysis and improvements to be reflected in the next generation process

[1441] Specifically, the device sends a notification to the user saying, "How did you like the picture book?", and the user enters feedback such as, "My child really enjoyed it. I'd like to add more animal species next time." The server analyzes this information and reflects it in the next generation process.

[1442] (Application example 2)

[1443] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1444] Conventional picture book generation systems lacked the ability to personalize the plot and character information entered by the user, making it particularly difficult to generate optimal content that reflected the user's emotions.Furthermore, there was no mechanism for incorporating feedback from the generated picture book into the next generation.

[1445] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for a user to input a summary, a means for automatically generating text and images from the summary using a generative AI model, a means for creating a reading material layout by combining the automatically generated text and images, and a means for recognizing the user's emotions using an emotion engine and optimizing the summary and character settings. This makes it possible to generate personalized reading material that is optimized for the user's emotions.

[1446] A "user" is an entity that uses the system to generate reading material, for example, someone who inputs information via a smart device.

[1447] The "summary" includes a brief summary of the story and information about the characters entered by the user.

[1448] A "generative AI model" is an artificial intelligence program that automatically generates text and images based on a summary entered by the user.

[1449] The "emotion engine" is a program that recognizes the user's emotions and optimizes the story outline and character settings based on those emotions.

[1450] "Sentence" refers to the text portion of the story generated by the generative AI model.

[1451] "Imagery" refers to visual illustrations or pictures related to the narrative generated by the generative AI model.

[1452] "Reading material layout" refers to creating a page layout for a reading material by combining automatically generated text and images.

[1453] "Printing" refers to the process of producing a reading material in a physical form based on the layout data of the generated reading material.

[1454] "Mail" means delivery of printed materials to the address specified by the user.

[1455] "Personalization" refers to providing content that is optimized to the user's emotions and preferences.

[1456] An embodiment of the present invention provides a system that allows users to easily create and view personalized reading material through a picture book streaming app using a smartphone or head-mounted display (HMD).

[1457] System Overview:

[1458] Hardware configuration:

[1459] Smartphone

[1460] Head-mounted display (HMD)

[1461] server

[1462] Software configuration:

[1463] Emotion Engine

[1464] Generative AI Models

[1465] Web frameworks (e.g., Flask, Django)

[1466] REST API

[1467] Process flow:

[1468] 1. Getting user input:

[1469] Using an application on a device (smartphone or HMD), the user inputs a summary of the story, for example, "One day, Peter the rabbit went out to search for a hidden treasure deep in the forest."

[1470] Additionally, the user can input their current emotions, choosing from options such as "happy," "sad," or "angry."

[1471] 2. Emotional engine optimization:

[1472] The server uses an emotion engine to recognize the user's emotion, and then checks whether the summary entered by the user is appropriate for that emotion.

[1473] If necessary, the sentiment engine suggests optimizing the summary based on the user's sentiment.

[1474] 3. Generative AI model for generating text and images:

[1475] The server receives the summary optimized by the emotion engine, and the generative AI model generates narrative text and images based on the summary.

[1476] For example, a text generation AI generates the sentence, "One day, Peter the rabbit went out to search for hidden treasure deep in the forest," and an image generation AI generates an illustration depicting that scene.

[1477] 4. Generating reading material arrangements:

[1478] The server combines the generated text and images to create the page layout of the reading material, including the title page and body pages.

[1479] 5. Getting feedback:

[1480] The system asks users for feedback on the generated reading material, including their feelings, which the emotion engine analyzes and reflects in the next generation.

[1481] Example prompt sentence:

[1482] "Generate a fun adventure story about Peter the Rabbit's search for hidden treasure."

[1483] This invention is a system that combines an emotion engine and a generative AI model to provide highly personalized reading material to users. By linking smart devices with a server, content can be generated based on the user's emotions, and further optimized content can be provided through feedback.

[1484] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1485] Step 1:

[1486] The device provides an interface for the user to input a story summary and their current emotions. Using this interface, the user can input a summary such as "One day, Peter the rabbit went out to search for hidden treasure deep in the forest" and the emotion "fun." This input information is sent to the server. The input is the user's story summary and emotions, and the output is the user's input data sent to the server.

[1487] Step 2:

[1488] The server sends the received user summary and emotion to the emotion engine. The emotion engine checks the suitability of the input summary based on the user's emotion and optimizes it if necessary. For example, if the user inputs "fun," the emotion engine suggests a title and character settings that increase the fun factor. In this step, the input is the user's summary and emotion, and the output is an optimized summary suggestion.

[1489] Step 3:

[1490] The server receives the summary optimized by the emotion engine and sends it to the generative AI model. The generative AI model generates story text and images based on this optimized summary. For example, the text generation part generates the sentence "One day, Peter the rabbit went out to search for hidden treasure deep in the forest," and the image generation part generates an illustration depicting that scene. The input is the optimized summary, and the output is the generated text and images.

[1491] Step 4:

[1492] The server combines the generated text and images to automatically create page layouts for the reading material, including title and body pages. For example, the title page displays the title "The Adventures of Peter Rabbit," and the body pages combine the generated text with corresponding illustrations. The input is the generated text and images, and the output is the page layout.

[1493] Step 5:

[1494] The server provides the layout data of the completed book to the user through the streaming app. If the user wishes, it can also send it to a printing service and mail the physical book to the user's address. In this step, the input is the layout data of the book, and the output is the provision of the book in streaming format and arrangement of a printing service.

[1495] Step 6:

[1496] The device sends a notification to the user requesting feedback on the generated reading material. After reading the reading material, the user enters their impressions in a feedback form. This feedback is sent to the server and reflected in the next generation. The input is the user's feedback, and the output is the feedback data analyzed by the emotion engine.

[1497] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1498] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1499] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1500] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1501] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1502] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1503] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1504] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1505] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1506] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1507] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1508] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1509] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1510] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1511] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1512] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1513] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1514] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1515] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1516] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1517] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1518] The following is further disclosed regarding the above embodiment.

[1519] (Claim 1)

[1520] a means for a user to input a synopsis;

[1521] A means for automatically generating text and illustrations from the synopsis using a generative AI model;

[1522] A method for creating a picture book layout by combining automatically generated text and illustrations,

[1523] means for transmitting the layout data of the picture book for printing and mailing;

[1524] A system including:

[1525] (Claim 2)

[1526] The system of claim 1, further comprising means for causing the user and specific characters to appear in the text and illustrations.

[1527] (Claim 3)

[1528] The system according to claim 1, further comprising means for serializing the generated picture book to create a plurality of picture books.

[1529] "Example 1"

[1530] (Claim 1)

[1531] a means for a user to input a synopsis;

[1532] A means for automatically generating text and illustrations from the synopsis using a generative AI model;

[1533] A method for creating a picture book layout by combining automatically generated text and illustrations,

[1534] means for transmitting the layout data of the picture book for printing and mailing;

[1535] A means for obtaining feedback from users regarding the generated picture book;

[1536] A system including:

[1537] (Claim 2)

[1538] 10. The system of claim 1, further comprising means for causing a particular character to appear in said text and illustrations.

[1539] (Claim 3)

[1540] The system according to claim 1, further comprising means for serializing the generated picture book to create a plurality of picture books.

[1541] "Application Example 1"

[1542] (Claim 1)

[1543] a means for a user to input a synopsis;

[1544] A means for automatically generating text and illustrations from the synopsis using a generative AI model;

[1545] A method for creating a picture book layout by combining automatically generated text and illustrations,

[1546] A means for digitally storing the generated picture book and sharing it with other users;

[1547] A means of distributing it as an e-book by linking with an external content distribution service,

[1548] A system including:

[1549] (Claim 2)

[1550] The system of claim 1, further comprising means for causing the user and specific characters to appear in the text and illustrations.

[1551] (Claim 3)

[1552] The system according to claim 1, further comprising means for serializing the generated picture book to create a plurality of picture books.

[1553] "Example 2: Combining Emotion Engines"

[1554] (Claim 1)

[1555] a means for a user to input a synopsis;

[1556] A means for automatically generating text and illustrations from the synopsis using a generative AI model;

[1557] A method for creating a picture book layout by combining automatically generated text and illustrations,

[1558] means for transmitting the layout data of the picture book for printing and mailing;

[1559] means for recognizing a user's emotion and optimizing the plot and setting based on the user's emotion;

[1560] A means for checking the generated text and illustrations based on the user's feelings and fine-tuning them as necessary;

[1561] A system including:

[1562] (Claim 2)

[1563] The system of claim 1, further comprising means for causing the user and specific characters to appear in the text and illustrations.

[1564] (Claim 3)

[1565] The system according to claim 1, further comprising means for serializing the generated picture book to create a plurality of picture books.

[1566] "Application example 2 when combining emotion engines"

[1567] (Claim 1)

[1568] a means for the user to input a summary;

[1569] means for automatically generating text and images from the summary using a generative AI model;

[1570] A means of creating a reading arrangement by combining automatically generated text and images;

[1571] means for transmitting layout data of said reading material for printing and mailing;

[1572] A means for recognizing a user's emotion using an emotion engine and optimizing the outline and character settings;

[1573] A system including:

[1574] (Claim 2)

[1575] 10. The system of claim 1, further comprising means for causing the user and a particular character to appear in the text and images.

[1576] (Claim 3)

[1577] 10. The system of claim 1, further comprising means for serializing the generated readings to create multiple readings. [Explanation of symbols]

[1578] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for a user to input a synopsis; A means for automatically generating text and illustrations from the synopsis using a generative AI model; A method for creating a picture book layout by combining automatically generated text and illustrations, means for transmitting the layout data of the picture book for printing and mailing; A system including:

2. The system of claim 1 further comprising means for causing the user and specific characters to appear in the text and illustrations.

3. The system according to claim 1 , further comprising means for serializing the generated picture books to create a plurality of picture books.

Images