System

Abstract

A system is provided.SOLUTION: This system is provided with a means for receiving image data transmitted by a user, a means for preprocessing the received image data, a means for generating characters by an image generating device with the preprocessed image data as input, a means for generating a story by a sentence generating device with the generated characters and background information as input, and a means for combining the story and the character information. A system comprising: means for automatically generating a page layout for a picture book; means for providing an electronic version of the generated picture book to a user; and means for receiving user customization information and generating a final version of the picture book that reflects the information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, children have limited opportunities to freely express their creativity and ideas, and self-expression through drawings is particularly underutilized at home and at school. Furthermore, capturing these drawings in a picture book requires time and money, making it an inaccessible method for many families. Furthermore, existing methods for creating picture books have limited educational value in deepening parent-child relationships and supporting children's development. To address these challenges, a system is needed that quickly and affordably converts children's drawings into picture books. [Means for solving the problem]

[0005] The present invention provides a system including: means for receiving image data sent by a user; means for preprocessing the received image data; means for generating characters using an image generation device with the preprocessed image data as input; means for generating a story using a text generation device with the generated characters and background information as input; means for automatically generating a page layout for a picture book by combining the story and character information; means for providing a user with an electronic version of the generated picture book; and means for receiving customization information from the user and generating a final picture book that reflects the information. This system allows for the rapid and affordable creation of high-quality, custom picture books from children's drawings, fostering children's creativity and expressiveness and deepening parent-child bonds. Furthermore, by including means for accepting orders for printed versions of the generated picture books and handling delivery procedures, and means for setting additional parameters related to the personalities and atmosphere of the characters for story generation, it is possible to provide services that meet a wider variety of needs.

[0006] A "user" is a person who provides images and customization information to create a picture book using the service.

[0007] "Image data" is a digital representation (e.g., JPEG, PNG) of a child's drawing.

[0008] "Means for receiving" refers to the process by which the server obtains image data and other information sent by the user.

[0009] "Preprocessing" is the process of converting received image data into a state suitable for AI processing, such as adjusting resolution and removing noise.

[0010] An "image generation device" is a device that identifies characters and backgrounds from preprocessed image data and generates character models and background data.

[0011] "Characters" are characters extracted and generated from image data provided by users.

[0012] "Background information" refers to background elements extracted and generated by the image generation device from areas other than the characters.

[0013] A "text generation device" is a device that automatically generates text based on generated character and background information.

[0014] A "story" is a story that weaves together characters and background information generated by a text generation device.

[0015] "Page layout" is the process of determining the design and composition of each page of a picture book based on information about the story and characters.

[0016] "Automatic generation" refers to the process of utilizing AI technology to mechanically create page layouts and integrate information without human intervention.

[0017] An "electronic version" is a picture book provided in digital format that users can view online.

[0018] "Customization information" refers to changes made by the user in response to individual requests, such as name changes or specific settings.

[0019] The "final version" is the completed picture book that reflects the user's customization information.

[0020] A "printed version" is a picture book that is provided in physical printed form on paper.

[0021] The "means for accepting orders" is the process of accepting and processing orders for printed versions from users.

[0022] The "means for carrying out delivery procedures" refers to the procedures for delivering the printed version of the picture book to a location designated by the user.

[0023] "Additional parameters" are setting values ​​used when generating a story, such as character personalities and background atmosphere. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0025] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0026] First, the terms used in the following description will be explained.

[0027] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0028] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0029] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0030] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0031] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0032] [First embodiment]

[0033] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0034] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0035] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0036] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0037] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0038] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0039] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0040] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0041] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0042] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0043] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0044] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] This invention is a system that quickly and affordably generates high-quality custom picture books based on user-provided drawings by children. The system consists of a series of processes: the user submits the drawings, the system classifies them into characters and backgrounds, creates a story based on the generated information, and finally provides the picture book.

[0046] Program processing

[0047] Process for users to post pictures

[0048] The user uses the website or application of the custom picture book creation service to upload a picture drawn by their child. The device then sends the image data of the picture to the server.

[0049] Picture pre-processing

[0050] The server processes the received image data, adjusting the image resolution and format to convert it into a state suitable for AI processing.

[0051] Character generation using image generation AI

[0052] The server inputs the preprocessed images into an image generation AI to identify elements in the picture (such as characters and backgrounds), and generates each element as an independent character model or background as needed.

[0053] Story generation using text generation AI

[0054] The server uses text generation AI to automatically generate a story based on the generated characters and background information. The AI ​​creates a storyline based on the characteristics of the characters and the atmosphere of the illustrations.

[0055] Picture book compilation and preview generation

[0056] The server combines the generated story with the original illustrations to create a page layout, and creates a digital preview that can be sent to the user's device for review.

[0057] User verification and customization

[0058] The user checks the preview data and, if necessary, makes corrections or additional customizations. The customization details are sent from the device to the server.

[0059] Final generation and delivery

[0060] The server generates the final picture book data that reflects the user's customizations and provides it for viewing on the web. Printed versions are also available as an optional paid option.

[0061] Specific examples

[0062] 1. The process of posting a picture

[0063] The parent user takes a photo of their child's drawing of a "blue dragon" with their smartphone and uploads it to the service's app.

[0064] 2. Picture preprocessing

[0065] After the server receives the uploaded image, it adjusts the resolution to 300 dpi and converts it to PNG format.

[0066] 3. Character Creation Using Image Generation AI

[0067] The server uses image generation AI to extract the "blue dragon" as the main character and converts its features into a digital model.

[0068] 4. Story Generation Using Text Generation AI

[0069] The server then has the text generation AI create a story with the "blue dragon" as the main character. For example, a story with the theme of "the blue dragon's adventure to save his friend" is generated.

[0070] 5. Picture book compilation and preview generation

[0071] The server integrates the generated story with the original illustrations, creates a page-by-page layout, and generates a digital preview that is sent to the user.

[0072] 6. User Verification and Customization

[0073] The user sees the preview and decides they want to change the dragon's name to "Drago." They make the changes in the app and submit the changes to the server.

[0074] 7. Final Generation and Delivery

[0075] The server generates the final version of the picture book data that reflects the user's changes and provides it for viewing on the web. If the user wants a printed version, they can order it for a fee.

[0076] In this way, children's drawings are quickly turned into engaging custom picture books.

[0077] The processing flow will be explained below.

[0078] Step 1:

[0079] Users access the service's website or app and log in. After logging in, they click the "Upload a picture" button, which opens a file selection dialog.

[0080] Step 2:

[0081] The user selects and uploads an image file of the picture drawn by the child, and the device sends the selected image file to the server.

[0082] Step 3:

[0083] The server checks the received image data, checking the image format and resolution, and displays an error message if they are inappropriate.

[0084] Step 4:

[0085] The server performs preprocessing on the image data, including adjusting the resolution and filtering noise. The preprocessed image data is then temporarily stored.

[0086] Step 5:

[0087] The server inputs the preprocessed image data into the image generation AI, which detects the main elements in the image (such as characters and backgrounds) and generates each element as an independent character model or background.

[0088] Step 6:

[0089] The server stores the generated character models and background data, which are then input into the text generation AI.

[0090] Step 7:

[0091] The server sets the necessary parameters for the text generation AI (e.g., character personality, atmosphere, etc.), and the text generation AI automatically generates a story based on the provided information.

[0092] Step 8:

[0093] The server combines the generated story with character models and background data to automatically generate a layout for each page.

[0094] Step 9:

[0095] The server creates a digital preview of the generated picture book, temporarily stores it, and generates a preview link and notifies the user.

[0096] Step 10:

[0097] Users click the link sent to them to see the digital preview, customize it as needed (e.g., change the character's name), and enter their details.

[0098] Step 11:

[0099] The terminal transmits the customization information entered by the user to the server.

[0100] Step 12:

[0101] The server reflects the received customization information in the final version of the picture book, generates the final version of the picture book data, and provides it for viewing on the web.

[0102] Step 13:

[0103] The user can check the final version of the picture book and download it if necessary. If the user wants a printed version, the terminal sends the order information to the server.

[0104] Step 14:

[0105] The server accepts the print order and initiates the necessary printing and delivery procedures. After the order is completed, the server notifies the user of the delivery status.

[0106] These steps allow us to quickly generate high-quality custom picture books from children's drawings.

[0107] Example 1

[0108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0109] In the conventional picture book creation process, it was difficult for users to easily and quickly generate custom picture books based on their children's drawings. Furthermore, there was a lack of automatic generation functions using digitalization and AI, forcing users to perform time-consuming and laborious manual work. This made it difficult to provide custom picture books quickly and with high quality.

[0110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0111] In this invention, the server includes means for receiving image data sent by a user, means for preprocessing the received image data, means for generating characters using an image generation device with the preprocessed image data as input, means for generating a story using a text generation device with the generated characters and background information as input, means for automatically generating a page layout for a picture book by combining the story and character information, means for providing the user with an electronic version of the generated picture book, means for receiving customization information from the user and generating a final version of the picture book that reflects the information, means for transmitting input data from the terminal to the server, and data processing and conversion means in the server. This enables the rapid automatic generation of a high-quality customized picture book from pictures provided by the user.

[0112] "User" refers to an individual or group that uses the system to send image data of children's drawings.

[0113] "Image data" refers to the data used to digitally store and send pictures drawn by children to the system.

[0114] "Preprocessing" refers to the process performed to convert received image data into a state suitable for AI processing. Specifically, this includes adjusting the resolution and converting the format.

[0115] An "image generation device" is a device or software that takes preprocessed image data as input, analyzes the image, and extracts and generates characters and backgrounds.

[0116] "Characters" are characters extracted from the image data sent by the user, and appear in the story of the picture book.

[0117] "Background information" refers to background elements and settings extracted from the image data sent by the user.

[0118] A "story generation device" is a device or software that automatically generates a story using generated characters and background information as input.

[0119] A "story" is a story that is automatically generated by the text generation device based on the generated characters and background information.

[0120] "Page layout" refers to the design that determines the layout of each page of a picture book by combining the generated story and character information.

[0121] "Electronic Edition" means a digital version of the generated picture book that can be viewed or downloaded by a user.

[0122] "Customization information" refers to information that a user inputs after checking a preview to reflect corrections or additional changes.

[0123] "Device" means the device used by a User to capture, upload, view, and customize Pictures, including smartphones and computers.

[0124] A "server" is a computer system that performs a series of processes, such as processing transmitted image data, preprocessing, character generation, story generation, and page layout creation.

[0125] "Data processing and conversion means" refers to software and hardware for a series of data conversion, analysis, and processing performed on the server.

[0126] This invention is a system that quickly and affordably generates high-quality custom picture books based on user-provided drawings by children. The system consists of a series of processes: the user submits the drawings, the system classifies them into characters and backgrounds, creates a story based on the generated information, and finally provides the picture book.

[0127] This system uses the following hardware and software to process and calculate data.

[0128] First, a user uses a device (such as a smartphone or PC) to photograph or scan a picture drawn by their child, and then uses the custom picture book creation service's website or application to send the image data to the server. This image data is sent to the server using an HTTP POST request. The server then uses Python and its image processing library (such as Pillow) to perform preprocessing, such as adjusting the resolution of the received image data to 300 dpi and converting it from JPEG to PNG format.

[0129] The server then inputs the preprocessed image data into an image generation AI (such as DeepArt or a GAN-based system) to extract important elements (characters and background) from the image. During this process, elements such as a blue dragon and a castle are identified, and these elements are then generated as independent character and background models.

[0130] Furthermore, the server automatically generates a story using a text generation AI (for example, OpenAI's GPT-4) based on the identified characters and background information. Specifically, there is an example where a prompt sentence such as "An adventure in which a blue dragon saves his friend" is input into the text generation AI to create a storyline.

[0131] The server then combines the generated story with the original illustrations to create a page-by-page layout. During this process, a dedicated layout engine is used to generate a digital preview of the book for the user to review. The preview data is then sent to the device as an HTTP response.

[0132] The user checks the digital preview on their device and makes any necessary corrections or customizations (for example, changing the dragon's name to "Drago"). The customizations are sent back to the server, which then generates the final picture book data reflecting the user's customizations.

[0133] The final picture book data is provided to users in a format that can be viewed online from the server. If the user wishes to purchase a printed version, they can order it for a fee and have it delivered.

[0134] Examples of specific prompts are:

[0135] Create a story starring your child's blue dragon and the dragon's adventures helping his friends.

[0136] In this way, a system is realized that quickly generates high-quality custom picture books based on user-provided illustrations.

[0137] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0138] Step 1:

[0139] A user uses the website or application of a custom picture book creation service to take a photo or scan a picture drawn by their child with a smartphone or PC and upload it. The device sends the image data to the server via an HTTP POST request. The input to this step is the image data sent from the user's device, and the output is the image data received by the server. Specifically, the user launches the application, takes a photo or selects an image from the gallery, and clicks the upload button.

[0140] Step 2:

[0141] The server preprocesses the received image data. First, it adjusts the image resolution to 300 dpi and converts the image format from JPEG to PNG. This is done using Python's Pillow library, for example. The input to this step is the raw image data received by the server, and the output is preprocessed image data with the resolution adjusted and format converted. Specifically, it resizes and converts the image using functions from the Pillow library.

[0142] Step 3:

[0143] The server inputs the preprocessed image data into an image generation AI, which identifies key elements in the image (characters and background) and generates independent digital models of each. The input for this step is the preprocessed image data, and the output is digital models of the identified characters and background. Specifically, the data is input into DeepArt or a GAN-based AI model to perform classification and model generation.

[0144] Step 4:

[0145] The server inputs a prompt into a text generation AI based on the identified characters and background, automatically generating a story. For example, the prompt "A blue dragon goes on an adventure to save his friends" is input into a text generation AI such as GPT-4. The input for this step is a digital model of the characters and background and the prompt, and the output is the generated story text. Specifically, the prompt and character data are input into the text generation AI, and the generated story is received in text format.

[0146] Step 5:

[0147] The server combines the generated story with the original illustrations to create a layout for each page. In this process, a dedicated layout engine is used to generate a digital preview of the picture book. The input for this step is the story text and a digital model of the original illustrations, and the output is the completed digital preview. Specifically, the layout engine automatically creates the page structure and generates the preview.

[0148] Step 6:

[0149] The user checks the digital preview and makes any necessary corrections or customizations. For example, they input the customization details, such as changing the dragon's name to "Drago," and send it to the server. The input in this step is the user's customization instructions, and the output is data sent to the server that has received the correction instructions. Specifically, the user selects the part to be corrected within the application, enters the changes, and sends them.

[0150] Step 7:

[0151] The server generates the final picture book data that reflects the user's customizations and provides it in a format that can be viewed on the web. If the user wishes, it can also accept orders for a printed version as a paid option and handles the delivery procedures. The input for this step is the customization information received from the user, and the output is the final picture book data and print order data. Specific operations include generating the final data, uploading it to the web, and managing the printing and delivery procedures.

[0152] (Application example 1)

[0153] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0154] Conventional custom picture book creation systems lack the ability to easily share content created by users with other users, and the process of creating a custom picture book is often cumbersome and time-consuming. This makes it difficult for users to easily and quickly create custom picture books based on their children's drawings and share them with other family and friends. Therefore, to improve the user experience, there is a need for a system that allows users to create high-quality picture books through a simple process and share those picture books with other users.

[0155] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0156] In this invention, the server includes means for receiving image data sent by a user, means for preprocessing the received image data, means for generating characters using an image generation device with the preprocessed image data as input, means for generating a story using a text generation device with the generated characters and background information as input, means for automatically generating a page layout for a picture book by combining the story and character information, means for providing the user with an electronic version of the generated picture book, means for receiving customization information from the user and generating a final version of the picture book that reflects the information, and means for sharing the generated picture book with other users within the application. This enables users to quickly create high-quality custom picture books based on their children's drawings and easily share them with other family and friends.

[0157] "Submitted image data" refers to user-generated visual information that a user uploads to the custom picture book creation system.

[0158] "Preprocessing" is the act of converting received image data into an appropriate format and resolution, making it suitable for subsequent processing steps.

[0159] An "image generation device" is a device that analyzes image data received as input and automatically generates characters and backgrounds.

[0160] A "text generation device" is a device that creates a story based on the characters and background information generated by an image generation device.

[0161] "Page layout" is a layout design that automatically arranges the structure of each page of a picture book by combining story and character information.

[0162] An "electronic version" is a picture book that is provided in digital format rather than in physical print form.

[0163] "Customization information" is information that reflects modifications and additional requirements made by the user.

[0164] The "final version" is a completed picture book that reflects the customization information provided by the user.

[0165] "Means for sharing" refers to the ability to send created custom picture books to other users within the application.

[0166] The present invention is a system for quickly and affordably generating high-quality custom picture books from user-supplied drawings by children. Specific methods for implementing the present invention are described below.

[0167] First, a user uses a smartphone app to take a picture of a drawing their child has made and upload it to the system. The uploaded image data is sent from the device to the server. The server then preprocesses the received image data and adjusts the image resolution and format. This preprocessing is performed using image processing software such as PIL (Python Imaging Library).

[0168] The preprocessed image data is then fed into an image generator (e.g., a system using a specific AI model) that analyzes the image data and defines the characters and background. For example, if a child's drawing of a "blue dragon" is uploaded, the image generator will extract the "blue dragon" from the drawing as the main character and generate a digital model of it.

[0169] The generated characters and background information are then passed to a text generator. The text generator automatically generates a story based on the characteristics of the characters and the atmosphere of the illustration. At this stage, a generative AI model is used. For example, a story with the theme of "a blue dragon's adventure to save his friend" is generated. An example prompt might be, "Generate a story featuring a child's drawing of a 'blue dragon.' This dragon is brave and embarks on an adventure to save a friend in trouble. Draw what difficulties he faces during his adventure and how he overcomes them."

[0170] The generated story and character information is compiled into a picture book using an automatic page layout generator. The server combines these to create a page-by-page structure and generate a digital preview. The user can check this preview on their smartphone and make corrections or additional customizations as needed. The user's customization information is then sent to the server, and the final picture book is generated.

[0171] The server then provides the user with an electronic version of the final generated picture book. The application also includes a means for sharing the generated custom picture book with other users. For example, the user can easily share the generated picture book with family and friends. The server also provides functionality for the user to order a printed version of the generated picture book and arrange for delivery, if desired.

[0172] In this way, the present invention provides a system that enables users to quickly and easily create high-quality custom picture books according to their needs, and further enables users to share these picture books.

[0173] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0174] Step 1:

[0175] The user takes a photo of a drawing made by their child through a smartphone app. The user then uploads this image data to the app. The input is the image data taken with the smartphone, and the output is the image data sent to the server. The specific action that occurs at this point is for the user to tap the "Upload Image" button on the app and select or take a photo of the drawing.

[0176] Step 2:

[0177] The server receives image data sent by the user. The input is the uploaded image data, and the output is image data that can be preprocessed. Specifically, the server receives the image data and saves it in storage.

[0178] Step 3:

[0179] The server preprocesses the received image data, including adjusting the resolution and converting the format. The input is the original image data, and the output is the preprocessed image data. Specifically, the server converts the image to RGB using PIL (Python Imaging Library) and resizes the image to 1024x1024.

[0180] Step 4:

[0181] Based on the preprocessed image data, the server generates the characters using an image generation device. The input is the preprocessed image data, and the output is a digital model of the character. Specifically, the server inputs the image data into the AI ​​model, identifies the elements of the picture, and generates the digital model.

[0182] Step 5:

[0183] Based on the generated characters and background information, the server generates a story using a text generator. The input is the characters and background information, and the output is the generated story. Specifically, the server provides a prompt to the generative AI model, which then creates the story. An example of a prompt is, "Generate a story featuring a child's drawing of a 'blue dragon.' This dragon is brave and sets out on an adventure to help a friend in trouble. Draw what difficulties he faces during his adventure and how he overcomes them."

[0184] Step 6:

[0185] The server automatically generates page layouts based on story and character information. The input is story and character information, and the output is the page layout of a picture book. Specifically, the server integrates the generated story with image elements and automatically formats the layout of each page.

[0186] Step 7:

[0187] The server provides the generated digital version of the picture book to the user. The input is the page layout and story, and the output is the digital version of the picture book data. Specifically, the server generates a digital file of the picture book and sends it via the app so that the user can view it.

[0188] Step 8:

[0189] The user checks the digital version of the picture book and inputs customization information. The input is the digital version of the picture book data, and the output is the customization information specified by the user. Specifically, the user makes modifications such as changing the names of characters and text through the app interface.

[0190] Step 9:

[0191] The server generates the final version of the picture book by reflecting the user's customization information. The input is the user's customization information, and the output is the final version of the picture book data. Specifically, the server applies the received customization information to the picture book pages to complete the final version.

[0192] Step 10:

[0193] The server provides a means for users to share the final version of the picture book with other users within the application. The input is the final version of the picture book data, and the output is the shared picture book data. Specifically, the server generates a sharing link for the picture book, allowing users to send it to family and friends.

[0194] These steps result in a system that allows users to quickly create high-quality custom picture books based on their children's drawings and easily share them with other family and friends.

[0195] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0196] This invention is a system that quickly and affordably generates high-quality custom picture books based on user-provided drawings by children. The system consists of a series of processes: the user submits a picture, the system classifies it into characters and backgrounds, creates a story based on the generated information, and finally provides the picture book. The system also incorporates an emotion engine that recognizes the user's emotions and has the ability to adjust the content of the story and the facial expressions of the characters based on the user's emotions.

[0197] Program processing

[0198] Process for users to post pictures

[0199] The user uses the website or application of the custom picture book creation service to upload a picture drawn by their child. The device then sends the image data of the picture to the server.

[0200] Picture pre-processing

[0201] The server processes the received image data, adjusting the image resolution and format to convert it into a state suitable for AI processing.

[0202] Character generation using image generation AI

[0203] The server inputs the preprocessed images into an image generation AI to identify elements in the picture (such as characters and backgrounds), and generates each element as an independent character model or background as needed.

[0204] Story generation using text generation AI

[0205] The server uses text generation AI to automatically generate a story based on the generated characters and background information. The AI ​​creates a storyline based on the characteristics of the characters and the atmosphere of the illustrations.

[0206] Recognizing user emotions with an emotion engine

[0207] The device collects the user's facial expressions and voice through a camera and microphone, and sends this data to a server.

[0208] The server uses an emotion engine to analyze the received facial expression data and voice data and determine the user's emotions.

[0209] Emotion-based story adjustment

[0210] The server then adjusts the story and the characters' expressions based on the determined emotional information. For example, if the user is enjoying the story, it will brighten up the story.

[0211] Picture book compilation and preview generation

[0212] The server combines the generated story with the original illustrations to create a page layout, and creates a digital preview that can be sent to the user's device for review.

[0213] User verification and customization

[0214] The user checks the preview data and, if necessary, makes corrections or additional customizations. The customization details are sent from the device to the server.

[0215] Final generation and delivery

[0216] The server generates the final picture book data that reflects the user's customizations and provides it for viewing on the web. Printed versions are also available as an optional paid option.

[0217] Specific examples

[0218] 1. The process of posting a picture

[0219] The parent user takes a photo of their child's drawing of a "blue dragon" with their smartphone and uploads it to the service's app.

[0220] 2. Picture preprocessing

[0221] After the server receives the uploaded image, it adjusts the resolution to 300 dpi and converts it to PNG format.

[0222] 3. Character Creation Using Image Generation AI

[0223] The server uses image generation AI to extract the "blue dragon" as the main character and converts its features into a digital model.

[0224] 4. Story Generation Using Text Generation AI

[0225] The server then has the text generation AI create a story with the "blue dragon" as the main character. For example, a story with the theme of "the blue dragon's adventure to save his friend" is generated.

[0226] 5. User Emotion Recognition by Emotion Engine

[0227] As the user reads parts of the story, emotions are collected through facial expressions and voice, and each emotional state is sent to the server.

[0228] The server analyzes the user's emotions through an emotion engine and adjusts the story and characters.

[0229] 6. Adjust your story based on emotions

[0230] If the server determines that the user is enjoying themselves, it will change the story to a brighter, more enjoyable one, and the character's facial expressions will also change accordingly.

[0231] 7. Picture book compilation and preview generation

[0232] The server combines the generated story, the original illustrations, and the emotional adjustments to create a page-by-page layout, creating a digital preview and sending it to the user.

[0233] 8. User Verification and Customization

[0234] The user checks the preview, makes any necessary changes, and then sends the customizations back to the server.

[0235] 9. Final Generation and Delivery

[0236] The server generates the final version of the picture book that reflects the user's changes and provides it on the web. If the user wishes, it also accepts an order for a printed version and begins the delivery process.

[0237] In this way, a system is provided that can quickly generate high-quality custom picture books from children's drawings and flexibly adjust the content of the story according to the user's emotions.

[0238] The processing flow will be explained below.

[0239] Step 1:

[0240] Users access the service's website or app and log in. After logging in, they click the "Upload a picture" button, which opens a file selection dialog.

[0241] Step 2:

[0242] The user selects and uploads an image file of the picture drawn by the child, and the device sends the selected image file to the server.

[0243] Step 3:

[0244] The server checks the received image data, checking the image format and resolution, and displays an error message if they are inappropriate.

[0245] Step 4:

[0246] The server performs preprocessing on the image data, including adjusting the resolution and filtering noise. The preprocessed image data is then temporarily stored.

[0247] Step 5:

[0248] The server inputs the preprocessed image data into the image generation AI, which detects the main elements in the image (such as characters and backgrounds) and generates each element as an independent character model or background.

[0249] Step 6:

[0250] The server stores the generated character models and background data, which are then input into the text generation AI.

[0251] Step 7:

[0252] The server sets the necessary parameters for the text generation AI (e.g., character personality, atmosphere, etc.), and the text generation AI automatically generates a story based on the provided information.

[0253] Step 8:

[0254] The device collects the user's facial expressions and voice through a camera and microphone, and transmits the emotional data to a server.

[0255] Step 9:

[0256] The server uses an emotion engine to analyze the received facial expression data and voice data and determine the user's emotions.

[0257] Step 10:

[0258] The server then adjusts the story and the characters' expressions based on the determined emotional information. For example, if the user is enjoying the story, it will brighten up the story.

[0259] Step 11:

[0260] The server combines the generated story with character models, background data, and the results of adjustments made by the emotion engine to automatically generate a layout for each page.

[0261] Step 12:

[0262] The server creates a digital preview of the generated picture book, temporarily stores it, and generates a preview link and notifies the user.

[0263] Step 13:

[0264] Users click the link sent to them to see the digital preview, customize it as needed (for example, change the character's name), and enter their details.

[0265] Step 14:

[0266] The terminal transmits the customization information entered by the user to the server.

[0267] Step 15:

[0268] The server reflects the received customization information in the final version of the picture book, generates the final version of the picture book data, and provides it for viewing on the web.

[0269] Step 16:

[0270] The user can check the final version of the picture book and download it if necessary. If the user wants a printed version, the terminal sends the order information to the server.

[0271] Step 17:

[0272] The server accepts the print order and initiates the necessary printing and delivery procedures. After the order is completed, the server notifies the user of the delivery status.

[0273] The above steps provide a system that can quickly generate high-quality custom picture books from children's drawings and flexibly adjust the story content according to the user's emotions.

[0274] Example 2

[0275] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0276] Conventional custom picture book generation systems have difficulty generating high-quality picture books quickly based on user-provided images. Furthermore, they lack the ability to adjust the story and character expressions to reflect the user's emotions, making it impossible to provide a personalized experience. This has led to problems such as low user satisfaction and low retention rates.

[0277] The specification processing by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving image data sent by a user, means for preprocessing the received image data, means for generating characters using an image generation device with the preprocessed image data as input, means for generating a story using a text generation device with the generated characters and background information as input, means for recognizing the user's emotional state, means for adjusting the story and character facial expressions based on the recognized emotions, means for automatically generating a page layout for a picture book by combining the story and character information, means for providing the user with an electronic version of the generated picture book, and means for receiving customization information from the user and generating a final picture book reflecting the information. This makes it possible to quickly process images provided by the user and adjust the story and character facial expressions according to the user's emotions.

[0278] A "user" is an end user who uses the custom picture book creation service to upload a picture drawn by a child and request the creation of a picture book.

[0279] A "server" is a central processing unit that performs various processes such as image processing, data analysis, and AI models.

[0280] "Image data" is a digital image file of a child's drawing provided by the user.

[0281] "Preprocessing" refers to initial data processing to facilitate subsequent AI processing, such as adjusting the image data resolution, converting formats, removing noise, and correcting colors.

[0282] An "image generation device" is an AI device that extracts characters and backgrounds from preprocessed image data and generates digital character models and background models.

[0283] "Characters" are characters or objects depicted in a user-provided picture.

[0284] "Background information" refers to information about the overall scenery and environment of the picture, excluding the characters.

[0285] A "text generation device" is an AI device that automatically generates a story based on generated characters and background information.

[0286] A "story" is an automatically generated story based on the characteristics and background information of the characters.

[0287] The "emotional state" is an emotional state such as joy, surprise, sadness, etc., that is analyzed from the user's facial expression and voice.

[0288] The "emotion engine" is an AI engine that analyzes the user's facial expressions and voice data to determine their emotional state.

[0289] "Page layout" refers to the layout of each page of a picture book, which is created by combining the generated story and character information.

[0290] "Electronic version" refers to the final version of the picture book data provided in digital format.

[0291] "Customization information" refers to the contents and settings of the picture book that the user checks in the preview and adds or changes as necessary.

[0292] The "final version" is the completed picture book data that is generated by reflecting the user's customization information.

[0293] This invention is a system that quickly generates high-quality custom picture books based on user-provided drawings by children. The system works in cooperation with the user, device, and server elements, and uses multiple AI models to pre-process images, generate characters, generate stories, and adjust content based on the user's emotions.

[0294] Hardware and software used

[0295] Terminal

[0296] A device is a device used by a user, such as a smartphone, tablet, or PC. Users take and upload pictures through these devices. Devices are equipped with cameras and microphones, and also have the ability to collect facial expressions and voice data from users.

[0297] server

[0298] The server is the center of the picture book generation system and has multiple AI models and engines for performing the following processes:

[0299] Image pre-processing engine: Performs pre-processing such as image resolution adjustment, noise reduction, and color correction.

[0300] Image generation AI model: Generates characters and backgrounds from preprocessed image data.

[0301] Text generation AI model: Generates a story based on characters and background information.

[0302] Emotion engine: Analyzes and determines emotions from the user's facial expressions and voice data.

[0303] Implementation Procedure

[0304] Users submit pictures

[0305] Users take a photo of their child's drawing using a smartphone or computer and upload the image file using the custom picture book creation service's website or application.

[0306] Picture pre-processing

[0307] The server passes the received image data to a pre-processing engine, which sets the resolution to 300 dpi, converts the image format to PNG, and performs noise removal and color correction to prepare the image for processing by the AI ​​model.

[0308] Character generation using image generation AI

[0309] The server inputs the preprocessed images into an image generation AI model to identify the main characters and background, and reconstructs the characters' features as digital models.

[0310] Story generation using text generation AI

[0311] The server uses a text generation AI model to generate a story based on the characters and background information. The AI ​​is given prompts like the following:

[0312] "This image depicts a blue dragon. Please create an adventure story with this dragon as the main character."

[0313] "This blue dragon is a brave character who likes to help his friends. Generate a story based on this setting."

[0314] Recognizing user emotions with an emotion engine

[0315] When users preview their custom picture books, their facial expressions and voice are collected using the device's camera and microphone, and this data is sent to the server in real time and analyzed by the server's emotion engine.

[0316] Emotion-based story adjustment

[0317] The server dynamically changes the story and the characters' expressions based on the analysis results of the emotion engine. If the user is enjoying the story, it will adjust the story to be more cheerful and fun.

[0318] Picture book compilation and preview generation

[0319] The server automatically generates a layout for each page by integrating information about the story and characters, and then creates a digital preview that is sent to the device.

[0320] User verification and customization

[0321] The user checks the preview and edits text or images as necessary. The customization information made on the device is sent to the server.

[0322] Final generation and delivery

[0323] The server generates the final picture book data that reflects the customization information and provides it in a format that can be viewed on the web. If the user requests a printed version, the server generates print data and sends an order to the printing company.

[0324] In this way, user-provided pictures can be quickly transformed into high-quality custom picture books, with content that can be flexibly tailored based on the user's emotions.

[0325] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0326] Step 1: How users post pictures

[0327] Users take a photo of a picture their child has drawn using a smartphone or computer, and then upload the image file using the website or application of the custom picture book creation service. Specifically, users click the "Upload Picture" button, select an image file, and upload it. They then press the "Send" button to send the image data from their device to the server. The input is the user's image data, and the output is the image data sent to the server.

[0328] Step 2: Pre-processing the image

[0329] The server receives image data sent from the device and passes it to the preprocessing engine. It adjusts the resolution of the received image data, for example, setting it to 300 dpi. It also converts the image format from common formats such as JPEG or BMP to PNG. It applies a noise reduction filter and performs color correction. The input is the received image data, and the output is the image data after preprocessing.

[0330] Step 3: Creating characters using image generation AI

[0331] The server inputs the preprocessed image data into the image generation AI model and performs image analysis. Specifically, it inputs a prompt to "extract the main characters and convert them into digital models" into the image generation AI model. The AI ​​identifies the main characters and background in the image and converts their features into digital models. The input is the preprocessed image data, and the output is the generated digital models of the characters and background.

[0332] Step 4: Story generation using text generation AI

[0333] The server uses a text generation AI model to generate a story based on the generated characters and background information. For example, a prompt such as "This image depicts a blue dragon. Please create an adventure story with this dragon as the main character" is input and the AI ​​generates the story. Specifically, based on the characteristics and background information of the characters, a story including the story flow, character actions, dialogue, etc. is automatically created. The input is the characters and background information, and the output is the generated story.

[0334] Step 5: Recognizing user emotions with the emotion engine

[0335] While viewing a preview of a custom picture book, the user uses the device's camera and microphone to collect facial expressions and voice. The device transmits this emotional data to the server in real time. The server's emotion engine analyzes this data and determines the user's emotional state (e.g., joy, surprise, sadness). The input is the user's facial and voice data, and the output is the analyzed emotional state.

[0336] Step 6: Adjust your story based on emotion

[0337] The server receives the analysis results from the emotion engine and adjusts the story and character expressions. For example, if it determines that the user is enjoying themselves, it changes the story to a more cheerful and fun development and adjusts the character expressions accordingly. The input is the analyzed emotional state, and the output is the adjusted story and character expressions.

[0338] Step 7: Compiling the picture book and generating a preview

[0339] The server integrates the generated story, character models, and backgrounds to create the layout for each page. Specifically, it determines the placement of text and images and automatically generates a layout including design elements for each page. Finally, it generates a digital preview and sends it to the device. The input is the adjusted story, character models, and backgrounds, and the output is a digital preview.

[0340] Step 8: Verify and customize users

[0341] The user checks the digital preview sent on their device and makes changes as necessary, such as editing text, changing images, and adjusting character positions. Once the user has finalized the customizations, the data is resent from their device to the server. The input is the digital preview, and the output is data reflecting the customizations.

[0342] Step 9: Final generation and serving

[0343] The server generates the final picture book data that reflects the customization information. It then provides it to the user in a format that can be viewed on the web. If the user requests a printed version, the server generates print data and sends an order to the printer. The input is data that reflects the customization details, and the output is the final picture book data. It also includes order data for delivering the printed version to the user.

[0344] (Application example 2)

[0345] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0346] There is a demand for improving the efficiency and accuracy of the work that field workers face, but conventional systems have the problem that the work content and instructions are uniform, making it difficult to respond flexibly while taking into account the stress level and concentration of each worker. Furthermore, work instructions can be either too detailed or too simple, which can lead to a decline in work efficiency. In particular, it has been difficult to grasp the emotional state of field workers in real time and provide appropriate instructions based on that.

[0347] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving image data sent by a user; means for preprocessing the received image data; means for generating characters using an image generation device with the preprocessed image data as input; means for generating a story using a text generation device with the generated characters and background information as input; emotion recognition means for analyzing the user's emotional state; means for adjusting the facial expressions of the story and characters based on the emotional state; means for automatically generating a page layout for a picture book by combining the story and character information; means for providing a user with an electronic version of the generated picture book; and means for receiving customization information from the user and generating a final version of the picture book reflecting the information. This makes it possible to provide appropriate work instructions that flexibly correspond to the emotional state of field workers.

[0348] A "user" is a person who uses the service to send image data.

[0349] "Image data" refers to image information sent by a user.

[0350] "Preprocessing" refers to a series of processes that convert received image data into a state suitable for analysis.

[0351] An "image generation device" is a device for generating characters based on pre-processed image data.

[0352] "Characters" are characters generated from image data sent by the user.

[0353] "Background information" refers to information that corresponds to the background of an image generated by an image generation device.

[0354] A "sentence generation device" is a device that automatically generates a story based on generated characters and background information.

[0355] A "story" is a storyline generated by a text generator.

[0356] "Emotion recognition means" refers to means for analyzing the emotional state of a user.

[0357] "Page layout" is the process of combining story and character information to determine the page structure of a picture book.

[0358] "Digital edition" refers to a picture book provided in digital format.

[0359] "Customization information" refers to modifications or additions requested by the user to the electronic version.

[0360] The "final version" refers to the final picture book data generated by reflecting the customization information.

[0361] This invention is a system that uses emotion recognition to provide appropriate work instructions to field workers based on smart glasses. Specific embodiments of this system will be described below.

[0362] Hardware and software used

[0363] Hardware: Smart glasses, webcam

[0364] Software: OpenCV (image processing), DeepFace (emotion recognition), Flask server (data transmission and reception)

[0365] Data processing and calculation

[0366] 1. User: A field worker wears smart glasses and takes a photo of the work site. This photo data is captured by the camera in the smart glasses.

[0367] 2. Terminal: The smart glasses transmit the acquired image data to the server using wireless communication technologies such as Wi-Fi or Bluetooth.

[0368] 3. Server: Preprocesses the received image data. OpenCV is used to adjust the image resolution and format, converting it into a suitable state for analysis.

[0369] 4. Server: Analyzes the preprocessed image data to identify the work content and instructions. It also recognizes emotions based on the worker's facial expressions and voice. It uses DeepFace to analyze the worker's emotional state.

[0370] 5. Terminal: The server generates appropriate work instructions based on the emotion recognition results. For example, if a worker is feeling stressed, a simple instruction will be displayed.

[0371] 6. Terminal: The generated work instructions are displayed on the smart glasses display and provided to the worker.

[0372] Specific examples

[0373] As an example, consider a scenario in which a worker in an automobile parts manufacturing plant uses smart glasses to install parts. The worker takes a photo of the work site with the camera in the smart glasses and sends the image to a server. The server processes the image to identify the installation location and uses DeepFace to analyze the worker's emotional state. If the worker is feeling stressed, the server generates a concise instruction and displays it on the smart glasses. This improves work efficiency and relieves the worker's stress.

[0374] Prompt Sentence Examples

[0375] "The user wears the smart glasses and takes photos of the work site. The system generates a manual of work procedures using the photos, and also implements a function to detect the wearer's stress level and adjust the instructions accordingly based on their emotions."

[0376] This invention makes it possible to provide flexible and appropriate work instructions based on the emotional state of field workers, thereby improving work efficiency and accuracy and reducing worker stress.

[0377] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0378] Step 1:

[0379] A user wears the smart glasses and takes a photo of the work site. The input is image data of the site, and the output is the captured image data, which is used in subsequent processing steps.

[0380] Step 2:

[0381] The device (smart glasses) sends the captured image data to the server. The input is the image data acquired in step 1, and the output is the image data sent to the server. Wi-Fi and Bluetooth are used as communication methods.

[0382] Step 3:

[0383] The server preprocesses the received image data. The input is the transmitted image data, and the output is the preprocessed image data. Specifically, it uses OpenCV to adjust the image resolution and convert it to the required format (e.g. PNG).

[0384] Step 4:

[0385] The server analyzes the preprocessed image data to identify the work content and instructions. The input is the preprocessed image data, and the output is data related to the work content and instructions. These elements are extracted using an image generation device.

[0386] Step 5:

[0387] The device (smart glasses) captures the worker's facial expressions and voice and sends them to the server. The input is the worker's facial expression data and voice data, and the output is the emotion data sent to the server.

[0388] Step 6:

[0389] The server analyzes the received emotion data and determines the worker's emotional state. The input is facial expression and voice data, and the output is the emotion recognition result. DeepFace is used to analyze the worker's emotional state.

[0390] Step 7:

[0391] The server generates appropriate work instructions based on the emotion recognition results. The inputs are the emotion recognition results and work content data, and the output is the generated work instructions. Based on the emotional state, it provides simple instructions to workers who are stressed and detailed instructions to workers who are calm.

[0392] Step 8:

[0393] The terminal (smart glasses) displays the generated work instructions. The input is the generated work instruction data, and the output is the work instructions displayed on the smart glasses display. The worker can proceed with the work according to these instructions.

[0394] Through the above processing steps, flexible work instructions are provided according to the emotional state of the field worker, thereby improving work efficiency and accuracy.

[0395] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0396] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0397] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0398] [Second embodiment]

[0399] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0400] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0401] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0402] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0403] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0404] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0405] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0406] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0407] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0408] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0409] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0410] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0411] This invention is a system that quickly and affordably generates high-quality custom picture books based on user-provided drawings by children. The system consists of a series of processes: the user submits the drawings, the system classifies them into characters and backgrounds, creates a story based on the generated information, and finally provides the picture book.

[0412] Program processing

[0413] Process for users to post pictures

[0414] The user uses the website or application of the custom picture book creation service to upload a picture drawn by their child. The device then sends the image data of the picture to the server.

[0415] Picture pre-processing

[0416] The server processes the received image data, adjusting the image resolution and format to convert it into a state suitable for AI processing.

[0417] Character generation using image generation AI

[0418] The server inputs the preprocessed images into an image generation AI to identify elements in the picture (such as characters and backgrounds), and generates each element as an independent character model or background as needed.

[0419] Story generation using text generation AI

[0420] The server uses text generation AI to automatically generate a story based on the generated characters and background information. The AI ​​creates a storyline based on the characteristics of the characters and the atmosphere of the illustrations.

[0421] Picture book compilation and preview generation

[0422] The server combines the generated story with the original illustrations to create a page layout, and creates a digital preview that can be sent to the user's device for review.

[0423] User verification and customization

[0424] The user checks the preview data and, if necessary, makes corrections or additional customizations. The customization details are sent from the device to the server.

[0425] Final generation and delivery

[0426] The server generates the final picture book data that reflects the user's customizations and provides it for viewing on the web. Printed versions are also available as an optional paid option.

[0427] Specific examples

[0428] 1. The process of posting a picture

[0429] The parent user takes a photo of their child's drawing of a "blue dragon" with their smartphone and uploads it to the service's app.

[0430] 2. Picture preprocessing

[0431] After the server receives the uploaded image, it adjusts the resolution to 300 dpi and converts it to PNG format.

[0432] 3. Character Creation Using Image Generation AI

[0433] The server uses image generation AI to extract the "blue dragon" as the main character and converts its features into a digital model.

[0434] 4. Story Generation Using Text Generation AI

[0435] The server then has the text generation AI create a story with the "blue dragon" as the main character. For example, a story with the theme of "the blue dragon's adventure to save his friend" is generated.

[0436] 5. Picture book compilation and preview generation

[0437] The server integrates the generated story with the original illustrations, creates a page-by-page layout, and generates a digital preview that is sent to the user.

[0438] 6. User Verification and Customization

[0439] The user sees the preview and decides they want to change the dragon's name to "Drago." They make the changes in the app and submit the changes to the server.

[0440] 7. Final Generation and Delivery

[0441] The server generates the final version of the picture book data that reflects the user's changes and provides it for viewing on the web. If the user wants a printed version, they can order it for a fee.

[0442] In this way, children's drawings are quickly turned into engaging custom picture books.

[0443] The processing flow will be explained below.

[0444] Step 1:

[0445] Users access the service's website or app and log in. After logging in, they click the "Upload a picture" button, which opens a file selection dialog.

[0446] Step 2:

[0447] The user selects and uploads an image file of the picture drawn by the child, and the device sends the selected image file to the server.

[0448] Step 3:

[0449] The server checks the received image data, checking the image format and resolution, and displays an error message if they are inappropriate.

[0450] Step 4:

[0451] The server performs preprocessing on the image data, including adjusting the resolution and filtering noise. The preprocessed image data is then temporarily stored.

[0452] Step 5:

[0453] The server inputs the preprocessed image data into the image generation AI, which detects the main elements in the image (such as characters and backgrounds) and generates each element as an independent character model or background.

[0454] Step 6:

[0455] The server stores the generated character models and background data, which are then input into the text generation AI.

[0456] Step 7:

[0457] The server sets the necessary parameters for the text generation AI (e.g., character personality, atmosphere, etc.), and the text generation AI automatically generates a story based on the provided information.

[0458] Step 8:

[0459] The server combines the generated story with character models and background data to automatically generate a layout for each page.

[0460] Step 9:

[0461] The server creates a digital preview of the generated picture book, temporarily stores it, and generates a preview link and notifies the user.

[0462] Step 10:

[0463] Users click the link sent to them to see the digital preview, customize it as needed (e.g., change the character's name), and enter their details.

[0464] Step 11:

[0465] The terminal transmits the customization information entered by the user to the server.

[0466] Step 12:

[0467] The server reflects the received customization information in the final version of the picture book, generates the final version of the picture book data, and provides it for viewing on the web.

[0468] Step 13:

[0469] The user can check the final version of the picture book and download it if necessary. If the user wants a printed version, the terminal sends the order information to the server.

[0470] Step 14:

[0471] The server accepts the print order and initiates the necessary printing and delivery procedures. After the order is completed, the server notifies the user of the delivery status.

[0472] These steps allow us to quickly generate high-quality custom picture books from children's drawings.

[0473] Example 1

[0474] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0475] In the conventional picture book creation process, it was difficult for users to easily and quickly generate custom picture books based on their children's drawings. Furthermore, there was a lack of automatic generation functions using digitalization and AI, forcing users to perform time-consuming and laborious manual work. This made it difficult to provide custom picture books quickly and with high quality.

[0476] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0477] In this invention, the server includes means for receiving image data sent by a user, means for preprocessing the received image data, means for generating characters using an image generation device with the preprocessed image data as input, means for generating a story using a text generation device with the generated characters and background information as input, means for automatically generating a page layout for a picture book by combining the story and character information, means for providing the user with an electronic version of the generated picture book, means for receiving customization information from the user and generating a final version of the picture book that reflects the information, means for transmitting input data from the terminal to the server, and data processing and conversion means in the server. This enables the rapid automatic generation of a high-quality customized picture book from pictures provided by the user.

[0478] "User" refers to an individual or group that uses the system to send image data of children's drawings.

[0479] "Image data" refers to the data used to digitally store and send pictures drawn by children to the system.

[0480] "Preprocessing" refers to the process performed to convert received image data into a state suitable for AI processing. Specifically, this includes adjusting the resolution and converting the format.

[0481] An "image generation device" is a device or software that takes preprocessed image data as input, analyzes the image, and extracts and generates characters and backgrounds.

[0482] "Characters" are characters extracted from the image data sent by the user, and appear in the story of the picture book.

[0483] "Background information" refers to background elements and settings extracted from the image data sent by the user.

[0484] A "story generation device" is a device or software that automatically generates a story using generated characters and background information as input.

[0485] A "story" is a story that is automatically generated by the text generation device based on the generated characters and background information.

[0486] "Page layout" refers to the design that determines the layout of each page of a picture book by combining the generated story and character information.

[0487] "Electronic Edition" means a digital version of the generated picture book that can be viewed or downloaded by a user.

[0488] "Customization information" refers to information that a user inputs after checking a preview to reflect corrections or additional changes.

[0489] "Device" means the device used by a User to capture, upload, view, and customize Pictures, including smartphones and computers.

[0490] A "server" is a computer system that performs a series of processes, such as processing transmitted image data, preprocessing, character generation, story generation, and page layout creation.

[0491] "Data processing and conversion means" refers to software and hardware for a series of data conversion, analysis, and processing performed on the server.

[0492] This invention is a system that quickly and affordably generates high-quality custom picture books based on user-provided drawings by children. The system consists of a series of processes: the user submits the drawings, the system classifies them into characters and backgrounds, creates a story based on the generated information, and finally provides the picture book.

[0493] This system uses the following hardware and software to process and calculate data.

[0494] First, a user uses a device (such as a smartphone or PC) to photograph or scan a picture drawn by their child, and then uses the custom picture book creation service's website or application to send the image data to the server. This image data is sent to the server using an HTTP POST request. The server then uses Python and its image processing library (such as Pillow) to perform preprocessing, such as adjusting the resolution of the received image data to 300 dpi and converting it from JPEG to PNG format.

[0495] The server then inputs the preprocessed image data into an image generation AI (such as DeepArt or a GAN-based system) to extract important elements (characters and background) from the image. During this process, elements such as a blue dragon and a castle are identified, and these elements are then generated as independent character and background models.

[0496] Furthermore, the server automatically generates a story using a text generation AI (for example, OpenAI's GPT-4) based on the identified characters and background information. Specifically, there is an example where a prompt sentence such as "An adventure in which a blue dragon saves his friend" is input into the text generation AI to create a storyline.

[0497] The server then combines the generated story with the original illustrations to create a page-by-page layout. During this process, a dedicated layout engine is used to generate a digital preview of the book for the user to review. The preview data is then sent to the device as an HTTP response.

[0498] The user checks the digital preview on their device and makes any necessary corrections or customizations (for example, changing the dragon's name to "Drago"). The customizations are sent back to the server, which then generates the final picture book data reflecting the user's customizations.

[0499] The final picture book data is provided to users in a format that can be viewed online from the server. If the user wishes to purchase a printed version, they can order it for a fee and have it delivered.

[0500] Examples of specific prompts are:

[0501] Create a story starring your child's blue dragon and the dragon's adventures helping his friends.

[0502] In this way, a system is realized that quickly generates high-quality custom picture books based on user-provided illustrations.

[0503] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0504] Step 1:

[0505] A user uses the website or application of a custom picture book creation service to take a photo or scan a picture drawn by their child with a smartphone or PC and upload it. The device sends the image data to the server via an HTTP POST request. The input to this step is the image data sent from the user's device, and the output is the image data received by the server. Specifically, the user launches the application, takes a photo or selects an image from the gallery, and clicks the upload button.

[0506] Step 2:

[0507] The server preprocesses the received image data. First, it adjusts the image resolution to 300 dpi and converts the image format from JPEG to PNG. This is done using Python's Pillow library, for example. The input to this step is the raw image data received by the server, and the output is preprocessed image data with the resolution adjusted and format converted. Specifically, it resizes and converts the image using functions from the Pillow library.

[0508] Step 3:

[0509] The server inputs the preprocessed image data into an image generation AI, which identifies key elements in the image (characters and background) and generates independent digital models of each. The input for this step is the preprocessed image data, and the output is digital models of the identified characters and background. Specifically, the data is input into DeepArt or a GAN-based AI model to perform classification and model generation.

[0510] Step 4:

[0511] The server inputs a prompt into a text generation AI based on the identified characters and background, automatically generating a story. For example, the prompt "A blue dragon goes on an adventure to save his friends" is input into a text generation AI such as GPT-4. The input for this step is a digital model of the characters and background and the prompt, and the output is the generated story text. Specifically, the prompt and character data are input into the text generation AI, and the generated story is received in text format.

[0512] Step 5:

[0513] The server combines the generated story with the original illustrations to create a layout for each page. In this process, a dedicated layout engine is used to generate a digital preview of the picture book. The input for this step is the story text and a digital model of the original illustrations, and the output is the completed digital preview. Specifically, the layout engine automatically creates the page structure and generates the preview.

[0514] Step 6:

[0515] The user checks the digital preview and makes any necessary corrections or customizations. For example, they input the customization details, such as changing the dragon's name to "Drago," and send it to the server. The input in this step is the user's customization instructions, and the output is data sent to the server that has received the correction instructions. Specifically, the user selects the part to be corrected within the application, enters the changes, and sends them.

[0516] Step 7:

[0517] The server generates the final picture book data that reflects the user's customizations and provides it in a format that can be viewed on the web. If the user wishes, it can also accept orders for a printed version as a paid option and handles the delivery procedures. The input for this step is the customization information received from the user, and the output is the final picture book data and print order data. Specific operations include generating the final data, uploading it to the web, and managing the printing and delivery procedures.

[0518] (Application example 1)

[0519] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0520] Conventional custom picture book creation systems lack the ability to easily share content created by users with other users, and the process of creating a custom picture book is often cumbersome and time-consuming. This makes it difficult for users to easily and quickly create custom picture books based on their children's drawings and share them with other family and friends. Therefore, to improve the user experience, there is a need for a system that allows users to create high-quality picture books through a simple process and share those picture books with other users.

[0521] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0522] In this invention, the server includes means for receiving image data sent by a user, means for preprocessing the received image data, means for generating characters using an image generation device with the preprocessed image data as input, means for generating a story using a text generation device with the generated characters and background information as input, means for automatically generating a page layout for a picture book by combining the story and character information, means for providing the user with an electronic version of the generated picture book, means for receiving customization information from the user and generating a final version of the picture book that reflects the information, and means for sharing the generated picture book with other users within the application. This enables users to quickly create high-quality custom picture books based on their children's drawings and easily share them with other family and friends.

[0523] "Submitted image data" refers to user-generated visual information that a user uploads to the custom picture book creation system.

[0524] "Preprocessing" is the act of converting received image data into an appropriate format and resolution, making it suitable for subsequent processing steps.

[0525] An "image generation device" is a device that analyzes image data received as input and automatically generates characters and backgrounds.

[0526] A "text generation device" is a device that creates a story based on the characters and background information generated by an image generation device.

[0527] "Page layout" is a layout design that automatically arranges the structure of each page of a picture book by combining story and character information.

[0528] An "electronic version" is a picture book that is provided in digital format rather than in physical print form.

[0529] "Customization information" is information that reflects modifications and additional requirements made by the user.

[0530] The "final version" is a completed picture book that reflects the customization information provided by the user.

[0531] "Means for sharing" refers to the ability to send created custom picture books to other users within the application.

[0532] The present invention is a system for quickly and affordably generating high-quality custom picture books from user-supplied drawings by children. Specific methods for implementing the present invention are described below.

[0533] First, a user uses a smartphone app to take a picture of a drawing their child has made and upload it to the system. The uploaded image data is sent from the device to the server. The server then preprocesses the received image data and adjusts the image resolution and format. This preprocessing is performed using image processing software such as PIL (Python Imaging Library).

[0534] The preprocessed image data is then fed into an image generator (e.g., a system using a specific AI model) that analyzes the image data and defines the characters and background. For example, if a child's drawing of a "blue dragon" is uploaded, the image generator will extract the "blue dragon" from the drawing as the main character and generate a digital model of it.

[0535] The generated characters and background information are then passed to a text generator. The text generator automatically generates a story based on the characteristics of the characters and the atmosphere of the illustration. At this stage, a generative AI model is used. For example, a story with the theme of "a blue dragon's adventure to save his friend" is generated. An example prompt might be, "Generate a story featuring a child's drawing of a 'blue dragon.' This dragon is brave and embarks on an adventure to save a friend in trouble. Draw what difficulties he faces during his adventure and how he overcomes them."

[0536] The generated story and character information is compiled into a picture book using an automatic page layout generator. The server combines these to create a page-by-page structure and generate a digital preview. The user can check this preview on their smartphone and make corrections or additional customizations as needed. The user's customization information is then sent to the server, and the final picture book is generated.

[0537] The server then provides the user with an electronic version of the final generated picture book. The application also includes a means for sharing the generated custom picture book with other users. For example, the user can easily share the generated picture book with family and friends. The server also provides functionality for the user to order a printed version of the generated picture book and arrange for delivery, if desired.

[0538] In this way, the present invention provides a system that enables users to quickly and easily create high-quality custom picture books according to their needs, and further enables users to share these picture books.

[0539] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0540] Step 1:

[0541] The user takes a photo of a drawing made by their child through a smartphone app. The user then uploads this image data to the app. The input is the image data taken with the smartphone, and the output is the image data sent to the server. The specific action that occurs at this point is for the user to tap the "Upload Image" button on the app and select or take a photo of the drawing.

[0542] Step 2:

[0543] The server receives image data sent by the user. The input is the uploaded image data, and the output is image data that can be preprocessed. Specifically, the server receives the image data and saves it in storage.

[0544] Step 3:

[0545] The server preprocesses the received image data, including adjusting the resolution and converting the format. The input is the original image data, and the output is the preprocessed image data. Specifically, the server converts the image to RGB using PIL (Python Imaging Library) and resizes the image to 1024x1024.

[0546] Step 4:

[0547] Based on the preprocessed image data, the server generates the characters using an image generation device. The input is the preprocessed image data, and the output is a digital model of the character. Specifically, the server inputs the image data into the AI ​​model, identifies the elements of the picture, and generates the digital model.

[0548] Step 5:

[0549] Based on the generated characters and background information, the server generates a story using a text generator. The input is the characters and background information, and the output is the generated story. Specifically, the server provides a prompt to the generative AI model, which then creates the story. An example of a prompt is, "Generate a story featuring a child's drawing of a 'blue dragon.' This dragon is brave and sets out on an adventure to help a friend in trouble. Draw what difficulties he faces during his adventure and how he overcomes them."

[0550] Step 6:

[0551] The server automatically generates page layouts based on story and character information. The input is story and character information, and the output is the page layout of a picture book. Specifically, the server integrates the generated story with image elements and automatically formats the layout of each page.

[0552] Step 7:

[0553] The server provides the generated digital version of the picture book to the user. The input is the page layout and story, and the output is the digital version of the picture book data. Specifically, the server generates a digital file of the picture book and sends it via the app so that the user can view it.

[0554] Step 8:

[0555] The user checks the digital version of the picture book and inputs customization information. The input is the digital version of the picture book data, and the output is the customization information specified by the user. Specifically, the user makes modifications such as changing the names of characters and text through the app interface.

[0556] Step 9:

[0557] The server generates the final version of the picture book by reflecting the user's customization information. The input is the user's customization information, and the output is the final version of the picture book data. Specifically, the server applies the received customization information to the picture book pages to complete the final version.

[0558] Step 10:

[0559] The server provides a means for users to share the final version of the picture book with other users within the application. The input is the final version of the picture book data, and the output is the shared picture book data. Specifically, the server generates a sharing link for the picture book, allowing users to send it to family and friends.

[0560] These steps result in a system that allows users to quickly create high-quality custom picture books based on their children's drawings and easily share them with other family and friends.

[0561] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0562] This invention is a system that quickly and affordably generates high-quality custom picture books based on user-provided drawings by children. The system consists of a series of processes: the user submits a picture, the system classifies it into characters and backgrounds, creates a story based on the generated information, and finally provides the picture book. The system also incorporates an emotion engine that recognizes the user's emotions and has the ability to adjust the content of the story and the facial expressions of the characters based on the user's emotions.

[0563] Program processing

[0564] Process for users to post pictures

[0565] The user uses the website or application of the custom picture book creation service to upload a picture drawn by their child. The device then sends the image data of the picture to the server.

[0566] Picture pre-processing

[0567] The server processes the received image data, adjusting the image resolution and format to convert it into a state suitable for AI processing.

[0568] Character generation using image generation AI

[0569] The server inputs the preprocessed images into an image generation AI to identify elements in the picture (such as characters and backgrounds), and generates each element as an independent character model or background as needed.

[0570] Story generation using text generation AI

[0571] The server uses text generation AI to automatically generate a story based on the generated characters and background information. The AI ​​creates a storyline based on the characteristics of the characters and the atmosphere of the illustrations.

[0572] Recognizing user emotions with an emotion engine

[0573] The device collects the user's facial expressions and voice through a camera and microphone, and sends this data to a server.

[0574] The server uses an emotion engine to analyze the received facial expression data and voice data and determine the user's emotions.

[0575] Emotion-based story adjustment

[0576] The server then adjusts the story and the characters' expressions based on the determined emotional information. For example, if the user is enjoying the story, it will brighten up the story.

[0577] Picture book compilation and preview generation

[0578] The server combines the generated story with the original illustrations to create a page layout, and creates a digital preview that can be sent to the user's device for review.

[0579] User verification and customization

[0580] The user checks the preview data and, if necessary, makes corrections or additional customizations. The customization details are sent from the device to the server.

[0581] Final generation and delivery

[0582] The server generates the final picture book data that reflects the user's customizations and provides it for viewing on the web. Printed versions are also available as an optional paid option.

[0583] Specific examples

[0584] 1. The process of posting a picture

[0585] The parent user takes a photo of their child's drawing of a "blue dragon" with their smartphone and uploads it to the service's app.

[0586] 2. Picture preprocessing

[0587] After the server receives the uploaded image, it adjusts the resolution to 300 dpi and converts it to PNG format.

[0588] 3. Character Creation Using Image Generation AI

[0589] The server uses image generation AI to extract the "blue dragon" as the main character and converts its features into a digital model.

[0590] 4. Story Generation Using Text Generation AI

[0591] The server then has the text generation AI create a story with the "blue dragon" as the main character. For example, a story with the theme of "the blue dragon's adventure to save his friend" is generated.

[0592] 5. User Emotion Recognition by Emotion Engine

[0593] As the user reads parts of the story, emotions are collected through facial expressions and voice, and each emotional state is sent to the server.

[0594] The server analyzes the user's emotions through an emotion engine and adjusts the story and characters.

[0595] 6. Adjust your story based on emotions

[0596] If the server determines that the user is enjoying themselves, it will change the story to a brighter, more enjoyable one, and the character's facial expressions will also change accordingly.

[0597] 7. Picture book compilation and preview generation

[0598] The server combines the generated story, the original illustrations, and the emotional adjustments to create a page-by-page layout, creating a digital preview and sending it to the user.

[0599] 8. User Verification and Customization

[0600] The user checks the preview, makes any necessary changes, and then sends the customizations back to the server.

[0601] 9. Final Generation and Delivery

[0602] The server generates the final version of the picture book that reflects the user's changes and provides it on the web. If the user wishes, it also accepts an order for a printed version and begins the delivery process.

[0603] In this way, a system is provided that can quickly generate high-quality custom picture books from children's drawings and flexibly adjust the content of the story according to the user's emotions.

[0604] The processing flow will be explained below.

[0605] Step 1:

[0606] Users access the service's website or app and log in. After logging in, they click the "Upload a picture" button, which opens a file selection dialog.

[0607] Step 2:

[0608] The user selects and uploads an image file of the picture drawn by the child, and the device sends the selected image file to the server.

[0609] Step 3:

[0610] The server checks the received image data, checking the image format and resolution, and displays an error message if they are inappropriate.

[0611] Step 4:

[0612] The server performs preprocessing on the image data, including adjusting the resolution and filtering noise. The preprocessed image data is then temporarily stored.

[0613] Step 5:

[0614] The server inputs the preprocessed image data into the image generation AI, which detects the main elements in the image (such as characters and backgrounds) and generates each element as an independent character model or background.

[0615] Step 6:

[0616] The server stores the generated character models and background data, which are then input into the text generation AI.

[0617] Step 7:

[0618] The server sets the necessary parameters for the text generation AI (e.g., character personality, atmosphere, etc.), and the text generation AI automatically generates a story based on the provided information.

[0619] Step 8:

[0620] The device collects the user's facial expressions and voice through a camera and microphone, and transmits the emotional data to a server.

[0621] Step 9:

[0622] The server uses an emotion engine to analyze the received facial expression data and voice data and determine the user's emotions.

[0623] Step 10:

[0624] The server then adjusts the story and the characters' expressions based on the determined emotional information. For example, if the user is enjoying the story, it will brighten up the story.

[0625] Step 11:

[0626] The server combines the generated story with character models, background data, and the results of adjustments made by the emotion engine to automatically generate a layout for each page.

[0627] Step 12:

[0628] The server creates a digital preview of the generated picture book, temporarily stores it, and generates a preview link and notifies the user.

[0629] Step 13:

[0630] Users click the link sent to them to see the digital preview, customize it as needed (for example, change the character's name), and enter their details.

[0631] Step 14:

[0632] The terminal transmits the customization information entered by the user to the server.

[0633] Step 15:

[0634] The server reflects the received customization information in the final version of the picture book, generates the final version of the picture book data, and provides it for viewing on the web.

[0635] Step 16:

[0636] The user can check the final version of the picture book and download it if necessary. If the user wants a printed version, the terminal sends the order information to the server.

[0637] Step 17:

[0638] The server accepts the print order and initiates the necessary printing and delivery procedures. After the order is completed, the server notifies the user of the delivery status.

[0639] The above steps provide a system that can quickly generate high-quality custom picture books from children's drawings and flexibly adjust the story content according to the user's emotions.

[0640] Example 2

[0641] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0642] Conventional custom picture book generation systems have difficulty generating high-quality picture books quickly based on user-provided images. Furthermore, they lack the ability to adjust the story and character expressions to reflect the user's emotions, making it impossible to provide a personalized experience. This has led to problems such as low user satisfaction and low retention rates.

[0643] The specification processing by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving image data sent by a user, means for preprocessing the received image data, means for generating characters using an image generation device with the preprocessed image data as input, means for generating a story using a text generation device with the generated characters and background information as input, means for recognizing the user's emotional state, means for adjusting the story and character facial expressions based on the recognized emotions, means for automatically generating a page layout for a picture book by combining the story and character information, means for providing the user with an electronic version of the generated picture book, and means for receiving customization information from the user and generating a final picture book reflecting the information. This makes it possible to quickly process images provided by the user and adjust the story and character facial expressions according to the user's emotions.

[0644] A "user" is an end user who uses the custom picture book creation service to upload a picture drawn by a child and request the creation of a picture book.

[0645] A "server" is a central processing unit that performs various processes such as image processing, data analysis, and AI models.

[0646] "Image data" is a digital image file of a child's drawing provided by the user.

[0647] "Preprocessing" refers to initial data processing to facilitate subsequent AI processing, such as adjusting the image data resolution, converting formats, removing noise, and correcting colors.

[0648] An "image generation device" is an AI device that extracts characters and backgrounds from preprocessed image data and generates digital character models and background models.

[0649] "Characters" are characters or objects depicted in a user-provided picture.

[0650] "Background information" refers to information about the overall scenery and environment of the picture, excluding the characters.

[0651] A "text generation device" is an AI device that automatically generates a story based on generated characters and background information.

[0652] A "story" is an automatically generated story based on the characteristics and background information of the characters.

[0653] The "emotional state" is an emotional state such as joy, surprise, sadness, etc., that is analyzed from the user's facial expression and voice.

[0654] The "emotion engine" is an AI engine that analyzes the user's facial expressions and voice data to determine their emotional state.

[0655] "Page layout" refers to the layout of each page of a picture book, which is created by combining the generated story and character information.

[0656] "Electronic version" refers to the final version of the picture book data provided in digital format.

[0657] "Customization information" refers to the contents and settings of the picture book that the user checks in the preview and adds or changes as necessary.

[0658] The "final version" is the completed picture book data that is generated by reflecting the user's customization information.

[0659] This invention is a system that quickly generates high-quality custom picture books based on user-provided drawings by children. The system works in cooperation with the user, device, and server elements, and uses multiple AI models to pre-process images, generate characters, generate stories, and adjust content based on the user's emotions.

[0660] Hardware and software used

[0661] Terminal

[0662] A device is a device used by a user, such as a smartphone, tablet, or PC. Users take and upload pictures through these devices. Devices are equipped with cameras and microphones, and also have the ability to collect facial expressions and voice data from users.

[0663] server

[0664] The server is the center of the picture book generation system and has multiple AI models and engines for performing the following processes:

[0665] Image pre-processing engine: Performs pre-processing such as image resolution adjustment, noise reduction, and color correction.

[0666] Image generation AI model: Generates characters and backgrounds from preprocessed image data.

[0667] Text generation AI model: Generates a story based on characters and background information.

[0668] Emotion engine: Analyzes and determines emotions from the user's facial expressions and voice data.

[0669] Implementation Procedure

[0670] Users submit pictures

[0671] Users take a photo of their child's drawing using a smartphone or computer and upload the image file using the custom picture book creation service's website or application.

[0672] Picture pre-processing

[0673] The server passes the received image data to a pre-processing engine, which sets the resolution to 300 dpi, converts the image format to PNG, and performs noise removal and color correction to prepare the image for processing by the AI ​​model.

[0674] Character generation using image generation AI

[0675] The server inputs the preprocessed images into an image generation AI model to identify the main characters and background, and reconstructs the characters' features as digital models.

[0676] Story generation using text generation AI

[0677] The server uses a text generation AI model to generate a story based on the characters and background information. The AI ​​is given prompts like the following:

[0678] "This image depicts a blue dragon. Please create an adventure story with this dragon as the main character."

[0679] "This blue dragon is a brave character who likes to help his friends. Generate a story based on this setting."

[0680] Recognizing user emotions with an emotion engine

[0681] When users preview their custom picture books, their facial expressions and voice are collected using the device's camera and microphone, and this data is sent to the server in real time and analyzed by the server's emotion engine.

[0682] Emotion-based story adjustment

[0683] The server dynamically changes the story and the characters' expressions based on the analysis results of the emotion engine. If the user is enjoying the story, it will adjust the story to be more cheerful and fun.

[0684] Picture book compilation and preview generation

[0685] The server automatically generates a layout for each page by integrating information about the story and characters, and then creates a digital preview that is sent to the device.

[0686] User verification and customization

[0687] The user checks the preview and edits text or images as necessary. The customization information made on the device is sent to the server.

[0688] Final generation and delivery

[0689] The server generates the final picture book data that reflects the customization information and provides it in a format that can be viewed on the web. If the user requests a printed version, the server generates print data and sends an order to the printing company.

[0690] In this way, user-provided pictures can be quickly transformed into high-quality custom picture books, with content that can be flexibly tailored based on the user's emotions.

[0691] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0692] Step 1: How users post pictures

[0693] Users take a photo of a picture their child has drawn using a smartphone or computer, and then upload the image file using the website or application of the custom picture book creation service. Specifically, users click the "Upload Picture" button, select an image file, and upload it. They then press the "Send" button to send the image data from their device to the server. The input is the user's image data, and the output is the image data sent to the server.

[0694] Step 2: Pre-processing the image

[0695] The server receives image data sent from the device and passes it to the preprocessing engine. It adjusts the resolution of the received image data, for example, setting it to 300 dpi. It also converts the image format from common formats such as JPEG or BMP to PNG. It applies a noise reduction filter and performs color correction. The input is the received image data, and the output is the image data after preprocessing.

[0696] Step 3: Creating characters using image generation AI

[0697] The server inputs the preprocessed image data into the image generation AI model and performs image analysis. Specifically, it inputs a prompt to "extract the main characters and convert them into digital models" into the image generation AI model. The AI ​​identifies the main characters and background in the image and converts their features into digital models. The input is the preprocessed image data, and the output is the generated digital models of the characters and background.

[0698] Step 4: Story generation using text generation AI

[0699] The server uses a text generation AI model to generate a story based on the generated characters and background information. For example, a prompt such as "This image depicts a blue dragon. Please create an adventure story with this dragon as the main character" is input and the AI ​​generates the story. Specifically, based on the characteristics and background information of the characters, a story including the story flow, character actions, dialogue, etc. is automatically created. The input is the characters and background information, and the output is the generated story.

[0700] Step 5: Recognizing user emotions with the emotion engine

[0701] While viewing a preview of a custom picture book, the user uses the device's camera and microphone to collect facial expressions and voice. The device transmits this emotional data to the server in real time. The server's emotion engine analyzes this data and determines the user's emotional state (e.g., joy, surprise, sadness). The input is the user's facial and voice data, and the output is the analyzed emotional state.

[0702] Step 6: Adjust your story based on emotion

[0703] The server receives the analysis results from the emotion engine and adjusts the story and character expressions. For example, if it determines that the user is enjoying themselves, it changes the story to a more cheerful and fun development and adjusts the character expressions accordingly. The input is the analyzed emotional state, and the output is the adjusted story and character expressions.

[0704] Step 7: Compiling the picture book and generating a preview

[0705] The server integrates the generated story, character models, and backgrounds to create the layout for each page. Specifically, it determines the placement of text and images and automatically generates a layout including design elements for each page. Finally, it generates a digital preview and sends it to the device. The input is the adjusted story, character models, and backgrounds, and the output is a digital preview.

[0706] Step 8: Verify and customize users

[0707] The user checks the digital preview sent on their device and makes changes as necessary, such as editing text, changing images, and adjusting character positions. Once the user has finalized the customizations, the data is resent from their device to the server. The input is the digital preview, and the output is data reflecting the customizations.

[0708] Step 9: Final generation and serving

[0709] The server generates the final picture book data that reflects the customization information. It then provides it to the user in a format that can be viewed on the web. If the user requests a printed version, the server generates print data and sends an order to the printer. The input is data that reflects the customization details, and the output is the final picture book data. It also includes order data for delivering the printed version to the user.

[0710] (Application example 2)

[0711] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0712] There is a demand for improving the efficiency and accuracy of the work that field workers face, but conventional systems have the problem that the work content and instructions are uniform, making it difficult to respond flexibly while taking into account the stress level and concentration of each worker. Furthermore, work instructions can be either too detailed or too simple, which can lead to a decline in work efficiency. In particular, it has been difficult to grasp the emotional state of field workers in real time and provide appropriate instructions based on that.

[0713] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving image data sent by a user; means for preprocessing the received image data; means for generating characters using an image generation device with the preprocessed image data as input; means for generating a story using a text generation device with the generated characters and background information as input; emotion recognition means for analyzing the user's emotional state; means for adjusting the facial expressions of the story and characters based on the emotional state; means for automatically generating a page layout for a picture book by combining the story and character information; means for providing a user with an electronic version of the generated picture book; and means for receiving customization information from the user and generating a final version of the picture book reflecting the information. This makes it possible to provide appropriate work instructions that flexibly correspond to the emotional state of field workers.

[0714] A "user" is a person who uses the service to send image data.

[0715] "Image data" refers to image information sent by a user.

[0716] "Preprocessing" refers to a series of processes that convert received image data into a state suitable for analysis.

[0717] An "image generation device" is a device for generating characters based on pre-processed image data.

[0718] "Characters" are characters generated from image data sent by the user.

[0719] "Background information" refers to information that corresponds to the background of an image generated by an image generation device.

[0720] A "sentence generation device" is a device that automatically generates a story based on generated characters and background information.

[0721] A "story" is a storyline generated by a text generator.

[0722] "Emotion recognition means" refers to means for analyzing the emotional state of a user.

[0723] "Page layout" is the process of combining story and character information to determine the page structure of a picture book.

[0724] "Digital edition" refers to a picture book provided in digital format.

[0725] "Customization information" refers to modifications or additions requested by the user to the electronic version.

[0726] The "final version" refers to the final picture book data generated by reflecting the customization information.

[0727] This invention is a system that uses emotion recognition to provide appropriate work instructions to field workers based on smart glasses. Specific embodiments of this system will be described below.

[0728] Hardware and software used

[0729] Hardware: Smart glasses, webcam

[0730] Software: OpenCV (image processing), DeepFace (emotion recognition), Flask server (data transmission and reception)

[0731] Data processing and calculation

[0732] 1. User: A field worker wears smart glasses and takes a photo of the work site. This photo data is captured by the camera in the smart glasses.

[0733] 2. Terminal: The smart glasses transmit the acquired image data to the server using wireless communication technologies such as Wi-Fi or Bluetooth.

[0734] 3. Server: Preprocesses the received image data. OpenCV is used to adjust the image resolution and format, converting it into a suitable state for analysis.

[0735] 4. Server: Analyzes the preprocessed image data to identify the work content and instructions. It also recognizes emotions based on the worker's facial expressions and voice. It uses DeepFace to analyze the worker's emotional state.

[0736] 5. Terminal: The server generates appropriate work instructions based on the emotion recognition results. For example, if a worker is feeling stressed, a simple instruction will be displayed.

[0737] 6. Terminal: The generated work instructions are displayed on the smart glasses display and provided to the worker.

[0738] Specific examples

[0739] As an example, consider a scenario in which a worker in an automobile parts manufacturing plant uses smart glasses to install parts. The worker takes a photo of the work site with the camera in the smart glasses and sends the image to a server. The server processes the image to identify the installation location and uses DeepFace to analyze the worker's emotional state. If the worker is feeling stressed, the server generates a concise instruction and displays it on the smart glasses. This improves work efficiency and relieves the worker's stress.

[0740] Prompt Sentence Examples

[0741] "The user wears the smart glasses and takes photos of the work site. The system generates a manual of work procedures using the photos, and also implements a function to detect the wearer's stress level and adjust the instructions accordingly based on their emotions."

[0742] This invention makes it possible to provide flexible and appropriate work instructions based on the emotional state of field workers, thereby improving work efficiency and accuracy and reducing worker stress.

[0743] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0744] Step 1:

[0745] A user wears the smart glasses and takes a photo of the work site. The input is image data of the site, and the output is the captured image data, which is used in subsequent processing steps.

[0746] Step 2:

[0747] The device (smart glasses) sends the captured image data to the server. The input is the image data acquired in step 1, and the output is the image data sent to the server. Wi-Fi and Bluetooth are used as communication methods.

[0748] Step 3:

[0749] The server preprocesses the received image data. The input is the transmitted image data, and the output is the preprocessed image data. Specifically, it uses OpenCV to adjust the image resolution and convert it to the required format (e.g. PNG).

[0750] Step 4:

[0751] The server analyzes the preprocessed image data to identify the work content and instructions. The input is the preprocessed image data, and the output is data related to the work content and instructions. These elements are extracted using an image generation device.

[0752] Step 5:

[0753] The device (smart glasses) captures the worker's facial expressions and voice and sends them to the server. The input is the worker's facial expression data and voice data, and the output is the emotion data sent to the server.

[0754] Step 6:

[0755] The server analyzes the received emotion data and determines the worker's emotional state. The input is facial expression and voice data, and the output is the emotion recognition result. DeepFace is used to analyze the worker's emotional state.

[0756] Step 7:

[0757] The server generates appropriate work instructions based on the emotion recognition results. The inputs are the emotion recognition results and work content data, and the output is the generated work instructions. Based on the emotional state, it provides simple instructions to workers who are stressed and detailed instructions to workers who are calm.

[0758] Step 8:

[0759] The terminal (smart glasses) displays the generated work instructions. The input is the generated work instruction data, and the output is the work instructions displayed on the smart glasses display. The worker can proceed with the work according to these instructions.

[0760] Through the above processing steps, flexible work instructions are provided according to the emotional state of the field worker, thereby improving work efficiency and accuracy.

[0761] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0762] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0763] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0764] [Third embodiment]

[0765] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0766] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0767] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0768] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0769] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0770] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0771] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0772] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0773] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0774] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0775] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0776] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0777] This invention is a system that quickly and affordably generates high-quality custom picture books based on user-provided drawings by children. The system consists of a series of processes: the user submits the drawings, the system classifies them into characters and backgrounds, creates a story based on the generated information, and finally provides the picture book.

[0778] Program processing

[0779] Process for users to post pictures

[0780] The user uses the website or application of the custom picture book creation service to upload a picture drawn by their child. The device then sends the image data of the picture to the server.

[0781] Picture pre-processing

[0782] The server processes the received image data, adjusting the image resolution and format to convert it into a state suitable for AI processing.

[0783] Character generation using image generation AI

[0784] The server inputs the preprocessed images into an image generation AI to identify elements in the picture (such as characters and backgrounds), and generates each element as an independent character model or background as needed.

[0785] Story generation using text generation AI

[0786] The server uses text generation AI to automatically generate a story based on the generated characters and background information. The AI ​​creates a storyline based on the characteristics of the characters and the atmosphere of the illustrations.

[0787] Picture book compilation and preview generation

[0788] The server combines the generated story with the original illustrations to create a page layout, and creates a digital preview that can be sent to the user's device for review.

[0789] User verification and customization

[0790] The user checks the preview data and, if necessary, makes corrections or additional customizations. The customization details are sent from the device to the server.

[0791] Final generation and delivery

[0792] The server generates the final picture book data that reflects the user's customizations and provides it for viewing on the web. Printed versions are also available as an optional paid option.

[0793] Specific examples

[0794] 1. The process of posting a picture

[0795] The parent user takes a photo of their child's drawing of a "blue dragon" with their smartphone and uploads it to the service's app.

[0796] 2. Picture preprocessing

[0797] After the server receives the uploaded image, it adjusts the resolution to 300 dpi and converts it to PNG format.

[0798] 3. Character Creation Using Image Generation AI

[0799] The server uses image generation AI to extract the "blue dragon" as the main character and converts its features into a digital model.

[0800] 4. Story Generation Using Text Generation AI

[0801] The server then has the text generation AI create a story with the "blue dragon" as the main character. For example, a story with the theme of "the blue dragon's adventure to save his friend" is generated.

[0802] 5. Picture book compilation and preview generation

[0803] The server integrates the generated story with the original illustrations, creates a page-by-page layout, and generates a digital preview that is sent to the user.

[0804] 6. User Verification and Customization

[0805] The user sees the preview and decides they want to change the dragon's name to "Drago." They make the changes in the app and submit the changes to the server.

[0806] 7. Final Generation and Delivery

[0807] The server generates the final version of the picture book data that reflects the user's changes and provides it for viewing on the web. If the user wants a printed version, they can order it for a fee.

[0808] In this way, children's drawings are quickly turned into engaging custom picture books.

[0809] The processing flow will be explained below.

[0810] Step 1:

[0811] Users access the service's website or app and log in. After logging in, they click the "Upload a picture" button, which opens a file selection dialog.

[0812] Step 2:

[0813] The user selects and uploads an image file of the picture drawn by the child, and the device sends the selected image file to the server.

[0814] Step 3:

[0815] The server checks the received image data, checking the image format and resolution, and displays an error message if they are inappropriate.

[0816] Step 4:

[0817] The server performs preprocessing on the image data, including adjusting the resolution and filtering noise. The preprocessed image data is then temporarily stored.

[0818] Step 5:

[0819] The server inputs the preprocessed image data into the image generation AI, which detects the main elements in the image (such as characters and backgrounds) and generates each element as an independent character model or background.

[0820] Step 6:

[0821] The server stores the generated character models and background data, which are then input into the text generation AI.

[0822] Step 7:

[0823] The server sets the necessary parameters for the text generation AI (e.g., character personality, atmosphere, etc.), and the text generation AI automatically generates a story based on the provided information.

[0824] Step 8:

[0825] The server combines the generated story with character models and background data to automatically generate a layout for each page.

[0826] Step 9:

[0827] The server creates a digital preview of the generated picture book, temporarily stores it, and generates a preview link and notifies the user.

[0828] Step 10:

[0829] Users click the link sent to them to see the digital preview, customize it as needed (e.g., change the character's name), and enter their details.

[0830] Step 11:

[0831] The terminal transmits the customization information entered by the user to the server.

[0832] Step 12:

[0833] The server reflects the received customization information in the final version of the picture book, generates the final version of the picture book data, and provides it for viewing on the web.

[0834] Step 13:

[0835] The user can check the final version of the picture book and download it if necessary. If the user wants a printed version, the terminal sends the order information to the server.

[0836] Step 14:

[0837] The server accepts the print order and initiates the necessary printing and delivery procedures. After the order is completed, the server notifies the user of the delivery status.

[0838] These steps allow us to quickly generate high-quality custom picture books from children's drawings.

[0839] Example 1

[0840] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0841] In the conventional picture book creation process, it was difficult for users to easily and quickly generate custom picture books based on their children's drawings. Furthermore, there was a lack of automatic generation functions using digitalization and AI, forcing users to perform time-consuming and laborious manual work. This made it difficult to provide custom picture books quickly and with high quality.

[0842] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0843] In this invention, the server includes means for receiving image data sent by a user, means for preprocessing the received image data, means for generating characters using an image generation device with the preprocessed image data as input, means for generating a story using a text generation device with the generated characters and background information as input, means for automatically generating a page layout for a picture book by combining the story and character information, means for providing the user with an electronic version of the generated picture book, means for receiving customization information from the user and generating a final version of the picture book that reflects the information, means for transmitting input data from the terminal to the server, and data processing and conversion means in the server. This enables the rapid automatic generation of a high-quality customized picture book from pictures provided by the user.

[0844] "User" refers to an individual or group that uses the system to send image data of children's drawings.

[0845] "Image data" refers to the data used to digitally store and send pictures drawn by children to the system.

[0846] "Preprocessing" refers to the process performed to convert received image data into a state suitable for AI processing. Specifically, this includes adjusting the resolution and converting the format.

[0847] An "image generation device" is a device or software that takes preprocessed image data as input, analyzes the image, and extracts and generates characters and backgrounds.

[0848] "Characters" are characters extracted from the image data sent by the user, and appear in the story of the picture book.

[0849] "Background information" refers to background elements and settings extracted from the image data sent by the user.

[0850] A "story generation device" is a device or software that automatically generates a story using generated characters and background information as input.

[0851] A "story" is a story that is automatically generated by the text generation device based on the generated characters and background information.

[0852] "Page layout" refers to the design that determines the layout of each page of a picture book by combining the generated story and character information.

[0853] "Electronic Edition" means a digital version of the generated picture book that can be viewed or downloaded by a user.

[0854] "Customization information" refers to information that a user inputs after checking a preview to reflect corrections or additional changes.

[0855] "Device" means the device used by a User to capture, upload, view, and customize Pictures, including smartphones and computers.

[0856] A "server" is a computer system that performs a series of processes, such as processing transmitted image data, preprocessing, character generation, story generation, and page layout creation.

[0857] "Data processing and conversion means" refers to software and hardware for a series of data conversion, analysis, and processing performed on the server.

[0858] This invention is a system that quickly and affordably generates high-quality custom picture books based on user-provided drawings by children. The system consists of a series of processes: the user submits the drawings, the system classifies them into characters and backgrounds, creates a story based on the generated information, and finally provides the picture book.

[0859] This system uses the following hardware and software to process and calculate data.

[0860] First, a user uses a device (such as a smartphone or PC) to photograph or scan a picture drawn by their child, and then uses the custom picture book creation service's website or application to send the image data to the server. This image data is sent to the server using an HTTP POST request. The server then uses Python and its image processing library (such as Pillow) to perform preprocessing, such as adjusting the resolution of the received image data to 300 dpi and converting it from JPEG to PNG format.

[0861] The server then inputs the preprocessed image data into an image generation AI (such as DeepArt or a GAN-based system) to extract important elements (characters and background) from the image. During this process, elements such as a blue dragon and a castle are identified, and these elements are then generated as independent character and background models.

[0862] Furthermore, the server automatically generates a story using a text generation AI (for example, OpenAI's GPT-4) based on the identified characters and background information. Specifically, there is an example where a prompt sentence such as "An adventure in which a blue dragon saves his friend" is input into the text generation AI to create a storyline.

[0863] The server then combines the generated story with the original illustrations to create a page-by-page layout. During this process, a dedicated layout engine is used to generate a digital preview of the book for the user to review. The preview data is then sent to the device as an HTTP response.

[0864] The user checks the digital preview on their device and makes any necessary corrections or customizations (for example, changing the dragon's name to "Drago"). The customizations are sent back to the server, which then generates the final picture book data reflecting the user's customizations.

[0865] The final picture book data is provided to users in a format that can be viewed online from the server. If the user wishes to purchase a printed version, they can order it for a fee and have it delivered.

[0866] Examples of specific prompts are:

[0867] Create a story starring your child's blue dragon and the dragon's adventures helping his friends.

[0868] In this way, a system is realized that quickly generates high-quality custom picture books based on user-provided illustrations.

[0869] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0870] Step 1:

[0871] A user uses the website or application of a custom picture book creation service to take a photo or scan a picture drawn by their child with a smartphone or PC and upload it. The device sends the image data to the server via an HTTP POST request. The input to this step is the image data sent from the user's device, and the output is the image data received by the server. Specifically, the user launches the application, takes a photo or selects an image from the gallery, and clicks the upload button.

[0872] Step 2:

[0873] The server preprocesses the received image data. First, it adjusts the image resolution to 300 dpi and converts the image format from JPEG to PNG. This is done using Python's Pillow library, for example. The input to this step is the raw image data received by the server, and the output is preprocessed image data with the resolution adjusted and format converted. Specifically, it resizes and converts the image using functions from the Pillow library.

[0874] Step 3:

[0875] The server inputs the preprocessed image data into an image generation AI, which identifies key elements in the image (characters and background) and generates independent digital models of each. The input for this step is the preprocessed image data, and the output is digital models of the identified characters and background. Specifically, the data is input into DeepArt or a GAN-based AI model to perform classification and model generation.

[0876] Step 4:

[0877] The server inputs a prompt into a text generation AI based on the identified characters and background, automatically generating a story. For example, the prompt "A blue dragon goes on an adventure to save his friends" is input into a text generation AI such as GPT-4. The input for this step is a digital model of the characters and background and the prompt, and the output is the generated story text. Specifically, the prompt and character data are input into the text generation AI, and the generated story is received in text format.

[0878] Step 5:

[0879] The server combines the generated story with the original illustrations to create a layout for each page. In this process, a dedicated layout engine is used to generate a digital preview of the picture book. The input for this step is the story text and a digital model of the original illustrations, and the output is the completed digital preview. Specifically, the layout engine automatically creates the page structure and generates the preview.

[0880] Step 6:

[0881] The user checks the digital preview and makes any necessary corrections or customizations. For example, they input the customization details, such as changing the dragon's name to "Drago," and send it to the server. The input in this step is the user's customization instructions, and the output is data sent to the server that has received the correction instructions. Specifically, the user selects the part to be corrected within the application, enters the changes, and sends them.

[0882] Step 7:

[0883] The server generates the final picture book data that reflects the user's customizations and provides it in a format that can be viewed on the web. If the user wishes, it can also accept orders for a printed version as a paid option and handles the delivery procedures. The input for this step is the customization information received from the user, and the output is the final picture book data and print order data. Specific operations include generating the final data, uploading it to the web, and managing the printing and delivery procedures.

[0884] (Application example 1)

[0885] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0886] Conventional custom picture book creation systems lack the ability to easily share content created by users with other users, and the process of creating a custom picture book is often cumbersome and time-consuming. This makes it difficult for users to easily and quickly create custom picture books based on their children's drawings and share them with other family and friends. Therefore, to improve the user experience, there is a need for a system that allows users to create high-quality picture books through a simple process and share those picture books with other users.

[0887] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0888] In this invention, the server includes means for receiving image data sent by a user, means for preprocessing the received image data, means for generating characters using an image generation device with the preprocessed image data as input, means for generating a story using a text generation device with the generated characters and background information as input, means for automatically generating a page layout for a picture book by combining the story and character information, means for providing the user with an electronic version of the generated picture book, means for receiving customization information from the user and generating a final version of the picture book that reflects the information, and means for sharing the generated picture book with other users within the application. This enables users to quickly create high-quality custom picture books based on their children's drawings and easily share them with other family and friends.

[0889] "Submitted image data" refers to user-generated visual information that a user uploads to the custom picture book creation system.

[0890] "Preprocessing" is the act of converting received image data into an appropriate format and resolution, making it suitable for subsequent processing steps.

[0891] An "image generation device" is a device that analyzes image data received as input and automatically generates characters and backgrounds.

[0892] A "text generation device" is a device that creates a story based on the characters and background information generated by an image generation device.

[0893] "Page layout" is a layout design that automatically arranges the structure of each page of a picture book by combining story and character information.

[0894] An "electronic version" is a picture book that is provided in digital format rather than in physical print form.

[0895] "Customization information" is information that reflects modifications and additional requirements made by the user.

[0896] The "final version" is a completed picture book that reflects the customization information provided by the user.

[0897] "Means for sharing" refers to the ability to send created custom picture books to other users within the application.

[0898] The present invention is a system for quickly and affordably generating high-quality custom picture books from user-supplied drawings by children. Specific methods for implementing the present invention are described below.

[0899] First, a user uses a smartphone app to take a picture of a drawing their child has made and upload it to the system. The uploaded image data is sent from the device to the server. The server then preprocesses the received image data and adjusts the image resolution and format. This preprocessing is performed using image processing software such as PIL (Python Imaging Library).

[0900] The preprocessed image data is then fed into an image generator (e.g., a system using a specific AI model) that analyzes the image data and defines the characters and background. For example, if a child's drawing of a "blue dragon" is uploaded, the image generator will extract the "blue dragon" from the drawing as the main character and generate a digital model of it.

[0901] The generated characters and background information are then passed to a text generator. The text generator automatically generates a story based on the characteristics of the characters and the atmosphere of the illustration. At this stage, a generative AI model is used. For example, a story with the theme of "a blue dragon's adventure to save his friend" is generated. An example prompt might be, "Generate a story featuring a child's drawing of a 'blue dragon.' This dragon is brave and embarks on an adventure to save a friend in trouble. Draw what difficulties he faces during his adventure and how he overcomes them."

[0902] The generated story and character information is compiled into a picture book using an automatic page layout generator. The server combines these to create a page-by-page structure and generate a digital preview. The user can check this preview on their smartphone and make corrections or additional customizations as needed. The user's customization information is then sent to the server, and the final picture book is generated.

[0903] The server then provides the user with an electronic version of the final generated picture book. The application also includes a means for sharing the generated custom picture book with other users. For example, the user can easily share the generated picture book with family and friends. The server also provides functionality for the user to order a printed version of the generated picture book and arrange for delivery, if desired.

[0904] In this way, the present invention provides a system that enables users to quickly and easily create high-quality custom picture books according to their needs, and further enables users to share these picture books.

[0905] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0906] Step 1:

[0907] The user takes a photo of a drawing made by their child through a smartphone app. The user then uploads this image data to the app. The input is the image data taken with the smartphone, and the output is the image data sent to the server. The specific action that occurs at this point is for the user to tap the "Upload Image" button on the app and select or take a photo of the drawing.

[0908] Step 2:

[0909] The server receives image data sent by the user. The input is the uploaded image data, and the output is image data that can be preprocessed. Specifically, the server receives the image data and saves it in storage.

[0910] Step 3:

[0911] The server preprocesses the received image data, including adjusting the resolution and converting the format. The input is the original image data, and the output is the preprocessed image data. Specifically, the server converts the image to RGB using PIL (Python Imaging Library) and resizes the image to 1024x1024.

[0912] Step 4:

[0913] Based on the preprocessed image data, the server generates the characters using an image generation device. The input is the preprocessed image data, and the output is a digital model of the character. Specifically, the server inputs the image data into the AI ​​model, identifies the elements of the picture, and generates the digital model.

[0914] Step 5:

[0915] Based on the generated characters and background information, the server generates a story using a text generator. The input is the characters and background information, and the output is the generated story. Specifically, the server provides a prompt to the generative AI model, which then creates the story. An example of a prompt is, "Generate a story featuring a child's drawing of a 'blue dragon.' This dragon is brave and sets out on an adventure to help a friend in trouble. Draw what difficulties he faces during his adventure and how he overcomes them."

[0916] Step 6:

[0917] The server automatically generates page layouts based on story and character information. The input is story and character information, and the output is the page layout of a picture book. Specifically, the server integrates the generated story with image elements and automatically formats the layout of each page.

[0918] Step 7:

[0919] The server provides the generated digital version of the picture book to the user. The input is the page layout and story, and the output is the digital version of the picture book data. Specifically, the server generates a digital file of the picture book and sends it via the app so that the user can view it.

[0920] Step 8:

[0921] The user checks the digital version of the picture book and inputs customization information. The input is the digital version of the picture book data, and the output is the customization information specified by the user. Specifically, the user makes modifications such as changing the names of characters and text through the app interface.

[0922] Step 9:

[0923] The server generates the final version of the picture book by reflecting the user's customization information. The input is the user's customization information, and the output is the final version of the picture book data. Specifically, the server applies the received customization information to the picture book pages to complete the final version.

[0924] Step 10:

[0925] The server provides a means for users to share the final version of the picture book with other users within the application. The input is the final version of the picture book data, and the output is the shared picture book data. Specifically, the server generates a sharing link for the picture book, allowing users to send it to family and friends.

[0926] These steps result in a system that allows users to quickly create high-quality custom picture books based on their children's drawings and easily share them with other family and friends.

[0927] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0928] This invention is a system that quickly and affordably generates high-quality custom picture books based on user-provided drawings by children. The system consists of a series of processes: the user submits a picture, the system classifies it into characters and backgrounds, creates a story based on the generated information, and finally provides the picture book. The system also incorporates an emotion engine that recognizes the user's emotions and has the ability to adjust the content of the story and the facial expressions of the characters based on the user's emotions.

[0929] Program processing

[0930] Process for users to post pictures

[0931] The user uses the website or application of the custom picture book creation service to upload a picture drawn by their child. The device then sends the image data of the picture to the server.

[0932] Picture pre-processing

[0933] The server processes the received image data, adjusting the image resolution and format to convert it into a state suitable for AI processing.

[0934] Character generation using image generation AI

[0935] The server inputs the preprocessed images into an image generation AI to identify elements in the picture (such as characters and backgrounds), and generates each element as an independent character model or background as needed.

[0936] Story generation using text generation AI

[0937] The server uses text generation AI to automatically generate a story based on the generated characters and background information. The AI ​​creates a storyline based on the characteristics of the characters and the atmosphere of the illustrations.

[0938] Recognizing user emotions with an emotion engine

[0939] The device collects the user's facial expressions and voice through a camera and microphone, and sends this data to a server.

[0940] The server uses an emotion engine to analyze the received facial expression data and voice data and determine the user's emotions.

[0941] Emotion-based story adjustment

[0942] The server then adjusts the story and the characters' expressions based on the determined emotional information. For example, if the user is enjoying the story, it will brighten up the story.

[0943] Picture book compilation and preview generation

[0944] The server combines the generated story with the original illustrations to create a page layout, and creates a digital preview that can be sent to the user's device for review.

[0945] User verification and customization

[0946] The user checks the preview data and, if necessary, makes corrections or additional customizations. The customization details are sent from the device to the server.

[0947] Final generation and delivery

[0948] The server generates the final picture book data that reflects the user's customizations and provides it for viewing on the web. Printed versions are also available as an optional paid option.

[0949] Specific examples

[0950] 1. The process of posting a picture

[0951] The parent user takes a photo of their child's drawing of a "blue dragon" with their smartphone and uploads it to the service's app.

[0952] 2. Picture preprocessing

[0953] After the server receives the uploaded image, it adjusts the resolution to 300 dpi and converts it to PNG format.

[0954] 3. Character Creation Using Image Generation AI

[0955] The server uses image generation AI to extract the "blue dragon" as the main character and converts its features into a digital model.

[0956] 4. Story Generation Using Text Generation AI

[0957] The server then has the text generation AI create a story with the "blue dragon" as the main character. For example, a story with the theme of "the blue dragon's adventure to save his friend" is generated.

[0958] 5. User Emotion Recognition by Emotion Engine

[0959] As the user reads parts of the story, emotions are collected through facial expressions and voice, and each emotional state is sent to the server.

[0960] The server analyzes the user's emotions through an emotion engine and adjusts the story and characters.

[0961] 6. Adjust your story based on emotions

[0962] If the server determines that the user is enjoying themselves, it will change the story to a brighter, more enjoyable one, and the character's facial expressions will also change accordingly.

[0963] 7. Picture book compilation and preview generation

[0964] The server combines the generated story, the original illustrations, and the emotional adjustments to create a page-by-page layout, creating a digital preview and sending it to the user.

[0965] 8. User Verification and Customization

[0966] The user checks the preview, makes any necessary changes, and then sends the customizations back to the server.

[0967] 9. Final Generation and Delivery

[0968] The server generates the final version of the picture book that reflects the user's changes and provides it on the web. If the user wishes, it also accepts an order for a printed version and begins the delivery process.

[0969] In this way, a system is provided that can quickly generate high-quality custom picture books from children's drawings and flexibly adjust the content of the story according to the user's emotions.

[0970] The processing flow will be explained below.

[0971] Step 1:

[0972] Users access the service's website or app and log in. After logging in, they click the "Upload a picture" button, which opens a file selection dialog.

[0973] Step 2:

[0974] The user selects and uploads an image file of the picture drawn by the child, and the device sends the selected image file to the server.

[0975] Step 3:

[0976] The server checks the received image data, checking the image format and resolution, and displays an error message if they are inappropriate.

[0977] Step 4:

[0978] The server performs preprocessing on the image data, including adjusting the resolution and filtering noise. The preprocessed image data is then temporarily stored.

[0979] Step 5:

[0980] The server inputs the preprocessed image data into the image generation AI, which detects the main elements in the image (such as characters and backgrounds) and generates each element as an independent character model or background.

[0981] Step 6:

[0982] The server stores the generated character models and background data, which are then input into the text generation AI.

[0983] Step 7:

[0984] The server sets the necessary parameters for the text generation AI (e.g., character personality, atmosphere, etc.), and the text generation AI automatically generates a story based on the provided information.

[0985] Step 8:

[0986] The device collects the user's facial expressions and voice through a camera and microphone, and transmits the emotional data to a server.

[0987] Step 9:

[0988] The server uses an emotion engine to analyze the received facial expression data and voice data and determine the user's emotions.

[0989] Step 10:

[0990] The server then adjusts the story and the characters' expressions based on the determined emotional information. For example, if the user is enjoying the story, it will brighten up the story.

[0991] Step 11:

[0992] The server combines the generated story with character models, background data, and the results of adjustments made by the emotion engine to automatically generate a layout for each page.

[0993] Step 12:

[0994] The server creates a digital preview of the generated picture book, temporarily stores it, and generates a preview link and notifies the user.

[0995] Step 13:

[0996] Users click the link sent to them to see the digital preview, customize it as needed (for example, change the character's name), and enter their details.

[0997] Step 14:

[0998] The terminal transmits the customization information entered by the user to the server.

[0999] Step 15:

[1000] The server reflects the received customization information in the final version of the picture book, generates the final version of the picture book data, and provides it for viewing on the web.

[1001] Step 16:

[1002] The user can check the final version of the picture book and download it if necessary. If the user wants a printed version, the terminal sends the order information to the server.

[1003] Step 17:

[1004] The server accepts the print order and initiates the necessary printing and delivery procedures. After the order is completed, the server notifies the user of the delivery status.

[1005] The above steps provide a system that can quickly generate high-quality custom picture books from children's drawings and flexibly adjust the story content according to the user's emotions.

[1006] Example 2

[1007] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1008] Conventional custom picture book generation systems have difficulty generating high-quality picture books quickly based on user-provided images. Furthermore, they lack the ability to adjust the story and character expressions to reflect the user's emotions, making it impossible to provide a personalized experience. This has led to problems such as low user satisfaction and low retention rates.

[1009] The specification processing by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving image data sent by a user, means for preprocessing the received image data, means for generating characters using an image generation device with the preprocessed image data as input, means for generating a story using a text generation device with the generated characters and background information as input, means for recognizing the user's emotional state, means for adjusting the story and character facial expressions based on the recognized emotions, means for automatically generating a page layout for a picture book by combining the story and character information, means for providing the user with an electronic version of the generated picture book, and means for receiving customization information from the user and generating a final picture book reflecting the information. This makes it possible to quickly process images provided by the user and adjust the story and character facial expressions according to the user's emotions.

[1010] A "user" is an end user who uses the custom picture book creation service to upload a picture drawn by a child and request the creation of a picture book.

[1011] A "server" is a central processing unit that performs various processes such as image processing, data analysis, and AI models.

[1012] "Image data" is a digital image file of a child's drawing provided by the user.

[1013] "Preprocessing" refers to initial data processing to facilitate subsequent AI processing, such as adjusting the image data resolution, converting formats, removing noise, and correcting colors.

[1014] An "image generation device" is an AI device that extracts characters and backgrounds from preprocessed image data and generates digital character models and background models.

[1015] "Characters" are characters or objects depicted in a user-provided picture.

[1016] "Background information" refers to information about the overall scenery and environment of the picture, excluding the characters.

[1017] A "text generation device" is an AI device that automatically generates a story based on generated characters and background information.

[1018] A "story" is an automatically generated story based on the characteristics and background information of the characters.

[1019] The "emotional state" is an emotional state such as joy, surprise, sadness, etc., that is analyzed from the user's facial expression and voice.

[1020] The "emotion engine" is an AI engine that analyzes the user's facial expressions and voice data to determine their emotional state.

[1021] "Page layout" refers to the layout of each page of a picture book, which is created by combining the generated story and character information.

[1022] "Electronic version" refers to the final version of the picture book data provided in digital format.

[1023] "Customization information" refers to the contents and settings of the picture book that the user checks in the preview and adds or changes as necessary.

[1024] The "final version" is the completed picture book data that is generated by reflecting the user's customization information.

[1025] This invention is a system that quickly generates high-quality custom picture books based on user-provided drawings by children. The system works in cooperation with the user, device, and server elements, and uses multiple AI models to pre-process images, generate characters, generate stories, and adjust content based on the user's emotions.

[1026] Hardware and software used

[1027] Terminal

[1028] A device is a device used by a user, such as a smartphone, tablet, or PC. Users take and upload pictures through these devices. Devices are equipped with cameras and microphones, and also have the ability to collect facial expressions and voice data from users.

[1029] server

[1030] The server is the center of the picture book generation system and has multiple AI models and engines for performing the following processes:

[1031] Image pre-processing engine: Performs pre-processing such as image resolution adjustment, noise reduction, and color correction.

[1032] Image generation AI model: Generates characters and backgrounds from preprocessed image data.

[1033] Text generation AI model: Generates a story based on characters and background information.

[1034] Emotion engine: Analyzes and determines emotions from the user's facial expressions and voice data.

[1035] Implementation Procedure

[1036] Users submit pictures

[1037] Users take a photo of their child's drawing using a smartphone or computer and upload the image file using the custom picture book creation service's website or application.

[1038] Picture pre-processing

[1039] The server passes the received image data to a pre-processing engine, which sets the resolution to 300 dpi, converts the image format to PNG, and performs noise removal and color correction to prepare the image for processing by the AI ​​model.

[1040] Character generation using image generation AI

[1041] The server inputs the preprocessed images into an image generation AI model to identify the main characters and background, and reconstructs the characters' features as digital models.

[1042] Story generation using text generation AI

[1043] The server uses a text generation AI model to generate a story based on the characters and background information. The AI ​​is given prompts like the following:

[1044] "This image depicts a blue dragon. Please create an adventure story with this dragon as the main character."

[1045] "This blue dragon is a brave character who likes to help his friends. Generate a story based on this setting."

[1046] Recognizing user emotions with an emotion engine

[1047] When users preview their custom picture books, their facial expressions and voice are collected using the device's camera and microphone, and this data is sent to the server in real time and analyzed by the server's emotion engine.

[1048] Emotion-based story adjustment

[1049] The server dynamically changes the story and the characters' expressions based on the analysis results of the emotion engine. If the user is enjoying the story, it will adjust the story to be more cheerful and fun.

[1050] Picture book compilation and preview generation

[1051] The server automatically generates a layout for each page by integrating information about the story and characters, and then creates a digital preview that is sent to the device.

[1052] User verification and customization

[1053] The user checks the preview and edits text or images as necessary. The customization information made on the device is sent to the server.

[1054] Final generation and delivery

[1055] The server generates the final picture book data that reflects the customization information and provides it in a format that can be viewed on the web. If the user requests a printed version, the server generates print data and sends an order to the printing company.

[1056] In this way, user-provided pictures can be quickly transformed into high-quality custom picture books, with content that can be flexibly tailored based on the user's emotions.

[1057] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1058] Step 1: How users post pictures

[1059] Users take a photo of a picture their child has drawn using a smartphone or computer, and then upload the image file using the website or application of the custom picture book creation service. Specifically, users click the "Upload Picture" button, select an image file, and upload it. They then press the "Send" button to send the image data from their device to the server. The input is the user's image data, and the output is the image data sent to the server.

[1060] Step 2: Pre-processing the image

[1061] The server receives image data sent from the device and passes it to the preprocessing engine. It adjusts the resolution of the received image data, for example, setting it to 300 dpi. It also converts the image format from common formats such as JPEG or BMP to PNG. It applies a noise reduction filter and performs color correction. The input is the received image data, and the output is the image data after preprocessing.

[1062] Step 3: Creating characters using image generation AI

[1063] The server inputs the preprocessed image data into the image generation AI model and performs image analysis. Specifically, it inputs a prompt to "extract the main characters and convert them into digital models" into the image generation AI model. The AI ​​identifies the main characters and background in the image and converts their features into digital models. The input is the preprocessed image data, and the output is the generated digital models of the characters and background.

[1064] Step 4: Story generation using text generation AI

[1065] The server uses a text generation AI model to generate a story based on the generated characters and background information. For example, a prompt such as "This image depicts a blue dragon. Please create an adventure story with this dragon as the main character" is input and the AI ​​generates the story. Specifically, based on the characteristics and background information of the characters, a story including the story flow, character actions, dialogue, etc. is automatically created. The input is the characters and background information, and the output is the generated story.

[1066] Step 5: Recognizing user emotions with the emotion engine

[1067] While viewing a preview of a custom picture book, the user uses the device's camera and microphone to collect facial expressions and voice. The device transmits this emotional data to the server in real time. The server's emotion engine analyzes this data and determines the user's emotional state (e.g., joy, surprise, sadness). The input is the user's facial and voice data, and the output is the analyzed emotional state.

[1068] Step 6: Adjust your story based on emotion

[1069] The server receives the analysis results from the emotion engine and adjusts the story and character expressions. For example, if it determines that the user is enjoying themselves, it changes the story to a more cheerful and fun development and adjusts the character expressions accordingly. The input is the analyzed emotional state, and the output is the adjusted story and character expressions.

[1070] Step 7: Compiling the picture book and generating a preview

[1071] The server integrates the generated story, character models, and backgrounds to create the layout for each page. Specifically, it determines the placement of text and images and automatically generates a layout including design elements for each page. Finally, it generates a digital preview and sends it to the device. The input is the adjusted story, character models, and backgrounds, and the output is a digital preview.

[1072] Step 8: Verify and customize users

[1073] The user checks the digital preview sent on their device and makes changes as necessary, such as editing text, changing images, and adjusting character positions. Once the user has finalized the customizations, the data is resent from their device to the server. The input is the digital preview, and the output is data reflecting the customizations.

[1074] Step 9: Final generation and serving

[1075] The server generates the final picture book data that reflects the customization information. It then provides it to the user in a format that can be viewed on the web. If the user requests a printed version, the server generates print data and sends an order to the printer. The input is data that reflects the customization details, and the output is the final picture book data. It also includes order data for delivering the printed version to the user.

[1076] (Application example 2)

[1077] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1078] There is a demand for improving the efficiency and accuracy of the work that field workers face, but conventional systems have the problem that the work content and instructions are uniform, making it difficult to respond flexibly while taking into account the stress level and concentration of each worker. Furthermore, work instructions can be either too detailed or too simple, which can lead to a decline in work efficiency. In particular, it has been difficult to grasp the emotional state of field workers in real time and provide appropriate instructions based on that.

[1079] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving image data sent by a user; means for preprocessing the received image data; means for generating characters using an image generation device with the preprocessed image data as input; means for generating a story using a text generation device with the generated characters and background information as input; emotion recognition means for analyzing the user's emotional state; means for adjusting the facial expressions of the story and characters based on the emotional state; means for automatically generating a page layout for a picture book by combining the story and character information; means for providing a user with an electronic version of the generated picture book; and means for receiving customization information from the user and generating a final version of the picture book reflecting the information. This makes it possible to provide appropriate work instructions that flexibly correspond to the emotional state of field workers.

[1080] A "user" is a person who uses the service to send image data.

[1081] "Image data" refers to image information sent by a user.

[1082] "Preprocessing" refers to a series of processes that convert received image data into a state suitable for analysis.

[1083] An "image generation device" is a device for generating characters based on pre-processed image data.

[1084] "Characters" are characters generated from image data sent by the user.

[1085] "Background information" refers to information that corresponds to the background of an image generated by an image generation device.

[1086] A "sentence generation device" is a device that automatically generates a story based on generated characters and background information.

[1087] A "story" is a storyline generated by a text generator.

[1088] "Emotion recognition means" refers to means for analyzing the emotional state of a user.

[1089] "Page layout" is the process of combining story and character information to determine the page structure of a picture book.

[1090] "Digital edition" refers to a picture book provided in digital format.

[1091] "Customization information" refers to modifications or additions requested by the user to the electronic version.

[1092] The "final version" refers to the final picture book data generated by reflecting the customization information.

[1093] This invention is a system that uses emotion recognition to provide appropriate work instructions to field workers based on smart glasses. Specific embodiments of this system will be described below.

[1094] Hardware and software used

[1095] Hardware: Smart glasses, webcam

[1096] Software: OpenCV (image processing), DeepFace (emotion recognition), Flask server (data transmission and reception)

[1097] Data processing and calculation

[1098] 1. User: A field worker wears smart glasses and takes a photo of the work site. This photo data is captured by the camera in the smart glasses.

[1099] 2. Terminal: The smart glasses transmit the acquired image data to the server using wireless communication technologies such as Wi-Fi or Bluetooth.

[1100] 3. Server: Preprocesses the received image data. OpenCV is used to adjust the image resolution and format, converting it into a suitable state for analysis.

[1101] 4. Server: Analyzes the preprocessed image data to identify the work content and instructions. It also recognizes emotions based on the worker's facial expressions and voice. It uses DeepFace to analyze the worker's emotional state.

[1102] 5. Terminal: The server generates appropriate work instructions based on the emotion recognition results. For example, if a worker is feeling stressed, a simple instruction will be displayed.

[1103] 6. Terminal: The generated work instructions are displayed on the smart glasses display and provided to the worker.

[1104] Specific examples

[1105] As an example, consider a scenario in which a worker in an automobile parts manufacturing plant uses smart glasses to install parts. The worker takes a photo of the work site with the camera in the smart glasses and sends the image to a server. The server processes the image to identify the installation location and uses DeepFace to analyze the worker's emotional state. If the worker is feeling stressed, the server generates a concise instruction and displays it on the smart glasses. This improves work efficiency and relieves the worker's stress.

[1106] Prompt Sentence Examples

[1107] "The user wears the smart glasses and takes photos of the work site. The system generates a manual of work procedures using the photos, and also implements a function to detect the wearer's stress level and adjust the instructions accordingly based on their emotions."

[1108] This invention makes it possible to provide flexible and appropriate work instructions based on the emotional state of field workers, thereby improving work efficiency and accuracy and reducing worker stress.

[1109] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1110] Step 1:

[1111] A user wears the smart glasses and takes a photo of the work site. The input is image data of the site, and the output is the captured image data, which is used in subsequent processing steps.

[1112] Step 2:

[1113] The device (smart glasses) sends the captured image data to the server. The input is the image data acquired in step 1, and the output is the image data sent to the server. Wi-Fi and Bluetooth are used as communication methods.

[1114] Step 3:

[1115] The server preprocesses the received image data. The input is the transmitted image data, and the output is the preprocessed image data. Specifically, it uses OpenCV to adjust the image resolution and convert it to the required format (e.g. PNG).

[1116] Step 4:

[1117] The server analyzes the preprocessed image data to identify the work content and instructions. The input is the preprocessed image data, and the output is data related to the work content and instructions. These elements are extracted using an image generation device.

[1118] Step 5:

[1119] The device (smart glasses) captures the worker's facial expressions and voice and sends them to the server. The input is the worker's facial expression data and voice data, and the output is the emotion data sent to the server.

[1120] Step 6:

[1121] The server analyzes the received emotion data and determines the worker's emotional state. The input is facial expression and voice data, and the output is the emotion recognition result. DeepFace is used to analyze the worker's emotional state.

[1122] Step 7:

[1123] The server generates appropriate work instructions based on the emotion recognition results. The inputs are the emotion recognition results and work content data, and the output is the generated work instructions. Based on the emotional state, it provides simple instructions to workers who are stressed and detailed instructions to workers who are calm.

[1124] Step 8:

[1125] The terminal (smart glasses) displays the generated work instructions. The input is the generated work instruction data, and the output is the work instructions displayed on the smart glasses display. The worker can proceed with the work according to these instructions.

[1126] Through the above processing steps, flexible work instructions are provided according to the emotional state of the field worker, thereby improving work efficiency and accuracy.

[1127] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1128] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1129] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1130] [Fourth embodiment]

[1131] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1132] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1133] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1134] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1135] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1136] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1137] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1138] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1139] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1140] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1141] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1142] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1143] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1144] This invention is a system that quickly and affordably generates high-quality custom picture books based on user-provided drawings by children. The system consists of a series of processes: the user submits the drawings, the system classifies them into characters and backgrounds, creates a story based on the generated information, and finally provides the picture book.

[1145] Program processing

[1146] Process for users to post pictures

[1147] The user uses the website or application of the custom picture book creation service to upload a picture drawn by their child. The device then sends the image data of the picture to the server.

[1148] Picture pre-processing

[1149] The server processes the received image data, adjusting the image resolution and format to convert it into a state suitable for AI processing.

[1150] Character generation using image generation AI

[1151] The server inputs the preprocessed images into an image generation AI to identify elements in the picture (such as characters and backgrounds), and generates each element as an independent character model or background as needed.

[1152] Story generation using text generation AI

[1153] The server uses text generation AI to automatically generate a story based on the generated characters and background information. The AI ​​creates a storyline based on the characteristics of the characters and the atmosphere of the illustrations.

[1154] Picture book compilation and preview generation

[1155] The server combines the generated story with the original illustrations to create a page layout, and creates a digital preview that can be sent to the user's device for review.

[1156] User verification and customization

[1157] The user checks the preview data and, if necessary, makes corrections or additional customizations. The customization details are sent from the device to the server.

[1158] Final generation and delivery

[1159] The server generates the final picture book data that reflects the user's customizations and provides it for viewing on the web. Printed versions are also available as an optional paid option.

[1160] Specific examples

[1161] 1. The process of posting a picture

[1162] The parent user takes a photo of their child's drawing of a "blue dragon" with their smartphone and uploads it to the service's app.

[1163] 2. Picture preprocessing

[1164] After the server receives the uploaded image, it adjusts the resolution to 300 dpi and converts it to PNG format.

[1165] 3. Character Creation Using Image Generation AI

[1166] The server uses image generation AI to extract the "blue dragon" as the main character and converts its features into a digital model.

[1167] 4. Story Generation Using Text Generation AI

[1168] The server then has the text generation AI create a story with the "blue dragon" as the main character. For example, a story with the theme of "the blue dragon's adventure to save his friend" is generated.

[1169] 5. Picture book compilation and preview generation

[1170] The server integrates the generated story with the original illustrations, creates a page-by-page layout, and generates a digital preview that is sent to the user.

[1171] 6. User Verification and Customization

[1172] The user sees the preview and decides they want to change the dragon's name to "Drago." They make the changes in the app and submit the changes to the server.

[1173] 7. Final Generation and Delivery

[1174] The server generates the final version of the picture book data that reflects the user's changes and provides it for viewing on the web. If the user wants a printed version, they can order it for a fee.

[1175] In this way, children's drawings are quickly turned into engaging custom picture books.

[1176] The processing flow will be explained below.

[1177] Step 1:

[1178] Users access the service's website or app and log in. After logging in, they click the "Upload a picture" button, which opens a file selection dialog.

[1179] Step 2:

[1180] The user selects and uploads an image file of the picture drawn by the child, and the device sends the selected image file to the server.

[1181] Step 3:

[1182] The server checks the received image data, checking the image format and resolution, and displays an error message if they are inappropriate.

[1183] Step 4:

[1184] The server performs preprocessing on the image data, including adjusting the resolution and filtering noise. The preprocessed image data is then temporarily stored.

[1185] Step 5:

[1186] The server inputs the preprocessed image data into the image generation AI, which detects the main elements in the image (such as characters and backgrounds) and generates each element as an independent character model or background.

[1187] Step 6:

[1188] The server stores the generated character models and background data, which are then input into the text generation AI.

[1189] Step 7:

[1190] The server sets the necessary parameters for the text generation AI (e.g., character personality, atmosphere, etc.), and the text generation AI automatically generates a story based on the provided information.

[1191] Step 8:

[1192] The server combines the generated story with character models and background data to automatically generate a layout for each page.

[1193] Step 9:

[1194] The server creates a digital preview of the generated picture book, temporarily stores it, and generates a preview link and notifies the user.

[1195] Step 10:

[1196] Users click the link sent to them to see the digital preview, customize it as needed (e.g., change the character's name), and enter their details.

[1197] Step 11:

[1198] The terminal transmits the customization information entered by the user to the server.

[1199] Step 12:

[1200] The server reflects the received customization information in the final version of the picture book, generates the final version of the picture book data, and provides it for viewing on the web.

[1201] Step 13:

[1202] The user can check the final version of the picture book and download it if necessary. If the user wants a printed version, the terminal sends the order information to the server.

[1203] Step 14:

[1204] The server accepts the print order and initiates the necessary printing and delivery procedures. After the order is completed, the server notifies the user of the delivery status.

[1205] These steps allow us to quickly generate high-quality custom picture books from children's drawings.

[1206] Example 1

[1207] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1208] In the conventional picture book creation process, it was difficult for users to easily and quickly generate custom picture books based on their children's drawings. Furthermore, there was a lack of automatic generation functions using digitalization and AI, forcing users to perform time-consuming and laborious manual work. This made it difficult to provide custom picture books quickly and with high quality.

[1209] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1210] In this invention, the server includes means for receiving image data sent by a user, means for preprocessing the received image data, means for generating characters using an image generation device with the preprocessed image data as input, means for generating a story using a text generation device with the generated characters and background information as input, means for automatically generating a page layout for a picture book by combining the story and character information, means for providing the user with an electronic version of the generated picture book, means for receiving customization information from the user and generating a final version of the picture book that reflects the information, means for transmitting input data from the terminal to the server, and data processing and conversion means in the server. This enables the rapid automatic generation of a high-quality customized picture book from pictures provided by the user.

[1211] "User" refers to an individual or group that uses the system to send image data of children's drawings.

[1212] "Image data" refers to the data used to digitally store and send pictures drawn by children to the system.

[1213] "Preprocessing" refers to the process performed to convert received image data into a state suitable for AI processing. Specifically, this includes adjusting the resolution and converting the format.

[1214] An "image generation device" is a device or software that takes preprocessed image data as input, analyzes the image, and extracts and generates characters and backgrounds.

[1215] "Characters" are characters extracted from the image data sent by the user, and appear in the story of the picture book.

[1216] "Background information" refers to background elements and settings extracted from the image data sent by the user.

[1217] A "story generation device" is a device or software that automatically generates a story using generated characters and background information as input.

[1218] A "story" is a story that is automatically generated by the text generation device based on the generated characters and background information.

[1219] "Page layout" refers to the design that determines the layout of each page of a picture book by combining the generated story and character information.

[1220] "Electronic Edition" means a digital version of the generated picture book that can be viewed or downloaded by a user.

[1221] "Customization information" refers to information that a user inputs after checking a preview to reflect corrections or additional changes.

[1222] "Device" means the device used by a User to capture, upload, view, and customize Pictures, including smartphones and computers.

[1223] A "server" is a computer system that performs a series of processes, such as processing transmitted image data, preprocessing, character generation, story generation, and page layout creation.

[1224] "Data processing and conversion means" refers to software and hardware for a series of data conversion, analysis, and processing performed on the server.

[1225] This invention is a system that quickly and affordably generates high-quality custom picture books based on user-provided drawings by children. The system consists of a series of processes: the user submits the drawings, the system classifies them into characters and backgrounds, creates a story based on the generated information, and finally provides the picture book.

[1226] This system uses the following hardware and software to process and calculate data.

[1227] First, a user uses a device (such as a smartphone or PC) to photograph or scan a picture drawn by their child, and then uses the custom picture book creation service's website or application to send the image data to the server. This image data is sent to the server using an HTTP POST request. The server then uses Python and its image processing library (such as Pillow) to perform preprocessing, such as adjusting the resolution of the received image data to 300 dpi and converting it from JPEG to PNG format.

[1228] The server then inputs the preprocessed image data into an image generation AI (such as DeepArt or a GAN-based system) to extract important elements (characters and background) from the image. During this process, elements such as a blue dragon and a castle are identified, and these elements are then generated as independent character and background models.

[1229] Furthermore, the server automatically generates a story using a text generation AI (for example, OpenAI's GPT-4) based on the identified characters and background information. Specifically, there is an example where a prompt sentence such as "An adventure in which a blue dragon saves his friend" is input into the text generation AI to create a storyline.

[1230] The server then combines the generated story with the original illustrations to create a page-by-page layout. During this process, a dedicated layout engine is used to generate a digital preview of the book for the user to review. The preview data is then sent to the device as an HTTP response.

[1231] The user checks the digital preview on their device and makes any necessary corrections or customizations (for example, changing the dragon's name to "Drago"). The customizations are sent back to the server, which then generates the final picture book data reflecting the user's customizations.

[1232] The final picture book data is provided to users in a format that can be viewed online from the server. If the user wishes to purchase a printed version, they can order it for a fee and have it delivered.

[1233] Examples of specific prompts are:

[1234] Create a story starring your child's blue dragon and the dragon's adventures helping his friends.

[1235] In this way, a system is realized that quickly generates high-quality custom picture books based on user-provided illustrations.

[1236] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1237] Step 1:

[1238] A user uses the website or application of a custom picture book creation service to take a photo or scan a picture drawn by their child with a smartphone or PC and upload it. The device sends the image data to the server via an HTTP POST request. The input to this step is the image data sent from the user's device, and the output is the image data received by the server. Specifically, the user launches the application, takes a photo or selects an image from the gallery, and clicks the upload button.

[1239] Step 2:

[1240] The server preprocesses the received image data. First, it adjusts the image resolution to 300 dpi and converts the image format from JPEG to PNG. This is done using Python's Pillow library, for example. The input to this step is the raw image data received by the server, and the output is preprocessed image data with the resolution adjusted and format converted. Specifically, it resizes and converts the image using functions from the Pillow library.

[1241] Step 3:

[1242] The server inputs the preprocessed image data into an image generation AI, which identifies key elements in the image (characters and background) and generates independent digital models of each. The input for this step is the preprocessed image data, and the output is digital models of the identified characters and background. Specifically, the data is input into DeepArt or a GAN-based AI model to perform classification and model generation.

[1243] Step 4:

[1244] The server inputs a prompt into a text generation AI based on the identified characters and background, automatically generating a story. For example, the prompt "A blue dragon goes on an adventure to save his friends" is input into a text generation AI such as GPT-4. The input for this step is a digital model of the characters and background and the prompt, and the output is the generated story text. Specifically, the prompt and character data are input into the text generation AI, and the generated story is received in text format.

[1245] Step 5:

[1246] The server combines the generated story with the original illustrations to create a layout for each page. In this process, a dedicated layout engine is used to generate a digital preview of the picture book. The input for this step is the story text and a digital model of the original illustrations, and the output is the completed digital preview. Specifically, the layout engine automatically creates the page structure and generates the preview.

[1247] Step 6:

[1248] The user checks the digital preview and makes any necessary corrections or customizations. For example, they input the customization details, such as changing the dragon's name to "Drago," and send it to the server. The input in this step is the user's customization instructions, and the output is data sent to the server that has received the correction instructions. Specifically, the user selects the part to be corrected within the application, enters the changes, and sends them.

[1249] Step 7:

[1250] The server generates the final picture book data that reflects the user's customizations and provides it in a format that can be viewed on the web. If the user wishes, it can also accept orders for a printed version as a paid option and handles the delivery procedures. The input for this step is the customization information received from the user, and the output is the final picture book data and print order data. Specific operations include generating the final data, uploading it to the web, and managing the printing and delivery procedures.

[1251] (Application example 1)

[1252] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1253] Conventional custom picture book creation systems lack the ability to easily share content created by users with other users, and the process of creating a custom picture book is often cumbersome and time-consuming. This makes it difficult for users to easily and quickly create custom picture books based on their children's drawings and share them with other family and friends. Therefore, to improve the user experience, there is a need for a system that allows users to create high-quality picture books through a simple process and share those picture books with other users.

[1254] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1255] In this invention, the server includes means for receiving image data sent by a user, means for preprocessing the received image data, means for generating characters using an image generation device with the preprocessed image data as input, means for generating a story using a text generation device with the generated characters and background information as input, means for automatically generating a page layout for a picture book by combining the story and character information, means for providing the user with an electronic version of the generated picture book, means for receiving customization information from the user and generating a final version of the picture book that reflects the information, and means for sharing the generated picture book with other users within the application. This enables users to quickly create high-quality custom picture books based on their children's drawings and easily share them with other family and friends.

[1256] "Submitted image data" refers to user-generated visual information that a user uploads to the custom picture book creation system.

[1257] "Preprocessing" is the act of converting received image data into an appropriate format and resolution, making it suitable for subsequent processing steps.

[1258] An "image generation device" is a device that analyzes image data received as input and automatically generates characters and backgrounds.

[1259] A "text generation device" is a device that creates a story based on the characters and background information generated by an image generation device.

[1260] "Page layout" is a layout design that automatically arranges the structure of each page of a picture book by combining story and character information.

[1261] An "electronic version" is a picture book that is provided in digital format rather than in physical print form.

[1262] "Customization information" is information that reflects modifications and additional requirements made by the user.

[1263] The "final version" is a completed picture book that reflects the customization information provided by the user.

[1264] "Means for sharing" refers to the ability to send created custom picture books to other users within the application.

[1265] The present invention is a system for quickly and affordably generating high-quality custom picture books from user-supplied drawings by children. Specific methods for implementing the present invention are described below.

[1266] First, a user uses a smartphone app to take a picture of a drawing their child has made and upload it to the system. The uploaded image data is sent from the device to the server. The server then preprocesses the received image data and adjusts the image resolution and format. This preprocessing is performed using image processing software such as PIL (Python Imaging Library).

[1267] The preprocessed image data is then fed into an image generator (e.g., a system using a specific AI model) that analyzes the image data and defines the characters and background. For example, if a child's drawing of a "blue dragon" is uploaded, the image generator will extract the "blue dragon" from the drawing as the main character and generate a digital model of it.

[1268] The generated characters and background information are then passed to a text generator. The text generator automatically generates a story based on the characteristics of the characters and the atmosphere of the illustration. At this stage, a generative AI model is used. For example, a story with the theme of "a blue dragon's adventure to save his friend" is generated. An example prompt might be, "Generate a story featuring a child's drawing of a 'blue dragon.' This dragon is brave and embarks on an adventure to save a friend in trouble. Draw what difficulties he faces during his adventure and how he overcomes them."

[1269] The generated story and character information is compiled into a picture book using an automatic page layout generator. The server combines these to create a page-by-page structure and generate a digital preview. The user can check this preview on their smartphone and make corrections or additional customizations as needed. The user's customization information is then sent to the server, and the final picture book is generated.

[1270] The server then provides the user with an electronic version of the final generated picture book. The application also includes a means for sharing the generated custom picture book with other users. For example, the user can easily share the generated picture book with family and friends. The server also provides functionality for the user to order a printed version of the generated picture book and arrange for delivery, if desired.

[1271] In this way, the present invention provides a system that enables users to quickly and easily create high-quality custom picture books according to their needs, and further enables users to share these picture books.

[1272] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1273] Step 1:

[1274] The user takes a photo of a drawing made by their child through a smartphone app. The user then uploads this image data to the app. The input is the image data taken with the smartphone, and the output is the image data sent to the server. The specific action that occurs at this point is for the user to tap the "Upload Image" button on the app and select or take a photo of the drawing.

[1275] Step 2:

[1276] The server receives image data sent by the user. The input is the uploaded image data, and the output is image data that can be preprocessed. Specifically, the server receives the image data and saves it in storage.

[1277] Step 3:

[1278] The server preprocesses the received image data, including adjusting the resolution and converting the format. The input is the original image data, and the output is the preprocessed image data. Specifically, the server converts the image to RGB using PIL (Python Imaging Library) and resizes the image to 1024x1024.

[1279] Step 4:

[1280] Based on the preprocessed image data, the server generates the characters using an image generation device. The input is the preprocessed image data, and the output is a digital model of the character. Specifically, the server inputs the image data into the AI ​​model, identifies the elements of the picture, and generates the digital model.

[1281] Step 5:

[1282] Based on the generated characters and background information, the server generates a story using a text generator. The input is the characters and background information, and the output is the generated story. Specifically, the server provides a prompt to the generative AI model, which then creates the story. An example of a prompt is, "Generate a story featuring a child's drawing of a 'blue dragon.' This dragon is brave and sets out on an adventure to help a friend in trouble. Draw what difficulties he faces during his adventure and how he overcomes them."

[1283] Step 6:

[1284] The server automatically generates page layouts based on story and character information. The input is story and character information, and the output is the page layout of a picture book. Specifically, the server integrates the generated story with image elements and automatically formats the layout of each page.

[1285] Step 7:

[1286] The server provides the generated digital version of the picture book to the user. The input is the page layout and story, and the output is the digital version of the picture book data. Specifically, the server generates a digital file of the picture book and sends it via the app so that the user can view it.

[1287] Step 8:

[1288] The user checks the digital version of the picture book and inputs customization information. The input is the digital version of the picture book data, and the output is the customization information specified by the user. Specifically, the user makes modifications such as changing the names of characters and text through the app interface.

[1289] Step 9:

[1290] The server generates the final version of the picture book by reflecting the user's customization information. The input is the user's customization information, and the output is the final version of the picture book data. Specifically, the server applies the received customization information to the picture book pages to complete the final version.

[1291] Step 10:

[1292] The server provides a means for users to share the final version of the picture book with other users within the application. The input is the final version of the picture book data, and the output is the shared picture book data. Specifically, the server generates a sharing link for the picture book, allowing users to send it to family and friends.

[1293] These steps result in a system that allows users to quickly create high-quality custom picture books based on their children's drawings and easily share them with other family and friends.

[1294] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1295] This invention is a system that quickly and affordably generates high-quality custom picture books based on user-provided drawings by children. The system consists of a series of processes: the user submits a picture, the system classifies it into characters and backgrounds, creates a story based on the generated information, and finally provides the picture book. The system also incorporates an emotion engine that recognizes the user's emotions and has the ability to adjust the content of the story and the facial expressions of the characters based on the user's emotions.

[1296] Program processing

[1297] Process for users to post pictures

[1298] The user uses the website or application of the custom picture book creation service to upload a picture drawn by their child. The device then sends the image data of the picture to the server.

[1299] Picture pre-processing

[1300] The server processes the received image data, adjusting the image resolution and format to convert it into a state suitable for AI processing.

[1301] Character generation using image generation AI

[1302] The server inputs the preprocessed images into an image generation AI to identify elements in the picture (such as characters and backgrounds), and generates each element as an independent character model or background as needed.

[1303] Story generation using text generation AI

[1304] The server uses text generation AI to automatically generate a story based on the generated characters and background information. The AI ​​creates a storyline based on the characteristics of the characters and the atmosphere of the illustrations.

[1305] Recognizing user emotions with an emotion engine

[1306] The device collects the user's facial expressions and voice through a camera and microphone, and sends this data to a server.

[1307] The server uses an emotion engine to analyze the received facial expression data and voice data and determine the user's emotions.

[1308] Emotion-based story adjustment

[1309] The server then adjusts the story and the characters' expressions based on the determined emotional information. For example, if the user is enjoying the story, it will brighten up the story.

[1310] Picture book compilation and preview generation

[1311] The server combines the generated story with the original illustrations to create a page layout, and creates a digital preview that can be sent to the user's device for review.

[1312] User verification and customization

[1313] The user checks the preview data and, if necessary, makes corrections or additional customizations. The customization details are sent from the device to the server.

[1314] Final generation and delivery

[1315] The server generates the final picture book data that reflects the user's customizations and provides it for viewing on the web. Printed versions are also available as an optional paid option.

[1316] Specific examples

[1317] 1. The process of posting a picture

[1318] The parent user takes a photo of their child's drawing of a "blue dragon" with their smartphone and uploads it to the service's app.

[1319] 2. Picture preprocessing

[1320] After the server receives the uploaded image, it adjusts the resolution to 300 dpi and converts it to PNG format.

[1321] 3. Character Creation Using Image Generation AI

[1322] The server uses image generation AI to extract the "blue dragon" as the main character and converts its features into a digital model.

[1323] 4. Story Generation Using Text Generation AI

[1324] The server then has the text generation AI create a story with the "blue dragon" as the main character. For example, a story with the theme of "the blue dragon's adventure to save his friend" is generated.

[1325] 5. User Emotion Recognition by Emotion Engine

[1326] As the user reads parts of the story, emotions are collected through facial expressions and voice, and each emotional state is sent to the server.

[1327] The server analyzes the user's emotions through an emotion engine and adjusts the story and characters.

[1328] 6. Adjust your story based on emotions

[1329] If the server determines that the user is enjoying themselves, it will change the story to a brighter, more enjoyable one, and the character's facial expressions will also change accordingly.

[1330] 7. Picture book compilation and preview generation

[1331] The server combines the generated story, the original illustrations, and the emotional adjustments to create a page-by-page layout, creating a digital preview and sending it to the user.

[1332] 8. User Verification and Customization

[1333] The user checks the preview, makes any necessary changes, and then sends the customizations back to the server.

[1334] 9. Final Generation and Delivery

[1335] The server generates the final version of the picture book that reflects the user's changes and provides it on the web. If the user wishes, it also accepts an order for a printed version and begins the delivery process.

[1336] In this way, a system is provided that can quickly generate high-quality custom picture books from children's drawings and flexibly adjust the content of the story according to the user's emotions.

[1337] The processing flow will be explained below.

[1338] Step 1:

[1339] Users access the service's website or app and log in. After logging in, they click the "Upload a picture" button, which opens a file selection dialog.

[1340] Step 2:

[1341] The user selects and uploads an image file of the picture drawn by the child, and the device sends the selected image file to the server.

[1342] Step 3:

[1343] The server checks the received image data, checking the image format and resolution, and displays an error message if they are inappropriate.

[1344] Step 4:

[1345] The server performs preprocessing on the image data, including adjusting the resolution and filtering noise. The preprocessed image data is then temporarily stored.

[1346] Step 5:

[1347] The server inputs the preprocessed image data into the image generation AI, which detects the main elements in the image (such as characters and backgrounds) and generates each element as an independent character model or background.

[1348] Step 6:

[1349] The server stores the generated character models and background data, which are then input into the text generation AI.

[1350] Step 7:

[1351] The server sets the necessary parameters for the text generation AI (e.g., character personality, atmosphere, etc.), and the text generation AI automatically generates a story based on the provided information.

[1352] Step 8:

[1353] The device collects the user's facial expressions and voice through a camera and microphone, and transmits the emotional data to a server.

[1354] Step 9:

[1355] The server uses an emotion engine to analyze the received facial expression data and voice data and determine the user's emotions.

[1356] Step 10:

[1357] The server then adjusts the story and the characters' expressions based on the determined emotional information. For example, if the user is enjoying the story, it will brighten up the story.

[1358] Step 11:

[1359] The server combines the generated story with character models, background data, and the results of adjustments made by the emotion engine to automatically generate a layout for each page.

[1360] Step 12:

[1361] The server creates a digital preview of the generated picture book, temporarily stores it, and generates a preview link and notifies the user.

[1362] Step 13:

[1363] Users click the link sent to them to see the digital preview, customize it as needed (for example, change the character's name), and enter their details.

[1364] Step 14:

[1365] The terminal transmits the customization information entered by the user to the server.

[1366] Step 15:

[1367] The server reflects the received customization information in the final version of the picture book, generates the final version of the picture book data, and provides it for viewing on the web.

[1368] Step 16:

[1369] The user can check the final version of the picture book and download it if necessary. If the user wants a printed version, the terminal sends the order information to the server.

[1370] Step 17:

[1371] The server accepts the print order and initiates the necessary printing and delivery procedures. After the order is completed, the server notifies the user of the delivery status.

[1372] The above steps provide a system that can quickly generate high-quality custom picture books from children's drawings and flexibly adjust the story content according to the user's emotions.

[1373] Example 2

[1374] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1375] Conventional custom picture book generation systems have difficulty generating high-quality picture books quickly based on user-provided images. Furthermore, they lack the ability to adjust the story and character expressions to reflect the user's emotions, making it impossible to provide a personalized experience. This has led to problems such as low user satisfaction and low retention rates.

[1376] The specification processing by the specification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving image data sent by a user, means for preprocessing the received image data, means for generating characters using an image generation device with the preprocessed image data as input, means for generating a story using a text generation device with the generated characters and background information as input, means for recognizing the user's emotional state, means for adjusting the story and character facial expressions based on the recognized emotions, means for automatically generating a page layout for a picture book by combining the story and character information, means for providing the user with an electronic version of the generated picture book, and means for receiving customization information from the user and generating a final picture book reflecting the information. This makes it possible to quickly process images provided by the user and adjust the story and character facial expressions according to the user's emotions.

[1377] A "user" is an end user who uses the custom picture book creation service to upload a picture drawn by a child and request the creation of a picture book.

[1378] A "server" is a central processing unit that performs various processes such as image processing, data analysis, and AI models.

[1379] "Image data" is a digital image file of a child's drawing provided by the user.

[1380] "Preprocessing" refers to initial data processing to facilitate subsequent AI processing, such as adjusting the image data resolution, converting formats, removing noise, and correcting colors.

[1381] An "image generation device" is an AI device that extracts characters and backgrounds from preprocessed image data and generates digital character models and background models.

[1382] "Characters" are characters or objects depicted in a user-provided picture.

[1383] "Background information" refers to information about the overall scenery and environment of the picture, excluding the characters.

[1384] A "text generation device" is an AI device that automatically generates a story based on generated characters and background information.

[1385] A "story" is an automatically generated story based on the characteristics and background information of the characters.

[1386] The "emotional state" is an emotional state such as joy, surprise, sadness, etc., that is analyzed from the user's facial expression and voice.

[1387] The "emotion engine" is an AI engine that analyzes the user's facial expressions and voice data to determine their emotional state.

[1388] "Page layout" refers to the layout of each page of a picture book, which is created by combining the generated story and character information.

[1389] "Electronic version" refers to the final version of the picture book data provided in digital format.

[1390] "Customization information" refers to the contents and settings of the picture book that the user checks in the preview and adds or changes as necessary.

[1391] The "final version" is the completed picture book data that is generated by reflecting the user's customization information.

[1392] This invention is a system that quickly generates high-quality custom picture books based on user-provided drawings by children. The system works in cooperation with the user, device, and server elements, and uses multiple AI models to pre-process images, generate characters, generate stories, and adjust content based on the user's emotions.

[1393] Hardware and software used

[1394] Terminal

[1395] A device is a device used by a user, such as a smartphone, tablet, or PC. Users take and upload pictures through these devices. Devices are equipped with cameras and microphones, and also have the ability to collect facial expressions and voice data from users.

[1396] server

[1397] The server is the center of the picture book generation system and has multiple AI models and engines for performing the following processes:

[1398] Image pre-processing engine: Performs pre-processing such as image resolution adjustment, noise reduction, and color correction.

[1399] Image generation AI model: Generates characters and backgrounds from preprocessed image data.

[1400] Text generation AI model: Generates a story based on characters and background information.

[1401] Emotion engine: Analyzes and determines emotions from the user's facial expressions and voice data.

[1402] Implementation Procedure

[1403] Users submit pictures

[1404] Users take a photo of their child's drawing using a smartphone or computer and upload the image file using the custom picture book creation service's website or application.

[1405] Picture pre-processing

[1406] The server passes the received image data to a pre-processing engine, which sets the resolution to 300 dpi, converts the image format to PNG, and performs noise removal and color correction to prepare the image for processing by the AI ​​model.

[1407] Character generation using image generation AI

[1408] The server inputs the preprocessed images into an image generation AI model to identify the main characters and background, and reconstructs the characters' features as digital models.

[1409] Story generation using text generation AI

[1410] The server uses a text generation AI model to generate a story based on the characters and background information. The AI ​​is given prompts like the following:

[1411] "This image depicts a blue dragon. Please create an adventure story with this dragon as the main character."

[1412] "This blue dragon is a brave character who likes to help his friends. Generate a story based on this setting."

[1413] Recognizing user emotions with an emotion engine

[1414] When users preview their custom picture books, their facial expressions and voice are collected using the device's camera and microphone, and this data is sent to the server in real time and analyzed by the server's emotion engine.

[1415] Emotion-based story adjustment

[1416] The server dynamically changes the story and the characters' expressions based on the analysis results of the emotion engine. If the user is enjoying the story, it will adjust the story to be more cheerful and fun.

[1417] Picture book compilation and preview generation

[1418] The server automatically generates a layout for each page by integrating information about the story and characters, and then creates a digital preview that is sent to the device.

[1419] User verification and customization

[1420] The user checks the preview and edits text or images as necessary. The customization information made on the device is sent to the server.

[1421] Final generation and delivery

[1422] The server generates the final picture book data that reflects the customization information and provides it in a format that can be viewed on the web. If the user requests a printed version, the server generates print data and sends an order to the printing company.

[1423] In this way, user-provided pictures can be quickly transformed into high-quality custom picture books, with content that can be flexibly tailored based on the user's emotions.

[1424] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1425] Step 1: How users post pictures

[1426] Users take a photo of a picture their child has drawn using a smartphone or computer, and then upload the image file using the website or application of the custom picture book creation service. Specifically, users click the "Upload Picture" button, select an image file, and upload it. They then press the "Send" button to send the image data from their device to the server. The input is the user's image data, and the output is the image data sent to the server.

[1427] Step 2: Pre-processing the image

[1428] The server receives image data sent from the device and passes it to the preprocessing engine. It adjusts the resolution of the received image data, for example, setting it to 300 dpi. It also converts the image format from common formats such as JPEG or BMP to PNG. It applies a noise reduction filter and performs color correction. The input is the received image data, and the output is the image data after preprocessing.

[1429] Step 3: Creating characters using image generation AI

[1430] The server inputs the preprocessed image data into the image generation AI model and performs image analysis. Specifically, it inputs a prompt to "extract the main characters and convert them into digital models" into the image generation AI model. The AI ​​identifies the main characters and background in the image and converts their features into digital models. The input is the preprocessed image data, and the output is the generated digital models of the characters and background.

[1431] Step 4: Story generation using text generation AI

[1432] The server uses a text generation AI model to generate a story based on the generated characters and background information. For example, a prompt such as "This image depicts a blue dragon. Please create an adventure story with this dragon as the main character" is input and the AI ​​generates the story. Specifically, based on the characteristics and background information of the characters, a story including the story flow, character actions, dialogue, etc. is automatically created. The input is the characters and background information, and the output is the generated story.

[1433] Step 5: Recognizing user emotions with the emotion engine

[1434] While viewing a preview of a custom picture book, the user uses the device's camera and microphone to collect facial expressions and voice. The device transmits this emotional data to the server in real time. The server's emotion engine analyzes this data and determines the user's emotional state (e.g., joy, surprise, sadness). The input is the user's facial and voice data, and the output is the analyzed emotional state.

[1435] Step 6: Adjust your story based on emotion

[1436] The server receives the analysis results from the emotion engine and adjusts the story and character expressions. For example, if it determines that the user is enjoying themselves, it changes the story to a more cheerful and fun development and adjusts the character expressions accordingly. The input is the analyzed emotional state, and the output is the adjusted story and character expressions.

[1437] Step 7: Compiling the picture book and generating a preview

[1438] The server integrates the generated story, character models, and backgrounds to create the layout for each page. Specifically, it determines the placement of text and images and automatically generates a layout including design elements for each page. Finally, it generates a digital preview and sends it to the device. The input is the adjusted story, character models, and backgrounds, and the output is a digital preview.

[1439] Step 8: Verify and customize users

[1440] The user checks the digital preview sent on their device and makes changes as necessary, such as editing text, changing images, and adjusting character positions. Once the user has finalized the customizations, the data is resent from their device to the server. The input is the digital preview, and the output is data reflecting the customizations.

[1441] Step 9: Final generation and serving

[1442] The server generates the final picture book data that reflects the customization information. It then provides it to the user in a format that can be viewed on the web. If the user requests a printed version, the server generates print data and sends an order to the printer. The input is data that reflects the customization details, and the output is the final picture book data. It also includes order data for delivering the printed version to the user.

[1443] (Application example 2)

[1444] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1445] There is a demand for improving the efficiency and accuracy of the work that field workers face, but conventional systems have the problem that the work content and instructions are uniform, making it difficult to respond flexibly while taking into account the stress level and concentration of each worker. Furthermore, work instructions can be either too detailed or too simple, which can lead to a decline in work efficiency. In particular, it has been difficult to grasp the emotional state of field workers in real time and provide appropriate instructions based on that.

[1446] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for receiving image data sent by a user; means for preprocessing the received image data; means for generating characters using an image generation device with the preprocessed image data as input; means for generating a story using a text generation device with the generated characters and background information as input; emotion recognition means for analyzing the user's emotional state; means for adjusting the facial expressions of the story and characters based on the emotional state; means for automatically generating a page layout for a picture book by combining the story and character information; means for providing a user with an electronic version of the generated picture book; and means for receiving customization information from the user and generating a final version of the picture book reflecting the information. This makes it possible to provide appropriate work instructions that flexibly correspond to the emotional state of field workers.

[1447] A "user" is a person who uses the service to send image data.

[1448] "Image data" refers to image information sent by a user.

[1449] "Preprocessing" refers to a series of processes that convert received image data into a state suitable for analysis.

[1450] An "image generation device" is a device for generating characters based on pre-processed image data.

[1451] "Characters" are characters generated from image data sent by the user.

[1452] "Background information" refers to information that corresponds to the background of an image generated by an image generation device.

[1453] A "sentence generation device" is a device that automatically generates a story based on generated characters and background information.

[1454] A "story" is a storyline generated by a text generator.

[1455] "Emotion recognition means" refers to means for analyzing the emotional state of a user.

[1456] "Page layout" is the process of combining story and character information to determine the page structure of a picture book.

[1457] "Digital edition" refers to a picture book provided in digital format.

[1458] "Customization information" refers to modifications or additions requested by the user to the electronic version.

[1459] The "final version" refers to the final picture book data generated by reflecting the customization information.

[1460] This invention is a system that uses emotion recognition to provide appropriate work instructions to field workers based on smart glasses. Specific embodiments of this system will be described below.

[1461] Hardware and software used

[1462] Hardware: Smart glasses, webcam

[1463] Software: OpenCV (image processing), DeepFace (emotion recognition), Flask server (data transmission and reception)

[1464] Data processing and calculation

[1465] 1. User: A field worker wears smart glasses and takes a photo of the work site. This photo data is captured by the camera in the smart glasses.

[1466] 2. Terminal: The smart glasses transmit the acquired image data to the server using wireless communication technologies such as Wi-Fi or Bluetooth.

[1467] 3. Server: Preprocesses the received image data. OpenCV is used to adjust the image resolution and format, converting it into a suitable state for analysis.

[1468] 4. Server: Analyzes the preprocessed image data to identify the work content and instructions. It also recognizes emotions based on the worker's facial expressions and voice. It uses DeepFace to analyze the worker's emotional state.

[1469] 5. Terminal: The server generates appropriate work instructions based on the emotion recognition results. For example, if a worker is feeling stressed, a simple instruction will be displayed.

[1470] 6. Terminal: The generated work instructions are displayed on the smart glasses display and provided to the worker.

[1471] Specific examples

[1472] As an example, consider a scenario in which a worker in an automobile parts manufacturing plant uses smart glasses to install parts. The worker takes a photo of the work site with the camera in the smart glasses and sends the image to a server. The server processes the image to identify the installation location and uses DeepFace to analyze the worker's emotional state. If the worker is feeling stressed, the server generates a concise instruction and displays it on the smart glasses. This improves work efficiency and relieves the worker's stress.

[1473] Prompt Sentence Examples

[1474] "The user wears the smart glasses and takes photos of the work site. The system generates a manual of work procedures using the photos, and also implements a function to detect the wearer's stress level and adjust the instructions accordingly based on their emotions."

[1475] This invention makes it possible to provide flexible and appropriate work instructions based on the emotional state of field workers, thereby improving work efficiency and accuracy and reducing worker stress.

[1476] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1477] Step 1:

[1478] A user wears the smart glasses and takes a photo of the work site. The input is image data of the site, and the output is the captured image data, which is used in subsequent processing steps.

[1479] Step 2:

[1480] The device (smart glasses) sends the captured image data to the server. The input is the image data acquired in step 1, and the output is the image data sent to the server. Wi-Fi and Bluetooth are used as communication methods.

[1481] Step 3:

[1482] The server preprocesses the received image data. The input is the transmitted image data, and the output is the preprocessed image data. Specifically, it uses OpenCV to adjust the image resolution and convert it to the required format (e.g. PNG).

[1483] Step 4:

[1484] The server analyzes the preprocessed image data to identify the work content and instructions. The input is the preprocessed image data, and the output is data related to the work content and instructions. These elements are extracted using an image generation device.

[1485] Step 5:

[1486] The device (smart glasses) captures the worker's facial expressions and voice and sends them to the server. The input is the worker's facial expression data and voice data, and the output is the emotion data sent to the server.

[1487] Step 6:

[1488] The server analyzes the received emotion data and determines the worker's emotional state. The input is facial expression and voice data, and the output is the emotion recognition result. DeepFace is used to analyze the worker's emotional state.

[1489] Step 7:

[1490] The server generates appropriate work instructions based on the emotion recognition results. The inputs are the emotion recognition results and work content data, and the output is the generated work instructions. Based on the emotional state, it provides simple instructions to workers who are stressed and detailed instructions to workers who are calm.

[1491] Step 8:

[1492] The terminal (smart glasses) displays the generated work instructions. The input is the generated work instruction data, and the output is the work instructions displayed on the smart glasses display. The worker can proceed with the work according to these instructions.

[1493] Through the above processing steps, flexible work instructions are provided according to the emotional state of the field worker, thereby improving work efficiency and accuracy.

[1494] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1495] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1496] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1497] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1498] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1499] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1500] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1501] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1502] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1503] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1504] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1505] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1506] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1507] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1508] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1509] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1510] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1511] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1512] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1513] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1514] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1515] The following is further disclosed regarding the above embodiment.

[1516] (Claim 1)

[1517] means for receiving image data sent by a user;

[1518] means for preprocessing the received image data;

[1519] means for generating characters using an image generation device using the preprocessed image data as input;

[1520] a means for generating a story by a text generation device using the generated characters and background information as input;

[1521] A method for automatically generating page layouts for picture books by combining story and character information;

[1522] a means for providing an electronic version of the generated picture book to a user;

[1523] a means for receiving customization information from a user and generating a final version of the picture book that reflects the information;

[1524] A system including:

[1525] (Claim 2)

[1526] 10. The system of claim 1, further comprising means for accepting orders for printed versions of the generated picture books and for carrying out delivery procedures.

[1527] (Claim 3)

[1528] 10. The system according to claim 1, further comprising means for setting additional parameters relating to the personalities and atmosphere of characters for story generation.

[1529] "Example 1"

[1530] (Claim 1)

[1531] means for receiving image data sent by a user;

[1532] means for preprocessing the received image data;

[1533] means for generating characters using an image generation device using the preprocessed image data as input;

[1534] a means for generating a story by a text generation device using the generated characters and background information as input;

[1535] A method for automatically generating page layouts for picture books by combining story and character information;

[1536] a means for providing an electronic version of the generated picture book to a user;

[1537] a means for receiving customization information from a user and generating a final version of the picture book that reflects the information;

[1538] means for transmitting input data from the terminal to the server;

[1539] data processing and conversion means in the server;

[1540] A system including:

[1541] (Claim 2)

[1542] The system of claim 1, which accepts orders for printed versions of the generated picture books and handles delivery procedures.

[1543] (Claim 3)

[1544] 10. The system according to claim 1, further comprising means for setting additional parameters relating to the personalities and atmosphere of characters for story generation.

[1545] "Application Example 1"

[1546] (Claim 1)

[1547] means for receiving image data sent by a user;

[1548] means for preprocessing the received image data;

[1549] means for generating characters using an image generation device using the preprocessed image data as input;

[1550] a means for generating a story by a text generation device using the generated characters and background information as input;

[1551] A method for automatically generating page layouts for picture books by combining story and character information;

[1552] a means for providing an electronic version of the generated picture book to a user;

[1553] a means for receiving customization information from a user and generating a final version of the picture book that reflects the information;

[1554] A means for sharing the generated picture book with other users within the application;

[1555] A system including:

[1556] (Claim 2)

[1557] 10. The system of claim 1, further comprising means for accepting orders for printed versions of the generated picture books and for carrying out delivery procedures.

[1558] (Claim 3)

[1559] 10. The system according to claim 1, further comprising means for setting additional parameters relating to the personalities and atmosphere of characters for story generation.

[1560] "Example 2: Combining Emotion Engines"

[1561] (Claim 1)

[1562] means for receiving image data sent by a user;

[1563] means for preprocessing the received image data;

[1564] means for generating characters using an image generation device using the preprocessed image data as input;

[1565] a means for generating a story by a text generation device using the generated characters and background information as input;

[1566] means for recognizing the emotional state of a user;

[1567] a way to adjust the narrative and character expressions based on perceived emotions;

[1568] A method for automatically generating page layouts for picture books by combining story and character information;

[1569] a means for providing an electronic version of the generated picture book to a user;

[1570] a means for receiving customization information from a user and generating a final version of the picture book that reflects the information;

[1571] A system including:

[1572] (Claim 2)

[1573] 10. The system of claim 1, further comprising means for accepting orders for printed versions of the generated picture books and for carrying out delivery procedures.

[1574] (Claim 3)

[1575] 10. The system according to claim 1, further comprising means for setting additional parameters relating to the personalities and atmosphere of characters for story generation.

[1576] "Application example 2 when combining emotion engines"

[1577] (Claim 1)

[1578] means for receiving image data sent by a user;

[1579] means for preprocessing the received image data;

[1580] means for generating characters using an image generation device using the preprocessed image data as input;

[1581] a means for generating a story by a text generation device using the generated characters and background information as input;

[1582] emotion recognition means for analyzing the emotional state of a user;

[1583] a way to adjust the narrative and characters' facial expressions based on their emotional state;

[1584] A method for automatically generating page layouts for picture books by combining story and character information;

[1585] a means for providing an electronic version of the generated picture book to a user;

[1586] a means for receiving customization information from a user and generating a final version of the picture book that reflects the information;

[1587] A system including:

[1588] (Claim 2)

[1589] 10. The system of claim 1, further comprising means for accepting orders for printed versions of the generated picture books and for carrying out delivery procedures.

[1590] (Claim 3)

[1591] 10. The system according to claim 1, further comprising means for setting additional parameters relating to the personalities and atmosphere of characters for story generation. [Explanation of symbols]

[1592] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving image data sent by a user; means for preprocessing the received image data; means for generating characters using an image generation device using the preprocessed image data as input; a means for generating a story by a text generation device using the generated characters and background information as input; A method for automatically generating page layouts for picture books by combining story and character information; a means for providing an electronic version of the generated picture book to a user; a means for receiving customization information from a user and generating a final version of the picture book that reflects the information; A system including:

2. 2. The system according to claim 1, further comprising means for accepting orders for printed versions of the created picture books and for carrying out delivery procedures.

3. 2. The system according to claim 1, further comprising means for setting additional parameters relating to the personalities and atmosphere of characters for story generation.

Images