System

Abstract

To provide a system for visually and effectively transmitting the contents of a sentence.SOLUTION: A system comprising: means for receiving and storing textual data; means for natural language processing the textual data to tokenize, part-of-speech tag, and parse the textual data; means for generating images using a machine learning model based on a result of the parsing; means for arranging the generated images according to an order to create a storyboard in a paper play format; and means for providing the paper play to a user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's world, many people find it difficult to understand long or complex texts. Reading and understanding specialized content, such as historical facts or scientific processes, is not only time-consuming but can also be stressful. While visual learning is highly effective, the prevalence of text-based information often reduces the efficiency of information comprehension. The present invention aims to solve these problems by providing a system that effectively conveys the content of text visually. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means. First, it includes a means for receiving and saving text data. Next, it includes a means for natural language processing the saved text data, where the natural language processing means includes means for tokenizing, part-of-speech tagging, and syntactic analysis of the text data. It further provides a means for generating images based on the analysis results, where the image generation means includes means for generating images using a machine learning model. It includes a means for arranging the generated images in order to create a storyboard in the form of a picture-story show. Finally, it includes a means for providing the picture-story show thus created to a user. These methods allow a user to visually understand the content of the text, thereby promoting understanding and reducing stress.

[0006] Understood. Below are definitions of important words included in the claims.

[0007] "Text data" is text information consisting of character strings entered by the user.

[0008] "Storage means" refers to a storage device or database for recording and storing input text data.

[0009] "Natural language processing" is a technology that allows computers to understand and analyze human language.

[0010] "Tokenization" is the process of dividing a sentence into its smallest units, such as words and phrases.

[0011] "Part-of-speech tagging" is the process of assigning grammatical roles, such as nouns and verbs, to tokenized elements.

[0012] "Syntax analysis" is the process of analyzing the grammatical structure of a sentence and clarifying the relationships between meanings.

[0013] "Image generating means" refers to technology or devices that create visual images based on input data.

[0014] A "machine learning model" is an algorithm that uses data to recognize patterns and make predictions or classifications.

[0015] A "kamishibai storyboard" is a visual representation of a story using multiple sequential images.

[0016] "Means for providing" refers to a method or device for delivering or displaying the generated picture story to a user.

[0017] The above are definitions of important terms contained in the claims. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention relates to a system that provides long or complex text in the form of a picture-story show that is easy for users to understand. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a storyboard in the form of a picture-story show, and finally provides it to the user.

[0040] How the system works

[0041] Receiving text data

[0042] A user inputs long text or text data through the system's interface, for example by copying and pasting text into a web form or application input field, or by typing directly into the input field. This input text data is sent to a server for initial processing.

[0043] Sentence analysis

[0044] The server stores the received text data and begins natural language processing. This process involves several steps. First, the text is tokenized, which is the process of dividing the text into the smallest units, such as words and phrases. Part-of-speech tagging is then performed, and each token is assigned a grammatical role. Next, syntactic analysis is performed to reveal the grammatical structure of the text. This allows the meaning of the text and the relationships between its parts to be analyzed.

[0045] Image generation

[0046] Based on the analysis results, the server generates an image. This process is carried out using a machine learning model. For example, DALL-E or Stable Diffusion can be used as a generative model. These models generate images with appropriate visual representations based on the content of the input text. For example, they can generate images such as "The Black Ships Arriving in Japan" or "Portrait of Sakamoto Ryoma."

[0047] Building a Kamishibai Frame

[0048] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard. This storyboard also includes a description for each image. The description is a concise summary of the main points of the analyzed text, making it easier for users to understand the content while looking at the images.

[0049] Provision to users

[0050] The completed Kamishibai is saved in digital format and provided to the user. The user can view the Kamishibai on their own device (PC or tablet). For example, by clicking on a link generated by the system, the Kamishibai can be viewed online. Furthermore, if necessary, the Kamishibai can be downloaded in PDF or slide format.

[0051] Specific examples

[0052] Example 1: History text

[0053] The user inputs a "text about Japan's Meiji Restoration" into the system. The server analyzes the text and identifies keywords such as "Meiji Restoration," "Sakamoto Ryoma," "arrival of the black ships," and "civilization and enlightenment." Based on these keywords, images of historical events and people are generated. For example, images such as "the arrival of the black ships in Japan," "scenes of Sakamoto Ryoma in action," and "the changing landscape of Japan with civilization and enlightenment" are created and presented in the form of a picture-story show with accompanying explanatory text.

[0054] Example 2: Scientific explanation

[0055] The user inputs a sentence about the mechanism of photosynthesis into the system. The server analyzes the sentence and identifies keywords such as "photosynthesis," "chlorophyll," "oxygen production," and "carbon dioxide absorption." Based on these keywords, it generates images that explain the photosynthetic process of plants. For example, it creates images of "how light hits a plant," "chemical reactions occurring inside a leaf," and "the scene where oxygen is released," and provides them in the form of a picture-story show with accompanying explanatory text.

[0056] The purpose of this system is to make it easier for users to visually understand long or complex sentences, thereby contributing to promoting comprehension and reducing stress.

[0057] The processing flow will be explained below.

[0058] Ok, I'll explain each process step in the format shown below.

[0059] Step 1:

[0060] A user accesses the system's interface and inputs long text or text data, for example by copying and pasting text into a web form or by typing it directly. At this point, the input data is sent to the server.

[0061] Step 2:

[0062] The server stores the received text data in a database. This initial storage is necessary for later analysis.

[0063] Step 3:

[0064] The server passes the stored text data to a natural language processing (NLP) library to begin analysis, which includes tokenization (dividing the text into words and phrases), part-of-speech tagging (assigning grammatical categories to each word), and syntactic analysis (analyzing the grammatical structure of the text).

[0065] Step 4:

[0066] The server uses the results of natural language processing to identify key sections of the text and important keywords. For example, if the text is about history, it might extract terms such as "Meiji Restoration," "Sakamoto Ryoma," and "Arrival of the Black Ships."

[0067] Step 5:

[0068] The server then generates an image for each section based on the analysis results. This process is done using machine learning models (e.g., DALL-E or Stable Diffusion), where you input the text content into the model and it generates an image based on that.

[0069] Step 6:

[0070] The server arranges the generated images in a sequence to create a picture-storyboard in the style of a picture story show. The order of the images is designed to follow the flow of the story in the text.

[0071] Step 7:

[0072] The server generates a description for each image, which is a brief summary of the main points of the text, to help users understand the content of the image when they view it.

[0073] Step 8:

[0074] The server stores the completed storyboard in digital format and generates a link to provide it to the user, which is a URL that the user can access to view the storyboard online.

[0075] Step 9:

[0076] Users can use their devices (PCs or tablets) to access the link provided by the server and watch the Kamishibai. The content is presented visually in an easy-to-understand way, allowing users to efficiently understand the content of the text.

[0077] Step 10:

[0078] Users can also download the Kamishibai as a PDF or slides if desired, which is convenient for offline viewing or presentations.

[0079] The above is a description of the specific processing steps of this system.

[0080] Example 1

[0081] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0082] There is a demand for a method for visualizing long or complex sentences in an efficient and easy-to-understand manner. The present invention aims to provide a system that presents text data entered by a user as a picture-storyboard format that is visually easy to understand.

[0083] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0084] In this invention, the server includes means for a user to input text data and send it through a system interface, means for the server to store the received text data and start natural language processing, means for the server to tokenize, tag parts of speech, and syntactically parse the text data, means for the server to generate prompts based on the analysis results and use them to generate visual representations using an image generation model, means for arranging the generated images in order to create a picture-storyboard in the style of a picture-story show, and means for the user to view, save, or download the storyboard in digital form. This makes it possible to provide complex text data to users in a visually easy-to-understand format.

[0085] "Means for inputting text data and transmitting it through a system interface" refers to a means by which a user can input long text or sentences and transmit them to a server via a web form or application.

[0086] The "means for saving received text data and starting natural language processing" refers to the means by which the server saves text data sent from the user and automatically starts the natural language processing process.

[0087] "Means for tokenizing, tagging parts of speech, and parsing text data" refers to the means by which the server divides text data into its smallest units, assigns grammatical roles to each word, and analyzes the grammatical structure and meaning of the entire text.

[0088] "Means for generating prompts based on the analysis results and using them to generate visual representations using an image generation model" refers to a means by which the server creates prompt sentences based on the results of the analyzed sentences, inputs them into a generative AI model, and generates appropriate images.

[0089] "Means for arranging generated images in a logical order and creating a picture-storyboard in the style of a picture-story show" refers to means by which the server arranges generated images in a logical order and adds explanatory text to each image to create a picture-storyboard in the style of a picture-story show.

[0090] "Means for users to view, save, or download storyboards in digital format" refers to means for users to view, digitally save, or download completed storyboards in picture-story show format on a device such as a computer or tablet.

[0091] This invention relates to a system that provides long or complex text in the form of a picture-story show that is easy for users to understand. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a storyboard in the form of a picture-story show, and finally provides it to the user.

[0092] How the system works

[0093] Receiving text data

[0094] A user inputs text data through the system interface, for example, by copying and pasting long sentences or text into a web form or application input field, or by typing and submitting the text data. This input text data is sent to the server for initial processing.

[0095] Sentence analysis

[0096] The server stores the received text data and starts natural language processing. Specifically, it performs the following steps:

[0097] 1. Tokenization: This is the process of breaking down the received text into its smallest units, such as words and phrases. This process is done using a language analysis engine or natural language processing library (e.g., SpaCy, NLTK).

[0098] 2. Part-of-speech tagging: This is the process of assigning a grammatical role to each token. For example, tagging the token "Sakamoto Ryoma" as a "noun."

[0099] 3. Syntactic analysis: This is the process of analyzing a sentence for its grammatical structure and meaning. This clarifies the relationships between tokens and the overall meaning of the sentence. Syntactic analysis is often performed using tree-based parsing algorithms.

[0100] Image generation

[0101] Based on the analysis results, the server generates a visual representation using an image generation model. This process involves the following steps:

[0102] 1. Take the identified keywords and phrases and turn them into prompts. For example, create a prompt like, "Generate an image of the Black Ships arriving in Japan."

[0103] 2. The created prompt sentence is input into a generative AI model (e.g., DALL-E, Stable Diffusion) to generate an appropriate image. The generated image is a visual representation based on the content of the sentence.

[0104] Building a Kamishibai Frame

[0105] The server arranges the generated images in a sequence to create a picture-storyboard, specifically:

[0106] 1. Arrange the images in a logical order. For example, arrange them in the order "Arrival of the Black Ships," "Sakamoto Ryoma in Action," and "Scenes of Westernization."

[0107] 2. A description is generated for each image. This description is a concise summary of the main points of the analyzed text. For example, it can be a specific description such as "This image shows the arrival of the Black Ships in Japan."

[0108] Provision to users

[0109] The completed Kamishibai is saved in digital format and provided to users. The specific steps are as follows:

[0110] 1. Users can view the Kamishibai on their own devices (PCs or tablets). By clicking on the link generated by the system, the Kamishibai will be displayed online.

[0111] 2. If necessary, users can download the Kamishibai in PDF or slide format. For example, by clicking the download button, the file will be saved to their device.

[0112] Specific examples

[0113] Example 1: History text

[0114] The user inputs a "text about Japan's modernization" into the system. The server analyzes the text and identifies keywords such as "modernization," "historical events," "industrial revolution," and "social change." Based on these keywords, images of historical events and people are generated. For example, images such as "a factory symbolic of modernization," "famous historical leaders," and "scenes of the industrial revolution" are created and presented in the form of a picture-story show with accompanying explanatory text.

[0115] Example prompt sentence:

[0116] Generate an image of an "Industrial Revolution factory scene."

[0117] Example 2: Scientific explanation

[0118] The user inputs a "text about the workings of ecosystems" into the system. The server analyzes the text and identifies keywords such as "ecosystem," "ecosystem," "biological interactions," and "natural environment." Based on these keywords, an image of the ecosystem is generated. For example, images such as "interactions between organisms," "changes in the natural environment," and "survival strategies of organisms" are created and presented in the form of a picture-story show with accompanying explanatory text.

[0119] Example prompt sentence:

[0120] Generate images of "scenes of interactions between organisms."

[0121] The system aims to make it easier for users to visually understand long or complex sentences, thereby contributing to promoting comprehension and reducing stress.

[0122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0123] Step 1: Enter and send text data

[0124] Users input and submit text data through the system interface, either by copying and pasting text into a web form or by typing it directly into an application input field. The input text data is then sent to the server.

[0125] Input: Text data entered by the user

[0126] Output: Raw text data sent to the server

[0127] Step 2: Saving and initial processing of text data

[0128] The server saves the received text data and starts initial processing. Specifically, it performs validation checks on the text data (e.g., removes unsupported characters) and saves it in a database (e.g., MySQL, PostgreSQL).

[0129] Input: Submitted text data

[0130] Output: Validated text data, data stored in a database

[0131] Step 3: Tokenization

[0132] The server tokenizes the stored text data, specifically by using a language analysis engine (e.g., SpaCy, NLTK) to split the text into words and phrases.

[0133] Input: Saved text data

[0134] Output: Tokenized word list

[0135] Step 4: Part-of-Speech Tagging

[0136] The server then tags the tokenized word list with part-of-speech tags, specifically assigning each word a grammatical role (e.g., noun, verb, adjective, etc.).

[0137] Input: tokenized word list

[0138] Output: A list of words with part-of-speech tags

[0139] Step 5: Parsing

[0140] The server uses the tagged word list to perform a syntactic analysis of the entire sentence, specifically by clarifying the relationships between words and analyzing the grammatical structure of the sentence (e.g., dependency analysis).

[0141] Input: A list of words with part-of-speech tags

[0142] Output: Parsed grammar structure data

[0143] Step 6: Prompt Generation

[0144] The server generates a prompt based on the results of the parsing, such as "Please describe the scene when the Black Ships arrive in Japan."

[0145] Input: Parsed grammar structure data

[0146] Output: prompt statement

[0147] Step 7: Image generation

[0148] The server uses the generated prompt sentence to input into an image generation model (e.g., DALL-E, Stable Diffusion) to generate a visual representation. Specifically, the server inputs the prompt sentence into the generative AI model and obtains the returned image.

[0149] Input: prompt statement

[0150] Output: Generated image data

[0151] Step 8: Build a storyboard

[0152] The server arranges the generated images in a sequence to create a picture-storyboard, and also generates explanatory text for each image.

[0153] Input: Generated image data

[0154] Output: Kamishibai style storyboard

[0155] Step 9: Provide to users

[0156] The server stores the completed Kamishibai in digital format and provides it to users by generating a link that allows them to view it online and download it as a PDF or slides if desired.

[0157] Input: Kamishibai-style storyboard

[0158] Output: A link or file that users can view and download

[0159] (Application example 1)

[0160] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0161] Today's consumers expect detailed product descriptions in online and virtual stores, but text alone is not enough to adequately convey the benefits and usage scenarios of products and services. Therefore, there is a need for a method to provide product descriptions in a visually easy-to-understand format. Furthermore, when users receive product information in long or complex sentences, it can take time to understand the content, which can be frustrating. To solve these problems, a system is needed that provides product descriptions in a picture-story show format.

[0162] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0163] In this invention, the server includes means for receiving and saving text data, means for natural language processing the text data, means for generating images based on the analysis results, means for arranging the generated images in order to create a storyboard in the form of a picture-story show, means for providing the picture-story show to the user, and means for visually providing product descriptions in the virtual store, which makes it easier for the user to visually understand the details of products and services, and enables effective product descriptions in the virtual store.

[0164] "Text data" refers to data of long or complex texts entered by the user.

[0165] The "means for saving" is a means having a function for saving input text data on a server or the like.

[0166] "Means for natural language processing" refers to means for processing text data such as tokenization, part-of-speech tagging, and syntactic analysis to analyze the meaning and structure of text.

[0167] "Generating means" refers to means that has the function of generating images or other visual content based on the analysis results.

[0168] The "means for arranging in order" is a means for arranging generated images and content in order in a storyboard format.

[0169] A "picture-storyboard" is a storyboard format that makes it visually easy to understand by displaying each image in order with corresponding explanatory text.

[0170] The "means of providing" is a means of ultimately providing the user with a picture-storyboard in a viewable format.

[0171] A "virtual store" is a virtual store that operates on the Internet, and refers to an environment where users can browse and purchase products online.

[0172] In this invention, the server is a system that receives and stores text data, analyzes it, generates images, and provides them to users in the form of a picture-story show. A specific embodiment of this system is described below.

[0173] First, a user inputs text data using a terminal. This text data is sent to a server through a web form or application input field. The server stores the received text data. The stored text data is then analyzed using natural language processing. This processing includes steps such as tokenization, part-of-speech tagging, and syntactic analysis. This allows the meaning and grammatical structure of the text to be revealed.

[0174] Based on the analysis results, the server uses a generative AI model to generate appropriate images. Examples of AI models used include DALL-E and Stable Diffusion. These models generate images that visually represent the content of the input text. The generated images are arranged in a sequence to create a picture-storyboard in the style of a picture-story show. This storyboard also includes explanatory text for each image, making it easier for users to visually understand the content.

[0175] Finally, users can view the Kamishibai on their own devices. They can do so by using a link generated by the system or by downloading it in PDF or slide format. This allows users to visually understand the product's features and usage scenarios in Kamishibai format.

[0176] As a specific example, if a user inputs the text data, "This product is a smartphone app that uses the latest AI technology. Its main functions are voice recognition and image recognition," the server analyzes this text and identifies keywords such as "the latest AI technology," "smartphone app," "voice recognition," and "image recognition." Based on this, it generates images such as futuristic technology images, smartphone screens, voice input images, and image analysis images, and incorporates these into a storyboard for presentation.

[0177] An example of a prompt is, "Create a presentation for a smartphone app using the latest AI technology. Generate images and explanatory text in a picture-story format that visually express specific functions and usage scenarios."

[0178] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0179] Step 1:

[0180] A user uses a terminal to input text data, which is then sent to a server via a web form or application input field.

[0181] Input: Text data entered by the user

[0182] Output: Text data sent to the server

[0183] Step 2:

[0184] The server stores the received text data, which is used for subsequent processing.

[0185] Input: Text data sent to the server

[0186] Output: Text data stored on the server

[0187] Step 3:

[0188] The server analyzes the stored text data using natural language processing techniques, which include steps such as tokenization, part-of-speech tagging, and syntactic analysis, to reveal the meaning and grammatical structure of the text.

[0189] Input: Text data stored on the server

[0190] Data processing: sentence tokenization, part-of-speech tagging, syntactic analysis

[0191] Output: Analysis results (tokens, part-of-speech tags, syntactic analysis results)

[0192] Step 4:

[0193] Based on the analysis results, the server uses a generative AI model, such as DALL-E or Stable Diffusion, to generate an appropriate image.

[0194] Input: Analysis results after natural language processing (keywords, context, etc.)

[0195] Data calculation: Image generation using generative AI models (e.g., DALL-E, Stable Diffusion)

[0196] Output: The generated image

[0197] Step 5:

[0198] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard, with each image accompanied by a description that summarizes the main points of the analysis results.

[0199] Input: Generated images, summary of analysis results

[0200] Data processing: Arranging images and creating descriptions

[0201] Output: Kamishibai style storyboard

[0202] Step 6:

[0203] The server generates a link to provide the completed Kamishibai to the user, who can then use this link to view the Kamishibai on their device. If desired, the Kamishibai can also be downloaded as a PDF or slides.

[0204] Input: Kamishibai-style storyboard

[0205] Data processing: Link generation, data conversion to PDF and slide format

[0206] Output: A link to the Kamishibai sent to the user and the downloadable data

[0207] As a concrete example, if a user inputs the text data "This product is a smartphone app that uses the latest AI technology. Its main functions are voice recognition and image recognition," the server analyzes this text and identifies keywords such as "the latest AI technology," "smartphone app," "voice recognition," and "image recognition." Based on this, it generates images such as futuristic technology images, smartphone screens, voice input images, and image analysis images, and incorporates these into a storyboard for presentation.

[0208] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0209] This invention combines a system that converts text data into a visually easy-to-understand picture-story show format with an emotion engine that recognizes the user's emotions. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a picture-story show format storyboard, and finally provides it to the user. In addition, it has the ability to recognize the user's emotions in real time and adjust the content based on those emotions.

[0210] How the system works

[0211] Receiving text data

[0212] A user inputs long text or text data through the system interface, for example by copying and pasting text into a web form or application input field, or by typing directly into the input field. This input text data is then sent to the server.

[0213] Sentence analysis

[0214] The server stores the received text data in a database and begins natural language processing, which includes tokenization (dividing the text into words and phrases), part-of-speech tagging (assigning grammatical categories to each word), and syntactic analysis (analyzing the grammatical structure of the text).

[0215] Image generation

[0216] Based on the analysis, the server generates an image for each section, using machine learning models (e.g., DALL-E and Stable Diffusion) to generate an appropriate image that visually represents the content of the text.

[0217] Building a Kamishibai Frame

[0218] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard. This storyboard also includes a description for each image. The description is a concise summary of the main points of the analyzed text, making it easier for users to understand the content while looking at the images.

[0219] Provision to users

[0220] The completed Kamishibai is saved in digital format and provided to the user. The user can view the Kamishibai on their own device (PC or tablet). For example, by clicking on a link generated by the system, the Kamishibai can be viewed online. Furthermore, if necessary, the Kamishibai can be downloaded in PDF or slide format.

[0221] Emotional engine regulation

[0222] emotion recognition

[0223] While the user is viewing the picture story, the system uses an emotion engine to recognize the user's emotions in real time. This emotion engine analyzes the user's emotional state using facial recognition and voice analysis technologies. For example, it can determine emotions such as joy, anger, sadness, and happiness based on data obtained from the user's webcam and microphone.

[0224] Content adjustment

[0225] The server dynamically adjusts the content of the picture story based on the user's recognized emotional state, highlighting sections that interest the user or providing additional explanations to deepen understanding. For example, if the user shows a confused expression, the system adds images and text that explain the content of that section in detail.

[0226] Specific examples

[0227] Example 1: History text

[0228] The user inputs a text about Japan's Meiji Restoration into the system. The server analyzes the text and identifies keywords such as "Meiji Restoration," "Sakamoto Ryoma," "arrival of the Black Ships," and "civilization and enlightenment." Based on these keywords, images of historical events and people are generated. For example, images such as "the arrival of the Black Ships in Japan," "scenes of Sakamoto Ryoma's exploits," and "the changing landscape of Japan as a result of civilization and enlightenment" are created and presented in the form of a picture-story show with accompanying explanatory text. Furthermore, specific background information and illustrations can be added to sections that the user finds interesting, deepening their understanding.

[0229] Example 2: Scientific explanation

[0230] The user inputs a sentence about the mechanism of photosynthesis into the system. The server analyzes the sentence and identifies keywords such as "photosynthesis," "chlorophyll," "oxygen production," and "carbon dioxide absorption." Based on these keywords, it generates images that explain the photosynthetic process in plants. For example, it creates images such as "how light hits a plant," "chemical reactions occurring inside a leaf," and "the scene where oxygen is released," and provides them in the form of a picture-story show with accompanying explanatory text. In addition, it can recognize the user's emotions and, if they are confused, provide more detailed illustrations or additional explanations to help them understand.

[0231] In this way, this system not only makes the content of a sentence easier to understand visually, but also achieves a greater effect of promoting comprehension by dynamically adjusting according to the user's emotions.

[0232] The processing flow will be explained below.

[0233] Ok, I'll explain each process step in the format shown below.

[0234] Step 1:

[0235] A user accesses the system's interface and inputs long or textual data, for example by copying and pasting text into a web form or by typing it directly. This input data is then sent to the server.

[0236] Step 2:

[0237] The server stores the received text data in a database. This initial storage is necessary for later analysis.

[0238] Step 3:

[0239] The server then passes the stored text data to a natural language processing (NLP) library to begin parsing it: tokenization, part-of-speech tagging, and syntactic analysis are performed to identify key points and sections of the text.

[0240] Step 4:

[0241] The server extracts key sections and important keywords from the analysis results, identifying keywords such as "Meiji Restoration," "Sakamoto Ryoma," and "Arrival of the Black Ships."

[0242] Step 5:

[0243] The server generates images for each section based on the extracted keywords, using machine learning models (e.g., DALL-E and Stable Diffusion), generating images such as "The Black Ships Arriving in Japan" or "Portrait of Ryoma Sakamoto."

[0244] Step 6:

[0245] The server arranges the generated images in a sequence to create a picture-storyboard in the style of a picture story show. The order of the images is designed to follow the flow of the story in the text.

[0246] Step 7:

[0247] The server generates a description for each image and adds it to the storyboard. The description is a concise summary of the main points of the text, making it easier for users to understand the content while viewing the image.

[0248] Step 8:

[0249] The server stores the completed storyboard in digital format and generates a link to provide it to the user, which is a URL that the user can access to view the storyboard online.

[0250] Step 9:

[0251] The server runs an emotion engine and provides a means to recognize the user's emotional state in real time, for example by capturing the user's facial expressions and voice via a webcam or microphone and running the emotion analysis algorithm on them.

[0252] Step 10:

[0253] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time, for example, detecting when they are confused or interested.

[0254] Step 11:

[0255] The server dynamically adjusts the content of the picture story based on the perceived emotional state, highlighting interesting sections and providing additional explanations to enhance understanding—for example, adding more detailed explanatory images or text if the user is confused.

[0256] Step 12:

[0257] Users can use their devices (PCs or tablets) to access the link provided by the server and watch the picture story. The content adjusted by the emotion engine is reflected in real time, and the content is presented visually in an easy-to-understand manner.

[0258] Step 13:

[0259] Users can also download the Kamishibai as a PDF or slides if desired, which is convenient for offline viewing or presentations.

[0260] The above is a description of the specific processing steps of this system.

[0261] Example 2

[0262] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0263] Conventional systems for promoting visual comprehension of text data have the problem of being unable to respond to changes in the user's emotions during processes such as text analysis and image generation. As a result, there is concern that the user's understanding and concentration may decline, resulting in insufficient learning effectiveness. The present invention proposes a system equipped with emotion recognition functionality to solve this problem and deepen the user's understanding.

[0264] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0265] In this invention, the server includes means for receiving and storing text data, means for natural language processing the text data, means for generating images based on the analysis results, means for arranging the generated images in order to create a storyboard in the form of a picture-story show, means for providing the picture-story show to the user, means for recognizing the user's emotions in real time, and means for dynamically adjusting the content of the picture-story show based on the recognized emotions.

[0266] This makes it possible to dynamically adjust the learning content according to the user's emotional state, thereby enhancing the effectiveness of promoting understanding.

[0267] "Text data" refers to all text information entered by the user, and includes the contents of any written document, from short sentences to long sentences.

[0268] "Natural language processing" refers to the technology that enables computers to understand and analyze human language, and includes the processes of tokenizing, part-of-speech tagging, and syntactic analysis of text.

[0269] "Image generation" refers to the process of creating appropriate visual elements based on analytical results, and is done using generative AI models.

[0270] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate new data, specifically using computer vision technology.

[0271] "Kamishibai format" refers to a technique in which multiple images are arranged in an orderly manner and displayed consecutively with corresponding explanatory text attached to each.

[0272] A "storyboard" is a visually structured format that combines a series of scenes or slides to effectively convey a story or information.

[0273] "User" refers to the entity that uses the System to input text data and receive the analysis results and generated content.

[0274] "Emotion recognition" refers to the technology of determining a user's emotional state from their facial expressions, voice, etc., and includes the process of detecting the user's psychological state.

[0275] "Dynamic adjustment" refers to technology that changes content and settings in real time based on the situation or condition, automatically adapting based on user responses.

[0276] This invention is a system that converts text data into a visually easy-to-understand picture-story show format, and also combines it with an emotion engine that recognizes the user's emotions. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a picture-story show format storyboard, and finally provides it to the user. In addition, it has the function of recognizing the user's emotions in real time and adjusting the content based on those emotions.

[0277] Receiving text data

[0278] Users input long text or text data through the system interface, for example by copying and pasting text into a web form or application input field, or by typing it directly. This input text data is then sent from the terminal to the server via the Internet.

[0279] Sentence analysis

[0280] The server stores the received text data in a database and begins natural language processing (NLP). This process includes tokenization (dividing the text into words and phrases), part-of-speech tagging (assigning grammatical categories to each word), and syntactic analysis (analyzing the grammatical structure of the text). Software such as Google Cloud Natural Language API and SpaCy are used for this. For example, in a historical text, keywords such as "Meiji Restoration," "Sakamoto Ryoma," and "Arrival of the Black Ships" are identified.

[0281] Image generation

[0282] Based on the analysis results, the server uses a generative AI model (e.g., DALL-E or Stable Diffusion) to generate images corresponding to each section. This generates an appropriate image that visually represents the content of the text. For example, an image that explains "the arrival of the Black Ships in Japan" is generated. The following prompt sentences are used:

[0283] Prompt statement:

[0284] "Generate an image illustrating the arrival of the black ships in Japan during the Meiji Restoration. The image should depict the historical context and importance of this event."

[0285] Building a Kamishibai Frame

[0286] The generated images are arranged in order by the server to create a storyboard in the style of a picture-story show. Each image is also given a description. For example, a description such as "This image depicts the arrival of the Black Ships in Japan in 1853" could be added.

[0287] Provision to users

[0288] The completed Kamishibai is saved in digital format and provided to the user. Users can view the Kamishibai on their own devices (PC or tablet) or view it online by clicking on a link generated by the system. Furthermore, if desired, users can download the Kamishibai in PDF or slide format.

[0289] emotion recognition

[0290] While the user is viewing the Kamishibai, the system uses an emotion engine to recognize the user's emotions in real time. The emotion engine analyzes the user's emotional state using facial recognition technology (e.g., OpenCV) and voice analysis technology (e.g., Google Speech-to-Text API). For example, if the user shows a confused expression, the system will determine this.

[0291] Content adjustment

[0292] The server dynamically adjusts the content of the picture story based on the user's recognized emotional state, highlighting sections that interest the user or providing additional explanations to deepen understanding. For example, if the user is confused, it adds images or text that explain the content in more detail.

[0293] As described above, this system not only makes text easier to understand visually, but also enhances learning effectiveness by dynamically adjusting according to the user's emotions.

[0294] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0295] Step 1:

[0296] The user inputs text data using the system's interface. They copy and paste long sentences or text data into the input fields of a web form or application, or type them directly. Once the input is complete, the device sends it to the server. The input is the text data created or collected by the user, and the output is the secure transmission of the text data to the server.

[0297] Step 2:

[0298] The server stores the received text data in a database and starts natural language processing (NLP). The input for this process is the text data provided by the user. In this step, the following specific data processing and calculations are performed:

[0299] Tokenization: Breaking text into words and phrases.

[0300] Part-of-speech tagging: attaching a grammatical category to each word.

[0301] Syntactic analysis: Analyzing the grammatical structure of a sentence.

[0302] The output is an analysis of keywords and grammatical structures. The software used is Google Cloud Natural Language API and SpaCy.

[0303] Step 3:

[0304] Based on the analysis results, the server generates an image using a generative AI model (e.g., DALL-E or Stable Diffusion). The input is the analysis results (keywords and grammatical structures) obtained in the previous step. Specific operations include:

[0305] Input a prompt sentence into the generative AI model.

[0306] Generates an image according to the prompt.

[0307] For example, based on the keywords "Meiji Restoration" and "Arrival of the Black Ships," an image that visually represents these is generated. The prompt used is "Generate an image illustrating the arrival of the black ships in Japan during the Meiji Restoration." The output is an image generated to visually represent the content of the text.

[0308] Step 4:

[0309] The generated images are arranged in a sequence by the server to create a picture-storyboard. The input to this step is the sequence of images generated in step 3 and their accompanying explanatory text. Specific operations include:

[0310] Determine the order of the images.

[0311] Add a description for each image.

[0312] Build a storyboard by integrating images and descriptions.

[0313] The output is a visually organized storyboard in the style of a picture story.

[0314] Step 5:

[0315] The completed Kamishibai is saved in digital format and provided to the user. The input for this step is the completed storyboard. Specific actions include:

[0316] Upload your storyboard online.

[0317] A link is generated and notified to the user.

[0318] If necessary, convert them into PDF or slide format and make them downloadable.

[0319] The output is a digital Kamishibai that is accessible to users.

[0320] Step 6:

[0321] While the user is viewing the Kamishibai, the system uses the emotion engine to recognize the user's emotions in real time. The input for this step is the user's facial expressions and voice data. The specific operation is as follows:

[0322] Data is collected in real time via webcam and microphone.

[0323] Analyze emotions using facial recognition technology (e.g., OpenCV) and voice analysis technology (e.g., Google Speech-to-Text API).

[0324] The output is data indicating the user's emotional state (e.g., joy, confusion, interest, etc.).

[0325] Step 7:

[0326] Based on the recognized user's emotional state, the server dynamically adjusts the content of the Kamishibai. The input for this step is the user's emotional data. Specific operations include:

[0327] Highlight certain sections depending on your emotional state.

[0328] Insert additional explanations and illustrations as needed.

[0329] For example, if the user is confused, more detailed images and explanations are added. The output is a picture-story show that is dynamically adjusted according to the user's emotions.

[0330] (Application example 2)

[0331] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0332] Conventional text data visualization systems simply provide information unilaterally, without considering the user's emotions or level of understanding. This often leaves users confused or disinterested, resulting in insufficient understanding of the information. Furthermore, the information provided visually is limited, and particularly in education and content distribution services, it is necessary to flexibly adjust the content based on the user's reaction.

[0333] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0334] In this invention, the server includes means for receiving and saving text data, means for natural language processing the text data, means for generating images based on the analysis results, means for arranging the generated images in order to create a storyboard in the form of a picture-story show, means for providing the picture-story show to the user, means for recognizing the user's emotions in real time, and means for adjusting the content based on the user's emotions. This makes it possible to dynamically adjust the provision of visual information according to the user's emotions and achieve a greater effect in promoting understanding.

[0335] The "means for receiving and saving text data" is a function for sending text data entered by the user to a server and saving it in a database.

[0336] "Means for natural language processing of text data" refers to a function for analyzing text data through a series of processes including tokenization, part-of-speech tagging, and syntactic analysis.

[0337] "Means for generating images based on analysis results" refers to a function that generates appropriate images using a machine learning engine such as a generative AI model based on the results of natural language processing.

[0338] "Means for arranging generated images in order and creating a storyboard in the form of a picture-story show" is a function for arranging multiple generated images in order as a storyboard and displaying them in the form of a picture-story show.

[0339] The "means for providing a picture story to a user" is a function for providing a picture story created in electronic form so that the user can view it.

[0340] "Means for recognizing user emotions in real time" refers to a function that uses sensors such as cameras and microphones to analyze emotions from the user's facial expressions and voice and recognize them in real time.

[0341] The "means for adjusting content based on user emotions" is a function that dynamically changes the content of the picture book based on the recognized user emotions, and provides additional explanations or detailed illustrations as needed.

[0342] The present invention is a system for converting text data into a visually easy-to-understand picture-story format, and includes a function for recognizing a user's emotions in real time and adjusting the content accordingly. Specific embodiments of the system are described below.

[0343] System configuration

[0344] The system consists of hardware such as a user terminal, server, camera, and microphone, as well as software components including a natural language processing engine, an image generation engine, and an emotion recognition engine.

[0345] Entering and saving text data

[0346] Users use their own devices to input text data into the system's interface, which is provided as an input field on a web form or a smartphone application. The input text data is sent to the server and stored in a database.

[0347] Sentence analysis

[0348] The server then performs natural language processing on the stored text data. It uses software such as spaCy, a natural language processing library, to tokenize the text data, tag it with parts of speech, and perform syntactic analysis. This process extracts keywords and themes from the text.

[0349] Image generation

[0350] Based on the analyzed keywords and themes, the server generates corresponding images using an image generation engine. Generative AI models used here include "DALL-E" and "Stable Diffusion." These models generate appropriate images based on the input prompt.

[0351] Creating a storyboard in the style of a picture story

[0352] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard, with each image accompanied by a brief description summarizing the main points of the analyzed text.

[0353] Provision to users

[0354] The completed storyboard in picture-storytelling format is saved in digital format and provided to users, who can view it on their own devices and download it as a PDF or slides if desired.

[0355] Emotion recognition and content regulation

[0356] While the user is viewing the Kamishibai, the server uses a camera and microphone to recognize the user's emotions in real time. The emotion recognition engine used uses the "Transformers" emotion recognition model. Based on the recognized emotions, the server dynamically adjusts the content of the Kamishibai and provides additional explanations and detailed illustrations to facilitate understanding.

[0357] Examples and prompts

[0358] For example, if a user inputs "historical text," keywords such as "Meiji Restoration," "famous revolutionary," and "background of arrival" are extracted, and DALL-E generates corresponding images. The generated images are arranged in order and provided to the user. If the user looks confused, a detailed explanation such as "The important background here is..." is added.

[0359] Example prompt sentence:

[0360] Meiji Restoration

[0361] Famous revolutionaries

[0362] Background to the arrival

[0363] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0364] Step 1:

[0365] The user inputs text data into the interface from their own device. This input data is sent to the server. The user can input, for example, educational texts or written works. This data is stored in the server's database.

[0366] Input: Text data entered by the user

[0367] Output: Text data sent to the server

[0368] Step 2:

[0369] The server then performs natural language processing on the stored text data. Specifically, it uses the natural language processing library "spaCy" to tokenize the text data, tag it with parts of speech, and perform syntactic analysis. This process extracts keywords and themes from the text.

[0370] Input: Saved text data

[0371] Output: Extracted keywords and themes

[0372] Step 3:

[0373] The server generates an image based on the analysis results. This process uses the generative AI model "DALL-E" or "Stable Diffusion." The server inputs the extracted keywords and themes into the generative AI model as prompts to generate an appropriate image.

[0374] Input: Extracted keywords and themes

[0375] Output: The generated image

[0376] Step 4:

[0377] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard, with each image accompanied by a brief description summarizing the main points of the analyzed text.

[0378] Input: Generated image

[0379] Output: Kamishibai style storyboard

[0380] Step 5:

[0381] The server then provides the completed storyboard in digital format to the user's device, where the user can view the story by clicking on a link.

[0382] Input: Kamishibai storyboard

[0383] Output: A digital picture story that can be viewed by users

[0384] Step 6:

[0385] While the user is viewing the picture story, the server uses a camera and microphone to recognize emotions in real time. This process uses the "Transformers" emotion recognition model. The recognized emotion data is sent to the server.

[0386] Input: User's facial expressions and voice data

[0387] Output: Recognized emotion data

[0388] Step 7:

[0389] The server dynamically adjusts the content of the picture story based on the recognized emotion data, for example, providing additional explanations or detailed illustrations if the user is confused.

[0390] Input: Recognized emotion data

[0391] Output: Adjusted Kamishibai content

[0392] This system not only provides users with information in a visually easy-to-understand format, but also allows them to receive content that is optimal for their emotions.

[0393] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0394] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0395] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0396] [Second embodiment]

[0397] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0398] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0399] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0400] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0401] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0402] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0403] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0404] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0405] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0406] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0407] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0408] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0409] This invention relates to a system that provides long or complex text in the form of a picture-story show that is easy for users to understand. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a storyboard in the form of a picture-story show, and finally provides it to the user.

[0410] How the system works

[0411] Receiving text data

[0412] A user inputs long text or text data through the system's interface, for example by copying and pasting text into a web form or application input field, or by typing directly into the input field. This input text data is sent to a server for initial processing.

[0413] Sentence analysis

[0414] The server stores the received text data and begins natural language processing. This process involves several steps. First, the text is tokenized, which is the process of dividing the text into the smallest units, such as words and phrases. Part-of-speech tagging is then performed, and each token is assigned a grammatical role. Next, syntactic analysis is performed to reveal the grammatical structure of the text. This allows the meaning of the text and the relationships between its parts to be analyzed.

[0415] Image generation

[0416] Based on the analysis results, the server generates an image. This process is carried out using a machine learning model. For example, DALL-E or Stable Diffusion can be used as a generative model. These models generate images with appropriate visual representations based on the content of the input text. For example, they can generate images such as "The Black Ships Arriving in Japan" or "Portrait of Sakamoto Ryoma."

[0417] Building a Kamishibai Frame

[0418] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard. This storyboard also includes a description for each image. The description is a concise summary of the main points of the analyzed text, making it easier for users to understand the content while looking at the images.

[0419] Provision to users

[0420] The completed Kamishibai is saved in digital format and provided to the user. The user can view the Kamishibai on their own device (PC or tablet). For example, by clicking on a link generated by the system, the Kamishibai can be viewed online. Furthermore, if necessary, the Kamishibai can be downloaded in PDF or slide format.

[0421] Specific examples

[0422] Example 1: History text

[0423] The user inputs a "text about Japan's Meiji Restoration" into the system. The server analyzes the text and identifies keywords such as "Meiji Restoration," "Sakamoto Ryoma," "arrival of the black ships," and "civilization and enlightenment." Based on these keywords, images of historical events and people are generated. For example, images such as "the arrival of the black ships in Japan," "scenes of Sakamoto Ryoma in action," and "the changing landscape of Japan with civilization and enlightenment" are created and presented in the form of a picture-story show with accompanying explanatory text.

[0424] Example 2: Scientific explanation

[0425] The user inputs a sentence about the mechanism of photosynthesis into the system. The server analyzes the sentence and identifies keywords such as "photosynthesis," "chlorophyll," "oxygen production," and "carbon dioxide absorption." Based on these keywords, it generates images that explain the photosynthetic process of plants. For example, it creates images of "how light hits a plant," "chemical reactions occurring inside a leaf," and "the scene where oxygen is released," and provides them in the form of a picture-story show with accompanying explanatory text.

[0426] The purpose of this system is to make it easier for users to visually understand long or complex sentences, thereby contributing to promoting comprehension and reducing stress.

[0427] The processing flow will be explained below.

[0428] Ok, I'll explain each process step in the format shown below.

[0429] Step 1:

[0430] A user accesses the system's interface and inputs long text or text data, for example by copying and pasting text into a web form or by typing it directly. At this point, the input data is sent to the server.

[0431] Step 2:

[0432] The server stores the received text data in a database. This initial storage is necessary for later analysis.

[0433] Step 3:

[0434] The server passes the stored text data to a natural language processing (NLP) library to begin analysis, which includes tokenization (dividing the text into words and phrases), part-of-speech tagging (assigning grammatical categories to each word), and syntactic analysis (analyzing the grammatical structure of the text).

[0435] Step 4:

[0436] The server uses the results of natural language processing to identify key sections of the text and important keywords. For example, if the text is about history, it might extract terms such as "Meiji Restoration," "Sakamoto Ryoma," and "Arrival of the Black Ships."

[0437] Step 5:

[0438] The server then generates an image for each section based on the analysis results. This process is done using machine learning models (e.g., DALL-E or Stable Diffusion), where you input the text content into the model and it generates an image based on that.

[0439] Step 6:

[0440] The server arranges the generated images in a sequence to create a picture-storyboard in the style of a picture story show. The order of the images is designed to follow the flow of the story in the text.

[0441] Step 7:

[0442] The server generates a description for each image, which is a brief summary of the main points of the text, to help users understand the content of the image when they view it.

[0443] Step 8:

[0444] The server stores the completed storyboard in digital format and generates a link to provide it to the user, which is a URL that the user can access to view the storyboard online.

[0445] Step 9:

[0446] Users can use their devices (PCs or tablets) to access the link provided by the server and watch the Kamishibai. The content is presented visually in an easy-to-understand way, allowing users to efficiently understand the content of the text.

[0447] Step 10:

[0448] Users can also download the Kamishibai as a PDF or slides if desired, which is convenient for offline viewing or presentations.

[0449] The above is a description of the specific processing steps of this system.

[0450] Example 1

[0451] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0452] There is a demand for a method for visualizing long or complex sentences in an efficient and easy-to-understand manner. The present invention aims to provide a system that presents text data entered by a user as a picture-storyboard format that is visually easy to understand.

[0453] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0454] In this invention, the server includes means for a user to input text data and send it through a system interface, means for the server to store the received text data and start natural language processing, means for the server to tokenize, tag parts of speech, and syntactically parse the text data, means for the server to generate prompts based on the analysis results and use them to generate visual representations using an image generation model, means for arranging the generated images in order to create a picture-storyboard in the style of a picture-story show, and means for the user to view, save, or download the storyboard in digital form. This makes it possible to provide complex text data to users in a visually easy-to-understand format.

[0455] "Means for inputting text data and transmitting it through a system interface" refers to a means by which a user can input long text or sentences and transmit them to a server via a web form or application.

[0456] The "means for saving received text data and starting natural language processing" refers to the means by which the server saves text data sent from the user and automatically starts the natural language processing process.

[0457] "Means for tokenizing, tagging parts of speech, and parsing text data" refers to the means by which the server divides text data into its smallest units, assigns grammatical roles to each word, and analyzes the grammatical structure and meaning of the entire text.

[0458] "Means for generating prompts based on the analysis results and using them to generate visual representations using an image generation model" refers to a means by which the server creates prompt sentences based on the results of the analyzed sentences, inputs them into a generative AI model, and generates appropriate images.

[0459] "Means for arranging generated images in a logical order and creating a picture-storyboard in the style of a picture-story show" refers to means by which the server arranges generated images in a logical order and adds explanatory text to each image to create a picture-storyboard in the style of a picture-story show.

[0460] "Means for users to view, save, or download storyboards in digital format" refers to means for users to view, digitally save, or download completed storyboards in picture-story show format on a device such as a computer or tablet.

[0461] This invention relates to a system that provides long or complex text in the form of a picture-story show that is easy for users to understand. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a storyboard in the form of a picture-story show, and finally provides it to the user.

[0462] How the system works

[0463] Receiving text data

[0464] A user inputs text data through the system interface, for example, by copying and pasting long sentences or text into a web form or application input field, or by typing and submitting the text data. This input text data is sent to the server for initial processing.

[0465] Sentence analysis

[0466] The server stores the received text data and starts natural language processing. Specifically, it performs the following steps:

[0467] 1. Tokenization: This is the process of breaking down the received text into its smallest units, such as words and phrases. This process is done using a language analysis engine or natural language processing library (e.g., SpaCy, NLTK).

[0468] 2. Part-of-speech tagging: This is the process of assigning a grammatical role to each token. For example, tagging the token "Sakamoto Ryoma" as a "noun."

[0469] 3. Syntactic analysis: This is the process of analyzing a sentence for its grammatical structure and meaning. This clarifies the relationships between tokens and the overall meaning of the sentence. Syntactic analysis is often performed using tree-based parsing algorithms.

[0470] Image generation

[0471] Based on the analysis results, the server generates a visual representation using an image generation model. This process involves the following steps:

[0472] 1. Take the identified keywords and phrases and turn them into prompts. For example, create a prompt like, "Generate an image of the Black Ships arriving in Japan."

[0473] 2. The created prompt sentence is input into a generative AI model (e.g., DALL-E, Stable Diffusion) to generate an appropriate image. The generated image is a visual representation based on the content of the sentence.

[0474] Building a Kamishibai Frame

[0475] The server arranges the generated images in a sequence to create a picture-storyboard, specifically:

[0476] 1. Arrange the images in a logical order. For example, arrange them in the order "Arrival of the Black Ships," "Sakamoto Ryoma in Action," and "Scenes of Westernization."

[0477] 2. A description is generated for each image. This description is a concise summary of the main points of the analyzed text. For example, it can be a specific description such as "This image shows the arrival of the Black Ships in Japan."

[0478] Provision to users

[0479] The completed Kamishibai is saved in digital format and provided to users. The specific steps are as follows:

[0480] 1. Users can view the Kamishibai on their own devices (PCs or tablets). By clicking on the link generated by the system, the Kamishibai will be displayed online.

[0481] 2. If necessary, users can download the Kamishibai in PDF or slide format. For example, by clicking the download button, the file will be saved to their device.

[0482] Specific examples

[0483] Example 1: History text

[0484] The user inputs a "text about Japan's modernization" into the system. The server analyzes the text and identifies keywords such as "modernization," "historical events," "industrial revolution," and "social change." Based on these keywords, images of historical events and people are generated. For example, images such as "a factory symbolic of modernization," "famous historical leaders," and "scenes of the industrial revolution" are created and presented in the form of a picture-story show with accompanying explanatory text.

[0485] Example prompt sentence:

[0486] Generate an image of an "Industrial Revolution factory scene."

[0487] Example 2: Scientific explanation

[0488] The user inputs a "text about the workings of ecosystems" into the system. The server analyzes the text and identifies keywords such as "ecosystem," "ecosystem," "biological interactions," and "natural environment." Based on these keywords, an image of the ecosystem is generated. For example, images such as "interactions between organisms," "changes in the natural environment," and "survival strategies of organisms" are created and presented in the form of a picture-story show with accompanying explanatory text.

[0489] Example prompt sentence:

[0490] Generate images of "scenes of interactions between organisms."

[0491] The system aims to make it easier for users to visually understand long or complex sentences, thereby contributing to promoting comprehension and reducing stress.

[0492] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0493] Step 1: Enter and send text data

[0494] Users input and submit text data through the system interface, either by copying and pasting text into a web form or by typing it directly into an application input field. The input text data is then sent to the server.

[0495] Input: Text data entered by the user

[0496] Output: Raw text data sent to the server

[0497] Step 2: Saving and initial processing of text data

[0498] The server saves the received text data and starts initial processing. Specifically, it performs validation checks on the text data (e.g., removes unsupported characters) and saves it in a database (e.g., MySQL, PostgreSQL).

[0499] Input: Submitted text data

[0500] Output: Validated text data, data stored in a database

[0501] Step 3: Tokenization

[0502] The server tokenizes the stored text data, specifically by using a language analysis engine (e.g., SpaCy, NLTK) to split the text into words and phrases.

[0503] Input: Saved text data

[0504] Output: Tokenized word list

[0505] Step 4: Part-of-Speech Tagging

[0506] The server then tags the tokenized word list with part-of-speech tags, specifically assigning each word a grammatical role (e.g., noun, verb, adjective, etc.).

[0507] Input: tokenized word list

[0508] Output: A list of words with part-of-speech tags

[0509] Step 5: Parsing

[0510] The server uses the tagged word list to perform a syntactic analysis of the entire sentence, specifically by clarifying the relationships between words and analyzing the grammatical structure of the sentence (e.g., dependency analysis).

[0511] Input: A list of words with part-of-speech tags

[0512] Output: Parsed grammar structure data

[0513] Step 6: Prompt Generation

[0514] The server generates a prompt based on the results of the parsing, such as "Please describe the scene when the Black Ships arrive in Japan."

[0515] Input: Parsed grammar structure data

[0516] Output: prompt statement

[0517] Step 7: Image generation

[0518] The server uses the generated prompt sentence to input into an image generation model (e.g., DALL-E, Stable Diffusion) to generate a visual representation. Specifically, the server inputs the prompt sentence into the generative AI model and obtains the returned image.

[0519] Input: prompt statement

[0520] Output: Generated image data

[0521] Step 8: Build a storyboard

[0522] The server arranges the generated images in a sequence to create a picture-storyboard, and also generates explanatory text for each image.

[0523] Input: Generated image data

[0524] Output: Kamishibai style storyboard

[0525] Step 9: Provide to users

[0526] The server stores the completed Kamishibai in digital format and provides it to users by generating a link that allows them to view it online and download it as a PDF or slides if desired.

[0527] Input: Kamishibai-style storyboard

[0528] Output: A link or file that users can view and download

[0529] (Application example 1)

[0530] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0531] Today's consumers expect detailed product descriptions in online and virtual stores, but text alone is not enough to adequately convey the benefits and usage scenarios of products and services. Therefore, there is a need for a method to provide product descriptions in a visually easy-to-understand format. Furthermore, when users receive product information in long or complex sentences, it can take time to understand the content, which can be frustrating. To solve these problems, a system is needed that provides product descriptions in a picture-story show format.

[0532] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0533] In this invention, the server includes means for receiving and saving text data, means for natural language processing the text data, means for generating images based on the analysis results, means for arranging the generated images in order to create a storyboard in the form of a picture-story show, means for providing the picture-story show to the user, and means for visually providing product descriptions in the virtual store, which makes it easier for the user to visually understand the details of products and services, and enables effective product descriptions in the virtual store.

[0534] "Text data" refers to data of long or complex texts entered by the user.

[0535] The "means for saving" is a means having a function for saving input text data on a server or the like.

[0536] "Means for natural language processing" refers to means for processing text data such as tokenization, part-of-speech tagging, and syntactic analysis to analyze the meaning and structure of text.

[0537] "Generating means" refers to means that has the function of generating images or other visual content based on the analysis results.

[0538] The "means for arranging in order" is a means for arranging generated images and content in order in a storyboard format.

[0539] A "picture-storyboard" is a storyboard format that makes it visually easy to understand by displaying each image in order with corresponding explanatory text.

[0540] The "means of providing" is a means of ultimately providing the user with a picture-storyboard in a viewable format.

[0541] A "virtual store" is a virtual store that operates on the Internet, and refers to an environment where users can browse and purchase products online.

[0542] In this invention, the server is a system that receives and stores text data, analyzes it, generates images, and provides them to users in the form of a picture-story show. A specific embodiment of this system is described below.

[0543] First, a user inputs text data using a terminal. This text data is sent to a server through a web form or application input field. The server stores the received text data. The stored text data is then analyzed using natural language processing. This processing includes steps such as tokenization, part-of-speech tagging, and syntactic analysis. This allows the meaning and grammatical structure of the text to be revealed.

[0544] Based on the analysis results, the server uses a generative AI model to generate appropriate images. Examples of AI models used include DALL-E and Stable Diffusion. These models generate images that visually represent the content of the input text. The generated images are arranged in a sequence to create a picture-storyboard in the style of a picture-story show. This storyboard also includes explanatory text for each image, making it easier for users to visually understand the content.

[0545] Finally, users can view the Kamishibai on their own devices. They can do so by using a link generated by the system or by downloading it in PDF or slide format. This allows users to visually understand the product's features and usage scenarios in Kamishibai format.

[0546] As a specific example, if a user inputs the text data, "This product is a smartphone app that uses the latest AI technology. Its main functions are voice recognition and image recognition," the server analyzes this text and identifies keywords such as "the latest AI technology," "smartphone app," "voice recognition," and "image recognition." Based on this, it generates images such as futuristic technology images, smartphone screens, voice input images, and image analysis images, and incorporates these into a storyboard for presentation.

[0547] An example of a prompt is, "Create a presentation for a smartphone app using the latest AI technology. Generate images and explanatory text in a picture-story format that visually express specific functions and usage scenarios."

[0548] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0549] Step 1:

[0550] A user uses a terminal to input text data, which is then sent to a server via a web form or application input field.

[0551] Input: Text data entered by the user

[0552] Output: Text data sent to the server

[0553] Step 2:

[0554] The server stores the received text data, which is used for subsequent processing.

[0555] Input: Text data sent to the server

[0556] Output: Text data stored on the server

[0557] Step 3:

[0558] The server analyzes the stored text data using natural language processing techniques, which include steps such as tokenization, part-of-speech tagging, and syntactic analysis, to reveal the meaning and grammatical structure of the text.

[0559] Input: Text data stored on the server

[0560] Data processing: sentence tokenization, part-of-speech tagging, syntactic analysis

[0561] Output: Analysis results (tokens, part-of-speech tags, syntactic analysis results)

[0562] Step 4:

[0563] Based on the analysis results, the server uses a generative AI model, such as DALL-E or Stable Diffusion, to generate an appropriate image.

[0564] Input: Analysis results after natural language processing (keywords, context, etc.)

[0565] Data calculation: Image generation using generative AI models (e.g., DALL-E, Stable Diffusion)

[0566] Output: The generated image

[0567] Step 5:

[0568] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard, with each image accompanied by a description that summarizes the main points of the analysis results.

[0569] Input: Generated images, summary of analysis results

[0570] Data processing: Arranging images and creating descriptions

[0571] Output: Kamishibai style storyboard

[0572] Step 6:

[0573] The server generates a link to provide the completed Kamishibai to the user, who can then use this link to view the Kamishibai on their device. If desired, the Kamishibai can also be downloaded as a PDF or slides.

[0574] Input: Kamishibai-style storyboard

[0575] Data processing: Link generation, data conversion to PDF and slide format

[0576] Output: A link to the Kamishibai sent to the user and the downloadable data

[0577] As a concrete example, if a user inputs the text data "This product is a smartphone app that uses the latest AI technology. Its main functions are voice recognition and image recognition," the server analyzes this text and identifies keywords such as "the latest AI technology," "smartphone app," "voice recognition," and "image recognition." Based on this, it generates images such as futuristic technology images, smartphone screens, voice input images, and image analysis images, and incorporates these into a storyboard for presentation.

[0578] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0579] This invention combines a system that converts text data into a visually easy-to-understand picture-story show format with an emotion engine that recognizes the user's emotions. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a picture-story show format storyboard, and finally provides it to the user. In addition, it has the ability to recognize the user's emotions in real time and adjust the content based on those emotions.

[0580] How the system works

[0581] Receiving text data

[0582] A user inputs long text or text data through the system interface, for example by copying and pasting text into a web form or application input field, or by typing directly into the input field. This input text data is then sent to the server.

[0583] Sentence analysis

[0584] The server stores the received text data in a database and begins natural language processing, which includes tokenization (dividing the text into words and phrases), part-of-speech tagging (assigning grammatical categories to each word), and syntactic analysis (analyzing the grammatical structure of the text).

[0585] Image generation

[0586] Based on the analysis, the server generates an image for each section, using machine learning models (e.g., DALL-E and Stable Diffusion) to generate an appropriate image that visually represents the content of the text.

[0587] Building a Kamishibai Frame

[0588] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard. This storyboard also includes a description for each image. The description is a concise summary of the main points of the analyzed text, making it easier for users to understand the content while looking at the images.

[0589] Provision to users

[0590] The completed Kamishibai is saved in digital format and provided to the user. The user can view the Kamishibai on their own device (PC or tablet). For example, by clicking on a link generated by the system, the Kamishibai can be viewed online. Furthermore, if necessary, the Kamishibai can be downloaded in PDF or slide format.

[0591] Emotional engine regulation

[0592] emotion recognition

[0593] While the user is viewing the picture story, the system uses an emotion engine to recognize the user's emotions in real time. This emotion engine analyzes the user's emotional state using facial recognition and voice analysis technologies. For example, it can determine emotions such as joy, anger, sadness, and happiness based on data obtained from the user's webcam and microphone.

[0594] Content adjustment

[0595] The server dynamically adjusts the content of the picture story based on the user's recognized emotional state, highlighting sections that interest the user or providing additional explanations to deepen understanding. For example, if the user shows a confused expression, the system adds images and text that explain the content of that section in detail.

[0596] Specific examples

[0597] Example 1: History text

[0598] The user inputs a text about Japan's Meiji Restoration into the system. The server analyzes the text and identifies keywords such as "Meiji Restoration," "Sakamoto Ryoma," "arrival of the Black Ships," and "civilization and enlightenment." Based on these keywords, images of historical events and people are generated. For example, images such as "the arrival of the Black Ships in Japan," "scenes of Sakamoto Ryoma's exploits," and "the changing landscape of Japan as a result of civilization and enlightenment" are created and presented in the form of a picture-story show with accompanying explanatory text. Furthermore, specific background information and illustrations can be added to sections that the user finds interesting, deepening their understanding.

[0599] Example 2: Scientific explanation

[0600] The user inputs a sentence about the mechanism of photosynthesis into the system. The server analyzes the sentence and identifies keywords such as "photosynthesis," "chlorophyll," "oxygen production," and "carbon dioxide absorption." Based on these keywords, it generates images that explain the photosynthetic process in plants. For example, it creates images such as "how light hits a plant," "chemical reactions occurring inside a leaf," and "the scene where oxygen is released," and provides them in the form of a picture-story show with accompanying explanatory text. In addition, it can recognize the user's emotions and, if they are confused, provide more detailed illustrations or additional explanations to help them understand.

[0601] In this way, this system not only makes the content of a sentence easier to understand visually, but also achieves a greater effect of promoting comprehension by dynamically adjusting according to the user's emotions.

[0602] The processing flow will be explained below.

[0603] Ok, I'll explain each process step in the format shown below.

[0604] Step 1:

[0605] A user accesses the system's interface and inputs long or textual data, for example by copying and pasting text into a web form or by typing it directly. This input data is then sent to the server.

[0606] Step 2:

[0607] The server stores the received text data in a database. This initial storage is necessary for later analysis.

[0608] Step 3:

[0609] The server then passes the stored text data to a natural language processing (NLP) library to begin parsing it: tokenization, part-of-speech tagging, and syntactic analysis are performed to identify key points and sections of the text.

[0610] Step 4:

[0611] The server extracts key sections and important keywords from the analysis results, identifying keywords such as "Meiji Restoration," "Sakamoto Ryoma," and "Arrival of the Black Ships."

[0612] Step 5:

[0613] The server generates images for each section based on the extracted keywords, using machine learning models (e.g., DALL-E and Stable Diffusion), generating images such as "The Black Ships Arriving in Japan" or "Portrait of Ryoma Sakamoto."

[0614] Step 6:

[0615] The server arranges the generated images in a sequence to create a picture-storyboard in the style of a picture story show. The order of the images is designed to follow the flow of the story in the text.

[0616] Step 7:

[0617] The server generates a description for each image and adds it to the storyboard. The description is a concise summary of the main points of the text, making it easier for users to understand the content while viewing the image.

[0618] Step 8:

[0619] The server stores the completed storyboard in digital format and generates a link to provide it to the user, which is a URL that the user can access to view the storyboard online.

[0620] Step 9:

[0621] The server runs an emotion engine and provides a means to recognize the user's emotional state in real time, for example by capturing the user's facial expressions and voice via a webcam or microphone and running the emotion analysis algorithm on them.

[0622] Step 10:

[0623] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time, for example, detecting when they are confused or interested.

[0624] Step 11:

[0625] The server dynamically adjusts the content of the picture story based on the perceived emotional state, highlighting interesting sections and providing additional explanations to enhance understanding—for example, adding more detailed explanatory images or text if the user is confused.

[0626] Step 12:

[0627] Users can use their devices (PCs or tablets) to access the link provided by the server and watch the picture story. The content adjusted by the emotion engine is reflected in real time, and the content is presented visually in an easy-to-understand manner.

[0628] Step 13:

[0629] Users can also download the Kamishibai as a PDF or slides if desired, which is convenient for offline viewing or presentations.

[0630] The above is a description of the specific processing steps of this system.

[0631] Example 2

[0632] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0633] Conventional systems for promoting visual comprehension of text data have the problem of being unable to respond to changes in the user's emotions during processes such as text analysis and image generation. As a result, there is concern that the user's understanding and concentration may decline, resulting in insufficient learning effectiveness. The present invention proposes a system equipped with emotion recognition functionality to solve this problem and deepen the user's understanding.

[0634] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0635] In this invention, the server includes means for receiving and storing text data, means for natural language processing the text data, means for generating images based on the analysis results, means for arranging the generated images in order to create a storyboard in the form of a picture-story show, means for providing the picture-story show to the user, means for recognizing the user's emotions in real time, and means for dynamically adjusting the content of the picture-story show based on the recognized emotions.

[0636] This makes it possible to dynamically adjust the learning content according to the user's emotional state, thereby enhancing the effectiveness of promoting understanding.

[0637] "Text data" refers to all text information entered by the user, and includes the contents of any written document, from short sentences to long sentences.

[0638] "Natural language processing" refers to the technology that enables computers to understand and analyze human language, and includes the processes of tokenizing, part-of-speech tagging, and syntactic analysis of text.

[0639] "Image generation" refers to the process of creating appropriate visual elements based on analytical results, and is done using generative AI models.

[0640] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate new data, specifically using computer vision technology.

[0641] "Kamishibai format" refers to a technique in which multiple images are arranged in an orderly manner and displayed consecutively with corresponding explanatory text attached to each.

[0642] A "storyboard" is a visually structured format that combines a series of scenes or slides to effectively convey a story or information.

[0643] "User" refers to the entity that uses the System to input text data and receive the analysis results and generated content.

[0644] "Emotion recognition" refers to the technology of determining a user's emotional state from their facial expressions, voice, etc., and includes the process of detecting the user's psychological state.

[0645] "Dynamic adjustment" refers to technology that changes content and settings in real time based on the situation or condition, automatically adapting based on user responses.

[0646] This invention is a system that converts text data into a visually easy-to-understand picture-story show format, and also combines it with an emotion engine that recognizes the user's emotions. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a picture-story show format storyboard, and finally provides it to the user. In addition, it has the function of recognizing the user's emotions in real time and adjusting the content based on those emotions.

[0647] Receiving text data

[0648] Users input long text or text data through the system interface, for example by copying and pasting text into a web form or application input field, or by typing it directly. This input text data is then sent from the terminal to the server via the Internet.

[0649] Sentence analysis

[0650] The server stores the received text data in a database and begins natural language processing (NLP). This process includes tokenization (dividing the text into words and phrases), part-of-speech tagging (assigning grammatical categories to each word), and syntactic analysis (analyzing the grammatical structure of the text). Software such as Google Cloud Natural Language API and SpaCy are used for this. For example, in a historical text, keywords such as "Meiji Restoration," "Sakamoto Ryoma," and "Arrival of the Black Ships" are identified.

[0651] Image generation

[0652] Based on the analysis results, the server uses a generative AI model (e.g., DALL-E or Stable Diffusion) to generate images corresponding to each section. This generates an appropriate image that visually represents the content of the text. For example, an image that explains "the arrival of the Black Ships in Japan" is generated. The following prompt sentences are used:

[0653] Prompt statement:

[0654] "Generate an image illustrating the arrival of the black ships in Japan during the Meiji Restoration. The image should depict the historical context and importance of this event."

[0655] Building a Kamishibai Frame

[0656] The generated images are arranged in order by the server to create a storyboard in the style of a picture-story show. Each image is also given a description. For example, a description such as "This image depicts the arrival of the Black Ships in Japan in 1853" could be added.

[0657] Provision to users

[0658] The completed Kamishibai is saved in digital format and provided to the user. Users can view the Kamishibai on their own devices (PC or tablet) or view it online by clicking on a link generated by the system. Furthermore, if desired, users can download the Kamishibai in PDF or slide format.

[0659] emotion recognition

[0660] While the user is viewing the Kamishibai, the system uses an emotion engine to recognize the user's emotions in real time. The emotion engine analyzes the user's emotional state using facial recognition technology (e.g., OpenCV) and voice analysis technology (e.g., Google Speech-to-Text API). For example, if the user shows a confused expression, the system will determine this.

[0661] Content adjustment

[0662] The server dynamically adjusts the content of the picture story based on the user's recognized emotional state, highlighting sections that interest the user or providing additional explanations to deepen understanding. For example, if the user is confused, it adds images or text that explain the content in more detail.

[0663] As described above, this system not only makes text easier to understand visually, but also enhances learning effectiveness by dynamically adjusting according to the user's emotions.

[0664] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0665] Step 1:

[0666] The user inputs text data using the system's interface. They copy and paste long sentences or text data into the input fields of a web form or application, or type them directly. Once the input is complete, the device sends it to the server. The input is the text data created or collected by the user, and the output is the secure transmission of the text data to the server.

[0667] Step 2:

[0668] The server stores the received text data in a database and starts natural language processing (NLP). The input for this process is the text data provided by the user. In this step, the following specific data processing and calculations are performed:

[0669] Tokenization: Breaking text into words and phrases.

[0670] Part-of-speech tagging: attaching a grammatical category to each word.

[0671] Syntactic analysis: Analyzing the grammatical structure of a sentence.

[0672] The output is an analysis of keywords and grammatical structures. The software used is Google Cloud Natural Language API and SpaCy.

[0673] Step 3:

[0674] Based on the analysis results, the server generates an image using a generative AI model (e.g., DALL-E or Stable Diffusion). The input is the analysis results (keywords and grammatical structures) obtained in the previous step. Specific operations include:

[0675] Input a prompt sentence into the generative AI model.

[0676] Generates an image according to the prompt.

[0677] For example, based on the keywords "Meiji Restoration" and "Arrival of the Black Ships," an image that visually represents these is generated. The prompt used is "Generate an image illustrating the arrival of the black ships in Japan during the Meiji Restoration." The output is an image generated to visually represent the content of the text.

[0678] Step 4:

[0679] The generated images are arranged in a sequence by the server to create a picture-storyboard. The input to this step is the sequence of images generated in step 3 and their accompanying explanatory text. Specific operations include:

[0680] Determine the order of the images.

[0681] Add a description for each image.

[0682] Build a storyboard by integrating images and descriptions.

[0683] The output is a visually organized storyboard in the style of a picture story.

[0684] Step 5:

[0685] The completed Kamishibai is saved in digital format and provided to the user. The input for this step is the completed storyboard. Specific actions include:

[0686] Upload your storyboard online.

[0687] A link is generated and notified to the user.

[0688] If necessary, convert them into PDF or slide format and make them downloadable.

[0689] The output is a digital Kamishibai that is accessible to users.

[0690] Step 6:

[0691] While the user is viewing the Kamishibai, the system uses the emotion engine to recognize the user's emotions in real time. The input for this step is the user's facial expressions and voice data. The specific operation is as follows:

[0692] Data is collected in real time via webcam and microphone.

[0693] Analyze emotions using facial recognition technology (e.g., OpenCV) and voice analysis technology (e.g., Google Speech-to-Text API).

[0694] The output is data indicating the user's emotional state (e.g., joy, confusion, interest, etc.).

[0695] Step 7:

[0696] Based on the recognized user's emotional state, the server dynamically adjusts the content of the Kamishibai. The input for this step is the user's emotional data. Specific operations include:

[0697] Highlight certain sections depending on your emotional state.

[0698] Insert additional explanations and illustrations as needed.

[0699] For example, if the user is confused, more detailed images and explanations are added. The output is a picture-story show that is dynamically adjusted according to the user's emotions.

[0700] (Application example 2)

[0701] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0702] Conventional text data visualization systems simply provide information unilaterally, without considering the user's emotions or level of understanding. This often leaves users confused or disinterested, resulting in insufficient understanding of the information. Furthermore, the information provided visually is limited, and particularly in education and content distribution services, it is necessary to flexibly adjust the content based on the user's reaction.

[0703] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0704] In this invention, the server includes means for receiving and saving text data, means for natural language processing the text data, means for generating images based on the analysis results, means for arranging the generated images in order to create a storyboard in the form of a picture-story show, means for providing the picture-story show to the user, means for recognizing the user's emotions in real time, and means for adjusting the content based on the user's emotions. This makes it possible to dynamically adjust the provision of visual information according to the user's emotions and achieve a greater effect in promoting understanding.

[0705] The "means for receiving and saving text data" is a function for sending text data entered by the user to a server and saving it in a database.

[0706] "Means for natural language processing of text data" refers to a function for analyzing text data through a series of processes including tokenization, part-of-speech tagging, and syntactic analysis.

[0707] "Means for generating images based on analysis results" refers to a function that generates appropriate images using a machine learning engine such as a generative AI model based on the results of natural language processing.

[0708] "Means for arranging generated images in order and creating a storyboard in the form of a picture-story show" is a function for arranging multiple generated images in order as a storyboard and displaying them in the form of a picture-story show.

[0709] The "means for providing a picture story to a user" is a function for providing a picture story created in electronic form so that the user can view it.

[0710] "Means for recognizing user emotions in real time" refers to a function that uses sensors such as cameras and microphones to analyze emotions from the user's facial expressions and voice and recognize them in real time.

[0711] The "means for adjusting content based on user emotions" is a function that dynamically changes the content of the picture book based on the recognized user emotions, and provides additional explanations or detailed illustrations as needed.

[0712] The present invention is a system for converting text data into a visually easy-to-understand picture-story format, and includes a function for recognizing a user's emotions in real time and adjusting the content accordingly. Specific embodiments of the system are described below.

[0713] System configuration

[0714] The system consists of hardware such as a user terminal, server, camera, and microphone, as well as software components including a natural language processing engine, an image generation engine, and an emotion recognition engine.

[0715] Entering and saving text data

[0716] Users use their own devices to input text data into the system's interface, which is provided as an input field on a web form or a smartphone application. The input text data is sent to the server and stored in a database.

[0717] Sentence analysis

[0718] The server then performs natural language processing on the stored text data. It uses software such as spaCy, a natural language processing library, to tokenize the text data, tag it with parts of speech, and perform syntactic analysis. This process extracts keywords and themes from the text.

[0719] Image generation

[0720] Based on the analyzed keywords and themes, the server generates corresponding images using an image generation engine. Generative AI models used here include "DALL-E" and "Stable Diffusion." These models generate appropriate images based on the input prompt.

[0721] Creating a storyboard in the style of a picture story

[0722] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard, with each image accompanied by a brief description summarizing the main points of the analyzed text.

[0723] Provision to users

[0724] The completed storyboard in picture-storytelling format is saved in digital format and provided to users, who can view it on their own devices and download it as a PDF or slides if desired.

[0725] Emotion recognition and content regulation

[0726] While the user is viewing the Kamishibai, the server uses a camera and microphone to recognize the user's emotions in real time. The emotion recognition engine used uses the "Transformers" emotion recognition model. Based on the recognized emotions, the server dynamically adjusts the content of the Kamishibai and provides additional explanations and detailed illustrations to facilitate understanding.

[0727] Examples and prompts

[0728] For example, if a user inputs "historical text," keywords such as "Meiji Restoration," "famous revolutionary," and "background of arrival" are extracted, and DALL-E generates corresponding images. The generated images are arranged in order and provided to the user. If the user looks confused, a detailed explanation such as "The important background here is..." is added.

[0729] Example prompt sentence:

[0730] Meiji Restoration

[0731] Famous revolutionaries

[0732] Background to the arrival

[0733] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0734] Step 1:

[0735] The user inputs text data into the interface from their own device. This input data is sent to the server. The user can input, for example, educational texts or written works. This data is stored in the server's database.

[0736] Input: Text data entered by the user

[0737] Output: Text data sent to the server

[0738] Step 2:

[0739] The server then performs natural language processing on the stored text data. Specifically, it uses the natural language processing library "spaCy" to tokenize the text data, tag it with parts of speech, and perform syntactic analysis. This process extracts keywords and themes from the text.

[0740] Input: Saved text data

[0741] Output: Extracted keywords and themes

[0742] Step 3:

[0743] The server generates an image based on the analysis results. This process uses the generative AI model "DALL-E" or "Stable Diffusion." The server inputs the extracted keywords and themes into the generative AI model as prompts to generate an appropriate image.

[0744] Input: Extracted keywords and themes

[0745] Output: The generated image

[0746] Step 4:

[0747] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard, with each image accompanied by a brief description summarizing the main points of the analyzed text.

[0748] Input: Generated image

[0749] Output: Kamishibai style storyboard

[0750] Step 5:

[0751] The server then provides the completed storyboard in digital format to the user's device, where the user can view the story by clicking on a link.

[0752] Input: Kamishibai storyboard

[0753] Output: A digital picture story that can be viewed by users

[0754] Step 6:

[0755] While the user is viewing the picture story, the server uses a camera and microphone to recognize emotions in real time. This process uses the "Transformers" emotion recognition model. The recognized emotion data is sent to the server.

[0756] Input: User's facial expressions and voice data

[0757] Output: Recognized emotion data

[0758] Step 7:

[0759] The server dynamically adjusts the content of the picture story based on the recognized emotion data, for example, providing additional explanations or detailed illustrations if the user is confused.

[0760] Input: Recognized emotion data

[0761] Output: Adjusted Kamishibai content

[0762] This system not only provides users with information in a visually easy-to-understand format, but also allows them to receive content that is optimal for their emotions.

[0763] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0764] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0765] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0766] [Third embodiment]

[0767] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0768] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0769] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0770] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0771] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0772] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0773] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0774] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0775] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0776] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0777] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0778] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0779] This invention relates to a system that provides long or complex text in the form of a picture-story show that is easy for users to understand. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a storyboard in the form of a picture-story show, and finally provides it to the user.

[0780] How the system works

[0781] Receiving text data

[0782] A user inputs long text or text data through the system's interface, for example by copying and pasting text into a web form or application input field, or by typing directly into the input field. This input text data is sent to a server for initial processing.

[0783] Sentence analysis

[0784] The server stores the received text data and begins natural language processing. This process involves several steps. First, the text is tokenized, which is the process of dividing the text into the smallest units, such as words and phrases. Part-of-speech tagging is then performed, and each token is assigned a grammatical role. Next, syntactic analysis is performed to reveal the grammatical structure of the text. This allows the meaning of the text and the relationships between its parts to be analyzed.

[0785] Image generation

[0786] Based on the analysis results, the server generates an image. This process is carried out using a machine learning model. For example, DALL-E or Stable Diffusion can be used as a generative model. These models generate images with appropriate visual representations based on the content of the input text. For example, they can generate images such as "The Black Ships Arriving in Japan" or "Portrait of Sakamoto Ryoma."

[0787] Building a Kamishibai Frame

[0788] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard. This storyboard also includes a description for each image. The description is a concise summary of the main points of the analyzed text, making it easier for users to understand the content while looking at the images.

[0789] Provision to users

[0790] The completed Kamishibai is saved in digital format and provided to the user. The user can view the Kamishibai on their own device (PC or tablet). For example, by clicking on a link generated by the system, the Kamishibai can be viewed online. Furthermore, if necessary, the Kamishibai can be downloaded in PDF or slide format.

[0791] Specific examples

[0792] Example 1: History text

[0793] The user inputs a "text about Japan's Meiji Restoration" into the system. The server analyzes the text and identifies keywords such as "Meiji Restoration," "Sakamoto Ryoma," "arrival of the black ships," and "civilization and enlightenment." Based on these keywords, images of historical events and people are generated. For example, images such as "the arrival of the black ships in Japan," "scenes of Sakamoto Ryoma in action," and "the changing landscape of Japan with civilization and enlightenment" are created and presented in the form of a picture-story show with accompanying explanatory text.

[0794] Example 2: Scientific explanation

[0795] The user inputs a sentence about the mechanism of photosynthesis into the system. The server analyzes the sentence and identifies keywords such as "photosynthesis," "chlorophyll," "oxygen production," and "carbon dioxide absorption." Based on these keywords, it generates images that explain the photosynthetic process of plants. For example, it creates images of "how light hits a plant," "chemical reactions occurring inside a leaf," and "the scene where oxygen is released," and provides them in the form of a picture-story show with accompanying explanatory text.

[0796] The purpose of this system is to make it easier for users to visually understand long or complex sentences, thereby contributing to promoting comprehension and reducing stress.

[0797] The processing flow will be explained below.

[0798] Ok, I'll explain each process step in the format shown below.

[0799] Step 1:

[0800] A user accesses the system's interface and inputs long text or text data, for example by copying and pasting text into a web form or by typing it directly. At this point, the input data is sent to the server.

[0801] Step 2:

[0802] The server stores the received text data in a database. This initial storage is necessary for later analysis.

[0803] Step 3:

[0804] The server passes the stored text data to a natural language processing (NLP) library to begin analysis, which includes tokenization (dividing the text into words and phrases), part-of-speech tagging (assigning grammatical categories to each word), and syntactic analysis (analyzing the grammatical structure of the text).

[0805] Step 4:

[0806] The server uses the results of natural language processing to identify key sections of the text and important keywords. For example, if the text is about history, it might extract terms such as "Meiji Restoration," "Sakamoto Ryoma," and "Arrival of the Black Ships."

[0807] Step 5:

[0808] The server then generates an image for each section based on the analysis results. This process is done using machine learning models (e.g., DALL-E or Stable Diffusion), where you input the text content into the model and it generates an image based on that.

[0809] Step 6:

[0810] The server arranges the generated images in a sequence to create a picture-storyboard in the style of a picture story show. The order of the images is designed to follow the flow of the story in the text.

[0811] Step 7:

[0812] The server generates a description for each image, which is a brief summary of the main points of the text, to help users understand the content of the image when they view it.

[0813] Step 8:

[0814] The server stores the completed storyboard in digital format and generates a link to provide it to the user, which is a URL that the user can access to view the storyboard online.

[0815] Step 9:

[0816] Users can use their devices (PCs or tablets) to access the link provided by the server and watch the Kamishibai. The content is presented visually in an easy-to-understand way, allowing users to efficiently understand the content of the text.

[0817] Step 10:

[0818] Users can also download the Kamishibai as a PDF or slides if desired, which is convenient for offline viewing or presentations.

[0819] The above is a description of the specific processing steps of this system.

[0820] Example 1

[0821] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0822] There is a demand for a method for visualizing long or complex sentences in an efficient and easy-to-understand manner. The present invention aims to provide a system that presents text data entered by a user as a picture-storyboard format that is visually easy to understand.

[0823] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0824] In this invention, the server includes means for a user to input text data and send it through a system interface, means for the server to store the received text data and start natural language processing, means for the server to tokenize, tag parts of speech, and syntactically parse the text data, means for the server to generate prompts based on the analysis results and use them to generate visual representations using an image generation model, means for arranging the generated images in order to create a picture-storyboard in the style of a picture-story show, and means for the user to view, save, or download the storyboard in digital form. This makes it possible to provide complex text data to users in a visually easy-to-understand format.

[0825] "Means for inputting text data and transmitting it through a system interface" refers to a means by which a user can input long text or sentences and transmit them to a server via a web form or application.

[0826] The "means for saving received text data and starting natural language processing" refers to the means by which the server saves text data sent from the user and automatically starts the natural language processing process.

[0827] "Means for tokenizing, tagging parts of speech, and parsing text data" refers to the means by which the server divides text data into its smallest units, assigns grammatical roles to each word, and analyzes the grammatical structure and meaning of the entire text.

[0828] "Means for generating prompts based on the analysis results and using them to generate visual representations using an image generation model" refers to a means by which the server creates prompt sentences based on the results of the analyzed sentences, inputs them into a generative AI model, and generates appropriate images.

[0829] "Means for arranging generated images in a logical order and creating a picture-storyboard in the style of a picture-story show" refers to means by which the server arranges generated images in a logical order and adds explanatory text to each image to create a picture-storyboard in the style of a picture-story show.

[0830] "Means for users to view, save, or download storyboards in digital format" refers to means for users to view, digitally save, or download completed storyboards in picture-story show format on a device such as a computer or tablet.

[0831] This invention relates to a system that provides long or complex text in the form of a picture-story show that is easy for users to understand. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a storyboard in the form of a picture-story show, and finally provides it to the user.

[0832] How the system works

[0833] Receiving text data

[0834] A user inputs text data through the system interface, for example, by copying and pasting long sentences or text into a web form or application input field, or by typing and submitting the text data. This input text data is sent to the server for initial processing.

[0835] Sentence analysis

[0836] The server stores the received text data and starts natural language processing. Specifically, it performs the following steps:

[0837] 1. Tokenization: This is the process of breaking down the received text into its smallest units, such as words and phrases. This process is done using a language analysis engine or natural language processing library (e.g., SpaCy, NLTK).

[0838] 2. Part-of-speech tagging: This is the process of assigning a grammatical role to each token. For example, tagging the token "Sakamoto Ryoma" as a "noun."

[0839] 3. Syntactic analysis: This is the process of analyzing a sentence for its grammatical structure and meaning. This clarifies the relationships between tokens and the overall meaning of the sentence. Syntactic analysis is often performed using tree-based parsing algorithms.

[0840] Image generation

[0841] Based on the analysis results, the server generates a visual representation using an image generation model. This process involves the following steps:

[0842] 1. Take the identified keywords and phrases and turn them into prompts. For example, create a prompt like, "Generate an image of the Black Ships arriving in Japan."

[0843] 2. The created prompt sentence is input into a generative AI model (e.g., DALL-E, Stable Diffusion) to generate an appropriate image. The generated image is a visual representation based on the content of the sentence.

[0844] Building a Kamishibai Frame

[0845] The server arranges the generated images in a sequence to create a picture-storyboard, specifically:

[0846] 1. Arrange the images in a logical order. For example, arrange them in the order "Arrival of the Black Ships," "Sakamoto Ryoma in Action," and "Scenes of Westernization."

[0847] 2. A description is generated for each image. This description is a concise summary of the main points of the analyzed text. For example, it can be a specific description such as "This image shows the arrival of the Black Ships in Japan."

[0848] Provision to users

[0849] The completed Kamishibai is saved in digital format and provided to users. The specific steps are as follows:

[0850] 1. Users can view the Kamishibai on their own devices (PCs or tablets). By clicking on the link generated by the system, the Kamishibai will be displayed online.

[0851] 2. If necessary, users can download the Kamishibai in PDF or slide format. For example, by clicking the download button, the file will be saved to their device.

[0852] Specific examples

[0853] Example 1: History text

[0854] The user inputs a "text about Japan's modernization" into the system. The server analyzes the text and identifies keywords such as "modernization," "historical events," "industrial revolution," and "social change." Based on these keywords, images of historical events and people are generated. For example, images such as "a factory symbolic of modernization," "famous historical leaders," and "scenes of the industrial revolution" are created and presented in the form of a picture-story show with accompanying explanatory text.

[0855] Example prompt sentence:

[0856] Generate an image of an "Industrial Revolution factory scene."

[0857] Example 2: Scientific explanation

[0858] The user inputs a "text about the workings of ecosystems" into the system. The server analyzes the text and identifies keywords such as "ecosystem," "ecosystem," "biological interactions," and "natural environment." Based on these keywords, an image of the ecosystem is generated. For example, images such as "interactions between organisms," "changes in the natural environment," and "survival strategies of organisms" are created and presented in the form of a picture-story show with accompanying explanatory text.

[0859] Example prompt sentence:

[0860] Generate images of "scenes of interactions between organisms."

[0861] The system aims to make it easier for users to visually understand long or complex sentences, thereby contributing to promoting comprehension and reducing stress.

[0862] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0863] Step 1: Enter and send text data

[0864] Users input and submit text data through the system interface, either by copying and pasting text into a web form or by typing it directly into an application input field. The input text data is then sent to the server.

[0865] Input: Text data entered by the user

[0866] Output: Raw text data sent to the server

[0867] Step 2: Saving and initial processing of text data

[0868] The server saves the received text data and starts initial processing. Specifically, it performs validation checks on the text data (e.g., removes unsupported characters) and saves it in a database (e.g., MySQL, PostgreSQL).

[0869] Input: Submitted text data

[0870] Output: Validated text data, data stored in a database

[0871] Step 3: Tokenization

[0872] The server tokenizes the stored text data, specifically by using a language analysis engine (e.g., SpaCy, NLTK) to split the text into words and phrases.

[0873] Input: Saved text data

[0874] Output: Tokenized word list

[0875] Step 4: Part-of-Speech Tagging

[0876] The server then tags the tokenized word list with part-of-speech tags, specifically assigning each word a grammatical role (e.g., noun, verb, adjective, etc.).

[0877] Input: tokenized word list

[0878] Output: A list of words with part-of-speech tags

[0879] Step 5: Parsing

[0880] The server uses the tagged word list to perform a syntactic analysis of the entire sentence, specifically by clarifying the relationships between words and analyzing the grammatical structure of the sentence (e.g., dependency analysis).

[0881] Input: A list of words with part-of-speech tags

[0882] Output: Parsed grammar structure data

[0883] Step 6: Prompt Generation

[0884] The server generates a prompt based on the results of the parsing, such as "Please describe the scene when the Black Ships arrive in Japan."

[0885] Input: Parsed grammar structure data

[0886] Output: prompt statement

[0887] Step 7: Image generation

[0888] The server uses the generated prompt sentence to input into an image generation model (e.g., DALL-E, Stable Diffusion) to generate a visual representation. Specifically, the server inputs the prompt sentence into the generative AI model and obtains the returned image.

[0889] Input: prompt statement

[0890] Output: Generated image data

[0891] Step 8: Build a storyboard

[0892] The server arranges the generated images in a sequence to create a picture-storyboard, and also generates explanatory text for each image.

[0893] Input: Generated image data

[0894] Output: Kamishibai style storyboard

[0895] Step 9: Provide to users

[0896] The server stores the completed Kamishibai in digital format and provides it to users by generating a link that allows them to view it online and download it as a PDF or slides if desired.

[0897] Input: Kamishibai-style storyboard

[0898] Output: A link or file that users can view and download

[0899] (Application example 1)

[0900] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0901] Today's consumers expect detailed product descriptions in online and virtual stores, but text alone is not enough to adequately convey the benefits and usage scenarios of products and services. Therefore, there is a need for a method to provide product descriptions in a visually easy-to-understand format. Furthermore, when users receive product information in long or complex sentences, it can take time to understand the content, which can be frustrating. To solve these problems, a system is needed that provides product descriptions in a picture-story show format.

[0902] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0903] In this invention, the server includes means for receiving and saving text data, means for natural language processing the text data, means for generating images based on the analysis results, means for arranging the generated images in order to create a storyboard in the form of a picture-story show, means for providing the picture-story show to the user, and means for visually providing product descriptions in the virtual store, which makes it easier for the user to visually understand the details of products and services, and enables effective product descriptions in the virtual store.

[0904] "Text data" refers to data of long or complex texts entered by the user.

[0905] The "means for saving" is a means having a function for saving input text data on a server or the like.

[0906] "Means for natural language processing" refers to means for processing text data such as tokenization, part-of-speech tagging, and syntactic analysis to analyze the meaning and structure of text.

[0907] "Generating means" refers to means that has the function of generating images or other visual content based on the analysis results.

[0908] The "means for arranging in order" is a means for arranging generated images and content in order in a storyboard format.

[0909] A "picture-storyboard" is a storyboard format that makes it visually easy to understand by displaying each image in order with corresponding explanatory text.

[0910] The "means of providing" is a means of ultimately providing the user with a picture-storyboard in a viewable format.

[0911] A "virtual store" is a virtual store that operates on the Internet, and refers to an environment where users can browse and purchase products online.

[0912] In this invention, the server is a system that receives and stores text data, analyzes it, generates images, and provides them to users in the form of a picture-story show. A specific embodiment of this system is described below.

[0913] First, a user inputs text data using a terminal. This text data is sent to a server through a web form or application input field. The server stores the received text data. The stored text data is then analyzed using natural language processing. This processing includes steps such as tokenization, part-of-speech tagging, and syntactic analysis. This allows the meaning and grammatical structure of the text to be revealed.

[0914] Based on the analysis results, the server uses a generative AI model to generate appropriate images. Examples of AI models used include DALL-E and Stable Diffusion. These models generate images that visually represent the content of the input text. The generated images are arranged in a sequence to create a picture-storyboard in the style of a picture-story show. This storyboard also includes explanatory text for each image, making it easier for users to visually understand the content.

[0915] Finally, users can view the Kamishibai on their own devices. They can do so by using a link generated by the system or by downloading it in PDF or slide format. This allows users to visually understand the product's features and usage scenarios in Kamishibai format.

[0916] As a specific example, if a user inputs the text data, "This product is a smartphone app that uses the latest AI technology. Its main functions are voice recognition and image recognition," the server analyzes this text and identifies keywords such as "the latest AI technology," "smartphone app," "voice recognition," and "image recognition." Based on this, it generates images such as futuristic technology images, smartphone screens, voice input images, and image analysis images, and incorporates these into a storyboard for presentation.

[0917] An example of a prompt is, "Create a presentation for a smartphone app using the latest AI technology. Generate images and explanatory text in a picture-story format that visually express specific functions and usage scenarios."

[0918] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0919] Step 1:

[0920] A user uses a terminal to input text data, which is then sent to a server via a web form or application input field.

[0921] Input: Text data entered by the user

[0922] Output: Text data sent to the server

[0923] Step 2:

[0924] The server stores the received text data, which is used for subsequent processing.

[0925] Input: Text data sent to the server

[0926] Output: Text data stored on the server

[0927] Step 3:

[0928] The server analyzes the stored text data using natural language processing techniques, which include steps such as tokenization, part-of-speech tagging, and syntactic analysis, to reveal the meaning and grammatical structure of the text.

[0929] Input: Text data stored on the server

[0930] Data processing: sentence tokenization, part-of-speech tagging, syntactic analysis

[0931] Output: Analysis results (tokens, part-of-speech tags, syntactic analysis results)

[0932] Step 4:

[0933] Based on the analysis results, the server uses a generative AI model, such as DALL-E or Stable Diffusion, to generate an appropriate image.

[0934] Input: Analysis results after natural language processing (keywords, context, etc.)

[0935] Data calculation: Image generation using generative AI models (e.g., DALL-E, Stable Diffusion)

[0936] Output: The generated image

[0937] Step 5:

[0938] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard, with each image accompanied by a description that summarizes the main points of the analysis results.

[0939] Input: Generated images, summary of analysis results

[0940] Data processing: Arranging images and creating descriptions

[0941] Output: Kamishibai style storyboard

[0942] Step 6:

[0943] The server generates a link to provide the completed Kamishibai to the user, who can then use this link to view the Kamishibai on their device. If desired, the Kamishibai can also be downloaded as a PDF or slides.

[0944] Input: Kamishibai-style storyboard

[0945] Data processing: Link generation, data conversion to PDF and slide format

[0946] Output: A link to the Kamishibai sent to the user and the downloadable data

[0947] As a concrete example, if a user inputs the text data "This product is a smartphone app that uses the latest AI technology. Its main functions are voice recognition and image recognition," the server analyzes this text and identifies keywords such as "the latest AI technology," "smartphone app," "voice recognition," and "image recognition." Based on this, it generates images such as futuristic technology images, smartphone screens, voice input images, and image analysis images, and incorporates these into a storyboard for presentation.

[0948] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0949] This invention combines a system that converts text data into a visually easy-to-understand picture-story show format with an emotion engine that recognizes the user's emotions. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a picture-story show format storyboard, and finally provides it to the user. In addition, it has the ability to recognize the user's emotions in real time and adjust the content based on those emotions.

[0950] How the system works

[0951] Receiving text data

[0952] A user inputs long text or text data through the system interface, for example by copying and pasting text into a web form or application input field, or by typing directly into the input field. This input text data is then sent to the server.

[0953] Sentence analysis

[0954] The server stores the received text data in a database and begins natural language processing, which includes tokenization (dividing the text into words and phrases), part-of-speech tagging (assigning grammatical categories to each word), and syntactic analysis (analyzing the grammatical structure of the text).

[0955] Image generation

[0956] Based on the analysis, the server generates an image for each section, using machine learning models (e.g., DALL-E and Stable Diffusion) to generate an appropriate image that visually represents the content of the text.

[0957] Building a Kamishibai Frame

[0958] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard. This storyboard also includes a description for each image. The description is a concise summary of the main points of the analyzed text, making it easier for users to understand the content while looking at the images.

[0959] Provision to users

[0960] The completed Kamishibai is saved in digital format and provided to the user. The user can view the Kamishibai on their own device (PC or tablet). For example, by clicking on a link generated by the system, the Kamishibai can be viewed online. Furthermore, if necessary, the Kamishibai can be downloaded in PDF or slide format.

[0961] Emotional engine regulation

[0962] emotion recognition

[0963] While the user is viewing the picture story, the system uses an emotion engine to recognize the user's emotions in real time. This emotion engine analyzes the user's emotional state using facial recognition and voice analysis technologies. For example, it can determine emotions such as joy, anger, sadness, and happiness based on data obtained from the user's webcam and microphone.

[0964] Content adjustment

[0965] The server dynamically adjusts the content of the picture story based on the user's recognized emotional state, highlighting sections that interest the user or providing additional explanations to deepen understanding. For example, if the user shows a confused expression, the system adds images and text that explain the content of that section in detail.

[0966] Specific examples

[0967] Example 1: History text

[0968] The user inputs a text about Japan's Meiji Restoration into the system. The server analyzes the text and identifies keywords such as "Meiji Restoration," "Sakamoto Ryoma," "arrival of the Black Ships," and "civilization and enlightenment." Based on these keywords, images of historical events and people are generated. For example, images such as "the arrival of the Black Ships in Japan," "scenes of Sakamoto Ryoma's exploits," and "the changing landscape of Japan as a result of civilization and enlightenment" are created and presented in the form of a picture-story show with accompanying explanatory text. Furthermore, specific background information and illustrations can be added to sections that the user finds interesting, deepening their understanding.

[0969] Example 2: Scientific explanation

[0970] The user inputs a sentence about the mechanism of photosynthesis into the system. The server analyzes the sentence and identifies keywords such as "photosynthesis," "chlorophyll," "oxygen production," and "carbon dioxide absorption." Based on these keywords, it generates images that explain the photosynthetic process in plants. For example, it creates images such as "how light hits a plant," "chemical reactions occurring inside a leaf," and "the scene where oxygen is released," and provides them in the form of a picture-story show with accompanying explanatory text. In addition, it can recognize the user's emotions and, if they are confused, provide more detailed illustrations or additional explanations to help them understand.

[0971] In this way, this system not only makes the content of a sentence easier to understand visually, but also achieves a greater effect of promoting comprehension by dynamically adjusting according to the user's emotions.

[0972] The processing flow will be explained below.

[0973] Ok, I'll explain each process step in the format shown below.

[0974] Step 1:

[0975] A user accesses the system's interface and inputs long or textual data, for example by copying and pasting text into a web form or by typing it directly. This input data is then sent to the server.

[0976] Step 2:

[0977] The server stores the received text data in a database. This initial storage is necessary for later analysis.

[0978] Step 3:

[0979] The server then passes the stored text data to a natural language processing (NLP) library to begin parsing it: tokenization, part-of-speech tagging, and syntactic analysis are performed to identify key points and sections of the text.

[0980] Step 4:

[0981] The server extracts key sections and important keywords from the analysis results, identifying keywords such as "Meiji Restoration," "Sakamoto Ryoma," and "Arrival of the Black Ships."

[0982] Step 5:

[0983] The server generates images for each section based on the extracted keywords, using machine learning models (e.g., DALL-E and Stable Diffusion), generating images such as "The Black Ships Arriving in Japan" or "Portrait of Ryoma Sakamoto."

[0984] Step 6:

[0985] The server arranges the generated images in a sequence to create a picture-storyboard in the style of a picture story show. The order of the images is designed to follow the flow of the story in the text.

[0986] Step 7:

[0987] The server generates a description for each image and adds it to the storyboard. The description is a concise summary of the main points of the text, making it easier for users to understand the content while viewing the image.

[0988] Step 8:

[0989] The server stores the completed storyboard in digital format and generates a link to provide it to the user, which is a URL that the user can access to view the storyboard online.

[0990] Step 9:

[0991] The server runs an emotion engine and provides a means to recognize the user's emotional state in real time, for example by capturing the user's facial expressions and voice via a webcam or microphone and running the emotion analysis algorithm on them.

[0992] Step 10:

[0993] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time, for example, detecting when they are confused or interested.

[0994] Step 11:

[0995] The server dynamically adjusts the content of the picture story based on the perceived emotional state, highlighting interesting sections and providing additional explanations to enhance understanding—for example, adding more detailed explanatory images or text if the user is confused.

[0996] Step 12:

[0997] Users can use their devices (PCs or tablets) to access the link provided by the server and watch the picture story. The content adjusted by the emotion engine is reflected in real time, and the content is presented visually in an easy-to-understand manner.

[0998] Step 13:

[0999] Users can also download the Kamishibai as a PDF or slides if desired, which is convenient for offline viewing or presentations.

[1000] The above is a description of the specific processing steps of this system.

[1001] Example 2

[1002] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1003] Conventional systems for promoting visual comprehension of text data have the problem of being unable to respond to changes in the user's emotions during processes such as text analysis and image generation. As a result, there is concern that the user's understanding and concentration may decline, resulting in insufficient learning effectiveness. The present invention proposes a system equipped with emotion recognition functionality to solve this problem and deepen the user's understanding.

[1004] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1005] In this invention, the server includes means for receiving and storing text data, means for natural language processing the text data, means for generating images based on the analysis results, means for arranging the generated images in order to create a storyboard in the form of a picture-story show, means for providing the picture-story show to the user, means for recognizing the user's emotions in real time, and means for dynamically adjusting the content of the picture-story show based on the recognized emotions.

[1006] This makes it possible to dynamically adjust the learning content according to the user's emotional state, thereby enhancing the effectiveness of promoting understanding.

[1007] "Text data" refers to all text information entered by the user, and includes the contents of any written document, from short sentences to long sentences.

[1008] "Natural language processing" refers to the technology that enables computers to understand and analyze human language, and includes the processes of tokenizing, part-of-speech tagging, and syntactic analysis of text.

[1009] "Image generation" refers to the process of creating appropriate visual elements based on analytical results, and is done using generative AI models.

[1010] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate new data, specifically using computer vision technology.

[1011] "Kamishibai format" refers to a technique in which multiple images are arranged in an orderly manner and displayed consecutively with corresponding explanatory text attached to each.

[1012] A "storyboard" is a visually structured format that combines a series of scenes or slides to effectively convey a story or information.

[1013] "User" refers to the entity that uses the System to input text data and receive the analysis results and generated content.

[1014] "Emotion recognition" refers to the technology of determining a user's emotional state from their facial expressions, voice, etc., and includes the process of detecting the user's psychological state.

[1015] "Dynamic adjustment" refers to technology that changes content and settings in real time based on the situation or condition, automatically adapting based on user responses.

[1016] This invention is a system that converts text data into a visually easy-to-understand picture-story show format, and also combines it with an emotion engine that recognizes the user's emotions. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a picture-story show format storyboard, and finally provides it to the user. In addition, it has the function of recognizing the user's emotions in real time and adjusting the content based on those emotions.

[1017] Receiving text data

[1018] Users input long text or text data through the system interface, for example by copying and pasting text into a web form or application input field, or by typing it directly. This input text data is then sent from the terminal to the server via the Internet.

[1019] Sentence analysis

[1020] The server stores the received text data in a database and begins natural language processing (NLP). This process includes tokenization (dividing the text into words and phrases), part-of-speech tagging (assigning grammatical categories to each word), and syntactic analysis (analyzing the grammatical structure of the text). Software such as Google Cloud Natural Language API and SpaCy are used for this. For example, in a historical text, keywords such as "Meiji Restoration," "Sakamoto Ryoma," and "Arrival of the Black Ships" are identified.

[1021] Image generation

[1022] Based on the analysis results, the server uses a generative AI model (e.g., DALL-E or Stable Diffusion) to generate images corresponding to each section. This generates an appropriate image that visually represents the content of the text. For example, an image that explains "the arrival of the Black Ships in Japan" is generated. The following prompt sentences are used:

[1023] Prompt statement:

[1024] "Generate an image illustrating the arrival of the black ships in Japan during the Meiji Restoration. The image should depict the historical context and importance of this event."

[1025] Building a Kamishibai Frame

[1026] The generated images are arranged in order by the server to create a storyboard in the style of a picture-story show. Each image is also given a description. For example, a description such as "This image depicts the arrival of the Black Ships in Japan in 1853" could be added.

[1027] Provision to users

[1028] The completed Kamishibai is saved in digital format and provided to the user. Users can view the Kamishibai on their own devices (PC or tablet) or view it online by clicking on a link generated by the system. Furthermore, if desired, users can download the Kamishibai in PDF or slide format.

[1029] emotion recognition

[1030] While the user is viewing the Kamishibai, the system uses an emotion engine to recognize the user's emotions in real time. The emotion engine analyzes the user's emotional state using facial recognition technology (e.g., OpenCV) and voice analysis technology (e.g., Google Speech-to-Text API). For example, if the user shows a confused expression, the system will determine this.

[1031] Content adjustment

[1032] The server dynamically adjusts the content of the picture story based on the user's recognized emotional state, highlighting sections that interest the user or providing additional explanations to deepen understanding. For example, if the user is confused, it adds images or text that explain the content in more detail.

[1033] As described above, this system not only makes text easier to understand visually, but also enhances learning effectiveness by dynamically adjusting according to the user's emotions.

[1034] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1035] Step 1:

[1036] The user inputs text data using the system's interface. They copy and paste long sentences or text data into the input fields of a web form or application, or type them directly. Once the input is complete, the device sends it to the server. The input is the text data created or collected by the user, and the output is the secure transmission of the text data to the server.

[1037] Step 2:

[1038] The server stores the received text data in a database and starts natural language processing (NLP). The input for this process is the text data provided by the user. In this step, the following specific data processing and calculations are performed:

[1039] Tokenization: Breaking text into words and phrases.

[1040] Part-of-speech tagging: attaching a grammatical category to each word.

[1041] Syntactic analysis: Analyzing the grammatical structure of a sentence.

[1042] The output is an analysis of keywords and grammatical structures. The software used is Google Cloud Natural Language API and SpaCy.

[1043] Step 3:

[1044] Based on the analysis results, the server generates an image using a generative AI model (e.g., DALL-E or Stable Diffusion). The input is the analysis results (keywords and grammatical structures) obtained in the previous step. Specific operations include:

[1045] Input a prompt sentence into the generative AI model.

[1046] Generates an image according to the prompt.

[1047] For example, based on the keywords "Meiji Restoration" and "Arrival of the Black Ships," an image that visually represents these is generated. The prompt used is "Generate an image illustrating the arrival of the black ships in Japan during the Meiji Restoration." The output is an image generated to visually represent the content of the text.

[1048] Step 4:

[1049] The generated images are arranged in a sequence by the server to create a picture-storyboard. The input to this step is the sequence of images generated in step 3 and their accompanying explanatory text. Specific operations include:

[1050] Determine the order of the images.

[1051] Add a description for each image.

[1052] Build a storyboard by integrating images and descriptions.

[1053] The output is a visually organized storyboard in the style of a picture story.

[1054] Step 5:

[1055] The completed Kamishibai is saved in digital format and provided to the user. The input for this step is the completed storyboard. Specific actions include:

[1056] Upload your storyboard online.

[1057] A link is generated and notified to the user.

[1058] If necessary, convert them into PDF or slide format and make them downloadable.

[1059] The output is a digital Kamishibai that is accessible to users.

[1060] Step 6:

[1061] While the user is viewing the Kamishibai, the system uses the emotion engine to recognize the user's emotions in real time. The input for this step is the user's facial expressions and voice data. The specific operation is as follows:

[1062] Data is collected in real time via webcam and microphone.

[1063] Analyze emotions using facial recognition technology (e.g., OpenCV) and voice analysis technology (e.g., Google Speech-to-Text API).

[1064] The output is data indicating the user's emotional state (e.g., joy, confusion, interest, etc.).

[1065] Step 7:

[1066] Based on the recognized user's emotional state, the server dynamically adjusts the content of the Kamishibai. The input for this step is the user's emotional data. Specific operations include:

[1067] Highlight certain sections depending on your emotional state.

[1068] Insert additional explanations and illustrations as needed.

[1069] For example, if the user is confused, more detailed images and explanations are added. The output is a picture-story show that is dynamically adjusted according to the user's emotions.

[1070] (Application example 2)

[1071] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1072] Conventional text data visualization systems simply provide information unilaterally, without considering the user's emotions or level of understanding. This often leaves users confused or disinterested, resulting in insufficient understanding of the information. Furthermore, the information provided visually is limited, and particularly in education and content distribution services, it is necessary to flexibly adjust the content based on the user's reaction.

[1073] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1074] In this invention, the server includes means for receiving and saving text data, means for natural language processing the text data, means for generating images based on the analysis results, means for arranging the generated images in order to create a storyboard in the form of a picture-story show, means for providing the picture-story show to the user, means for recognizing the user's emotions in real time, and means for adjusting the content based on the user's emotions. This makes it possible to dynamically adjust the provision of visual information according to the user's emotions and achieve a greater effect in promoting understanding.

[1075] The "means for receiving and saving text data" is a function for sending text data entered by the user to a server and saving it in a database.

[1076] "Means for natural language processing of text data" refers to a function for analyzing text data through a series of processes including tokenization, part-of-speech tagging, and syntactic analysis.

[1077] "Means for generating images based on analysis results" refers to a function that generates appropriate images using a machine learning engine such as a generative AI model based on the results of natural language processing.

[1078] "Means for arranging generated images in order and creating a storyboard in the form of a picture-story show" is a function for arranging multiple generated images in order as a storyboard and displaying them in the form of a picture-story show.

[1079] The "means for providing a picture story to a user" is a function for providing a picture story created in electronic form so that the user can view it.

[1080] "Means for recognizing user emotions in real time" refers to a function that uses sensors such as cameras and microphones to analyze emotions from the user's facial expressions and voice and recognize them in real time.

[1081] The "means for adjusting content based on user emotions" is a function that dynamically changes the content of the picture book based on the recognized user emotions, and provides additional explanations or detailed illustrations as needed.

[1082] The present invention is a system for converting text data into a visually easy-to-understand picture-story format, and includes a function for recognizing a user's emotions in real time and adjusting the content accordingly. Specific embodiments of the system are described below.

[1083] System configuration

[1084] The system consists of hardware such as a user terminal, server, camera, and microphone, as well as software components including a natural language processing engine, an image generation engine, and an emotion recognition engine.

[1085] Entering and saving text data

[1086] Users use their own devices to input text data into the system's interface, which is provided as an input field on a web form or a smartphone application. The input text data is sent to the server and stored in a database.

[1087] Sentence analysis

[1088] The server then performs natural language processing on the stored text data. It uses software such as spaCy, a natural language processing library, to tokenize the text data, tag it with parts of speech, and perform syntactic analysis. This process extracts keywords and themes from the text.

[1089] Image generation

[1090] Based on the analyzed keywords and themes, the server generates corresponding images using an image generation engine. Generative AI models used here include "DALL-E" and "Stable Diffusion." These models generate appropriate images based on the input prompt.

[1091] Creating a storyboard in the style of a picture story

[1092] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard, with each image accompanied by a brief description summarizing the main points of the analyzed text.

[1093] Provision to users

[1094] The completed storyboard in picture-storytelling format is saved in digital format and provided to users, who can view it on their own devices and download it as a PDF or slides if desired.

[1095] Emotion recognition and content regulation

[1096] While the user is viewing the Kamishibai, the server uses a camera and microphone to recognize the user's emotions in real time. The emotion recognition engine used uses the "Transformers" emotion recognition model. Based on the recognized emotions, the server dynamically adjusts the content of the Kamishibai and provides additional explanations and detailed illustrations to facilitate understanding.

[1097] Examples and prompts

[1098] For example, if a user inputs "historical text," keywords such as "Meiji Restoration," "famous revolutionary," and "background of arrival" are extracted, and DALL-E generates corresponding images. The generated images are arranged in order and provided to the user. If the user looks confused, a detailed explanation such as "The important background here is..." is added.

[1099] Example prompt sentence:

[1100] Meiji Restoration

[1101] Famous revolutionaries

[1102] Background to the arrival

[1103] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1104] Step 1:

[1105] The user inputs text data into the interface from their own device. This input data is sent to the server. The user can input, for example, educational texts or written works. This data is stored in the server's database.

[1106] Input: Text data entered by the user

[1107] Output: Text data sent to the server

[1108] Step 2:

[1109] The server then performs natural language processing on the stored text data. Specifically, it uses the natural language processing library "spaCy" to tokenize the text data, tag it with parts of speech, and perform syntactic analysis. This process extracts keywords and themes from the text.

[1110] Input: Saved text data

[1111] Output: Extracted keywords and themes

[1112] Step 3:

[1113] The server generates an image based on the analysis results. This process uses the generative AI model "DALL-E" or "Stable Diffusion." The server inputs the extracted keywords and themes into the generative AI model as prompts to generate an appropriate image.

[1114] Input: Extracted keywords and themes

[1115] Output: The generated image

[1116] Step 4:

[1117] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard, with each image accompanied by a brief description summarizing the main points of the analyzed text.

[1118] Input: Generated image

[1119] Output: Kamishibai style storyboard

[1120] Step 5:

[1121] The server then provides the completed storyboard in digital format to the user's device, where the user can view the story by clicking on a link.

[1122] Input: Kamishibai storyboard

[1123] Output: A digital picture story that can be viewed by users

[1124] Step 6:

[1125] While the user is viewing the picture story, the server uses a camera and microphone to recognize emotions in real time. This process uses the "Transformers" emotion recognition model. The recognized emotion data is sent to the server.

[1126] Input: User's facial expressions and voice data

[1127] Output: Recognized emotion data

[1128] Step 7:

[1129] The server dynamically adjusts the content of the picture story based on the recognized emotion data, for example, providing additional explanations or detailed illustrations if the user is confused.

[1130] Input: Recognized emotion data

[1131] Output: Adjusted Kamishibai content

[1132] This system not only provides users with information in a visually easy-to-understand format, but also allows them to receive content that is optimal for their emotions.

[1133] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1134] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1135] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1136] [Fourth embodiment]

[1137] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1138] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1139] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1140] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1141] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1142] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1143] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1144] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1145] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1146] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1147] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1148] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1149] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1150] This invention relates to a system that provides long or complex text in the form of a picture-story show that is easy for users to understand. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a storyboard in the form of a picture-story show, and finally provides it to the user.

[1151] How the system works

[1152] Receiving text data

[1153] A user inputs long text or text data through the system's interface, for example by copying and pasting text into a web form or application input field, or by typing directly into the input field. This input text data is sent to a server for initial processing.

[1154] Sentence analysis

[1155] The server stores the received text data and begins natural language processing. This process involves several steps. First, the text is tokenized, which is the process of dividing the text into the smallest units, such as words and phrases. Part-of-speech tagging is then performed, and each token is assigned a grammatical role. Next, syntactic analysis is performed to reveal the grammatical structure of the text. This allows the meaning of the text and the relationships between its parts to be analyzed.

[1156] Image generation

[1157] Based on the analysis results, the server generates an image. This process is carried out using a machine learning model. For example, DALL-E or Stable Diffusion can be used as a generative model. These models generate images with appropriate visual representations based on the content of the input text. For example, they can generate images such as "The Black Ships Arriving in Japan" or "Portrait of Sakamoto Ryoma."

[1158] Building a Kamishibai Frame

[1159] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard. This storyboard also includes a description for each image. The description is a concise summary of the main points of the analyzed text, making it easier for users to understand the content while looking at the images.

[1160] Provision to users

[1161] The completed Kamishibai is saved in digital format and provided to the user. The user can view the Kamishibai on their own device (PC or tablet). For example, by clicking on a link generated by the system, the Kamishibai can be viewed online. Furthermore, if necessary, the Kamishibai can be downloaded in PDF or slide format.

[1162] Specific examples

[1163] Example 1: History text

[1164] The user inputs a "text about Japan's Meiji Restoration" into the system. The server analyzes the text and identifies keywords such as "Meiji Restoration," "Sakamoto Ryoma," "arrival of the black ships," and "civilization and enlightenment." Based on these keywords, images of historical events and people are generated. For example, images such as "the arrival of the black ships in Japan," "scenes of Sakamoto Ryoma in action," and "the changing landscape of Japan with civilization and enlightenment" are created and presented in the form of a picture-story show with accompanying explanatory text.

[1165] Example 2: Scientific explanation

[1166] The user inputs a sentence about the mechanism of photosynthesis into the system. The server analyzes the sentence and identifies keywords such as "photosynthesis," "chlorophyll," "oxygen production," and "carbon dioxide absorption." Based on these keywords, it generates images that explain the photosynthetic process of plants. For example, it creates images of "how light hits a plant," "chemical reactions occurring inside a leaf," and "the scene where oxygen is released," and provides them in the form of a picture-story show with accompanying explanatory text.

[1167] The purpose of this system is to make it easier for users to visually understand long or complex sentences, thereby contributing to promoting comprehension and reducing stress.

[1168] The processing flow will be explained below.

[1169] Ok, I'll explain each process step in the format shown below.

[1170] Step 1:

[1171] A user accesses the system's interface and inputs long text or text data, for example by copying and pasting text into a web form or by typing it directly. At this point, the input data is sent to the server.

[1172] Step 2:

[1173] The server stores the received text data in a database. This initial storage is necessary for later analysis.

[1174] Step 3:

[1175] The server passes the stored text data to a natural language processing (NLP) library to begin analysis, which includes tokenization (dividing the text into words and phrases), part-of-speech tagging (assigning grammatical categories to each word), and syntactic analysis (analyzing the grammatical structure of the text).

[1176] Step 4:

[1177] The server uses the results of natural language processing to identify key sections of the text and important keywords. For example, if the text is about history, it might extract terms such as "Meiji Restoration," "Sakamoto Ryoma," and "Arrival of the Black Ships."

[1178] Step 5:

[1179] The server then generates an image for each section based on the analysis results. This process is done using machine learning models (e.g., DALL-E or Stable Diffusion), where you input the text content into the model and it generates an image based on that.

[1180] Step 6:

[1181] The server arranges the generated images in a sequence to create a picture-storyboard in the style of a picture story show. The order of the images is designed to follow the flow of the story in the text.

[1182] Step 7:

[1183] The server generates a description for each image, which is a brief summary of the main points of the text, to help users understand the content of the image when they view it.

[1184] Step 8:

[1185] The server stores the completed storyboard in digital format and generates a link to provide it to the user, which is a URL that the user can access to view the storyboard online.

[1186] Step 9:

[1187] Users can use their devices (PCs or tablets) to access the link provided by the server and watch the Kamishibai. The content is presented visually in an easy-to-understand way, allowing users to efficiently understand the content of the text.

[1188] Step 10:

[1189] Users can also download the Kamishibai as a PDF or slides if desired, which is convenient for offline viewing or presentations.

[1190] The above is a description of the specific processing steps of this system.

[1191] Example 1

[1192] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1193] There is a demand for a method for visualizing long or complex sentences in an efficient and easy-to-understand manner. The present invention aims to provide a system that presents text data entered by a user as a picture-storyboard format that is visually easy to understand.

[1194] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1195] In this invention, the server includes means for a user to input text data and send it through a system interface, means for the server to store the received text data and start natural language processing, means for the server to tokenize, tag parts of speech, and syntactically parse the text data, means for the server to generate prompts based on the analysis results and use them to generate visual representations using an image generation model, means for arranging the generated images in order to create a picture-storyboard in the style of a picture-story show, and means for the user to view, save, or download the storyboard in digital form. This makes it possible to provide complex text data to users in a visually easy-to-understand format.

[1196] "Means for inputting text data and transmitting it through a system interface" refers to a means by which a user can input long text or sentences and transmit them to a server via a web form or application.

[1197] The "means for saving received text data and starting natural language processing" refers to the means by which the server saves text data sent from the user and automatically starts the natural language processing process.

[1198] "Means for tokenizing, tagging parts of speech, and parsing text data" refers to the means by which the server divides text data into its smallest units, assigns grammatical roles to each word, and analyzes the grammatical structure and meaning of the entire text.

[1199] "Means for generating prompts based on the analysis results and using them to generate visual representations using an image generation model" refers to a means by which the server creates prompt sentences based on the results of the analyzed sentences, inputs them into a generative AI model, and generates appropriate images.

[1200] "Means for arranging generated images in a logical order and creating a picture-storyboard in the style of a picture-story show" refers to means by which the server arranges generated images in a logical order and adds explanatory text to each image to create a picture-storyboard in the style of a picture-story show.

[1201] "Means for users to view, save, or download storyboards in digital format" refers to means for users to view, digitally save, or download completed storyboards in picture-story show format on a device such as a computer or tablet.

[1202] This invention relates to a system that provides long or complex text in the form of a picture-story show that is easy for users to understand. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a storyboard in the form of a picture-story show, and finally provides it to the user.

[1203] How the system works

[1204] Receiving text data

[1205] A user inputs text data through the system interface, for example, by copying and pasting long sentences or text into a web form or application input field, or by typing and submitting the text data. This input text data is sent to the server for initial processing.

[1206] Sentence analysis

[1207] The server stores the received text data and starts natural language processing. Specifically, it performs the following steps:

[1208] 1. Tokenization: This is the process of breaking down the received text into its smallest units, such as words and phrases. This process is done using a language analysis engine or natural language processing library (e.g., SpaCy, NLTK).

[1209] 2. Part-of-speech tagging: This is the process of assigning a grammatical role to each token. For example, tagging the token "Sakamoto Ryoma" as a "noun."

[1210] 3. Syntactic analysis: This is the process of analyzing a sentence for its grammatical structure and meaning. This clarifies the relationships between tokens and the overall meaning of the sentence. Syntactic analysis is often performed using tree-based parsing algorithms.

[1211] Image generation

[1212] Based on the analysis results, the server generates a visual representation using an image generation model. This process involves the following steps:

[1213] 1. Take the identified keywords and phrases and turn them into prompts. For example, create a prompt like, "Generate an image of the Black Ships arriving in Japan."

[1214] 2. The created prompt sentence is input into a generative AI model (e.g., DALL-E, Stable Diffusion) to generate an appropriate image. The generated image is a visual representation based on the content of the sentence.

[1215] Building a Kamishibai Frame

[1216] The server arranges the generated images in a sequence to create a picture-storyboard, specifically:

[1217] 1. Arrange the images in a logical order. For example, arrange them in the order "Arrival of the Black Ships," "Sakamoto Ryoma in Action," and "Scenes of Westernization."

[1218] 2. A description is generated for each image. This description is a concise summary of the main points of the analyzed text. For example, it can be a specific description such as "This image shows the arrival of the Black Ships in Japan."

[1219] Provision to users

[1220] The completed Kamishibai is saved in digital format and provided to users. The specific steps are as follows:

[1221] 1. Users can view the Kamishibai on their own devices (PCs or tablets). By clicking on the link generated by the system, the Kamishibai will be displayed online.

[1222] 2. If necessary, users can download the Kamishibai in PDF or slide format. For example, by clicking the download button, the file will be saved to their device.

[1223] Specific examples

[1224] Example 1: History text

[1225] The user inputs a "text about Japan's modernization" into the system. The server analyzes the text and identifies keywords such as "modernization," "historical events," "industrial revolution," and "social change." Based on these keywords, images of historical events and people are generated. For example, images such as "a factory symbolic of modernization," "famous historical leaders," and "scenes of the industrial revolution" are created and presented in the form of a picture-story show with accompanying explanatory text.

[1226] Example prompt sentence:

[1227] Generate an image of an "Industrial Revolution factory scene."

[1228] Example 2: Scientific explanation

[1229] The user inputs a "text about the workings of ecosystems" into the system. The server analyzes the text and identifies keywords such as "ecosystem," "ecosystem," "biological interactions," and "natural environment." Based on these keywords, an image of the ecosystem is generated. For example, images such as "interactions between organisms," "changes in the natural environment," and "survival strategies of organisms" are created and presented in the form of a picture-story show with accompanying explanatory text.

[1230] Example prompt sentence:

[1231] Generate images of "scenes of interactions between organisms."

[1232] The system aims to make it easier for users to visually understand long or complex sentences, thereby contributing to promoting comprehension and reducing stress.

[1233] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1234] Step 1: Enter and send text data

[1235] Users input and submit text data through the system interface, either by copying and pasting text into a web form or by typing it directly into an application input field. The input text data is then sent to the server.

[1236] Input: Text data entered by the user

[1237] Output: Raw text data sent to the server

[1238] Step 2: Saving and initial processing of text data

[1239] The server saves the received text data and starts initial processing. Specifically, it performs validation checks on the text data (e.g., removes unsupported characters) and saves it in a database (e.g., MySQL, PostgreSQL).

[1240] Input: Submitted text data

[1241] Output: Validated text data, data stored in a database

[1242] Step 3: Tokenization

[1243] The server tokenizes the stored text data, specifically by using a language analysis engine (e.g., SpaCy, NLTK) to split the text into words and phrases.

[1244] Input: Saved text data

[1245] Output: Tokenized word list

[1246] Step 4: Part-of-Speech Tagging

[1247] The server then tags the tokenized word list with part-of-speech tags, specifically assigning each word a grammatical role (e.g., noun, verb, adjective, etc.).

[1248] Input: tokenized word list

[1249] Output: A list of words with part-of-speech tags

[1250] Step 5: Parsing

[1251] The server uses the tagged word list to perform a syntactic analysis of the entire sentence, specifically by clarifying the relationships between words and analyzing the grammatical structure of the sentence (e.g., dependency analysis).

[1252] Input: A list of words with part-of-speech tags

[1253] Output: Parsed grammar structure data

[1254] Step 6: Prompt Generation

[1255] The server generates a prompt based on the results of the parsing, such as "Please describe the scene when the Black Ships arrive in Japan."

[1256] Input: Parsed grammar structure data

[1257] Output: prompt statement

[1258] Step 7: Image generation

[1259] The server uses the generated prompt sentence to input into an image generation model (e.g., DALL-E, Stable Diffusion) to generate a visual representation. Specifically, the server inputs the prompt sentence into the generative AI model and obtains the returned image.

[1260] Input: prompt statement

[1261] Output: Generated image data

[1262] Step 8: Build a storyboard

[1263] The server arranges the generated images in a sequence to create a picture-storyboard, and also generates explanatory text for each image.

[1264] Input: Generated image data

[1265] Output: Kamishibai style storyboard

[1266] Step 9: Provide to users

[1267] The server stores the completed Kamishibai in digital format and provides it to users by generating a link that allows them to view it online and download it as a PDF or slides if desired.

[1268] Input: Kamishibai-style storyboard

[1269] Output: A link or file that users can view and download

[1270] (Application example 1)

[1271] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1272] Today's consumers expect detailed product descriptions in online and virtual stores, but text alone is not enough to adequately convey the benefits and usage scenarios of products and services. Therefore, there is a need for a method to provide product descriptions in a visually easy-to-understand format. Furthermore, when users receive product information in long or complex sentences, it can take time to understand the content, which can be frustrating. To solve these problems, a system is needed that provides product descriptions in a picture-story show format.

[1273] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1274] In this invention, the server includes means for receiving and saving text data, means for natural language processing the text data, means for generating images based on the analysis results, means for arranging the generated images in order to create a storyboard in the form of a picture-story show, means for providing the picture-story show to the user, and means for visually providing product descriptions in the virtual store, which makes it easier for the user to visually understand the details of products and services, and enables effective product descriptions in the virtual store.

[1275] "Text data" refers to data of long or complex texts entered by the user.

[1276] The "means for saving" is a means having a function for saving input text data on a server or the like.

[1277] "Means for natural language processing" refers to means for processing text data such as tokenization, part-of-speech tagging, and syntactic analysis to analyze the meaning and structure of text.

[1278] "Generating means" refers to means that has the function of generating images or other visual content based on the analysis results.

[1279] The "means for arranging in order" is a means for arranging generated images and content in order in a storyboard format.

[1280] A "picture-storyboard" is a storyboard format that makes it visually easy to understand by displaying each image in order with corresponding explanatory text.

[1281] The "means of providing" is a means of ultimately providing the user with a picture-storyboard in a viewable format.

[1282] A "virtual store" is a virtual store that operates on the Internet, and refers to an environment where users can browse and purchase products online.

[1283] In this invention, the server is a system that receives and stores text data, analyzes it, generates images, and provides them to users in the form of a picture-story show. A specific embodiment of this system is described below.

[1284] First, a user inputs text data using a terminal. This text data is sent to a server through a web form or application input field. The server stores the received text data. The stored text data is then analyzed using natural language processing. This processing includes steps such as tokenization, part-of-speech tagging, and syntactic analysis. This allows the meaning and grammatical structure of the text to be revealed.

[1285] Based on the analysis results, the server uses a generative AI model to generate appropriate images. Examples of AI models used include DALL-E and Stable Diffusion. These models generate images that visually represent the content of the input text. The generated images are arranged in a sequence to create a picture-storyboard in the style of a picture-story show. This storyboard also includes explanatory text for each image, making it easier for users to visually understand the content.

[1286] Finally, users can view the Kamishibai on their own devices. They can do so by using a link generated by the system or by downloading it in PDF or slide format. This allows users to visually understand the product's features and usage scenarios in Kamishibai format.

[1287] As a specific example, if a user inputs the text data, "This product is a smartphone app that uses the latest AI technology. Its main functions are voice recognition and image recognition," the server analyzes this text and identifies keywords such as "the latest AI technology," "smartphone app," "voice recognition," and "image recognition." Based on this, it generates images such as futuristic technology images, smartphone screens, voice input images, and image analysis images, and incorporates these into a storyboard for presentation.

[1288] An example of a prompt is, "Create a presentation for a smartphone app using the latest AI technology. Generate images and explanatory text in a picture-story format that visually express specific functions and usage scenarios."

[1289] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1290] Step 1:

[1291] A user uses a terminal to input text data, which is then sent to a server via a web form or application input field.

[1292] Input: Text data entered by the user

[1293] Output: Text data sent to the server

[1294] Step 2:

[1295] The server stores the received text data, which is used for subsequent processing.

[1296] Input: Text data sent to the server

[1297] Output: Text data stored on the server

[1298] Step 3:

[1299] The server analyzes the stored text data using natural language processing techniques, which include steps such as tokenization, part-of-speech tagging, and syntactic analysis, to reveal the meaning and grammatical structure of the text.

[1300] Input: Text data stored on the server

[1301] Data processing: sentence tokenization, part-of-speech tagging, syntactic analysis

[1302] Output: Analysis results (tokens, part-of-speech tags, syntactic analysis results)

[1303] Step 4:

[1304] Based on the analysis results, the server uses a generative AI model, such as DALL-E or Stable Diffusion, to generate an appropriate image.

[1305] Input: Analysis results after natural language processing (keywords, context, etc.)

[1306] Data calculation: Image generation using generative AI models (e.g., DALL-E, Stable Diffusion)

[1307] Output: The generated image

[1308] Step 5:

[1309] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard, with each image accompanied by a description that summarizes the main points of the analysis results.

[1310] Input: Generated images, summary of analysis results

[1311] Data processing: Arranging images and creating descriptions

[1312] Output: Kamishibai style storyboard

[1313] Step 6:

[1314] The server generates a link to provide the completed Kamishibai to the user, who can then use this link to view the Kamishibai on their device. If desired, the Kamishibai can also be downloaded as a PDF or slides.

[1315] Input: Kamishibai-style storyboard

[1316] Data processing: Link generation, data conversion to PDF and slide format

[1317] Output: A link to the Kamishibai sent to the user and the downloadable data

[1318] As a concrete example, if a user inputs the text data "This product is a smartphone app that uses the latest AI technology. Its main functions are voice recognition and image recognition," the server analyzes this text and identifies keywords such as "the latest AI technology," "smartphone app," "voice recognition," and "image recognition." Based on this, it generates images such as futuristic technology images, smartphone screens, voice input images, and image analysis images, and incorporates these into a storyboard for presentation.

[1319] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1320] This invention combines a system that converts text data into a visually easy-to-understand picture-story show format with an emotion engine that recognizes the user's emotions. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a picture-story show format storyboard, and finally provides it to the user. In addition, it has the ability to recognize the user's emotions in real time and adjust the content based on those emotions.

[1321] How the system works

[1322] Receiving text data

[1323] A user inputs long text or text data through the system interface, for example by copying and pasting text into a web form or application input field, or by typing directly into the input field. This input text data is then sent to the server.

[1324] Sentence analysis

[1325] The server stores the received text data in a database and begins natural language processing, which includes tokenization (dividing the text into words and phrases), part-of-speech tagging (assigning grammatical categories to each word), and syntactic analysis (analyzing the grammatical structure of the text).

[1326] Image generation

[1327] Based on the analysis, the server generates an image for each section, using machine learning models (e.g., DALL-E and Stable Diffusion) to generate an appropriate image that visually represents the content of the text.

[1328] Building a Kamishibai Frame

[1329] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard. This storyboard also includes a description for each image. The description is a concise summary of the main points of the analyzed text, making it easier for users to understand the content while looking at the images.

[1330] Provision to users

[1331] The completed Kamishibai is saved in digital format and provided to the user. The user can view the Kamishibai on their own device (PC or tablet). For example, by clicking on a link generated by the system, the Kamishibai can be viewed online. Furthermore, if necessary, the Kamishibai can be downloaded in PDF or slide format.

[1332] Emotional engine regulation

[1333] emotion recognition

[1334] While the user is viewing the picture story, the system uses an emotion engine to recognize the user's emotions in real time. This emotion engine analyzes the user's emotional state using facial recognition and voice analysis technologies. For example, it can determine emotions such as joy, anger, sadness, and happiness based on data obtained from the user's webcam and microphone.

[1335] Content adjustment

[1336] The server dynamically adjusts the content of the picture story based on the user's recognized emotional state, highlighting sections that interest the user or providing additional explanations to deepen understanding. For example, if the user shows a confused expression, the system adds images and text that explain the content of that section in detail.

[1337] Specific examples

[1338] Example 1: History text

[1339] The user inputs a text about Japan's Meiji Restoration into the system. The server analyzes the text and identifies keywords such as "Meiji Restoration," "Sakamoto Ryoma," "arrival of the Black Ships," and "civilization and enlightenment." Based on these keywords, images of historical events and people are generated. For example, images such as "the arrival of the Black Ships in Japan," "scenes of Sakamoto Ryoma's exploits," and "the changing landscape of Japan as a result of civilization and enlightenment" are created and presented in the form of a picture-story show with accompanying explanatory text. Furthermore, specific background information and illustrations can be added to sections that the user finds interesting, deepening their understanding.

[1340] Example 2: Scientific explanation

[1341] The user inputs a sentence about the mechanism of photosynthesis into the system. The server analyzes the sentence and identifies keywords such as "photosynthesis," "chlorophyll," "oxygen production," and "carbon dioxide absorption." Based on these keywords, it generates images that explain the photosynthetic process in plants. For example, it creates images such as "how light hits a plant," "chemical reactions occurring inside a leaf," and "the scene where oxygen is released," and provides them in the form of a picture-story show with accompanying explanatory text. In addition, it can recognize the user's emotions and, if they are confused, provide more detailed illustrations or additional explanations to help them understand.

[1342] In this way, this system not only makes the content of a sentence easier to understand visually, but also achieves a greater effect of promoting comprehension by dynamically adjusting according to the user's emotions.

[1343] The processing flow will be explained below.

[1344] Ok, I'll explain each process step in the format shown below.

[1345] Step 1:

[1346] A user accesses the system's interface and inputs long or textual data, for example by copying and pasting text into a web form or by typing it directly. This input data is then sent to the server.

[1347] Step 2:

[1348] The server stores the received text data in a database. This initial storage is necessary for later analysis.

[1349] Step 3:

[1350] The server then passes the stored text data to a natural language processing (NLP) library to begin parsing it: tokenization, part-of-speech tagging, and syntactic analysis are performed to identify key points and sections of the text.

[1351] Step 4:

[1352] The server extracts key sections and important keywords from the analysis results, identifying keywords such as "Meiji Restoration," "Sakamoto Ryoma," and "Arrival of the Black Ships."

[1353] Step 5:

[1354] The server generates images for each section based on the extracted keywords, using machine learning models (e.g., DALL-E and Stable Diffusion), generating images such as "The Black Ships Arriving in Japan" or "Portrait of Ryoma Sakamoto."

[1355] Step 6:

[1356] The server arranges the generated images in a sequence to create a picture-storyboard in the style of a picture story show. The order of the images is designed to follow the flow of the story in the text.

[1357] Step 7:

[1358] The server generates a description for each image and adds it to the storyboard. The description is a concise summary of the main points of the text, making it easier for users to understand the content while viewing the image.

[1359] Step 8:

[1360] The server stores the completed storyboard in digital format and generates a link to provide it to the user, which is a URL that the user can access to view the storyboard online.

[1361] Step 9:

[1362] The server runs an emotion engine and provides a means to recognize the user's emotional state in real time, for example by capturing the user's facial expressions and voice via a webcam or microphone and running the emotion analysis algorithm on them.

[1363] Step 10:

[1364] The emotion engine analyzes the user's facial expressions and voice to recognize their emotional state in real time, for example, detecting when they are confused or interested.

[1365] Step 11:

[1366] The server dynamically adjusts the content of the picture story based on the perceived emotional state, highlighting interesting sections and providing additional explanations to enhance understanding—for example, adding more detailed explanatory images or text if the user is confused.

[1367] Step 12:

[1368] Users can use their devices (PCs or tablets) to access the link provided by the server and watch the picture story. The content adjusted by the emotion engine is reflected in real time, and the content is presented visually in an easy-to-understand manner.

[1369] Step 13:

[1370] Users can also download the Kamishibai as a PDF or slides if desired, which is convenient for offline viewing or presentations.

[1371] The above is a description of the specific processing steps of this system.

[1372] Example 2

[1373] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1374] Conventional systems for promoting visual comprehension of text data have the problem of being unable to respond to changes in the user's emotions during processes such as text analysis and image generation. As a result, there is concern that the user's understanding and concentration may decline, resulting in insufficient learning effectiveness. The present invention proposes a system equipped with emotion recognition functionality to solve this problem and deepen the user's understanding.

[1375] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1376] In this invention, the server includes means for receiving and storing text data, means for natural language processing the text data, means for generating images based on the analysis results, means for arranging the generated images in order to create a storyboard in the form of a picture-story show, means for providing the picture-story show to the user, means for recognizing the user's emotions in real time, and means for dynamically adjusting the content of the picture-story show based on the recognized emotions.

[1377] This makes it possible to dynamically adjust the learning content according to the user's emotional state, thereby enhancing the effectiveness of promoting understanding.

[1378] "Text data" refers to all text information entered by the user, and includes the contents of any written document, from short sentences to long sentences.

[1379] "Natural language processing" refers to the technology that enables computers to understand and analyze human language, and includes the processes of tokenizing, part-of-speech tagging, and syntactic analysis of text.

[1380] "Image generation" refers to the process of creating appropriate visual elements based on analytical results, and is done using generative AI models.

[1381] A "generative AI model" refers to an algorithm or system that uses artificial intelligence to generate new data, specifically using computer vision technology.

[1382] "Kamishibai format" refers to a technique in which multiple images are arranged in an orderly manner and displayed consecutively with corresponding explanatory text attached to each.

[1383] A "storyboard" is a visually structured format that combines a series of scenes or slides to effectively convey a story or information.

[1384] "User" refers to the entity that uses the System to input text data and receive the analysis results and generated content.

[1385] "Emotion recognition" refers to the technology of determining a user's emotional state from their facial expressions, voice, etc., and includes the process of detecting the user's psychological state.

[1386] "Dynamic adjustment" refers to technology that changes content and settings in real time based on the situation or condition, automatically adapting based on user responses.

[1387] This invention is a system that converts text data into a visually easy-to-understand picture-story show format, and also combines it with an emotion engine that recognizes the user's emotions. This system receives text data, performs natural language processing, generates images based on the analysis results, creates a picture-story show format storyboard, and finally provides it to the user. In addition, it has the function of recognizing the user's emotions in real time and adjusting the content based on those emotions.

[1388] Receiving text data

[1389] Users input long text or text data through the system interface, for example by copying and pasting text into a web form or application input field, or by typing it directly. This input text data is then sent from the terminal to the server via the Internet.

[1390] Sentence analysis

[1391] The server stores the received text data in a database and begins natural language processing (NLP). This process includes tokenization (dividing the text into words and phrases), part-of-speech tagging (assigning grammatical categories to each word), and syntactic analysis (analyzing the grammatical structure of the text). Software such as Google Cloud Natural Language API and SpaCy are used for this. For example, in a historical text, keywords such as "Meiji Restoration," "Sakamoto Ryoma," and "Arrival of the Black Ships" are identified.

[1392] Image generation

[1393] Based on the analysis results, the server uses a generative AI model (e.g., DALL-E or Stable Diffusion) to generate images corresponding to each section. This generates an appropriate image that visually represents the content of the text. For example, an image that explains "the arrival of the Black Ships in Japan" is generated. The following prompt sentences are used:

[1394] Prompt statement:

[1395] "Generate an image illustrating the arrival of the black ships in Japan during the Meiji Restoration. The image should depict the historical context and importance of this event."

[1396] Building a Kamishibai Frame

[1397] The generated images are arranged in order by the server to create a storyboard in the style of a picture-story show. Each image is also given a description. For example, a description such as "This image depicts the arrival of the Black Ships in Japan in 1853" could be added.

[1398] Provision to users

[1399] The completed Kamishibai is saved in digital format and provided to the user. Users can view the Kamishibai on their own devices (PC or tablet) or view it online by clicking on a link generated by the system. Furthermore, if desired, users can download the Kamishibai in PDF or slide format.

[1400] emotion recognition

[1401] While the user is viewing the Kamishibai, the system uses an emotion engine to recognize the user's emotions in real time. The emotion engine analyzes the user's emotional state using facial recognition technology (e.g., OpenCV) and voice analysis technology (e.g., Google Speech-to-Text API). For example, if the user shows a confused expression, the system will determine this.

[1402] Content adjustment

[1403] The server dynamically adjusts the content of the picture story based on the user's recognized emotional state, highlighting sections that interest the user or providing additional explanations to deepen understanding. For example, if the user is confused, it adds images or text that explain the content in more detail.

[1404] As described above, this system not only makes text easier to understand visually, but also enhances learning effectiveness by dynamically adjusting according to the user's emotions.

[1405] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1406] Step 1:

[1407] The user inputs text data using the system's interface. They copy and paste long sentences or text data into the input fields of a web form or application, or type them directly. Once the input is complete, the device sends it to the server. The input is the text data created or collected by the user, and the output is the secure transmission of the text data to the server.

[1408] Step 2:

[1409] The server stores the received text data in a database and starts natural language processing (NLP). The input for this process is the text data provided by the user. In this step, the following specific data processing and calculations are performed:

[1410] Tokenization: Breaking text into words and phrases.

[1411] Part-of-speech tagging: attaching a grammatical category to each word.

[1412] Syntactic analysis: Analyzing the grammatical structure of a sentence.

[1413] The output is an analysis of keywords and grammatical structures. The software used is Google Cloud Natural Language API and SpaCy.

[1414] Step 3:

[1415] Based on the analysis results, the server generates an image using a generative AI model (e.g., DALL-E or Stable Diffusion). The input is the analysis results (keywords and grammatical structures) obtained in the previous step. Specific operations include:

[1416] Input a prompt sentence into the generative AI model.

[1417] Generates an image according to the prompt.

[1418] For example, based on the keywords "Meiji Restoration" and "Arrival of the Black Ships," an image that visually represents these is generated. The prompt used is "Generate an image illustrating the arrival of the black ships in Japan during the Meiji Restoration." The output is an image generated to visually represent the content of the text.

[1419] Step 4:

[1420] The generated images are arranged in a sequence by the server to create a picture-storyboard. The input to this step is the sequence of images generated in step 3 and their accompanying explanatory text. Specific operations include:

[1421] Determine the order of the images.

[1422] Add a description for each image.

[1423] Build a storyboard by integrating images and descriptions.

[1424] The output is a visually organized storyboard in the style of a picture story.

[1425] Step 5:

[1426] The completed Kamishibai is saved in digital format and provided to the user. The input for this step is the completed storyboard. Specific actions include:

[1427] Upload your storyboard online.

[1428] A link is generated and notified to the user.

[1429] If necessary, convert them into PDF or slide format and make them downloadable.

[1430] The output is a digital Kamishibai that is accessible to users.

[1431] Step 6:

[1432] While the user is viewing the Kamishibai, the system uses the emotion engine to recognize the user's emotions in real time. The input for this step is the user's facial expressions and voice data. The specific operation is as follows:

[1433] Data is collected in real time via webcam and microphone.

[1434] Analyze emotions using facial recognition technology (e.g., OpenCV) and voice analysis technology (e.g., Google Speech-to-Text API).

[1435] The output is data indicating the user's emotional state (e.g., joy, confusion, interest, etc.).

[1436] Step 7:

[1437] Based on the recognized user's emotional state, the server dynamically adjusts the content of the Kamishibai. The input for this step is the user's emotional data. Specific operations include:

[1438] Highlight certain sections depending on your emotional state.

[1439] Insert additional explanations and illustrations as needed.

[1440] For example, if the user is confused, more detailed images and explanations are added. The output is a picture-story show that is dynamically adjusted according to the user's emotions.

[1441] (Application example 2)

[1442] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1443] Conventional text data visualization systems simply provide information unilaterally, without considering the user's emotions or level of understanding. This often leaves users confused or disinterested, resulting in insufficient understanding of the information. Furthermore, the information provided visually is limited, and particularly in education and content distribution services, it is necessary to flexibly adjust the content based on the user's reaction.

[1444] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1445] In this invention, the server includes means for receiving and saving text data, means for natural language processing the text data, means for generating images based on the analysis results, means for arranging the generated images in order to create a storyboard in the form of a picture-story show, means for providing the picture-story show to the user, means for recognizing the user's emotions in real time, and means for adjusting the content based on the user's emotions. This makes it possible to dynamically adjust the provision of visual information according to the user's emotions and achieve a greater effect in promoting understanding.

[1446] The "means for receiving and saving text data" is a function for sending text data entered by the user to a server and saving it in a database.

[1447] "Means for natural language processing of text data" refers to a function for analyzing text data through a series of processes including tokenization, part-of-speech tagging, and syntactic analysis.

[1448] "Means for generating images based on analysis results" refers to a function that generates appropriate images using a machine learning engine such as a generative AI model based on the results of natural language processing.

[1449] "Means for arranging generated images in order and creating a storyboard in the form of a picture-story show" is a function for arranging multiple generated images in order as a storyboard and displaying them in the form of a picture-story show.

[1450] The "means for providing a picture story to a user" is a function for providing a picture story created in electronic form so that the user can view it.

[1451] "Means for recognizing user emotions in real time" refers to a function that uses sensors such as cameras and microphones to analyze emotions from the user's facial expressions and voice and recognize them in real time.

[1452] The "means for adjusting content based on user emotions" is a function that dynamically changes the content of the picture book based on the recognized user emotions, and provides additional explanations or detailed illustrations as needed.

[1453] The present invention is a system for converting text data into a visually easy-to-understand picture-story format, and includes a function for recognizing a user's emotions in real time and adjusting the content accordingly. Specific embodiments of the system are described below.

[1454] System configuration

[1455] The system consists of hardware such as a user terminal, server, camera, and microphone, as well as software components including a natural language processing engine, an image generation engine, and an emotion recognition engine.

[1456] Entering and saving text data

[1457] Users use their own devices to input text data into the system's interface, which is provided as an input field on a web form or a smartphone application. The input text data is sent to the server and stored in a database.

[1458] Sentence analysis

[1459] The server then performs natural language processing on the stored text data. It uses software such as spaCy, a natural language processing library, to tokenize the text data, tag it with parts of speech, and perform syntactic analysis. This process extracts keywords and themes from the text.

[1460] Image generation

[1461] Based on the analyzed keywords and themes, the server generates corresponding images using an image generation engine. Generative AI models used here include "DALL-E" and "Stable Diffusion." These models generate appropriate images based on the input prompt.

[1462] Creating a storyboard in the style of a picture story

[1463] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard, with each image accompanied by a brief description summarizing the main points of the analyzed text.

[1464] Provision to users

[1465] The completed storyboard in picture-storytelling format is saved in digital format and provided to users, who can view it on their own devices and download it as a PDF or slides if desired.

[1466] Emotion recognition and content regulation

[1467] While the user is viewing the Kamishibai, the server uses a camera and microphone to recognize the user's emotions in real time. The emotion recognition engine used uses the "Transformers" emotion recognition model. Based on the recognized emotions, the server dynamically adjusts the content of the Kamishibai and provides additional explanations and detailed illustrations to facilitate understanding.

[1468] Examples and prompts

[1469] For example, if a user inputs "historical text," keywords such as "Meiji Restoration," "famous revolutionary," and "background of arrival" are extracted, and DALL-E generates corresponding images. The generated images are arranged in order and provided to the user. If the user looks confused, a detailed explanation such as "The important background here is..." is added.

[1470] Example prompt sentence:

[1471] Meiji Restoration

[1472] Famous revolutionaries

[1473] Background to the arrival

[1474] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1475] Step 1:

[1476] The user inputs text data into the interface from their own device. This input data is sent to the server. The user can input, for example, educational texts or written works. This data is stored in the server's database.

[1477] Input: Text data entered by the user

[1478] Output: Text data sent to the server

[1479] Step 2:

[1480] The server then performs natural language processing on the stored text data. Specifically, it uses the natural language processing library "spaCy" to tokenize the text data, tag it with parts of speech, and perform syntactic analysis. This process extracts keywords and themes from the text.

[1481] Input: Saved text data

[1482] Output: Extracted keywords and themes

[1483] Step 3:

[1484] The server generates an image based on the analysis results. This process uses the generative AI model "DALL-E" or "Stable Diffusion." The server inputs the extracted keywords and themes into the generative AI model as prompts to generate an appropriate image.

[1485] Input: Extracted keywords and themes

[1486] Output: The generated image

[1487] Step 4:

[1488] The generated images are arranged in a sequence by the server to create a picture-storyboard style storyboard, with each image accompanied by a brief description summarizing the main points of the analyzed text.

[1489] Input: Generated image

[1490] Output: Kamishibai style storyboard

[1491] Step 5:

[1492] The server then provides the completed storyboard in digital format to the user's device, where the user can view the story by clicking on a link.

[1493] Input: Kamishibai storyboard

[1494] Output: A digital picture story that can be viewed by users

[1495] Step 6:

[1496] While the user is viewing the picture story, the server uses a camera and microphone to recognize emotions in real time. This process uses the "Transformers" emotion recognition model. The recognized emotion data is sent to the server.

[1497] Input: User's facial expressions and voice data

[1498] Output: Recognized emotion data

[1499] Step 7:

[1500] The server dynamically adjusts the content of the picture story based on the recognized emotion data, for example, providing additional explanations or detailed illustrations if the user is confused.

[1501] Input: Recognized emotion data

[1502] Output: Adjusted Kamishibai content

[1503] This system not only provides users with information in a visually easy-to-understand format, but also allows them to receive content that is optimal for their emotions.

[1504] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1505] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1506] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1507] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1508] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1509] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1510] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1511] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1512] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1513] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1514] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1515] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1516] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1517] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1518] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1519] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1520] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1521] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1522] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1523] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1524] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1525] The following is further disclosed regarding the above embodiment.

[1526] Understood. Below are some suggestions for drafting patent claims that take into account the features of the system.

[1527] (Claim 1)

[1528] a means for receiving and storing text data;

[1529] A means for natural language processing text data;

[1530] means for generating an image based on the analysis results;

[1531] a means for arranging the generated images in a sequence to create a storyboard in the form of a picture-story show;

[1532] A means for providing a picture story to a user;

[1533] A system including:

[1534] (Claim 2)

[1535] 2. The system of claim 1, wherein the natural language processing means includes means for tokenizing, part-of-speech tagging, and syntactically parsing the text data.

[1536] (Claim 3)

[1537] 10. The system of claim 1, wherein the image generating means comprises means for generating the image using a machine learning model.

[1538] The above is the proposed draft of the claims.

[1539] "Example 1"

[1540] (Claim 1)

[1541] means for a user to input text data and transmit it through a system interface;

[1542] A means for the server to store the received text data and start natural language processing;

[1543] means for the server to tokenize, tag parts of speech, and parse the text data;

[1544] a means for the server to generate prompts based on the analysis results and use the prompts to generate a visual representation using an image generation model;

[1545] A means for arranging the generated images in order to create a storyboard in the form of a picture-story show;

[1546] a means for users to view, save, or download the storyboard in digital form;

[1547] A system including:

[1548] (Claim 2)

[1549] 2. The system of claim 1, wherein the natural language processing means includes means for tokenizing, part-of-speech tagging, and syntactically parsing the text data.

[1550] (Claim 3)

[1551] 10. The system of claim 1, wherein the image generating means comprises means for generating the image using a machine learning model.

[1552] "Application Example 1"

[1553] (Claim 1)

[1554] a means for receiving and storing text data;

[1555] A means for natural language processing text data;

[1556] means for generating an image based on the analysis results;

[1557] a means for arranging the generated images in a sequence to create a storyboard in the form of a picture-story show;

[1558] A means for providing a picture story to a user;

[1559] a means for visually providing product descriptions in a virtual store;

[1560] A system including:

[1561] (Claim 2)

[1562] 2. The system of claim 1, wherein the natural language processing means includes means for tokenizing, part-of-speech tagging, and syntactically parsing the text data.

[1563] (Claim 3)

[1564] 10. The system of claim 1, wherein the image generating means includes means for generating the image using a generative AI model.

[1565] "Example 2: Combining Emotion Engines"

[1566] (Claim 1)

[1567] a means for receiving and storing text data;

[1568] A means for natural language processing text data;

[1569] means for generating an image based on the analysis results;

[1570] a means for arranging the generated images in a sequence to create a storyboard in the form of a picture-story show;

[1571] A means for providing a picture story to a user;

[1572] means for recognizing user emotions in real time;

[1573] A means for dynamically adjusting the content of the picture story based on the recognized emotion;

[1574] A system including:

[1575] (Claim 2)

[1576] 2. The system of claim 1, wherein the natural language processing means includes means for tokenizing, part-of-speech tagging, and syntactically parsing the text data.

[1577] (Claim 3)

[1578] 10. The system of claim 1, wherein the image generating means comprises means for generating the image using a machine learning model.

[1579] "Application example 2 when combining emotion engines"

[1580] (Claim 1)

[1581] a means for receiving and storing text data;

[1582] A means for natural language processing text data;

[1583] means for generating an image based on the analysis results;

[1584] a means for arranging the generated images in a sequence to create a storyboard in the form of a picture-story show;

[1585] A means for providing a picture story to a user;

[1586] means for recognizing user emotions in real time;

[1587] means for adjusting content based on user sentiment;

[1588] A system including:

[1589] (Claim 2)

[1590] 2. The system of claim 1, wherein the natural language processing means includes means for tokenizing, part-of-speech tagging, and syntactically parsing the text data.

[1591] (Claim 3)

[1592] 10. The system of claim 1, wherein the image generating means comprises means for generating the image using a machine learning model. [Explanation of symbols]

[1593] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for receiving and storing text data; A means for natural language processing text data; means for generating an image based on the analysis results; a means for arranging the generated images in a sequence to create a storyboard in the form of a picture-story show; A means for providing a picture story to a user; A system including:

2. 2. The system of claim 1, wherein the natural language processing means includes means for tokenizing, part-of-speech tagging, and syntactically parsing the text data.

3. The system of claim 1 , wherein the image generating means comprises means for generating the image using a machine learning model.

Images