system

Abstract

We provide the system. [Solution] A natural language processing means that analyzes text data and divides it into nouns and punctuation units, A speech synthesis means that generates speech data in multiple languages ​​based on analyzed text data, Image generation preparation means for preparing data for image generation based on analyzed text data, An image generation means that generates images corresponding to each scene in real time based on data for image generation, A synchronized playback means that outputs audio data and generated images in sync, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the modern digital picture book market, there are few interactive products with a multilingual voice reading function and dynamically generated visual expressions. Therefore, children cannot obtain sufficient immersion when experiencing stories, and there is also a lack of educational elements. In addition, conventional digital content has limited methods for users to interactively operate, and it is difficult to achieve flexible expressions suitable for a wide range of age groups.

Means for Solving the Problems

[0005] The present invention solves the above problems by providing a system that includes a natural language processing means for analyzing text data and dividing it into nouns and punctuation units, a speech synthesis means for generating audio data in multiple languages ​​based on the analyzed text data, an image generation preparation means for preparing data for image generation based on the analyzed text data, an image generation means for generating images corresponding to each scene in real time based on the image generation data, and a synchronized playback means for outputting audio data and generated images in sync. Furthermore, by including a playback control means for controlling the playback of audio and images through user interface operations, and a text adjustment means for converting and displaying text data in easy-to-understand language according to user settings, the system provides a flexible user experience and can accommodate a wide range of age groups.

[0006] "Natural language processing methods" refer to techniques that analyze text data and divide it into nouns and punctuation marks.

[0007] "Speech synthesis means" refers to a technology that generates speech data in multiple languages ​​based on analyzed text data.

[0008] "Image generation preparation means" refers to a technique for preparing data for image generation based on analyzed text data.

[0009] "Image generation means" refers to a technology that generates images corresponding to each scene in real time based on image generation data.

[0010] "Synchronized playback means" refers to a technology that synchronizes audio data and generated images for output.

[0011] "Playback control means" refers to technology that controls the playback of audio and images through user interface operations.

[0012] "Text adjustment means" refers to a technology that converts and displays text data in a gentler language according to the user's settings. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is an interactive system that reads aloud stories or materials selected by the user and displays automatically generated images based on that content. The user accesses this system using a terminal and selects their preferred content. After selecting content, the terminal sends the selected content and associated language information to the server.

[0035] The server analyzes the selected story text using natural language processing techniques, dividing it into nouns and punctuation marks. Based on each element of the analyzed text, the server uses speech synthesis techniques to generate audio data in the user's specified language. The audio data generated at this stage is ready for use in subsequent processes.

[0036] Next, the server uses the image generation preparation means to prepare image generation data based on the analyzed text data. This data is used to define the visual elements of each scene based on nouns and important phrases. Based on this prepared data, the image generation means generates images corresponding to each scene in real time.

[0037] The device receives audio files and image generation data from the server. A synchronized playback mechanism for audio and images is activated, allowing the user to enjoy a seamless narrative experience both visually and aurally. For example, in the story of "Momotaro," when the user makes a selection, the device can sequentially display scenes such as the peach floating in the river, along with an audio reading of "Momotaro." Audio and image playback can be controlled by the user, allowing them to stop or skip playback.

[0038] Furthermore, for users who require explanations in simpler language, the system allows for text adjustment to convert the content into age-appropriate language. This system enriches stories and materials both visually and aurally, enabling users to experience and learn from a wider variety of content.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] The user launches the application using their device and selects the stories or materials they wish to view. The device then sends the selected content and the set language information to the server.

[0042] Step 2:

[0043] The server retrieves the text data of the selected story based on the received request. This text data is then analyzed using natural language processing techniques and divided into nouns and punctuation marks.

[0044] Step 3:

[0045] The server generates audio data in the specified language using speech synthesis based on the analyzed text data. The generated audio data corresponds to each element of the story.

[0046] Step 4:

[0047] The server uses image generation preparation means to prepare base data for image generation based on the analyzed text data. This data includes the visual elements required for each scene.

[0048] Step 5:

[0049] The server sends audio data and base data for image generation to the terminal.

[0050] Step 6:

[0051] The terminal activates its audio playback engine to play the received audio data and begins outputting audio.

[0052] Step 7:

[0053] The device runs an image generation engine and, based on image generation data, generates images for each scene in real time and displays them to the user. By synchronizing with audio playback, it provides a seamless narrative experience.

[0054] Step 8:

[0055] Users control audio and image playback through their device. They can pause and skip playback, among other actions. Additionally, a feature is available to convert text data into simplified language for easier understanding, if needed.

[0056] (Example 1)

[0057] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0058] There is a need for systems that allow users to experience stories and materials in a rich auditory and visual way, but conventional technology has made it difficult to reproduce story content in real time using audio and video. Furthermore, there has been a lack of interfaces that allow users to enjoy stories at their own pace, as well as functions to translate the content into simpler language suitable for specific age groups. This has hindered a wider range of users from intuitively and comfortably enjoying the information.

[0059] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0060] In this invention, the server includes a natural language processing means for analyzing text data and dividing it into nouns and punctuation units, a speech synthesis means for generating audio data in multiple languages ​​based on the analyzed text data, and an image generation means for generating images corresponding to each scene in real time based on data for image generation. This allows users to enjoy stories and materials aurally and visually in real time.

[0061] "Natural language processing techniques" are technologies that analyze text data and divide it into constituent units such as nouns and punctuation marks.

[0062] "Speech synthesis means" refers to a technology that generates speech data in multiple languages ​​based on analyzed text data.

[0063] "Image generation data preparation method" refers to a technology that prepares the data necessary for image generation based on analyzed text data.

[0064] "Image generation means" refers to a technology that generates images corresponding to each scene in real time based on image generation data.

[0065] A "synchronous playback method" is a technology that outputs generated audio data and images in sync.

[0066] A "selection interface" is a user interface that allows users to easily select stories or materials.

[0067] A "prompt generation method" is a technology that automatically generates prompt text for image generation based on nouns and important phrases.

[0068] "Playback control means" refers to a function that controls the playback of audio and images through user operation.

[0069] "Text adjustment means" refers to a technology that converts and displays text data in a gentler language according to the user's settings.

[0070] This invention is a system that generates and displays audio and images in real time based on a user's selection of stories or materials. The user can access this system using a general-purpose terminal device. The terminal communicates with the server and transmits the necessary data according to the user's selection.

[0071] The server uses natural language processing techniques to analyze the received text data. Specifically, it uses the Python language and the Natural Language Toolkit (NLTK) library to divide the text into units such as nouns and punctuation marks. For example, in the story of "Momotaro," it can extract nouns such as "Momotaro," "peach," and "river."

[0072] Next, the server utilizes speech synthesis technology to generate audio data based on the analyzed text data. It uses a speech synthesis engine such as Google® Text-to-Speech API to create an audio file in the specified language.

[0073] Furthermore, the server activates the prompt generation means using the image generation data preparation means. It generates prompt sentences for image generation from nouns and important phrases and sends them to the generation AI model. It utilizes OpenAI's DALL-E model to generate images based on the prompt sentences in real time. For example, an image is generated using a prompt sentence based on the scene of "a peach floating in a river."

[0074] As a concrete example, a prompt based on the story of Momotaro would be presented as follows: "Read the story of Momotaro aloud and generate images based on each key scene. Use nouns and important phrases to determine the visual elements."

[0075] The generated audio data and images are transmitted to the terminal and provided to the user via a synchronized playback mechanism. The terminal plays the audio and images in sync, providing the user with a seamless narrative experience through both sight and sound. A playback control mechanism is also included, allowing the user to stop or skip playback. Furthermore, a text adjustment mechanism allows the content to be displayed in user-friendly language according to user preferences.

[0076] This system allows users to experience the story's content through multiple senses, providing them with educational and entertaining value.

[0077] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0078] Step 1:

[0079] The user selects the content they want to read from stories and materials through the user interface on their device. The user's content selection information is received as input. The device sends the selected content information to the server. A request containing metadata for the selected content is generated as output.

[0080] Step 2:

[0081] The server analyzes the received text data using natural language processing techniques. The input is text data sent from the terminal. The server utilizes the Python language and the NLTK library to divide the text into nouns and punctuation units. Specifically, it tokenizes the text and performs syntactic analysis. The output is the analyzed text data.

[0082] Step 3:

[0083] The server generates audio data using speech synthesis based on the parsed text data. The parsed text data is used as input. A speech synthesis engine (e.g., Google Text-to-Speech API) is used to create an audio file in the specified language. Specifically, the process involves calling the speech synthesis engine's API to generate the audio file. The output is the generated audio data file.

[0084] Step 4:

[0085] The server uses image generation data preparation means to generate prompt sentences for image generation based on the analyzed text data, utilizing prompt generation means. The input is the analyzed text data. Nouns and important phrases are extracted, and prompt sentences are formed based on these. Specifically, the prompt sentences are created by embedding templates based on the text analysis results. The output is the generated prompt sentences.

[0086] Step 5:

[0087] The server uses a generative AI model to generate images in real time based on prompt text, representing various scenarios. The input is the generated prompt text. It utilizes OpenAI's DALL-E model to generate image data from the prompt text. Specifically, it sends the prompt text to the generative AI model and retrieves the returned image data. The output is the generated image data.

[0088] Step 6:

[0089] The terminal receives audio and image data files sent from the server and plays them in sync using a synchronous playback mechanism. The input consists of audio and image data files. The playback application on the terminal presents the audio and images to the user in sync. The output allows the user to experience the content visually and aurally. The specific operation involves adjusting the data stream and starting playback in a synchronous state.

[0090] Step 7:

[0091] The user controls audio and image playback using the device's playback control mechanisms. Input is the user's action events (e.g., start playback, stop playback, skip). The playback control function is used to adjust the timing of audio and images, enabling display at the user's pace. Specifically, the operation involves changing the playback state in response to the user's input events. Output is the display of content according to the playback sequence desired by the user.

[0092] Step 8:

[0093] The server uses text adjustment tools to convert and display text data in a gentler language according to the user's settings. The input consists of the user's settings and the original text data. The text adjustment engine processes the text to be used in a way that is appropriate for the user's age and level of comprehension. Specifically, it reconstructs the text based on an appropriate language model. The output is text expressed in a gentle manner that suits the user's settings.

[0094] (Application Example 1)

[0095] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0096] To provide an interactive content experience, it is necessary to efficiently convert text information into audio and visual information and present it seamlessly to the user. However, traditional methods struggle with multilingual support and real-time image generation, limiting the user experience. Furthermore, establishing an intuitive interface that users can operate easily and accommodating users of different ages and languages ​​remains a challenge.

[0097] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0098] In this invention, the server includes language processing means for analyzing text data and dividing it into noun and punctuation units, speech generation means for generating audio data in multiple languages ​​based on the analyzed text data, and visual data preparation means for preparing data for image generation based on the analyzed text data. This enables users to seamlessly experience selected content through audio and visual information.

[0099] "Text data" refers to a collection of strings containing information written in natural language.

[0100] "Language processing means" refers to a device or software that has the function of analyzing text data and dividing it into nouns and punctuation marks.

[0101] "Speech generation means" refers to a device or software that has the function of generating speech data in different languages ​​based on analyzed text data.

[0102] "Visual data preparation means" refers to a device or software that has the function of preparing data for image generation based on analyzed text data.

[0103] "Image generation means" refers to a device or software that has the function of generating visual information corresponding to each scene in real time based on visual data.

[0104] "Synchronized output means" refers to a device or software that has the function of outputting audio data and generated visual information in a synchronized manner.

[0105] "Output control means" refers to a device or software that has the function of controlling the playback of audio and visual information through user operation.

[0106] "Language adjustment means" refers to a device or software that has the function of converting text data into simplified language according to the user's settings and displaying it.

[0107] The system that realizes this invention consists of a user terminal, a server, and a generative AI model. The user operates the terminal and can select desired content through the provided interface. Information about the selected content is transmitted to the server via the internet.

[0108] The server analyzes the received text data using pre-built natural language processing capabilities and divides it into nouns and punctuation marks. Based on the information obtained from this analysis, the server generates audio data in the selected language using speech generation capabilities. For this speech generation, speech synthesis software such as Google Cloud Text-to-Speech is used.

[0109] Next, the server uses visual data preparation means to prepare data for image generation from the analyzed text data. This includes identifying visual elements based on important nouns and phrases. Based on this visual data, a generative AI model such as DALL-E is used to generate visual information corresponding to each scene in real time.

[0110] The generated audio data and visual information are seamlessly integrated using a synchronous output mechanism and transmitted to the user's terminal. This allows the user to enjoy a rich content experience that appeals to both sight and sound. Furthermore, the user can control the playback of the audio and visual information at their preferred timing using an output control mechanism.

[0111] For example, if the story of "Little Red Riding Hood" is selected, the server will send a prompt to DALL-E to generate a scene in which "a character wearing a red hood walks through the forest." An example of this prompt would be, "Please generate an image of the scene in the forest where Little Red Riding Hood encounters the wolf."

[0112] This system configuration makes multilingual, visually and aurally interactive content easily accessible to users.

[0113] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0114] Step 1:

[0115] The user accesses the system using a terminal and selects the desired story from the content selection screen. This input data includes the story's title and detailed information. The selected content information is sent to the server.

[0116] Step 2:

[0117] The server retrieves the text data of the received content and performs analysis using natural language processing. The analysis divides the text into nouns and punctuation marks. This process generates the analyzed text data.

[0118] Step 3:

[0119] The server creates audio data using a speech generation method based on the analyzed text data. This process involves speech synthesis in the configured language, for example, using Google Cloud Text-to-Speech. The output of this step is an audio data file.

[0120] Step 4:

[0121] The server then uses the analyzed data to prepare data for image generation using a visual data preparation system. Specifically, it extracts important nouns and phrases and identifies visual elements based on them. The output of this process is a visual dataset.

[0122] Step 5:

[0123] The server passes visual data to a generation AI model (e.g., DALL-E) to form prompt sentences. Based on these prompt sentences, the AI ​​generates visual information corresponding to each scene. The generated image data is then output.

[0124] Step 6:

[0125] The generated audio data and visual information are integrated through a synchronous output means and transmitted to the user terminal. The terminal seamlessly plays the received audio and images, presenting the content to the user. The output of this step is a visual and auditory content experience.

[0126] Step 7:

[0127] Users can control audio and image playback using the device's output control mechanisms. Specific actions include starting and stopping playback, and skipping scenes. This allows users to freely customize their content experience.

[0128] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0129] This invention is a system that recognizes user emotions and provides an interactive experience based on that information. The user launches an application using a terminal and selects their preferred story or material. Based on this selection, the terminal sends the necessary data to a server.

[0130] The server analyzes the selected content using natural language processing and divides it into nouns and punctuation marks. Using these analysis results, the server activates a speech synthesis system to generate multilingual audio data. At this stage, the generated audio data is prepared for subsequent synchronized playback.

[0131] Next, the analyzed text data is processed by the image generation preparation mechanism. The server prepares image generation data to clarify the visual elements for each scene. Based on this preparation data, the image generation mechanism generates images for each scene in real time.

[0132] In addition, the emotion engine runs on the device and recognizes the user's emotions in real time. This emotion data is sent to a server and reflected in the playback of audio and images. For example, if the user expresses surprise, some audio and images are adjusted based on the emotion recognized by the emotion engine. By emphasizing the surprised scene, a more immersive experience is provided.

[0133] The device uses all data received from the server to play audio and images in sync. Users can control playback through the device, pausing and skipping. A feature is also available where text data is converted into gentle language based on the user's emotions recognized by the emotion engine. This feature allows users to receive an optimal content experience tailored to their emotions at any given time.

[0134] By combining emotion recognition technology, this system further enhances the delivery of visual and auditory content, enabling personalized experiences for users.

[0135] The following describes the processing flow.

[0136] Step 1:

[0137] The user launches the application using their device and selects the stories or materials they wish to view. Based on this selection, the device sends content information and user settings to the server.

[0138] Step 2:

[0139] The server retrieves the text data of the selected story according to the received request. This text data is then analyzed using natural language processing techniques and divided into nouns and punctuation marks.

[0140] Step 3:

[0141] Based on the analysis results, the server generates audio data using a multilingual speech synthesis system. This audio data is configured according to specifications for later synchronous playback.

[0142] Step 4:

[0143] The server uses an image generation preparation mechanism to prepare the necessary image generation data based on the analyzed text data. This data defines the visual elements for each scene.

[0144] Step 5:

[0145] The server sends the completed audio data and the base data for image generation to the terminal.

[0146] Step 6:

[0147] The device activates its audio playback engine and plays the received audio data. The user is then provided with the audio of the story.

[0148] Step 7:

[0149] The device activates an image generation engine and generates images for each scene in real time from image generation data. The generated images are displayed to the user in sync with the audio.

[0150] Step 8:

[0151] An emotion engine operates on the device, recognizing the user's emotions in real time from their facial expressions and voice. The recognized emotion data is used by the server and for image and voice control.

[0152] Step 9:

[0153] Based on the recognized user's emotions, the server adjusts the content of audio and images. For example, if the user shows a sad expression, the system will be controlled to emphasize and play happy scenes.

[0154] Step 10:

[0155] Users can control playback, including pausing and skipping, using the device's interface. Furthermore, a text adjustment function that responds to emotions displays text in gentler language as needed. In this way, users enjoy a personalized, interactive experience.

[0156] (Example 2)

[0157] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0158] Traditional interactive content delivery systems have struggled to provide experiences that take user emotions into account. Therefore, the challenge lies in appropriately adjusting content according to user emotions to achieve a more personalized and immersive experience.

[0159] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0160] In this invention, the server includes a natural language processing means for analyzing text data and dividing it into nouns and punctuation units, a speech synthesis means for generating audio data in multiple languages ​​based on the analyzed text data, and an image generation preparation means for preparing data for image generation based on the analyzed text data. This enables a more personalized and interactive content experience by recognizing the user's emotions in real time and adjusting the audio and images based on those emotions.

[0161] "Natural language processing" is a technology that analyzes text data, divides it into elements such as words and punctuation, and understands the meaning and context of each element.

[0162] "Speech synthesis" is a technology that generates speech in a specified language based on text data.

[0163] "Image generation preparation" is the process of preparing the necessary visual information for each scene based on the analyzed text data.

[0164] "Image generation" is a technology in which a computer generates images corresponding to each scene in real time, based on prepared data.

[0165] "Synchronized playback" is a technology that plays generated audio and images simultaneously, providing users with a seamless viewing experience.

[0166] "Emotion recognition" is a technology that analyzes a user's facial expressions and tone of voice to identify their emotions in real time.

[0167] This invention is a system that provides an interactive content experience based on the user's emotions. The system mainly consists of terminals and servers, which work together in cooperation.

[0168] The terminal is a device used by the user to launch applications, and it allows the user to select content such as stories and documents using its operating interface. The content selected by the user is sent to the server via the network.

[0169] The server analyzes the received content using natural language processing tools (e.g., SpaCy, NLTK) and divides the text data into nouns and punctuation marks. Based on this analysis, multilingual audio data is generated by speech synthesis software (e.g., Amazon Polly, Google Cloud Text-to-Speech).

[0170] Furthermore, the server utilizes an image generation preparation engine to prepare scene-specific image generation data based on the analyzed text data. Then, it generates images in real time using an AI image generation model (e.g., DALL-E 2, Stable Diffusion). This process is executed by creating prompt statements. An example of a prompt statement is, "Recreate the story of the adventure in the magical forest with visuals and sound that enhance surprise and emotion."

[0171] Furthermore, the device recognizes the user's emotions in real time from their facial expressions and voice through an emotion engine (e.g., Microsoft® Azure® Emotion API). This emotion data is sent to a server and reflected in the audio and image data. For example, if the user expresses surprise, the audio tone and image effects of the corresponding scene are adjusted to provide a more immersive experience.

[0172] Finally, the device uses all the data to play audio and images in sync. Users can control playback, pausing and skipping. It also translates displayed text into gentle language based on the user's emotions, providing an optimal content experience tailored to the user.

[0173] This system configuration makes it possible to provide users with personalized, interactive entertainment.

[0174] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0175] Step 1:

[0176] The user launches the application on their device and selects stories or materials of interest through the provided interface. The user's selection is the input, and the device sends this selection information to the server in a structured format. The selected content data is obtained as output.

[0177] Step 2:

[0178] The server analyzes the received content data using a natural language processing engine (e.g., SpaCy or NLTK). The input data is in text format and is divided into nouns, verbs, and punctuation marks. This allows the grammatical structure to be analyzed, and structured text data is output.

[0179] Step 3:

[0180] The server generates audio data using a text-to-speech system (e.g., Amazon Polly, Google Cloud Text-to-Speech) based on the analysis results. The analyzed text data is used as input, and synthesized audio data in the specified language is output. This audio data is temporarily stored for later playback.

[0181] Step 4:

[0182] The server operates an image generation preparation engine based on text data, identifying visual elements for each scene. The input data is the result of text analysis, and it outputs data used as prompts for image generation. This output data is used as prompts for the image generation model.

[0183] Step 5:

[0184] The server generates images in real time using an AI image generation model (e.g., DALL-E 2, Stable Diffusion). Prompt data created during the generation preparation process serves as input, and image data corresponding to each scene is output. This image data is also stored, similar to the audio data.

[0185] Step 6:

[0186] The device uses a built-in emotion engine (e.g., Microsoft Azure Emotion API) to determine the user's emotions in real time from information obtained from the camera and microphone. User voice and facial expression data are used as input, and recognized emotion data is output. This emotion data is later reflected in the playback data.

[0187] Step 7:

[0188] The server receives emotion data sent by the user and reflects it in audio and image data. It performs adjustments such as volume and color adjustments based on emotion. Emotion data is used as input, and the adjusted audio and image data are output.

[0189] Step 8:

[0190] The terminal synchronizes and plays audio and image data sent from the server. User playback operations are used as input data, and playback controls such as pausing and skipping audio and video are output. At this stage, the user can also use an emotion-responsive text conversion function, which converts the text into gentler language.

[0191] (Application Example 2)

[0192] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0193] While modern entertainment experiences are incredibly rich in visual and auditory elements, providing interactive content that dynamically changes in response to individual users' emotions is still not fully achieved. In particular, tailoring content based on user emotions to deliver a personalized and immersive experience is a crucial challenge.

[0194] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0195] In this invention, the server includes language processing means for analyzing text data and dividing it into words and segments; speech generation means for generating audio information in multiple languages ​​based on the analyzed text data; and image preparation means for preparing visual data based on the analyzed text data. This enables a more personalized and interactive content experience by recognizing the user's emotions in real time and dynamically adjusting the audio and video according to those emotions.

[0196] "Language processing means" refers to technologies that analyze text data and divide the data according to words and boundaries.

[0197] "Speech generation means" refers to a technology that generates multilingual speech information based on analyzed text data.

[0198] "Image preparation means" refers to a technology that prepares visual data based on analyzed text data, thereby facilitating video generation.

[0199] "Image generation means" refers to a technology that generates video corresponding to each scene in real time based on visual data.

[0200] A "synchronous playback method" is a technology that outputs audio information and generated video simultaneously.

[0201] "Emotion recognition means" refers to technology that detects a user's emotions and acquires that information in real time.

[0202] "Emotional response technology" refers to a technology that dynamically adjusts audio and video information based on the user's emotional information.

[0203] "Playback control means" refers to technology that controls the playback of audio and video through user interface operations.

[0204] "Text adjustment means" refers to a technology that converts text data into a softer expression according to the user's settings and displays it accordingly.

[0205] To implement this invention, a server and a terminal are required. The terminal has a user interface and is used by the user to select content. The selected text data is sent to the server. The server uses language processing means to analyze the text data according to words and segments. The analysis results are converted into multilingual audio information by speech generation means. In addition, image preparation means are used to prepare visual data, and based on this, image generation means generate video in real time.

[0206] The device uses emotion recognition to analyze the user's facial expressions and voice, acquiring the user's emotions in real time. This emotion data is sent to a server, where emotion response mechanisms dynamically adjust the audio and video information according to the user's emotions. This enables a more immersive experience.

[0207] The audio and video being played are output simultaneously by the synchronized playback means, and the user can control their playback through the playback control means. For example, playback can be paused or skipped. In addition, the text adjustment means converts the text to a softer expression according to the user's settings and displays it.

[0208] For example, if a user who has selected horror content feels a level of fear, the background music will change and the video will become darker in accordance with their emotion, allowing the user to have a more intense fear experience. An example of a prompt message is, "If the user shows emotion of surprise, increase the volume."

[0209] This system integrates cutting-edge generative AI models, such as emotion recognition using TENSORFLOW®, speech synthesis using Google TTS and AWS® Polly, and image generation using Stable Diffusion, to provide users with personalized and interactive content experiences.

[0210] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0211] Step 1:

[0212] The terminal sends the content selected by the user to the server. The input is the story or material chosen by the user, and the output is the data sent to the server. The data processing performed here is the digitization and transmission of the content selection information.

[0213] Step 2:

[0214] The server uses language processing tools to analyze the received text data, dividing it into words and segments. The input is user-selected content data, and the output is the analyzed text data. Specifically, its operation involves structuring the data using natural language processing.

[0215] Step 3:

[0216] The server generates multilingual audio information using a speech generation system based on the analyzed text data. The input is the analyzed text data, and the output is audio information. In this step, the speech synthesis engine is started and audio data is generated according to the language selection.

[0217] Step 4:

[0218] The server uses an image preparation mechanism to prepare visual data from text data. The input is the parsed text data, and the output is visual data for image generation. Specifically, it constructs the data for the visual elements required for each scene.

[0219] Step 5:

[0220] The server generates video in real time based on visual data using an image generation means. The input is visual data, and the output is the generated video. This step involves deploying an image generation model and generating the required visuals.

[0221] Step 6:

[0222] The device uses emotion recognition technology to analyze the user's facial expressions and voice, detecting emotions in real time. The input is real-time user data, and the output is emotional information. Specifically, it measures emotional data through the camera and microphone and uses a model to infer it.

[0223] Step 7:

[0224] The server adjusts audio and video information using emotion-responsive mechanisms based on emotional information. The input is recognized emotional information, and the output is adjusted audio and video. This operation includes adjusting the tone of the audio and changing the brightness of the video.

[0225] Step 8:

[0226] The device simultaneously outputs synchronized audio and video using a synchronized playback mechanism. The input is the synchronized audio and video data, and the output is the user's viewing experience. Specific operations include synchronizing the audio and video in time before outputting them.

[0227] Step 9:

[0228] The user controls audio and video playback as desired through the interface using playback control means. Input is user operation, and output is the response of the playback control. This operation includes controls such as pause, skip, and resume.

[0229] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0230] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0231] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0232] [Second Embodiment]

[0233] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0234] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0235] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0236] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0237] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0238] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0239] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0240] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0241] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0242] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0243] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0244] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0245] This invention is an interactive system that reads aloud stories or materials selected by the user and displays automatically generated images based on that content. The user accesses this system using a terminal and selects their preferred content. After selecting content, the terminal sends the selected content and associated language information to the server.

[0246] The server analyzes the selected story text using natural language processing techniques, dividing it into nouns and punctuation marks. Based on each element of the analyzed text, the server uses speech synthesis techniques to generate audio data in the user's specified language. The audio data generated at this stage is ready for use in subsequent processes.

[0247] Next, the server uses the image generation preparation means to prepare image generation data based on the analyzed text data. This data is used to define the visual elements of each scene based on nouns and important phrases. Based on this prepared data, the image generation means generates images corresponding to each scene in real time.

[0248] The device receives audio files and image generation data from the server. A synchronized playback mechanism for audio and images is activated, allowing the user to enjoy a seamless narrative experience both visually and aurally. For example, in the story of "Momotaro," when the user makes a selection, the device can sequentially display scenes such as the peach floating in the river, along with an audio reading of "Momotaro." Audio and image playback can be controlled by the user, allowing them to stop or skip playback.

[0249] Furthermore, for users who require explanations in simpler language, the system allows for text adjustment to convert the content into age-appropriate language. This system enriches stories and materials both visually and aurally, enabling users to experience and learn from a wider variety of content.

[0250] The following describes the processing flow.

[0251] Step 1:

[0252] The user launches the application using their device and selects the stories or materials they wish to view. The device then sends the selected content and the set language information to the server.

[0253] Step 2:

[0254] The server retrieves the text data of the selected story based on the received request. This text data is then analyzed using natural language processing techniques and divided into nouns and punctuation marks.

[0255] Step 3:

[0256] The server generates audio data in the specified language using speech synthesis based on the analyzed text data. The generated audio data corresponds to each element of the story.

[0257] Step 4:

[0258] The server uses image generation preparation means to prepare base data for image generation based on the analyzed text data. This data includes the visual elements required for each scene.

[0259] Step 5:

[0260] The server sends audio data and base data for image generation to the terminal.

[0261] Step 6:

[0262] The terminal activates its audio playback engine to play the received audio data and begins outputting audio.

[0263] Step 7:

[0264] The device runs an image generation engine and, based on image generation data, generates images for each scene in real time and displays them to the user. By synchronizing with audio playback, it provides a seamless narrative experience.

[0265] Step 8:

[0266] Users control audio and image playback through their device. They can pause and skip playback, among other actions. Additionally, a feature is available to convert text data into simplified language for easier understanding, if needed.

[0267] (Example 1)

[0268] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0269] There is a need for systems that allow users to experience stories and materials in a rich auditory and visual way, but conventional technology has made it difficult to reproduce story content in real time using audio and video. Furthermore, there has been a lack of interfaces that allow users to enjoy stories at their own pace, as well as functions to translate the content into simpler language suitable for specific age groups. This has hindered a wider range of users from intuitively and comfortably enjoying the information.

[0270] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0271] In this invention, the server includes a natural language processing means for analyzing text data and dividing it into nouns and punctuation units, a speech synthesis means for generating audio data in multiple languages ​​based on the analyzed text data, and an image generation means for generating images corresponding to each scene in real time based on data for image generation. This allows users to enjoy stories and materials aurally and visually in real time.

[0272] "Natural language processing techniques" are technologies that analyze text data and divide it into constituent units such as nouns and punctuation marks.

[0273] "Speech synthesis means" refers to a technology that generates speech data in multiple languages ​​based on analyzed text data.

[0274] "Image generation data preparation method" refers to a technology that prepares the data necessary for image generation based on analyzed text data.

[0275] "Image generation means" refers to a technology that generates images corresponding to each scene in real time based on image generation data.

[0276] A "synchronous playback method" is a technology that outputs generated audio data and images in sync.

[0277] A "selection interface" is a user interface that allows users to easily select stories or materials.

[0278] A "prompt generation method" is a technology that automatically generates prompt text for image generation based on nouns and important phrases.

[0279] "Playback control means" refers to a function that controls the playback of audio and images through user operation.

[0280] "Text adjustment means" refers to a technology that converts and displays text data in a gentler language according to the user's settings.

[0281] This invention is a system that generates and displays audio and images in real time based on a user's selection of stories or materials. The user can access this system using a general-purpose terminal device. The terminal communicates with the server and transmits the necessary data according to the user's selection.

[0282] The server uses natural language processing techniques to analyze the received text data. Specifically, it uses the Python language and the Natural Language Toolkit (NLTK) library to divide the text into units such as nouns and punctuation marks. For example, in the story of "Momotaro," it can extract nouns such as "Momotaro," "peach," and "river."

[0283] Next, the server utilizes voice synthesis means to generate voice data based on the analyzed text data. Using a voice synthesis engine such as the Google Text-to-Speech API, an audio file is created in the specified language.

[0284] Furthermore, the server activates the prompt generation means using image generation data preparation means. A prompt sentence for image generation is generated from nouns and important phrases and sent to the generation AI model. Utilizing OpenAI's DALL-E model, an image based on the prompt sentence is generated in real time. For example, an image is generated using a prompt sentence based on the scene of "peaches floating on a river".

[0285] As a specific example, a prompt sentence based on the story of "Momotaro" is shown as follows. "Read the story of Momotaro aloud and generate an image based on each main scene. Utilize nouns and important phrases to determine visual elements."

[0286] The generated voice data and image are sent to the terminal and provided to the user by the synchronous playback means. The terminal synchronously plays the voice and the image, providing the user with a seamless storytelling experience both visually and auditorily. It also has playback control means, enabling the user to stop or skip the playback. Additionally, using the text adjustment means, it is possible to display the content in kind words according to the user's settings.

[0287] With this system, the user can experience the content of the story in multiple senses and spend an educational and entertaining valuable time.

[0288] The flow of the specific process in Example 1 will be described using FIG. 11.

[0289] Step 1:

[0290] The user selects the content they want to read from stories and materials through the user interface on their device. The user's content selection information is received as input. The device sends the selected content information to the server. A request containing metadata for the selected content is generated as output.

[0291] Step 2:

[0292] The server analyzes the received text data using natural language processing techniques. The input is text data sent from the terminal. The server utilizes the Python language and the NLTK library to divide the text into nouns and punctuation units. Specifically, it tokenizes the text and performs syntactic analysis. The output is the analyzed text data.

[0293] Step 3:

[0294] The server generates audio data using speech synthesis based on the parsed text data. The parsed text data is used as input. A speech synthesis engine (e.g., Google Text-to-Speech API) is used to create an audio file in the specified language. Specifically, the process involves calling the speech synthesis engine's API to generate the audio file. The output is the generated audio data file.

[0295] Step 4:

[0296] The server uses image generation data preparation means to generate prompt sentences for image generation based on the analyzed text data, utilizing prompt generation means. The input is the analyzed text data. Nouns and important phrases are extracted, and prompt sentences are formed based on these. Specifically, the prompt sentences are created by embedding templates based on the text analysis results. The output is the generated prompt sentences.

[0297] Step 5:

[0298] The server uses a generative AI model to generate images in real time based on prompt text, representing various scenarios. The input is the generated prompt text. It utilizes OpenAI's DALL-E model to generate image data from the prompt text. Specifically, it sends the prompt text to the generative AI model and retrieves the returned image data. The output is the generated image data.

[0299] Step 6:

[0300] The terminal receives audio and image data files sent from the server and plays them in sync using a synchronous playback mechanism. The input consists of audio and image data files. The playback application on the terminal presents the audio and images to the user in sync. The output allows the user to experience the content visually and aurally. The specific operation involves adjusting the data stream and starting playback in a synchronous state.

[0301] Step 7:

[0302] The user controls audio and image playback using the device's playback control mechanisms. Input is the user's action events (e.g., start playback, stop playback, skip). The playback control function is used to adjust the timing of audio and images, enabling display at the user's pace. Specifically, the operation involves changing the playback state in response to the user's input events. Output is the display of content according to the playback sequence desired by the user.

[0303] Step 8:

[0304] The server uses text adjustment means to convert the text data to be displayed into gentle words according to the user's settings and display it. The input is the user's setting information and the original text data. The text adjustment engine performs processing to convert the diction according to the user's age and comprehension level. The specific operation is to reconstruct the text based on an appropriate language model. As output, text with a gentle expression suitable for the user's settings is displayed.

[0305] (Application Example 1)

[0306] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0307] In order to provide an interactive content experience, it is necessary to efficiently convert text information into audio and visual information and seamlessly present it to the user. However, with conventional methods, it is difficult to support multiple languages and generate images in real time, and the user experience is limited. Furthermore, establishing an interface that the user can intuitively operate and corresponding to users of different ages and languages is also an issue.

[0308] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0309] In this invention, the server includes language processing means for analyzing text data and delimiting it into units of nouns and punctuation marks, voice generation means for generating voice data in multiple languages based on the analyzed text data, and visual data preparation means for preparing data for image generation based on the analyzed text data. Thereby, it becomes possible for the user to seamlessly experience the content selected by the user through voice and visual information.

[0310] "Text data" is a collection of character strings including information described in natural language.

[0311] "Language processing means" refers to a device or software that has the function of analyzing text data and dividing it into nouns and punctuation marks.

[0312] "Speech generation means" refers to a device or software that has the function of generating speech data in different languages ​​based on analyzed text data.

[0313] "Visual data preparation means" refers to a device or software that has the function of preparing data for image generation based on analyzed text data.

[0314] "Image generation means" refers to a device or software that has the function of generating visual information corresponding to each scene in real time based on visual data.

[0315] "Synchronized output means" refers to a device or software that has the function of outputting audio data and generated visual information in a synchronized manner.

[0316] "Output control means" refers to a device or software that has the function of controlling the playback of audio and visual information through user operation.

[0317] "Language adjustment means" refers to a device or software that has the function of converting text data into simplified language according to the user's settings and displaying it.

[0318] The system that realizes this invention consists of a user terminal, a server, and a generative AI model. The user operates the terminal and can select desired content through the provided interface. Information about the selected content is transmitted to the server via the internet.

[0319] The server analyzes the received text data using pre-built natural language processing capabilities and divides it into nouns and punctuation marks. Based on the information obtained from this analysis, the server generates audio data in the selected language using speech generation capabilities. For this speech generation, speech synthesis software such as Google Cloud Text-to-Speech is used.

[0320] Next, the server uses visual data preparation means to prepare data for image generation from the analyzed text data. This includes identifying visual elements based on important nouns and phrases. Based on this visual data, a generative AI model such as DALL-E is used to generate visual information corresponding to each scene in real time.

[0321] The generated audio data and visual information are seamlessly integrated using a synchronous output mechanism and transmitted to the user's terminal. This allows the user to enjoy a rich content experience that appeals to both sight and sound. Furthermore, the user can control the playback of the audio and visual information at their preferred timing using an output control mechanism.

[0322] For example, if the story of "Little Red Riding Hood" is selected, the server will send a prompt to DALL-E to generate a scene in which "a character wearing a red hood walks through the forest." An example of this prompt would be, "Please generate an image of the scene in the forest where Little Red Riding Hood encounters the wolf."

[0323] This system configuration makes multilingual, visually and aurally interactive content easily accessible to users.

[0324] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0325] Step 1:

[0326] The user accesses the system using a terminal and selects the desired story from the content selection screen. This input data includes the story's title and detailed information. The selected content information is sent to the server.

[0327] Step 2:

[0328] The server retrieves the text data of the received content and performs analysis using natural language processing. The analysis divides the text into nouns and punctuation marks. This process generates the analyzed text data.

[0329] Step 3:

[0330] The server creates audio data using a speech generation method based on the analyzed text data. This process involves speech synthesis in the configured language, for example, using Google Cloud Text-to-Speech. The output of this step is an audio data file.

[0331] Step 4:

[0332] The server then uses the analyzed data to prepare data for image generation using a visual data preparation system. Specifically, it extracts important nouns and phrases and identifies visual elements based on them. The output of this process is a visual dataset.

[0333] Step 5:

[0334] The server passes visual data to a generation AI model (e.g., DALL-E) to form prompt sentences. Based on these prompt sentences, the AI ​​generates visual information corresponding to each scene. The generated image data is then output.

[0335] Step 6:

[0336] The generated audio data and visual information are integrated through a synchronous output means and transmitted to the user terminal. The terminal seamlessly plays the received audio and images, presenting the content to the user. The output of this step is a visual and auditory content experience.

[0337] Step 7:

[0338] Users can control audio and image playback using the device's output control mechanisms. Specific actions include starting and stopping playback, and skipping scenes. This allows users to freely customize their content experience.

[0339] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0340] This invention is a system that recognizes user emotions and provides an interactive experience based on that information. The user launches an application using a terminal and selects their preferred story or material. Based on this selection, the terminal sends the necessary data to the server.

[0341] The server analyzes the selected content using natural language processing and divides it into nouns and punctuation marks. Using these analysis results, the server activates a speech synthesis system to generate multilingual audio data. At this stage, the generated audio data is prepared for synchronized playback that will occur later.

[0342] Next, the analyzed text data is processed by the image generation preparation mechanism. The server prepares image generation data to clarify the visual elements for each scene. Based on this preparation data, the image generation mechanism generates images for each scene in real time.

[0343] In addition, the emotion engine runs on the device and recognizes the user's emotions in real time. This emotion data is sent to a server and reflected in the playback of audio and images. For example, if the user expresses surprise, some audio and images are adjusted based on the emotion recognized by the emotion engine. By emphasizing the surprised scene, a more immersive experience is provided.

[0344] The device uses all data received from the server to play audio and images in sync. Users can control playback through the device, pausing and skipping. A feature is also available where text data is converted into gentle language based on the user's emotions recognized by the emotion engine. This feature allows users to receive an optimal content experience tailored to their emotions at any given time.

[0345] By combining emotion recognition technology, this system further enhances the delivery of visual and auditory content, enabling personalized experiences for users.

[0346] The following describes the processing flow.

[0347] Step 1:

[0348] The user launches the application using their device and selects the stories or materials they wish to view. Based on this selection, the device sends content information and user settings to the server.

[0349] Step 2:

[0350] The server retrieves the text data of the selected story according to the received request. This text data is then analyzed using natural language processing techniques and divided into nouns and punctuation marks.

[0351] Step 3:

[0352] Based on the analysis results, the server generates audio data using a multilingual speech synthesis system. This audio data is configured according to specifications for later synchronous playback.

[0353] Step 4:

[0354] The server uses an image generation preparation mechanism to prepare the necessary image generation data based on the analyzed text data. This data defines the visual elements for each scene.

[0355] Step 5:

[0356] The server sends the completed audio data and the base data for image generation to the terminal.

[0357] Step 6:

[0358] The device activates its audio playback engine and plays the received audio data. The user is then provided with the audio of the story.

[0359] Step 7:

[0360] The device activates an image generation engine and generates images for each scene in real time from image generation data. The generated images are displayed to the user in sync with the audio.

[0361] Step 8:

[0362] An emotion engine operates on the device, recognizing the user's emotions in real time from their facial expressions and voice. The recognized emotion data is used by the server and for image and voice control.

[0363] Step 9:

[0364] Based on the recognized user's emotions, the server adjusts the content of audio and images. For example, if the user shows a sad expression, the system will be controlled to emphasize and play happy scenes.

[0365] Step 10:

[0366] Users can control playback, including pausing and skipping, using the device's interface. Furthermore, a text adjustment function that responds to emotions displays text in gentler language as needed. In this way, users enjoy a personalized, interactive experience.

[0367] (Example 2)

[0368] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0369] Traditional interactive content delivery systems have struggled to provide experiences that take user emotions into account. Therefore, the challenge lies in appropriately adjusting content according to user emotions to achieve a more personalized and immersive experience.

[0370] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0371] In this invention, the server includes a natural language processing means for analyzing text data and dividing it into nouns and punctuation units, a speech synthesis means for generating audio data in multiple languages ​​based on the analyzed text data, and an image generation preparation means for preparing data for image generation based on the analyzed text data. This enables a more personalized and interactive content experience by recognizing the user's emotions in real time and adjusting the audio and images based on those emotions.

[0372] "Natural language processing" is a technology that analyzes text data, divides it into elements such as words and punctuation, and understands the meaning and context of each element.

[0373] "Speech synthesis" is a technology that generates speech in a specified language based on text data.

[0374] "Image generation preparation" is the process of preparing the necessary visual information for each scene based on the analyzed text data.

[0375] "Image generation" is a technology in which a computer generates images corresponding to each scene in real time, based on prepared data.

[0376] "Synchronized playback" is a technology that plays generated audio and images simultaneously, providing users with a seamless viewing experience.

[0377] "Emotion recognition" is a technology that analyzes a user's facial expressions and tone of voice to identify their emotions in real time.

[0378] This invention is a system that provides an interactive content experience based on the user's emotions. The system mainly consists of terminals and servers, which work together in cooperation.

[0379] The terminal is a device used by the user to launch applications, and it allows the user to select content such as stories and documents using its operating interface. The content selected by the user is sent to the server via the network.

[0380] The server analyzes the received content using natural language processing tools (e.g., SpaCy, NLTK) and divides the text data into nouns and punctuation marks. Based on this analysis, multilingual audio data is generated by speech synthesis software (e.g., Amazon Polly, Google Cloud Text-to-Speech).

[0381] Furthermore, the server utilizes an image generation preparation engine to prepare scene-specific image generation data based on the analyzed text data. Then, it generates images in real time using an AI image generation model (e.g., DALL-E 2, Stable Diffusion). This process is executed by creating prompt statements. An example of a prompt statement is, "Recreate the story of the adventure in the magical forest with visuals and sound that enhance surprise and emotion."

[0382] Furthermore, the device recognizes the user's emotions in real time from their facial expressions and voice through an emotion engine (e.g., Microsoft Azure Emotion API). This emotion data is sent to a server and reflected in the audio and image data. For example, if the user expresses surprise, the audio tone and image effects of the corresponding scene are adjusted to provide a more immersive experience.

[0383] Finally, the device uses all the data to play audio and images in sync. Users can control playback, pausing and skipping. It also translates displayed text into gentle language based on the user's emotions, providing an optimal content experience tailored to the user.

[0384] This system configuration makes it possible to provide users with personalized, interactive entertainment.

[0385] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0386] Step 1:

[0387] The user launches the application on their device and selects stories or materials of interest through the provided interface. The user's selection is the input, and the device sends this selection information to the server in a structured format. The selected content data is obtained as output.

[0388] Step 2:

[0389] The server analyzes the received content data using a natural language processing engine (e.g., SpaCy or NLTK). The data received as input is in text format and is divided into nouns, verbs, and punctuation marks. This allows the grammatical structure to be analyzed, and structured text data is output.

[0390] Step 3:

[0391] The server generates audio data using a text-to-speech system (e.g., Amazon Polly, Google Cloud Text-to-Speech) based on the analysis results. The analyzed text data is used as input, and synthesized audio data in the specified language is output. This audio data is temporarily stored for later playback.

[0392] Step 4:

[0393] The server operates an image generation preparation engine based on text data, identifying visual elements for each scene. The input data is the result of text analysis, and it outputs data used as prompts for image generation. This output data is used as prompts for the image generation model.

[0394] Step 5:

[0395] The server generates images in real time using an AI image generation model (e.g., DALL-E 2, Stable Diffusion). Prompt data created during the generation preparation process serves as input, and image data corresponding to each scene is output. This image data is also stored, similar to the audio data.

[0396] Step 6:

[0397] The device uses a built-in emotion engine (e.g., Microsoft Azure Emotion API) to determine the user's emotions in real time from information obtained from the camera and microphone. User voice and facial expression data are used as input, and recognized emotion data is output. This emotion data is later reflected in the playback data.

[0398] Step 7:

[0399] The server receives emotion data sent by the user and reflects it in audio and image data. It performs adjustments such as volume and color adjustments based on emotion. Emotion data is used as input, and the adjusted audio and image data are output.

[0400] Step 8:

[0401] The terminal synchronizes and plays audio and image data sent from the server. User playback operations are used as input data, and playback controls such as pausing and skipping audio and video are output. At this stage, the user can also use an emotion-responsive text conversion function, which converts the text into gentler language.

[0402] (Application Example 2)

[0403] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0404] While modern entertainment experiences are incredibly rich in visual and auditory elements, providing interactive content that dynamically changes in response to individual users' emotions is still not fully achieved. In particular, tailoring content based on user emotions to deliver a personalized and immersive experience is a crucial challenge.

[0405] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0406] In this invention, the server includes language processing means for analyzing text data and dividing it into words and segments; speech generation means for generating audio information in multiple languages ​​based on the analyzed text data; and image preparation means for preparing visual data based on the analyzed text data. This enables a more personalized and interactive content experience by recognizing the user's emotions in real time and dynamically adjusting the audio and video according to those emotions.

[0407] "Language processing means" refers to technologies that analyze text data and divide the data according to words and boundaries.

[0408] "Speech generation means" refers to a technology that generates multilingual speech information based on analyzed text data.

[0409] "Image preparation means" refers to a technology that prepares visual data based on analyzed text data, thereby facilitating video generation.

[0410] "Image generation means" refers to a technology that generates video corresponding to each scene in real time based on visual data.

[0411] A "synchronous playback method" is a technology that outputs audio information and generated video simultaneously.

[0412] "Emotion recognition means" refers to technology that detects a user's emotions and acquires that information in real time.

[0413] "Emotional response technology" refers to a technology that dynamically adjusts audio and video information based on the user's emotional information.

[0414] "Playback control means" refers to technology that controls the playback of audio and video through user interface operations.

[0415] "Text adjustment means" refers to a technology that converts text data into a softer expression according to the user's settings and displays it accordingly.

[0416] To implement this invention, a server and a terminal are required. The terminal has a user interface and is used by the user to select content. The selected text data is sent to the server. The server uses language processing means to analyze the text data according to words and segments. The analysis results are converted into multilingual audio information by speech generation means. In addition, image preparation means are used to prepare visual data, and based on this, image generation means generate video in real time.

[0417] The device uses emotion recognition to analyze the user's facial expressions and voice, acquiring the user's emotions in real time. This emotion data is sent to a server, where emotion response mechanisms dynamically adjust the audio and video information according to the user's emotions. This enables a more immersive experience.

[0418] The audio and video being played are output simultaneously by the synchronized playback means, and the user can control their playback through the playback control means. For example, playback can be paused or skipped. In addition, the text adjustment means converts the text to a softer expression according to the user's settings and displays it.

[0419] For example, if a user who has selected horror content feels a level of fear, the background music will change and the video will become darker in accordance with their emotion, allowing the user to have a more intense fear experience. An example of a prompt message is, "If the user shows emotion of surprise, increase the volume."

[0420] This system integrates cutting-edge generative AI models, such as emotion recognition using TensorFlow, speech synthesis using Google TTS and AWS Polly, and image generation using Stable Diffusion, to provide users with personalized and interactive content experiences.

[0421] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0422] Step 1:

[0423] The terminal sends the content selected by the user to the server. The input is the story or material chosen by the user, and the output is the data sent to the server. The data processing performed here is the digitization and transmission of the content selection information.

[0424] Step 2:

[0425] The server uses language processing tools to analyze the received text data, dividing it into words and segments. The input is user-selected content data, and the output is the analyzed text data. Specifically, it structures the data using natural language processing.

[0426] Step 3:

[0427] The server generates multilingual audio information using a speech generation system based on the analyzed text data. The input is the analyzed text data, and the output is audio information. In this step, the speech synthesis engine is started and audio data is generated according to the language selection.

[0428] Step 4:

[0429] The server uses an image preparation mechanism to prepare visual data from text data. The input is the parsed text data, and the output is visual data for image generation. Specifically, it constructs the data for the visual elements required for each scene.

[0430] Step 5:

[0431] The server generates video in real time based on visual data using an image generation mechanism. The input is visual data, and the output is the generated video. This step involves deploying an image generation model and generating the required visuals.

[0432] Step 6:

[0433] The device uses emotion recognition technology to analyze the user's facial expressions and voice, detecting emotions in real time. The input is real-time user data, and the output is emotional information. Specifically, it measures emotional data through the camera and microphone and uses a model to infer it.

[0434] Step 7:

[0435] The server adjusts audio and video information using emotion-responsive mechanisms based on emotional information. The input is recognized emotional information, and the output is adjusted audio and video. This operation includes adjusting the tone of the audio and changing the brightness of the video.

[0436] Step 8:

[0437] The device simultaneously outputs synchronized audio and video using a synchronized playback mechanism. The input is the synchronized audio and video data, and the output is the user's viewing experience. Specific operations include synchronizing the audio and video in time before outputting them.

[0438] Step 9:

[0439] The user controls audio and video playback as desired through the interface using playback control means. Input is user operation, and output is the response of the playback control. This operation includes controls such as pause, skip, and resume.

[0440] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0441] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0442] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0443] [Third Embodiment]

[0444] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0445] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0446] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0447] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0448] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0449] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0450] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0451] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0452] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0453] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0454] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0455] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0456] This invention is an interactive system that reads aloud stories or materials selected by the user and displays automatically generated images based on that content. The user accesses this system using a terminal and selects their preferred content. After selecting content, the terminal sends the selected content and associated language information to the server.

[0457] The server analyzes the selected story text using natural language processing techniques, dividing it into nouns and punctuation marks. Based on each element of the analyzed text, the server uses speech synthesis techniques to generate audio data in the user's specified language. The audio data generated at this stage is ready for use in subsequent processes.

[0458] Next, the server uses the image generation preparation means to prepare image generation data based on the analyzed text data. This data is used to define the visual elements of each scene based on nouns and important phrases. Based on this prepared data, the image generation means generates images corresponding to each scene in real time.

[0459] The device receives audio files and image generation data from the server. A synchronized playback mechanism for audio and images is activated, allowing the user to enjoy a seamless narrative experience both visually and aurally. For example, in the story of "Momotaro," when the user makes a selection, the device can sequentially display scenes such as the peach floating in the river, along with an audio reading of "Momotaro." Audio and image playback can be controlled by the user, allowing them to stop or skip playback.

[0460] Furthermore, for users who require explanations in simpler language, the system allows for text adjustment to convert the content into age-appropriate language. This system enriches stories and materials both visually and aurally, enabling users to experience and learn from a wider variety of content.

[0461] The following describes the processing flow.

[0462] Step 1:

[0463] The user launches the application using their device and selects the stories or materials they wish to view. The device then sends the selected content and the set language information to the server.

[0464] Step 2:

[0465] The server retrieves the text data of the selected story based on the received request. This text data is then analyzed using natural language processing techniques and divided into nouns and punctuation marks.

[0466] Step 3:

[0467] The server generates audio data in the specified language using speech synthesis based on the analyzed text data. The generated audio data corresponds to each element of the story.

[0468] Step 4:

[0469] The server uses image generation preparation means to prepare base data for image generation based on the analyzed text data. This data includes the visual elements required for each scene.

[0470] Step 5:

[0471] The server sends audio data and base data for image generation to the terminal.

[0472] Step 6:

[0473] The terminal activates its audio playback engine to play the received audio data and begins outputting audio.

[0474] Step 7:

[0475] The device runs an image generation engine and, based on image generation data, generates images for each scene in real time and displays them to the user. By synchronizing with audio playback, it provides a seamless narrative experience.

[0476] Step 8:

[0477] Users control audio and image playback through their device. They can pause and skip playback, among other actions. Additionally, a feature is available to convert text data into simplified language for easier understanding, if needed.

[0478] (Example 1)

[0479] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0480] There is a need for systems that allow users to experience stories and materials in a rich auditory and visual way, but conventional technology has made it difficult to reproduce story content in real time using audio and video. Furthermore, there has been a lack of interfaces that allow users to enjoy stories at their own pace, as well as functions to translate the content into simpler language suitable for specific age groups. This has hindered a wider range of users from intuitively and comfortably enjoying the information.

[0481] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0482] In this invention, the server includes a natural language processing means for analyzing text data and dividing it into nouns and punctuation units, a speech synthesis means for generating audio data in multiple languages ​​based on the analyzed text data, and an image generation means for generating images corresponding to each scene in real time based on data for image generation. This allows users to enjoy stories and materials aurally and visually in real time.

[0483] "Natural language processing techniques" are technologies that analyze text data and divide it into constituent units such as nouns and punctuation marks.

[0484] "Speech synthesis means" refers to a technology that generates speech data in multiple languages ​​based on analyzed text data.

[0485] "Image generation data preparation method" refers to a technology that prepares the data necessary for image generation based on analyzed text data.

[0486] "Image generation means" refers to a technology that generates images corresponding to each scene in real time based on image generation data.

[0487] A "synchronous playback method" is a technology that outputs generated audio data and images in sync.

[0488] A "selection interface" is a user interface that allows users to easily select stories or materials.

[0489] A "prompt generation method" is a technology that automatically generates prompt text for image generation based on nouns and important phrases.

[0490] "Playback control means" refers to a function that controls the playback of audio and images through user operation.

[0491] "Text adjustment means" refers to a technology that converts and displays text data in a gentler language according to the user's settings.

[0492] This invention is a system that generates and displays audio and images in real time based on a user's selection of stories or materials. The user can access this system using a general-purpose terminal device. The terminal communicates with the server and transmits the necessary data according to the user's selection.

[0493] The server uses natural language processing techniques to analyze the received text data. Specifically, it uses the Python language and the Natural Language Toolkit (NLTK) library to divide the text into units such as nouns and punctuation marks. For example, in the story of "Momotaro," it can extract nouns such as "Momotaro," "peach," and "river."

[0494] Next, the server utilizes speech synthesis technology to generate audio data based on the analyzed text data. It uses a speech synthesis engine such as the Google Text-to-Speech API to create an audio file in the specified language.

[0495] Furthermore, the server activates the prompt generation mechanism using the image generation data preparation mechanism. It generates prompt sentences for image generation from nouns and important phrases and sends them to the generation AI model. Utilizing OpenAI's DALL-E model, it generates images in real time based on the prompt sentences. For example, an image is generated using a prompt sentence based on the scene of "peaches floating in a river."

[0496] As a concrete example, a prompt based on the story of Momotaro would be presented as follows: "Read the story of Momotaro aloud and generate images based on each key scene. Use nouns and important phrases to determine the visual elements."

[0497] The generated audio data and images are transmitted to the terminal and provided to the user via a synchronized playback mechanism. The terminal plays the audio and images in sync, providing the user with a seamless narrative experience through both sight and sound. A playback control mechanism is also included, allowing the user to stop or skip playback. Furthermore, a text adjustment mechanism allows the content to be displayed in user-friendly language according to user preferences.

[0498] This system allows users to experience the story's content through multiple senses, providing them with educational and entertaining value.

[0499] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0500] Step 1:

[0501] The user selects the content they want to read from stories and materials through the user interface on their device. The user's content selection information is received as input. The device sends the selected content information to the server. A request containing metadata for the selected content is generated as output.

[0502] Step 2:

[0503] The server analyzes the received text data using natural language processing techniques. The input is text data sent from the terminal. The server utilizes the Python language and the NLTK library to divide the text into nouns and punctuation units. Specifically, it tokenizes the text and performs syntactic analysis. The output is the analyzed text data.

[0504] Step 3:

[0505] The server generates audio data using speech synthesis based on the parsed text data. The parsed text data is used as input. A speech synthesis engine (e.g., Google Text-to-Speech API) is used to create an audio file in the specified language. Specifically, the process involves calling the speech synthesis engine's API to generate the audio file. The output is the generated audio data file.

[0506] Step 4:

[0507] The server uses image generation data preparation means to generate prompt sentences for image generation based on the analyzed text data, utilizing prompt generation means. The input is the analyzed text data. Nouns and important phrases are extracted, and prompt sentences are formed based on these. Specifically, the prompt sentences are created by embedding templates based on the text analysis results. The output is the generated prompt sentences.

[0508] Step 5:

[0509] The server uses a generative AI model to generate images in real time based on prompt text, representing various scenarios. The input is the generated prompt text. It utilizes OpenAI's DALL-E model to generate image data from the prompt text. Specifically, it sends the prompt text to the generative AI model and retrieves the returned image data. The output is the generated image data.

[0510] Step 6:

[0511] The terminal receives audio and image data files sent from the server and plays them in sync using a synchronous playback mechanism. The input consists of audio and image data files. The playback application on the terminal presents the audio and images to the user in sync. The output allows the user to experience the content visually and aurally. The specific operation involves adjusting the data stream and starting playback in a synchronous state.

[0512] Step 7:

[0513] The user controls audio and image playback using the device's playback control mechanisms. Input is the user's action events (e.g., start playback, stop playback, skip). The playback control function is used to adjust the timing of audio and images, enabling display at the user's pace. Specifically, the operation involves changing the playback state in response to the user's input events. Output is the display of content according to the playback sequence desired by the user.

[0514] Step 8:

[0515] The server uses text adjustment tools to convert and display text data in a gentler language according to the user's settings. The input consists of the user's settings and the original text data. The text adjustment engine processes the text to be used in a way that is appropriate for the user's age and level of comprehension. Specifically, it reconstructs the text based on an appropriate language model. The output is text expressed in a gentle manner that suits the user's settings.

[0516] (Application Example 1)

[0517] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0518] To provide an interactive content experience, it is necessary to efficiently convert text information into audio and visual information and present it seamlessly to the user. However, traditional methods struggle with multilingual support and real-time image generation, limiting the user experience. Furthermore, establishing an intuitive interface that users can operate easily and accommodating users of different ages and languages ​​remains a challenge.

[0519] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0520] In this invention, the server includes language processing means for analyzing text data and dividing it into noun and punctuation units, speech generation means for generating audio data in multiple languages ​​based on the analyzed text data, and visual data preparation means for preparing data for image generation based on the analyzed text data. This enables users to seamlessly experience selected content through audio and visual information.

[0521] "Text data" refers to a collection of strings containing information written in natural language.

[0522] "Language processing means" refers to a device or software that has the function of analyzing text data and dividing it into nouns and punctuation marks.

[0523] "Speech generation means" refers to a device or software that has the function of generating speech data in different languages ​​based on analyzed text data.

[0524] "Visual data preparation means" refers to a device or software that has the function of preparing data for image generation based on analyzed text data.

[0525] "Image generation means" refers to a device or software that has the function of generating visual information corresponding to each scene in real time based on visual data.

[0526] "Synchronized output means" refers to a device or software that has the function of outputting audio data and generated visual information in a synchronized manner.

[0527] "Output control means" refers to a device or software that has the function of controlling the playback of audio and visual information through user operation.

[0528] "Language adjustment means" refers to a device or software that has the function of converting text data into simplified language according to the user's settings and displaying it.

[0529] The system that realizes this invention consists of a user terminal, a server, and a generative AI model. The user operates the terminal and can select desired content through the provided interface. Information about the selected content is transmitted to the server via the internet.

[0530] The server analyzes the received text data using pre-built natural language processing capabilities and divides it into nouns and punctuation marks. Based on the information obtained from this analysis, the server generates audio data in the selected language using speech generation capabilities. For this speech generation, speech synthesis software such as Google Cloud Text-to-Speech is used.

[0531] Next, the server uses visual data preparation means to prepare data for image generation from the analyzed text data. This includes identifying visual elements based on important nouns and phrases. Based on this visual data, a generative AI model such as DALL-E is used to generate visual information corresponding to each scene in real time.

[0532] The generated audio data and visual information are seamlessly integrated using a synchronous output mechanism and transmitted to the user's terminal. This allows the user to enjoy a rich content experience that appeals to both sight and sound. Furthermore, the user can control the playback of the audio and visual information at their preferred timing using an output control mechanism.

[0533] For example, if the story of "Little Red Riding Hood" is selected, the server will send a prompt to DALL-E to generate a scene in which "a character wearing a red hood walks through the forest." An example of this prompt would be, "Please generate an image of the scene in the forest where Little Red Riding Hood encounters the wolf."

[0534] This system configuration makes multilingual, visually and aurally interactive content easily accessible to users.

[0535] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0536] Step 1:

[0537] The user accesses the system using a terminal and selects the desired story from the content selection screen. This input data includes the story's title and detailed information. The selected content information is sent to the server.

[0538] Step 2:

[0539] The server retrieves the text data of the received content and performs analysis using natural language processing. The analysis divides the text into nouns and punctuation marks. This process generates the analyzed text data.

[0540] Step 3:

[0541] The server creates audio data using a speech generation method based on the analyzed text data. This process involves speech synthesis in the configured language, for example, using Google Cloud Text-to-Speech. The output of this step is an audio data file.

[0542] Step 4:

[0543] The server then uses the analyzed data to prepare data for image generation using a visual data preparation system. Specifically, it extracts important nouns and phrases and identifies visual elements based on them. The output of this process is a visual dataset.

[0544] Step 5:

[0545] The server passes visual data to a generation AI model (e.g., DALL-E) to form prompt sentences. Based on these prompt sentences, the AI ​​generates visual information corresponding to each scene. The generated image data is then output.

[0546] Step 6:

[0547] The generated audio data and visual information are integrated through a synchronous output means and transmitted to the user terminal. The terminal seamlessly plays the received audio and images, presenting the content to the user. The output of this step is a visual and auditory content experience.

[0548] Step 7:

[0549] Users can control audio and image playback using the device's output control mechanisms. Specific actions include starting and stopping playback, and skipping scenes. This allows users to freely customize their content experience.

[0550] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0551] This invention is a system that recognizes user emotions and provides an interactive experience based on that information. The user launches an application using a terminal and selects their preferred story or material. Based on this selection, the terminal sends the necessary data to the server.

[0552] The server analyzes the selected content using natural language processing and divides it into nouns and punctuation marks. Using these analysis results, the server activates a speech synthesis system to generate multilingual audio data. At this stage, the generated audio data is prepared for synchronized playback that will occur later.

[0553] Next, the analyzed text data is processed by the image generation preparation mechanism. The server prepares image generation data to clarify the visual elements for each scene. Based on this preparation data, the image generation mechanism generates images for each scene in real time.

[0554] In addition, the emotion engine runs on the device and recognizes the user's emotions in real time. This emotion data is sent to a server and reflected in the playback of audio and images. For example, if the user expresses surprise, some audio and images are adjusted based on the emotion recognized by the emotion engine. By emphasizing the surprised scene, a more immersive experience is provided.

[0555] The device uses all data received from the server to play audio and images in sync. Users can control playback through the device, pausing and skipping. A feature is also available where text data is converted into gentle language based on the user's emotions recognized by the emotion engine. This feature allows users to receive an optimal content experience tailored to their emotions at any given time.

[0556] By combining emotion recognition technology, this system further enhances the delivery of visual and auditory content, enabling personalized experiences for users.

[0557] The following describes the processing flow.

[0558] Step 1:

[0559] The user launches the application using their device and selects the stories or materials they wish to view. Based on this selection, the device sends content information and user settings to the server.

[0560] Step 2:

[0561] The server retrieves the text data of the selected story according to the received request. This text data is then analyzed using natural language processing techniques and divided into nouns and punctuation marks.

[0562] Step 3:

[0563] Based on the analysis results, the server generates audio data using a multilingual speech synthesis system. This audio data is configured according to specifications for later synchronous playback.

[0564] Step 4:

[0565] The server uses an image generation preparation mechanism to prepare the necessary image generation data based on the analyzed text data. This data defines the visual elements for each scene.

[0566] Step 5:

[0567] The server sends the completed audio data and the base data for image generation to the terminal.

[0568] Step 6:

[0569] The device activates its audio playback engine and plays the received audio data. The user is then provided with the audio of the story.

[0570] Step 7:

[0571] The device activates an image generation engine and generates images for each scene in real time from image generation data. The generated images are displayed to the user in sync with the audio.

[0572] Step 8:

[0573] An emotion engine operates on the device, recognizing the user's emotions in real time from their facial expressions and voice. The recognized emotion data is used by the server and for image and voice control.

[0574] Step 9:

[0575] Based on the recognized user's emotions, the server adjusts the content of audio and images. For example, if the user shows a sad expression, the system will be controlled to emphasize and play happy scenes.

[0576] Step 10:

[0577] Users can control playback, including pausing and skipping, using the device's interface. Furthermore, a text adjustment function that responds to emotions displays text in gentler language as needed. In this way, users enjoy a personalized, interactive experience.

[0578] (Example 2)

[0579] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0580] Traditional interactive content delivery systems have struggled to provide experiences that take user emotions into account. Therefore, the challenge lies in appropriately adjusting content according to user emotions to achieve a more personalized and immersive experience.

[0581] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0582] In this invention, the server includes a natural language processing means for analyzing text data and dividing it into nouns and punctuation units, a speech synthesis means for generating audio data in multiple languages ​​based on the analyzed text data, and an image generation preparation means for preparing data for image generation based on the analyzed text data. This enables a more personalized and interactive content experience by recognizing the user's emotions in real time and adjusting the audio and images based on those emotions.

[0583] "Natural language processing" is a technology that analyzes text data, divides it into elements such as words and punctuation, and understands the meaning and context of each element.

[0584] "Speech synthesis" is a technology that generates speech in a specified language based on text data.

[0585] "Image generation preparation" is the process of preparing the necessary visual information for each scene based on the analyzed text data.

[0586] "Image generation" is a technology in which a computer generates images corresponding to each scene in real time, based on prepared data.

[0587] "Synchronized playback" is a technology that plays generated audio and images simultaneously, providing users with a seamless viewing experience.

[0588] "Emotion recognition" is a technology that analyzes a user's facial expressions and tone of voice to identify their emotions in real time.

[0589] This invention is a system that provides an interactive content experience based on the user's emotions. The system mainly consists of terminals and servers, which work together in cooperation.

[0590] The terminal is a device used by the user to launch applications, and it allows the user to select content such as stories and documents using its operating interface. The content selected by the user is sent to the server via the network.

[0591] The server analyzes the received content using natural language processing tools (e.g., SpaCy, NLTK) and divides the text data into nouns and punctuation marks. Based on this analysis, multilingual audio data is generated by speech synthesis software (e.g., Amazon Polly, Google Cloud Text-to-Speech).

[0592] Furthermore, the server utilizes an image generation preparation engine to prepare scene-specific image generation data based on the analyzed text data. Then, it generates images in real time using an AI image generation model (e.g., DALL-E 2, Stable Diffusion). This process is executed by creating prompt statements. An example of a prompt statement is, "Recreate the story of the adventure in the magical forest with visuals and sound that enhance surprise and emotion."

[0593] Furthermore, the device recognizes the user's emotions in real time from their facial expressions and voice through an emotion engine (e.g., Microsoft Azure Emotion API). This emotion data is sent to a server and reflected in the audio and image data. For example, if the user expresses surprise, the audio tone and image effects of the corresponding scene are adjusted to provide a more immersive experience.

[0594] Finally, the device uses all the data to play audio and images in sync. Users can control playback, pausing and skipping. It also translates displayed text into gentle language based on the user's emotions, providing an optimal content experience tailored to the user.

[0595] This system configuration makes it possible to provide users with personalized, interactive entertainment.

[0596] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0597] Step 1:

[0598] The user launches the application on their device and selects stories or materials of interest through the provided interface. The user's selection is the input, and the device sends this selection information to the server in a structured format. The selected content data is obtained as output.

[0599] Step 2:

[0600] The server analyzes the received content data using a natural language processing engine (e.g., SpaCy or NLTK). The data received as input is in text format and is divided into nouns, verbs, and punctuation marks. This allows the grammatical structure to be analyzed, and structured text data is output.

[0601] Step 3:

[0602] The server generates audio data using a text-to-speech system (e.g., Amazon Polly, Google Cloud Text-to-Speech) based on the analysis results. The analyzed text data is used as input, and synthesized audio data in the specified language is output. This audio data is temporarily stored for later playback.

[0603] Step 4:

[0604] The server operates an image generation preparation engine based on text data, identifying visual elements for each scene. The input data is the result of text analysis, and it outputs data used as prompts for image generation. This output data is used as prompts for the image generation model.

[0605] Step 5:

[0606] The server generates images in real time using an AI image generation model (e.g., DALL-E 2, Stable Diffusion). Prompt data created during the generation preparation process serves as input, and image data corresponding to each scene is output. This image data is also stored, similar to the audio data.

[0607] Step 6:

[0608] The device uses a built-in emotion engine (e.g., Microsoft Azure Emotion API) to determine the user's emotions in real time from information obtained from the camera and microphone. User voice and facial expression data are used as input, and recognized emotion data is output. This emotion data is later reflected in the playback data.

[0609] Step 7:

[0610] The server receives emotion data sent by the user and reflects it in audio and image data. It performs adjustments such as volume and color adjustments based on emotion. Emotion data is used as input, and the adjusted audio and image data are output.

[0611] Step 8:

[0612] The terminal synchronizes and plays audio and image data sent from the server. User playback operations are used as input data, and playback controls such as pausing and skipping audio and video are output. At this stage, the user can also use an emotion-responsive text conversion function, which converts the text into gentler language.

[0613] (Application Example 2)

[0614] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0615] While modern entertainment experiences are incredibly rich in visual and auditory elements, providing interactive content that dynamically changes in response to individual users' emotions is still not fully achieved. In particular, tailoring content based on user emotions to deliver a personalized and immersive experience is a crucial challenge.

[0616] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0617] In this invention, the server includes language processing means for analyzing text data and dividing it into words and segments; speech generation means for generating audio information in multiple languages ​​based on the analyzed text data; and image preparation means for preparing visual data based on the analyzed text data. This enables a more personalized and interactive content experience by recognizing the user's emotions in real time and dynamically adjusting the audio and video according to those emotions.

[0618] "Language processing means" refers to technologies that analyze text data and divide the data according to words and boundaries.

[0619] "Speech generation means" refers to a technology that generates multilingual speech information based on analyzed text data.

[0620] "Image preparation means" refers to a technology that prepares visual data based on analyzed text data, thereby facilitating video generation.

[0621] "Image generation means" refers to a technology that generates video corresponding to each scene in real time based on visual data.

[0622] A "synchronous playback method" is a technology that outputs audio information and generated video simultaneously.

[0623] "Emotion recognition means" refers to technology that detects a user's emotions and acquires that information in real time.

[0624] "Emotional response technology" refers to a technology that dynamically adjusts audio and video information based on the user's emotional information.

[0625] "Playback control means" refers to technology that controls the playback of audio and video through user interface operations.

[0626] "Text adjustment means" refers to a technology that converts text data into a softer expression according to the user's settings and displays it accordingly.

[0627] To implement this invention, a server and a terminal are required. The terminal has a user interface and is used by the user to select content. The selected text data is sent to the server. The server uses language processing means to analyze the text data according to words and segments. The analysis results are converted into multilingual audio information by speech generation means. In addition, image preparation means are used to prepare visual data, and based on this, image generation means generate video in real time.

[0628] The device uses emotion recognition to analyze the user's facial expressions and voice, acquiring the user's emotions in real time. This emotion data is sent to a server, where emotion response mechanisms dynamically adjust the audio and video information according to the user's emotions. This enables a more immersive experience.

[0629] The audio and video being played are output simultaneously by the synchronized playback means, and the user can control their playback through the playback control means. For example, playback can be paused or skipped. In addition, the text adjustment means converts the text to a softer expression according to the user's settings and displays it.

[0630] For example, if a user who has selected horror content feels a level of fear, the background music will change and the video will become darker in accordance with their emotion, allowing the user to have a more intense fear experience. An example of a prompt message is, "If the user shows emotion of surprise, increase the volume."

[0631] This system integrates cutting-edge generative AI models, such as emotion recognition using TensorFlow, speech synthesis using Google TTS and AWS Polly, and image generation using Stable Diffusion, to provide users with personalized and interactive content experiences.

[0632] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0633] Step 1:

[0634] The terminal sends the content selected by the user to the server. The input is the story or material chosen by the user, and the output is the data sent to the server. The data processing performed here is the digitization and transmission of the content selection information.

[0635] Step 2:

[0636] The server uses language processing tools to analyze the received text data, dividing it into words and segments. The input is user-selected content data, and the output is the analyzed text data. Specifically, it structures the data using natural language processing.

[0637] Step 3:

[0638] The server generates multilingual audio information using a speech generation system based on the analyzed text data. The input is the analyzed text data, and the output is audio information. In this step, the speech synthesis engine is started and audio data is generated according to the language selection.

[0639] Step 4:

[0640] The server uses an image preparation mechanism to prepare visual data from text data. The input is the parsed text data, and the output is visual data for image generation. Specifically, it constructs the data for the visual elements required for each scene.

[0641] Step 5:

[0642] The server generates video in real time based on visual data using an image generation mechanism. The input is visual data, and the output is the generated video. This step involves deploying an image generation model and generating the required visuals.

[0643] Step 6:

[0644] The device uses emotion recognition technology to analyze the user's facial expressions and voice, detecting emotions in real time. The input is real-time user data, and the output is emotional information. Specifically, it measures emotional data through the camera and microphone and uses a model to infer it.

[0645] Step 7:

[0646] The server adjusts audio and video information using emotion-responsive mechanisms based on emotional information. The input is recognized emotional information, and the output is adjusted audio and video. This operation includes adjusting the tone of the audio and changing the brightness of the video.

[0647] Step 8:

[0648] The device simultaneously outputs synchronized audio and video using a synchronized playback mechanism. The input is the synchronized audio and video data, and the output is the user's viewing experience. Specific operations include synchronizing the audio and video in time before outputting them.

[0649] Step 9:

[0650] The user controls audio and video playback as desired through the interface using playback control means. Input is user operation, and output is the response of the playback control. This operation includes controls such as pause, skip, and resume.

[0651] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0652] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0653] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0654] [Fourth Embodiment]

[0655] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0656] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0657] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0658] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0659] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0660] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0661] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0662] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0663] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0664] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0665] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0666] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0667] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] This invention is an interactive system that reads aloud stories or materials selected by the user and displays automatically generated images based on that content. The user accesses this system using a terminal and selects their preferred content. After selecting content, the terminal sends the selected content and associated language information to the server.

[0669] The server analyzes the selected story text using natural language processing techniques, dividing it into nouns and punctuation marks. Based on each element of the analyzed text, the server uses speech synthesis techniques to generate audio data in the user's specified language. The audio data generated at this stage is ready for use in subsequent processes.

[0670] Next, the server uses the image generation preparation means to prepare image generation data based on the analyzed text data. This data is used to define the visual elements of each scene based on nouns and important phrases. Based on this prepared data, the image generation means generates images corresponding to each scene in real time.

[0671] The device receives audio files and image generation data from the server. A synchronized playback mechanism for audio and images is activated, allowing the user to enjoy a seamless narrative experience both visually and aurally. For example, in the story of "Momotaro," when the user makes a selection, the device can sequentially display scenes such as the peach floating in the river, along with an audio reading of "Momotaro." Audio and image playback can be controlled by the user, allowing them to stop or skip playback.

[0672] Furthermore, for users who require explanations in simpler language, the system allows for text adjustment to convert the content into age-appropriate language. This system enriches stories and materials both visually and aurally, enabling users to experience and learn from a wider variety of content.

[0673] The following describes the processing flow.

[0674] Step 1:

[0675] The user launches the application using their device and selects the stories or materials they wish to view. The device then sends the selected content and the set language information to the server.

[0676] Step 2:

[0677] The server retrieves the text data of the selected story based on the received request. This text data is then analyzed using natural language processing techniques and divided into nouns and punctuation marks.

[0678] Step 3:

[0679] The server generates audio data in the specified language using speech synthesis based on the analyzed text data. The generated audio data corresponds to each element of the story.

[0680] Step 4:

[0681] The server uses image generation preparation means to prepare base data for image generation based on the analyzed text data. This data includes the visual elements required for each scene.

[0682] Step 5:

[0683] The server sends audio data and base data for image generation to the terminal.

[0684] Step 6:

[0685] The terminal activates its audio playback engine to play the received audio data and begins outputting audio.

[0686] Step 7:

[0687] The device runs an image generation engine and, based on image generation data, generates images for each scene in real time and displays them to the user. By synchronizing with audio playback, it provides a seamless narrative experience.

[0688] Step 8:

[0689] Users control audio and image playback through their device. They can pause and skip playback, among other actions. Additionally, a feature is available to convert text data into simplified language for easier understanding, if needed.

[0690] (Example 1)

[0691] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0692] There is a need for systems that allow users to experience stories and materials in a rich auditory and visual way, but conventional technology has made it difficult to reproduce story content in real time using audio and video. Furthermore, there has been a lack of interfaces that allow users to enjoy stories at their own pace, as well as functions to translate the content into simpler language suitable for specific age groups. This has hindered a wider range of users from intuitively and comfortably enjoying the information.

[0693] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0694] In this invention, the server includes a natural language processing means for analyzing text data and dividing it into nouns and punctuation units, a speech synthesis means for generating audio data in multiple languages ​​based on the analyzed text data, and an image generation means for generating images corresponding to each scene in real time based on data for image generation. This allows users to enjoy stories and materials aurally and visually in real time.

[0695] "Natural language processing techniques" are technologies that analyze text data and divide it into constituent units such as nouns and punctuation marks.

[0696] "Speech synthesis means" refers to a technology that generates speech data in multiple languages ​​based on analyzed text data.

[0697] "Image generation data preparation method" refers to a technology that prepares the data necessary for image generation based on analyzed text data.

[0698] "Image generation means" refers to a technology that generates images corresponding to each scene in real time based on image generation data.

[0699] A "synchronous playback method" is a technology that outputs generated audio data and images in sync.

[0700] A "selection interface" is a user interface that allows users to easily select stories or materials.

[0701] A "prompt generation method" is a technology that automatically generates prompt text for image generation based on nouns and important phrases.

[0702] "Playback control means" refers to a function that controls the playback of audio and images through user operation.

[0703] "Text adjustment means" refers to a technology that converts and displays text data in a gentler language according to the user's settings.

[0704] This invention is a system that generates and displays audio and images in real time based on a user's selection of stories or materials. The user can access this system using a general-purpose terminal device. The terminal communicates with the server and transmits the necessary data according to the user's selection.

[0705] The server uses natural language processing techniques to analyze the received text data. Specifically, it uses the Python language and the Natural Language Toolkit (NLTK) library to divide the text into units such as nouns and punctuation marks. For example, in the story of "Momotaro," it can extract nouns such as "Momotaro," "peach," and "river."

[0706] Next, the server utilizes speech synthesis technology to generate audio data based on the analyzed text data. It uses a speech synthesis engine such as the Google Text-to-Speech API to create an audio file in the specified language.

[0707] Furthermore, the server activates the prompt generation mechanism using the image generation data preparation mechanism. It generates prompt sentences for image generation from nouns and important phrases and sends them to the generation AI model. Utilizing OpenAI's DALL-E model, it generates images in real time based on the prompt sentences. For example, an image is generated using a prompt sentence based on the scene of "peaches floating in a river."

[0708] As a concrete example, a prompt based on the story of Momotaro would be presented as follows: "Read the story of Momotaro aloud and generate images based on each key scene. Use nouns and important phrases to determine the visual elements."

[0709] The generated audio data and images are transmitted to the terminal and provided to the user via a synchronized playback mechanism. The terminal plays the audio and images in sync, providing the user with a seamless narrative experience through both sight and sound. A playback control mechanism is also included, allowing the user to stop or skip playback. Furthermore, a text adjustment mechanism allows the content to be displayed in user-friendly language according to user preferences.

[0710] This system allows users to experience the story's content through multiple senses, providing them with educational and entertaining value.

[0711] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0712] Step 1:

[0713] The user selects the content they want to read from stories and materials through the user interface on their device. The user's content selection information is received as input. The device sends the selected content information to the server. A request containing metadata for the selected content is generated as output.

[0714] Step 2:

[0715] The server analyzes the received text data using natural language processing techniques. The input is text data sent from the terminal. The server utilizes the Python language and the NLTK library to divide the text into nouns and punctuation units. Specifically, it tokenizes the text and performs syntactic analysis. The output is the analyzed text data.

[0716] Step 3:

[0717] The server generates audio data using speech synthesis based on the parsed text data. The parsed text data is used as input. A speech synthesis engine (e.g., Google Text-to-Speech API) is used to create an audio file in the specified language. Specifically, the process involves calling the speech synthesis engine's API to generate the audio file. The output is the generated audio data file.

[0718] Step 4:

[0719] The server uses image generation data preparation means to generate prompt sentences for image generation based on the analyzed text data, utilizing prompt generation means. The input is the analyzed text data. Nouns and important phrases are extracted, and prompt sentences are formed based on these. Specifically, the prompt sentences are created by embedding templates based on the text analysis results. The output is the generated prompt sentences.

[0720] Step 5:

[0721] The server uses a generative AI model to generate images in real time based on prompt text, representing various scenarios. The input is the generated prompt text. It utilizes OpenAI's DALL-E model to generate image data from the prompt text. Specifically, it sends the prompt text to the generative AI model and retrieves the returned image data. The output is the generated image data.

[0722] Step 6:

[0723] The terminal receives audio and image data files sent from the server and plays them in sync using a synchronous playback mechanism. The input consists of audio and image data files. The playback application on the terminal presents the audio and images to the user in sync. The output allows the user to experience the content visually and aurally. The specific operation involves adjusting the data stream and starting playback in a synchronous state.

[0724] Step 7:

[0725] The user controls audio and image playback using the device's playback control mechanisms. Input is the user's action events (e.g., start playback, stop playback, skip). The playback control function is used to adjust the timing of audio and images, enabling display at the user's pace. Specifically, the operation involves changing the playback state in response to the user's input events. Output is the display of content according to the playback sequence desired by the user.

[0726] Step 8:

[0727] The server uses text adjustment tools to convert and display text data in a gentler language according to the user's settings. The input consists of the user's settings and the original text data. The text adjustment engine processes the text to be used in a way that is appropriate for the user's age and level of comprehension. Specifically, it reconstructs the text based on an appropriate language model. The output is text expressed in a gentle manner that suits the user's settings.

[0728] (Application Example 1)

[0729] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0730] To provide an interactive content experience, it is necessary to efficiently convert text information into audio and visual information and present it seamlessly to the user. However, traditional methods struggle with multilingual support and real-time image generation, limiting the user experience. Furthermore, establishing an intuitive interface that users can operate easily and accommodating users of different ages and languages ​​remains a challenge.

[0731] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0732] In this invention, the server includes language processing means for analyzing text data and dividing it into noun and punctuation units, speech generation means for generating audio data in multiple languages ​​based on the analyzed text data, and visual data preparation means for preparing data for image generation based on the analyzed text data. This enables users to seamlessly experience selected content through audio and visual information.

[0733] "Text data" refers to a collection of strings containing information written in natural language.

[0734] "Language processing means" refers to a device or software that has the function of analyzing text data and dividing it into nouns and punctuation marks.

[0735] "Speech generation means" refers to a device or software that has the function of generating speech data in different languages ​​based on analyzed text data.

[0736] "Visual data preparation means" refers to a device or software that has the function of preparing data for image generation based on analyzed text data.

[0737] "Image generation means" refers to a device or software that has the function of generating visual information corresponding to each scene in real time based on visual data.

[0738] "Synchronized output means" refers to a device or software that has the function of outputting audio data and generated visual information in a synchronized manner.

[0739] "Output control means" refers to a device or software that has the function of controlling the playback of audio and visual information through user operation.

[0740] "Language adjustment means" refers to a device or software that has the function of converting text data into simplified language according to the user's settings and displaying it.

[0741] The system that realizes this invention consists of a user terminal, a server, and a generative AI model. The user operates the terminal and can select desired content through the provided interface. Information about the selected content is transmitted to the server via the internet.

[0742] The server analyzes the received text data using pre-built natural language processing capabilities and divides it into nouns and punctuation marks. Based on the information obtained from this analysis, the server generates audio data in the selected language using speech generation capabilities. For this speech generation, speech synthesis software such as Google Cloud Text-to-Speech is used.

[0743] Next, the server uses visual data preparation means to prepare data for image generation from the analyzed text data. This includes identifying visual elements based on important nouns and phrases. Based on this visual data, a generative AI model such as DALL-E is used to generate visual information corresponding to each scene in real time.

[0744] The generated audio data and visual information are seamlessly integrated using a synchronous output mechanism and transmitted to the user's terminal. This allows the user to enjoy a rich content experience that appeals to both sight and sound. Furthermore, the user can control the playback of the audio and visual information at their preferred timing using an output control mechanism.

[0745] For example, if the story of "Little Red Riding Hood" is selected, the server will send a prompt to DALL-E to generate a scene in which "a character wearing a red hood walks through the forest." An example of this prompt would be, "Please generate an image of the scene in the forest where Little Red Riding Hood encounters the wolf."

[0746] This system configuration makes multilingual, visually and aurally interactive content easily accessible to users.

[0747] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0748] Step 1:

[0749] The user accesses the system using a terminal and selects the desired story from the content selection screen. This input data includes the story's title and detailed information. The selected content information is sent to the server.

[0750] Step 2:

[0751] The server retrieves the text data of the received content and performs analysis using natural language processing. The analysis divides the text into nouns and punctuation marks. This process generates the analyzed text data.

[0752] Step 3:

[0753] The server creates audio data using a speech generation method based on the analyzed text data. This process involves speech synthesis in the configured language, for example, using Google Cloud Text-to-Speech. The output of this step is an audio data file.

[0754] Step 4:

[0755] The server then uses the analyzed data to prepare data for image generation using a visual data preparation system. Specifically, it extracts important nouns and phrases and identifies visual elements based on them. The output of this process is a visual dataset.

[0756] Step 5:

[0757] The server passes visual data to a generation AI model (e.g., DALL-E) to form prompt sentences. Based on these prompt sentences, the AI ​​generates visual information corresponding to each scene. The generated image data is then output.

[0758] Step 6:

[0759] The generated audio data and visual information are integrated through a synchronous output means and transmitted to the user terminal. The terminal seamlessly plays the received audio and images, presenting the content to the user. The output of this step is a visual and auditory content experience.

[0760] Step 7:

[0761] Users can control audio and image playback using the device's output control mechanisms. Specific actions include starting and stopping playback, and skipping scenes. This allows users to freely customize their content experience.

[0762] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0763] This invention is a system that recognizes user emotions and provides an interactive experience based on that information. The user launches an application using a terminal and selects their preferred story or material. Based on this selection, the terminal sends the necessary data to the server.

[0764] The server analyzes the selected content using natural language processing and divides it into nouns and punctuation marks. Using these analysis results, the server activates a speech synthesis system to generate multilingual audio data. At this stage, the generated audio data is prepared for synchronized playback that will occur later.

[0765] Next, the analyzed text data is processed by the image generation preparation mechanism. The server prepares image generation data to clarify the visual elements for each scene. Based on this preparation data, the image generation mechanism generates images for each scene in real time.

[0766] In addition, the emotion engine runs on the device and recognizes the user's emotions in real time. This emotion data is sent to a server and reflected in the playback of audio and images. For example, if the user expresses surprise, some audio and images are adjusted based on the emotion recognized by the emotion engine. By emphasizing the surprised scene, a more immersive experience is provided.

[0767] The device uses all data received from the server to play audio and images in sync. Users can control playback through the device, pausing and skipping. A feature is also available where text data is converted into gentle language based on the user's emotions recognized by the emotion engine. This feature allows users to receive an optimal content experience tailored to their emotions at any given time.

[0768] By combining emotion recognition technology, this system further enhances the delivery of visual and auditory content, enabling personalized experiences for users.

[0769] The following describes the processing flow.

[0770] Step 1:

[0771] The user launches the application using their device and selects the stories or materials they wish to view. Based on this selection, the device sends content information and user settings to the server.

[0772] Step 2:

[0773] The server retrieves the text data of the selected story according to the received request. This text data is then analyzed using natural language processing techniques and divided into nouns and punctuation marks.

[0774] Step 3:

[0775] Based on the analysis results, the server generates audio data using a multilingual speech synthesis system. This audio data is configured according to specifications for later synchronous playback.

[0776] Step 4:

[0777] The server uses an image generation preparation mechanism to prepare the necessary image generation data based on the analyzed text data. This data defines the visual elements for each scene.

[0778] Step 5:

[0779] The server sends the completed audio data and the base data for image generation to the terminal.

[0780] Step 6:

[0781] The device activates its audio playback engine and plays the received audio data. The user is then provided with the audio of the story.

[0782] Step 7:

[0783] The device activates an image generation engine and generates images for each scene in real time from image generation data. The generated images are displayed to the user in sync with the audio.

[0784] Step 8:

[0785] An emotion engine operates on the device, recognizing the user's emotions in real time from their facial expressions and voice. The recognized emotion data is used by the server and for image and voice control.

[0786] Step 9:

[0787] Based on the recognized user's emotions, the server adjusts the content of audio and images. For example, if the user shows a sad expression, the system will be controlled to emphasize and play happy scenes.

[0788] Step 10:

[0789] Users can control playback, including pausing and skipping, using the device's interface. Furthermore, a text adjustment function that responds to emotions displays text in gentler language as needed. In this way, users enjoy a personalized, interactive experience.

[0790] (Example 2)

[0791] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0792] Traditional interactive content delivery systems have struggled to provide experiences that take user emotions into account. Therefore, the challenge lies in appropriately adjusting content according to user emotions to achieve a more personalized and immersive experience.

[0793] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0794] In this invention, the server includes a natural language processing means for analyzing text data and dividing it into nouns and punctuation units, a speech synthesis means for generating audio data in multiple languages ​​based on the analyzed text data, and an image generation preparation means for preparing data for image generation based on the analyzed text data. This enables a more personalized and interactive content experience by recognizing the user's emotions in real time and adjusting the audio and images based on those emotions.

[0795] "Natural language processing" is a technology that analyzes text data, divides it into elements such as words and punctuation, and understands the meaning and context of each element.

[0796] "Speech synthesis" is a technology that generates speech in a specified language based on text data.

[0797] "Image generation preparation" is the process of preparing the necessary visual information for each scene based on the analyzed text data.

[0798] "Image generation" is a technology in which a computer generates images corresponding to each scene in real time, based on prepared data.

[0799] "Synchronized playback" is a technology that plays generated audio and images simultaneously, providing users with a seamless viewing experience.

[0800] "Emotion recognition" is a technology that analyzes a user's facial expressions and tone of voice to identify their emotions in real time.

[0801] This invention is a system that provides an interactive content experience based on the user's emotions. The system mainly consists of terminals and servers, which work together in cooperation.

[0802] The terminal is a device used by the user to launch applications, and it allows the user to select content such as stories and documents using its operating interface. The content selected by the user is sent to the server via the network.

[0803] The server analyzes the received content using natural language processing tools (e.g., SpaCy, NLTK) and divides the text data into nouns and punctuation marks. Based on this analysis, multilingual audio data is generated by speech synthesis software (e.g., Amazon Polly, Google Cloud Text-to-Speech).

[0804] Furthermore, the server utilizes an image generation preparation engine to prepare scene-specific image generation data based on the analyzed text data. Then, it generates images in real time using an AI image generation model (e.g., DALL-E 2, Stable Diffusion). This process is executed by creating prompt statements. An example of a prompt statement is, "Recreate the story of the adventure in the magical forest with visuals and sound that enhance surprise and emotion."

[0805] Furthermore, the device recognizes the user's emotions in real time from their facial expressions and voice through an emotion engine (e.g., Microsoft Azure Emotion API). This emotion data is sent to a server and reflected in the audio and image data. For example, if the user expresses surprise, the audio tone and image effects of the corresponding scene are adjusted to provide a more immersive experience.

[0806] Finally, the device uses all the data to play audio and images in sync. Users can control playback, pausing and skipping. It also translates displayed text into gentle language based on the user's emotions, providing an optimal content experience tailored to the user.

[0807] This system configuration makes it possible to provide users with personalized, interactive entertainment.

[0808] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0809] Step 1:

[0810] The user launches the application on their device and selects stories or materials of interest through the provided interface. The user's selection is the input, and the device sends this selection information to the server in a structured format. The selected content data is obtained as output.

[0811] Step 2:

[0812] The server analyzes the received content data using a natural language processing engine (e.g., SpaCy or NLTK). The data received as input is in text format and is divided into nouns, verbs, and punctuation marks. This allows the grammatical structure to be analyzed, and structured text data is output.

[0813] Step 3:

[0814] The server generates audio data using a text-to-speech system (e.g., Amazon Polly, Google Cloud Text-to-Speech) based on the analysis results. The analyzed text data is used as input, and synthesized audio data in the specified language is output. This audio data is temporarily stored for later playback.

[0815] Step 4:

[0816] The server operates an image generation preparation engine based on text data, identifying visual elements for each scene. The input data is the result of text analysis, and it outputs data used as prompts for image generation. This output data is used as prompts for the image generation model.

[0817] Step 5:

[0818] The server generates images in real time using an AI image generation model (e.g., DALL-E 2, Stable Diffusion). Prompt data created during the generation preparation process serves as input, and image data corresponding to each scene is output. This image data is also stored, similar to the audio data.

[0819] Step 6:

[0820] The device uses a built-in emotion engine (e.g., Microsoft Azure Emotion API) to determine the user's emotions in real time from information obtained from the camera and microphone. User voice and facial expression data are used as input, and recognized emotion data is output. This emotion data is later reflected in the playback data.

[0821] Step 7:

[0822] The server receives emotion data sent by the user and reflects it in audio and image data. It performs adjustments such as volume and color adjustments based on emotion. Emotion data is used as input, and the adjusted audio and image data are output.

[0823] Step 8:

[0824] The terminal synchronizes and plays audio and image data sent from the server. User playback operations are used as input data, and playback controls such as pausing and skipping audio and video are output. At this stage, the user can also use an emotion-responsive text conversion function, which converts the text into gentler language.

[0825] (Application Example 2)

[0826] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0827] While modern entertainment experiences are incredibly rich in visual and auditory elements, providing interactive content that dynamically changes in response to individual users' emotions is still not fully achieved. In particular, tailoring content based on user emotions to deliver a personalized and immersive experience is a crucial challenge.

[0828] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0829] In this invention, the server includes language processing means for analyzing text data and dividing it into words and segments; speech generation means for generating audio information in multiple languages ​​based on the analyzed text data; and image preparation means for preparing visual data based on the analyzed text data. This enables a more personalized and interactive content experience by recognizing the user's emotions in real time and dynamically adjusting the audio and video according to those emotions.

[0830] "Language processing means" refers to technologies that analyze text data and divide the data according to words and boundaries.

[0831] "Speech generation means" refers to a technology that generates multilingual speech information based on analyzed text data.

[0832] "Image preparation means" refers to a technology that prepares visual data based on analyzed text data, thereby facilitating video generation.

[0833] "Image generation means" refers to a technology that generates video corresponding to each scene in real time based on visual data.

[0834] A "synchronous playback method" is a technology that outputs audio information and generated video simultaneously.

[0835] "Emotion recognition means" refers to technology that detects a user's emotions and acquires that information in real time.

[0836] "Emotional response technology" refers to a technology that dynamically adjusts audio and video information based on the user's emotional information.

[0837] "Playback control means" refers to technology that controls the playback of audio and video through user interface operations.

[0838] "Text adjustment means" refers to a technology that converts text data into a softer expression according to the user's settings and displays it accordingly.

[0839] To implement this invention, a server and a terminal are required. The terminal has a user interface and is used by the user to select content. The selected text data is sent to the server. The server uses language processing means to analyze the text data according to words and segments. The analysis results are converted into multilingual audio information by speech generation means. In addition, image preparation means are used to prepare visual data, and based on this, image generation means generate video in real time.

[0840] The device uses emotion recognition to analyze the user's facial expressions and voice, acquiring the user's emotions in real time. This emotion data is sent to a server, where emotion response mechanisms dynamically adjust the audio and video information according to the user's emotions. This enables a more immersive experience.

[0841] The audio and video being played are output simultaneously by the synchronized playback means, and the user can control their playback through the playback control means. For example, playback can be paused or skipped. In addition, the text adjustment means converts the text to a softer expression according to the user's settings and displays it.

[0842] For example, if a user who has selected horror content feels a level of fear, the background music will change and the video will become darker in accordance with their emotion, allowing the user to have a more intense fear experience. An example of a prompt message is, "If the user shows emotion of surprise, increase the volume."

[0843] This system integrates cutting-edge generative AI models, such as emotion recognition using TensorFlow, speech synthesis using Google TTS and AWS Polly, and image generation using Stable Diffusion, to provide users with personalized and interactive content experiences.

[0844] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0845] Step 1:

[0846] The terminal sends the content selected by the user to the server. The input is the story or material chosen by the user, and the output is the data sent to the server. The data processing performed here is the digitization and transmission of the content selection information.

[0847] Step 2:

[0848] The server uses language processing tools to analyze the received text data, dividing it into words and segments. The input is user-selected content data, and the output is the analyzed text data. Specifically, it structures the data using natural language processing.

[0849] Step 3:

[0850] The server generates multilingual audio information using a speech generation system based on the analyzed text data. The input is the analyzed text data, and the output is audio information. In this step, the speech synthesis engine is started and audio data is generated according to the language selection.

[0851] Step 4:

[0852] The server uses an image preparation mechanism to prepare visual data from text data. The input is the parsed text data, and the output is visual data for image generation. Specifically, it constructs the data for the visual elements required for each scene.

[0853] Step 5:

[0854] The server generates video in real time based on visual data using an image generation mechanism. The input is visual data, and the output is the generated video. This step involves deploying an image generation model and generating the required visuals.

[0855] Step 6:

[0856] The device uses emotion recognition technology to analyze the user's facial expressions and voice, detecting emotions in real time. The input is real-time user data, and the output is emotional information. Specifically, it measures emotional data through the camera and microphone and uses a model to infer it.

[0857] Step 7:

[0858] The server adjusts audio and video information using emotion-responsive mechanisms based on emotional information. The input is recognized emotional information, and the output is adjusted audio and video. This operation includes adjusting the tone of the audio and changing the brightness of the video.

[0859] Step 8:

[0860] The device simultaneously outputs synchronized audio and video using a synchronized playback mechanism. The input is the synchronized audio and video data, and the output is the user's viewing experience. Specific operations include synchronizing the audio and video in time before outputting them.

[0861] Step 9:

[0862] The user controls audio and video playback as desired through the interface using playback control means. Input is user operation, and output is the response of the playback control. This operation includes controls such as pause, skip, and resume.

[0863] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0864] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0865] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0866] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0867] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0868] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0869] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0870] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0871] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0872] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0873] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0874] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0875] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0876] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0877] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0878] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0879] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0880] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0881] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0882] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0883] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0884] The following is further disclosed regarding the embodiments described above.

[0885] (Claim 1)

[0886] A natural language processing means that analyzes text data and divides it into nouns and punctuation units,

[0887] A speech synthesis means that generates speech data in multiple languages ​​based on analyzed text data,

[0888] Image generation preparation means for preparing data for image generation based on analyzed text data,

[0889] An image generation means that generates images corresponding to each scene in real time based on data for image generation,

[0890] A synchronized playback means that outputs audio data and generated images in sync,

[0891] A system that includes this.

[0892] (Claim 2)

[0893] The system according to claim 1, comprising playback control means for controlling the playback of audio and images through user interface operations.

[0894] (Claim 3)

[0895] The system according to claim 1, comprising a text adjustment means for converting and displaying text data in a user-friendly language according to user settings.

[0896] "Example 1"

[0897] (Claim 1)

[0898] A natural language processing means that analyzes text data and divides it into nouns and punctuation units,

[0899] A speech synthesis means that generates speech data in multiple languages ​​based on analyzed text data,

[0900] Image generation data preparation means for preparing data for image generation based on analyzed text data,

[0901] An image generation means that generates images corresponding to each scene in real time based on data for image generation,

[0902] A synchronized playback means that outputs audio data and generated images in sync,

[0903] A selection interface for users to choose stories and materials,

[0904] A prompt generation means that automatically generates prompt sentences for image generation based on nouns and important phrases,

[0905] A system that includes this.

[0906] (Claim 2)

[0907] The system according to claim 1, comprising playback control means for controlling the playback of audio and images through user interface operations.

[0908] (Claim 3)

[0909] The system according to claim 1, comprising a text adjustment means for converting and displaying text data in a user-friendly language according to user settings.

[0910] "Application Example 1"

[0911] (Claim 1)

[0912] A language processing means that analyzes text data and divides it into noun and punctuation units,

[0913] A speech generation means that generates speech data in multiple languages ​​based on analyzed text data,

[0914] A visual data preparation means that prepares data for image generation based on analyzed text data,

[0915] An image generation means that generates visual information corresponding to each scene in real time based on visual data,

[0916] A synchronized output means that outputs audio data and generated visual information in a synchronized manner,

[0917] A system that includes this.

[0918] (Claim 2)

[0919] The system according to claim 1, comprising output control means for controlling the playback of audio and visual information by user operation.

[0920] (Claim 3)

[0921] The system according to claim 1, comprising a language adjustment means for converting and displaying text data in a user-friendly language according to user settings.

[0922] "Example 2 of combining an emotion engine"

[0923] (Claim 1)

[0924] A natural language processing means that analyzes text data and divides it into nouns and punctuation units,

[0925] A speech synthesis means that generates speech data in multiple languages ​​based on analyzed text data,

[0926] Image generation preparation means for preparing data for image generation based on analyzed text data,

[0927] An image generation means that generates images corresponding to each scene in real time based on data for image generation,

[0928] A synchronized playback means that outputs audio data and generated images in sync,

[0929] An emotion recognition means that recognizes the user's emotions in real time, analyzes the emotion data, and reflects it in audio and images,

[0930] A system that includes this.

[0931] (Claim 2)

[0932] The system according to claim 1, comprising playback control means for controlling the playback of audio and images via a user interface.

[0933] (Claim 3)

[0934] The system according to claim 1, comprising a text adjustment means for converting and displaying text data in a user-friendly language according to user settings.

[0935] "Application example 2 when combining with an emotional engine"

[0936] (Claim 1)

[0937] A language processing means that analyzes text data and divides it into words and segment units,

[0938] A speech generation means that generates speech information in multiple languages ​​based on analyzed text data,

[0939] Image preparation means for preparing visual data based on analyzed text data,

[0940] An image generation means that dynamically generates images corresponding to each scene based on visual data,

[0941] A synchronized playback means that outputs audio information and generated video in sync,

[0942] A means for detecting the user's emotions,

[0943] An emotion-responsive means that adjusts audio and video information based on the user's emotional information,

[0944] A system that includes this.

[0945] (Claim 2)

[0946] The system according to claim 1, comprising playback control means for controlling the playback of audio and video by user interface operation.

[0947] (Claim 3)

[0948] The system according to claim 1, comprising text adjustment means for converting and displaying text data in a softer style according to user settings. [Explanation of Symbols]

[0949] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A natural language processing means that analyzes text data and divides it into nouns and punctuation units, A speech synthesis means that generates speech data in multiple languages ​​based on analyzed text data, Image generation preparation means for preparing data for image generation based on analyzed text data, An image generation means that generates images corresponding to each scene in real time based on data for image generation, A synchronized playback means that outputs audio data and generated images in sync, A system that includes this.

2. The system according to claim 1, comprising playback control means for controlling the playback of audio and images through user interface operations.

3. The system according to claim 1, comprising a text adjustment means for converting and displaying text data in a user-friendly language according to user settings.

Images