system

Abstract

We provide the system. [Solution] Interface means for receiving prompts, A generation unit means that automatically generates presentation materials, speech scripts, and demo videos using a generation model based on received prompts, A provisioning unit means that provides generated content to the user, A video receiving unit means for receiving practice videos taken by the user, An analysis unit means analyzes the received practice video and generates feedback on audio and visual performance, A training system that generates instructional videos including feedback and provides them to users, A system that includes this.

Description

Technical Field

[0004] , , ,

[0005] , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Speeches and presentations are important communication means for individuals and companies, but there are problems that preparing them takes a great deal of time and effort. Also, since receiving professional guidance is costly, there is a problem that it cannot be easily used when needed. Furthermore, there is also an aspect that it is difficult to create materials suitable for TPO, manuscripts that move the heart, and preparation of assumed Q&A sets. There is a desire to provide a system that solves these problems and efficiently and effectively prepares and improves speeches and presentations.

Means for Solving the Problems

[0005] This invention provides a system comprising an interface unit for receiving prompts and a generation unit that automatically generates presentation materials, speech scripts, and demo videos using a generation model based on the received prompts. This system also includes an analysis unit and a teaching unit that receive practice videos from the user, analyze the videos to generate feedback, and create instructional videos. Furthermore, by providing the generated content to the user, it is possible to effectively support the preparation and improvement of presentations. Thus, the solution of this invention is to efficiently support the improvement of presentation skills without requiring specialized instruction.

[0006] A "prompt" is text information that a user enters to specify the purpose and content of a presentation.

[0007] The "interface unit" is the part that has communication functions for receiving prompts and videos from the user and sending them to the server.

[0008] A "generative model" is an AI algorithm that automatically creates presentation materials, speech scripts, and demo videos based on input prompts.

[0009] The "generation unit" is the part that uses a generative model to generate presentation materials, speech scripts, and demo videos from prompts.

[0010] The "delivery section" refers to the part that has the function of distributing or displaying the generated content to the user.

[0011] The "video receiving unit" is the part that receives practice videos taken by the user and sends them to the server.

[0012] The "analysis unit" is the part that analyzes the received practice videos and performs evaluations regarding audio and visual performance.

[0013] "Feedback" refers to information that provides specific advice and suggestions for improving the user's presentation skills, based on the video analysis results.

[0014] The "Instruction Department" is the part of the system that creates instructional videos based on the feedback generated by the Analysis Department and provides them to users. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention is a system that automatically generates presentation materials, speech scripts, and demo videos based on user prompt input, and provides feedback through analysis of practice videos recorded by the user. This system mainly consists of three elements: a server, a terminal, and the user.

[0037] First, the user uses a terminal to enter prompts that include the presentation's theme and objectives. These prompts play a crucial role in providing specific information based on the user's needs. Next, the terminal sends these prompts to the server. Based on the received prompts, the server uses a generative model to generate presentation materials, a speech script, and a demo video. This allows the user to quickly receive content tailored to their needs.

[0038] When users practice using generated materials and scripts, they record their presentations with their devices. These practice videos are important data for improving user performance. The devices upload the recorded practice videos to the server.

[0039] Uploaded videos are analyzed by an analysis unit on the server, which examines items such as volume, speaking speed, gaze, and posture. Based on this analysis, the server generates feedback. This feedback includes specific suggestions and advice to improve the user's presentation skills.

[0040] Ultimately, the server creates instructional videos based on the generated feedback and provides them to the user via the terminal. These instructional videos contain advice that can help improve the user's performance, allowing them to enhance the quality of their next presentation.

[0041] As a concrete example, consider preparing a speech for a wedding. The user enters the prompt "wedding speech groom's friend," and the server generates a speech script and slides based on that information. When the user practices the speech and uploads a video recorded on their device to the server, feedback is generated regarding the volume of their voice and their demeanor. The user can then use this feedback to practice further and make improvements.

[0042] This system allows users to effectively prepare and improve their presentations themselves, eliminating the need to rely on expensive professional guidance.

[0043] The following describes the processing flow.

[0044] Step 1:

[0045] The user will use their device to enter detailed information about the purpose and theme of the presentation as prompts.

[0046] Step 2:

[0047] The terminal receives the entered prompt, formats it, and prepares to send it to the server.

[0048] Step 3:

[0049] The terminal sends the prepared prompt data to the server.

[0050] Step 4:

[0051] The server analyzes the received prompt data and passes it to the generative model as input data.

[0052] Step 5:

[0053] The server automatically generates presentation materials, speech scripts, and demo videos based on prompts using a generative model.

[0054] Step 6:

[0055] The server organizes the generated content and sends it to the terminal.

[0056] Step 7:

[0057] The terminal provides the user with content received from the server and displays it so that the user can review the content.

[0058] Step 8:

[0059] Users review the generated materials and drafts and practice their presentations based on them.

[0060] Step 9:

[0061] Users record the practice process on their devices.

[0062] Step 10:

[0063] The user selects the best practice video they've recorded and uploads it to the server via their device.

[0064] Step 11:

[0065] The server receives the uploaded practice video and begins analyzing its content using the video analysis unit.

[0066] Step 12:

[0067] The server identifies items such as audio volume, speaking speed, gaze, and posture from the video and generates feedback based on the analysis results.

[0068] Step 13:

[0069] The server incorporates the generated feedback into the instructional video and sends it to the user's device, including areas for improvement.

[0070] Step 14:

[0071] The terminal provides users with instructional videos received from the server, allowing them to review areas for improvement and use that information in their next practice session.

[0072] (Example 1)

[0073] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0074] When preparing presentations and speeches, there is a need for automated content generation tailored to individual user needs and circumstances, as well as appropriate feedback during practice. However, a comprehensive system to efficiently achieve this has not existed until now. Furthermore, it has been difficult to provide users with specific guidance methods to improve their performance without expert instruction.

[0075] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0076] In this invention, the server includes an interface unit means for receiving input prompts, a generation unit means for automatically generating materials, manuscripts, and videos using a generation model based on the received prompts, and a provision unit means for providing the generated content to the user. This allows the user to efficiently prepare their presentations, receive feedback based on their practice results, and improve their skills.

[0077] The "interface unit that receives input prompts" is a component that receives text and instructions from the user, and it is the starting point for initiating processing based on that information.

[0078] The "generative unit that automatically generates materials, manuscripts, and videos using a generative model" refers to the part that has the function of generating presentation materials, speech manuscripts, and video content using machine learning algorithms based on received prompts.

[0079] The "content delivery unit that provides generated content to the user" is a component that has the role of sending and presenting the content created by the generation unit to the user.

[0080] The "video receiving unit that receives videos shot by the user" is a component that receives video files created by the user and acquires data for subsequent analysis processing.

[0081] The "analysis unit that analyzes received videos and generates performance feedback" is a part that analyzes video data and evaluates user performance to create feedback that includes areas for improvement and advice.

[0082] The "instructional unit that generates and provides instructional videos including feedback" is a component that creates instructional materials based on analysis results and provides users with the information they need to make improvements.

[0083] A "terminal device with a user interface" is an interactive device that allows users to input prompts or videos, and has the functionality to send and receive that data within the system.

[0084] This invention is an information processing system that enables users to efficiently prepare and practice presentations. The system consists of three elements: a server, a terminal, and the user.

[0085] The user uses the terminal to enter prompts indicating the theme and purpose of the presentation. An example of a prompt might be, "Create slides and a speech script for a new product launch." These prompts are sent to the server through the input interface.

[0086] The server uses a generative AI model based on received prompts to generate presentation materials, speech scripts, and demo videos. Specifically, the server utilizes a generative model based on a machine learning platform, for example. This generative model automatically analyzes prompts and generates related information, enabling the creation of high-quality content in a short amount of time.

[0087] The generated content is sent to the terminal via the service provider and displayed for the user to visually review. The user can then use this to practice their actual presentations.

[0088] During practice, users record their performance using the device's camera. This recorded video is uploaded to the server via a video receiving unit.

[0089] The server processes uploaded videos in its analysis unit. The analysis unit meticulously evaluates performance aspects such as voice, eye gaze, speaking speed, and posture. The data obtained from this analysis is used to generate feedback, which includes specific advice and suggestions for improving the user's skills.

[0090] Ultimately, the server creates instructional videos based on the feedback and provides them to users via their terminals through the instructional team. These instructional videos visually demonstrate areas for improvement and recommended techniques, allowing users to enhance the quality of their next presentations.

[0091] In this way, users can prepare and improve their presentations on their own, without the need for expert assistance.

[0092] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0093] Step 1:

[0094] The user enters prompts related to presentations and speeches through the terminal. These entered prompts are processed by the terminal's interface and sent to the server as text data. At this point, a specific prompt might be something like "Create slides and speech script for the new product launch."

[0095] Step 2:

[0096] The server calls a generative AI model based on prompts received from the terminal and performs data analysis. This analysis generates presentation materials, speech scripts, and demo videos that correspond to the content of the prompts. The generative AI model used by the server utilizes machine learning algorithms, for example, to produce highly reliable output. As a result of the analysis, content tailored to the user's requests is generated.

[0097] Step 3:

[0098] The generated content is sent to the terminal via the server's delivery unit. The terminal visually presents this received data to the user, who then reviews and downloads the content. At this time, presentation slides or speech scripts are displayed on the terminal, and the user can use them as practice material.

[0099] Step 4:

[0100] Users practice their presentations using their devices and record the process as a video. The recorded videos are treated as important data for analyzing the user's performance. In this step, actions and the flow of the speech during practice are recorded in detail.

[0101] Step 5:

[0102] The device uploads the recorded practice videos to the server via a video receiver. This video data is converted to an appropriate format and prepared for analysis on the server.

[0103] Step 6:

[0104] The server processes the received video in its analysis unit, performing a detailed analysis of attributes such as volume, speed, gaze, and posture. The analysis results are used to create feedback to improve the user's presentation skills. This process reveals the user's specific strengths and areas for improvement.

[0105] Step 7:

[0106] Based on the data obtained from the analysis, the server generates feedback and creates instructional videos. These instructional videos include specific points and advice to help users improve their skills.

[0107] Step 8:

[0108] The server sends the generated feedback and instructional videos to the device. The device receives this and provides it to the user so that they can use it to improve their presentation. As a result, the user has the material to take effective measures for their next presentation.

[0109] (Application Example 1)

[0110] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0111] In today's busy society, there is a need for efficient ways for individuals to prepare for presentations and speeches. While specialized and personalized instruction is desired, its high cost necessitates more accessible and effective support. Furthermore, there is a lack of technology-driven practice environments and difficulty in providing feedback tailored to individual skills and needs.

[0112] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0113] In this invention, the server includes communication means for receiving prompts, generation means for automatically generating materials, manuscripts, and videos using generation technology based on the received prompts, and speech recognition means for recognizing voice input through a robot and engaging in dialogue based on the individual's practice content. This makes it possible for individuals to easily practice, receive feedback, and improve their skills.

[0114] A "communication method" is an interface that exchanges data between a terminal and a server in order to send a prompt to the server.

[0115] "Generation means" refers to a function that automatically creates materials, manuscripts, and videos using generation technology based on received prompts.

[0116] "Means of delivery" refers to an interface for presenting generated content to an individual.

[0117] "Video receiving means" refers to a function that allows a server to receive practice videos filmed by individuals.

[0118] The "analysis means" is a function that analyzes the received practice video and generates feedback based on audio and visual information.

[0119] "Instructional tools" refer to a function that creates instructional videos based on feedback and provides them to individuals.

[0120] "Voice recognition means" refers to a function that recognizes voice input through a robot and engages in dialogue based on the individual's practice content.

[0121] "Adjustment means" refers to a function that adjusts the generated content and feedback according to voice instructions.

[0122] A "personal terminal device" is a device that enables the transmission of personal prompts and practice videos.

[0123] "Target audience" refers to the collective term for the audience and stakeholders who will receive the generated content.

[0124] "Characteristics" refer to the distinctive features and properties of the target audience or event, and are an important element in content creation.

[0125] To implement this invention, the user first uses a terminal device to input prompts that include a theme and purpose. These prompts serve as information to convey the user's intentions in detail. The terminal transmits these prompts to a server via a communication means. Based on the received prompts, the server uses generation technology to generate presentation materials, speech scripts, and demo videos.

[0126] The generated content is delivered to the user via a distribution method. The user practices using the generated materials and manuscripts, and uploads the practice videos to the server via a video receiving method.

[0127] The server analyzes practice videos using analytical tools and generates feedback based on speaking speed, visual information, and audio information. This feedback is provided to the individual as an instructional video using instructional tools. Furthermore, it has speech recognition capabilities that recognize voice input through the robot, allowing individuals to interact directly with the robot during practice.

[0128] A concrete example is when a user enters a prompt such as "wedding speech, groom's friend," and the server generates a relevant speech script and slides, providing feedback on their content. By receiving this feedback in audio format, the user can improve their practice.

[0129] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0130] Step 1:

[0131] The user enters prompts using a terminal device. The input consists of text data related to the presentation's theme and purpose. The terminal formats this data into a prompt message and transmits it to the server via a communication method.

[0132] Step 2:

[0133] The server receives the received prompt message as input and processes the data using a generation AI model. Specifically, it analyzes the prompt message and automatically generates presentation materials, speech scripts, and demo videos based on the theme. The output of this step is the generated content files.

[0134] Step 3:

[0135] The server compiles the generated content and transmits it to the terminal via a delivery method. The terminal displays these content files to the user, allowing the user to view the details.

[0136] Step 4:

[0137] The user practices based on this content. The device records the user's video and audio during practice and saves the data as a video file. This video file serves as input for the next step.

[0138] Step 5:

[0139] The terminal uploads saved practice videos to the server via a video receiving device. The server receives the video file as input and performs data calculations using an analysis device. It analyzes audio volume, speaking speed, visual information, etc., and generates feedback information. The output of this step is specific feedback information regarding the user's performance.

[0140] Step 6:

[0141] The server creates instructional videos based on the generated feedback information. Using instructional tools, it generates content that visually and audibly showcases areas for improvement for the user and sends it to the device. Users can then refer to these instructional videos and make improvements for their next presentation.

[0142] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0143] This invention is a system that recognizes user emotions and prepares and improves presentations based on those emotions. The system can analyze user emotions through prompts and practice videos and reflect this in the content of the presentation. It mainly consists of three elements: a server, a terminal, and a user, and is characterized by including an emotion analysis unit equipped with an emotion engine.

[0144] First, the user uses a terminal to input prompts related to the presentation's theme and purpose. These prompts are crucial as foundational data for reflecting the user's emotions. The terminal sends the entered prompts to the server. The server receives the prompts and recognizes the user's emotions in its emotion analysis unit. The emotion engine uses natural language processing technology to analyze the user's emotions from their input.

[0145] Next, the server uses a generative model to generate presentation materials, speech scripts, and demo videos based on the acquired sentiment data. The generated content takes the user's emotions into account. This process allows users to receive content that effectively reflects their own feelings.

[0146] The user practices their presentation using the generated materials and records a video of the practice session on their device. The recorded video is uploaded to the server via the device. The server analyzes the practice video, identifying changes in emotion in addition to audio and visual information. Based on this analysis, appropriate feedback is generated for the user.

[0147] This feedback includes specific advice to improve the user's presentation skills and is provided as instructional videos. These instructional videos, which include sentiment analysis results, help users gain a deeper understanding of the relationship between their emotions and their performance.

[0148] Specifically, when a user is preparing a wedding speech and enters "wedding speech groom's friend nervous" into the prompt, the server generates content that takes nervousness into account. Analysis of the practice video includes suggestions regarding speaking speed and volume to alleviate nervousness, providing emotionally responsive feedback. In this way, emotionally responsive support maximizes the effectiveness of the presentation.

[0149] The following describes the processing flow.

[0150] Step 1:

[0151] The user enters prompts on their device that include information about the presentation's theme and purpose. For example, they might type "wedding speech groom's friend nervous".

[0152] Step 2:

[0153] The terminal sends the entered prompt to the server. This prompt becomes important data for emotion recognition.

[0154] Step 3:

[0155] The server passes the received prompt to the emotion analysis unit, which uses natural language processing technology to recognize the user's emotions from the prompt. This is where emotions like "tension" are extracted.

[0156] Step 4:

[0157] Based on the recognized emotions, the server uses a generative model to generate presentation materials, speech scripts, and demo videos. This content is then adjusted to reflect the user's emotions.

[0158] Step 5:

[0159] The server sends the generated content to the terminal, which then displays it to the user. The user reviews the content and prepares to use it in the next step.

[0160] Step 6:

[0161] Users practice their speeches using materials and scripts displayed on their devices, and record themselves doing so on video. These practice videos contain important data, including the user's emotional state.

[0162] Step 7:

[0163] Users upload practice videos they've recorded to the server via their device. These videos are necessary to evaluate the user's practical presentation skills.

[0164] Step 8:

[0165] The server activates the video analysis unit to analyze the practice video, detecting changes in the user's emotions in addition to volume, speaking speed, gaze, and posture.

[0166] Step 9:

[0167] Based on the analysis results, the server generates feedback to improve the user's presentation skills. This feedback also includes emotionally responsive advice.

[0168] Step 10:

[0169] The server creates instructional videos and incorporates the generated feedback. These videos help users understand the connection between their emotions and their presentation skills.

[0170] Step 11:

[0171] The device provides users with instructional videos received from the server, enabling them to improve their next presentation through feedback. Users then use this as a reference for further practice and adjustments.

[0172] (Example 2)

[0173] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0174] In recent years, presentations have become an important element in a variety of situations, but many people find it difficult to effectively convey their emotions. Furthermore, emotions such as nervousness and anxiety can also affect the quality of a presentation. For this reason, there is a need for a system that prepares and improves effective presentations that reflect the user's emotions.

[0175] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0176] In this invention, the server includes an information input unit means for receiving information input, an analysis engine means for analyzing the input information and recognizing the emotions of the person who entered it, and a generation device means for automatically creating presentation materials, presentation documents, and demonstration videos using generation technology based on the analysis engine. This makes it possible to automatically support the preparation and improvement of presentations while reflecting the user's emotions.

[0177] The "information input section" is where users can input information related to the presentation's theme and purpose.

[0178] An "analysis engine" is a device that processes received information and uses natural language processing techniques to generate emotion data in order to identify emotions.

[0179] The "generation device" is the part that has the function of creating presentation materials, presentation documents, and demonstration videos using emotional data obtained from the analysis engine.

[0180] The "communication section" is the part that provides the generated content to the user and makes it available for use.

[0181] The "video receiving unit" is an element used to transmit practice videos recorded by the user to the server.

[0182] The "analysis device" is the part that analyzes the received practice video, identifies changes in audio and visual information, and creates feedback.

[0183] The "instructional device" is the part that generates and provides instructional videos incorporating feedback to the user.

[0184] A "personal information device" is a device that allows a user to send prompts and video data to a server.

[0185] This invention is a system that incorporates user emotions to assist in the preparation and improvement of presentations, and mainly consists of three elements: a server, a terminal, and the user. It utilizes a generative AI model to analyze the user's emotions through prompt text and reflect the results in the presentation.

[0186] First, the user uses their device to input prompts related to the presentation's theme and purpose. At this point, they input prompt text via a text editor or dedicated application on their device, such as "wedding speech groom's friend nervous." This forms the basis of the emotion recognition process described later.

[0187] The terminal sends the entered prompt to the server, which uses an emotion analysis engine to analyze the user's emotions from the prompt. This analysis uses software that implements natural language processing technology. Specifically, a Python library running on the server extracts emotions from text data and provides that data as input to a generative model.

[0188] Next, the server uses a generative AI model to generate presentation materials, speech scripts, and demo videos that reflect the user's emotions. Here, for example, natural language models and image generation models are used to generate information that matches the user's input and emotional data. The generated content is then provided to the user again via the terminal, allowing the user to practice their presentation.

[0189] After the user records a practice video on their device, the device uploads the video to a server. The server analyzes this practice video, using speech recognition and visual analysis technologies to thoroughly check the audio, visual information, and emotional changes in the presentation. The feedback obtained from this analysis is provided to the user as specific advice.

[0190] For example, when a user is preparing a wedding speech, if they enter the prompt "wedding speech groom's friend nervous," the server will create content tailored to that emotion and generate advice on tone and speaking speed to alleviate nervousness based on the analysis of a practice video. This system enables users to prepare presentations that reflect their emotions, maximizing their effectiveness.

[0191] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0192] Step 1:

[0193] The user uses a terminal to enter prompts about the presentation's theme and purpose.

[0194] In terms of specific actions, the user inputs information in text format into a designated application on the terminal using the keyboard and sends a prompt message such as "wedding speech groom's friend nervous".

[0195] Input: Text input of a prompt message by the user.

[0196] Output: Prompt message data sent from the terminal to the server.

[0197] Step 2:

[0198] The terminal sends a prompt message to the server.

[0199] The terminal sends the prompt text entered by the user as a request to the server via its network communication function.

[0200] Input: Prompt message data held by the terminal.

[0201] Output: Prompt message data reached the server.

[0202] Step 3:

[0203] The server passes the received prompt message to the sentiment analysis engine, which then analyzes the user's emotions.

[0204] The server uses natural language processing libraries such as Python to analyze the prompt text and extract sentiment data.

[0205] Input: Prompt statement data.

[0206] Output: User sentiment data.

[0207] Step 4:

[0208] The server generates presentation materials, speech scripts, and demo videos using an AI model based on emotional data.

[0209] The generative AI model outputs content that reflects the user's emotions, creating a presentation tailored to the user.

[0210] Input: User sentiment data.

[0211] Output: Generated presentation materials, speech transcript, and demo video data.

[0212] Step 5:

[0213] The server provides the generated content to the user through the terminal.

[0214] Transfer the generated results to the user's device and make them available for download or streaming.

[0215] Input: Generated presentation and video data.

[0216] Output: Content data provided to the user's device.

[0217] Step 6:

[0218] Users practice their presentations on their devices and record practice videos.

[0219] Users practice their presentations using their device's camera function and record themselves doing so.

[0220] Input: Actual presentation performance by users.

[0221] Output: Recorded practice video data.

[0222] Step 7:

[0223] The device uploads the recorded practice video to the server.

[0224] The user operates the device to send the video file to the server.

[0225] Input: Practice video data saved on the device.

[0226] Output: Practice video data received by the server.

[0227] Step 8:

[0228] The server analyzes the received practice videos, analyzing audio and visual information to generate feedback.

[0229] The server utilizes speech recognition and video analysis technologies to identify areas for improvement in the presentation.

[0230] Input: Practice video data.

[0231] Output: Feedback data based on analysis results.

[0232] Step 9:

[0233] The server generates instructional videos, including feedback, and provides them to the user.

[0234] The server generates instructional videos incorporating the advice and provides them to the user's device.

[0235] Input: Feedback data.

[0236] Output: Video data for instruction.

[0237] (Application Example 2)

[0238] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0239] Existing presentation preparation systems often lack sufficient feedback that adapts to user emotions, making it difficult to address situations where users feel nervous or anxious. Furthermore, real-time emotion analysis and feedback are rarely seamless, making it difficult to immediately obtain specific advice to improve presentation quality.

[0240] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0241] In this invention, the server includes an information exchange unit means for receiving prompts, a creation unit means for automatically generating presentation materials, speaking patterns, and demonstration videos using a generation model based on the received prompts, and an emotion analysis unit means for analyzing the user's emotions in real time and providing immediate feedback. This enables the preparation of presentations that are tailored to the user's emotions and realizes the provision of appropriate feedback through real-time emotion analysis.

[0242] The "information exchange unit" is an interface that receives prompts from the user and exchanges information with other system components.

[0243] The "creation unit" is a component that uses a generative model based on received prompts to automatically generate presentation materials and demonstration videos suitable for the user.

[0244] The "distribution section" is the means by which generated information is provided to users, and it is the part that plays the role of disseminating information in a form that is easily accessible to users.

[0245] The "video receiving unit" is a system component that receives the training video shot by the user and sends it to the next analysis step.

[0246] The "analysis unit" is the part that analyzes the received practice video and provides functions to generate feedback that encourages users to improve their audio and visual performance.

[0247] The "Education Department" is a section that promotes education by creating educational videos based on improvement suggestions provided by the Analysis Department and making them available to users.

[0248] The "emotional analysis unit" is an analysis component that analyzes the user's emotions in real time and provides immediate feedback based on the results.

[0249] To implement this invention, a terminal device equipped with a user interface is used to receive prompts from the user. The user first inputs the theme and intention of their presentation as a prompt. This prompt is sent to the server via the information exchange unit. The server analyzes the received prompt and uses the creation unit to utilize a generation AI model to automatically generate presentation materials and demonstration videos suitable for the user.

[0250] The generated materials are provided to the user through the distribution unit. The user then uses the provided materials to perform exercises and records the exercises via a terminal device. The recorded video is then transmitted to the server from the video receiving unit.

[0251] The server analyzes the received video in its analysis unit and generates feedback on audio and visual performance. This feedback information is then processed by the education department into educational videos and distributed to users.

[0252] Furthermore, an emotion analysis unit is provided, which analyzes the user's emotions in real time and provides immediate feedback based on the analysis results. This entire process utilizes natural language processing technologies such as Google® Cloud Natural Language API, and GPT-based AI models are used as the generative AI models.

[0253] As a concrete example, when a user is preparing a presentation to introduce a new product, they can enter "new product presentation audience engaging" into the prompt. Based on this prompt, the generated materials and videos will be tailored to the user's objectives, and feedback based on emotional analysis will provide appropriate information, contributing to the improvement of the user's presentation skills.

[0254] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0255] Step 1:

[0256] The user inputs information about the presentation's theme and intent as prompt messages into the terminal. These prompt messages are sent to the server via the information exchange unit. The input process involves receiving text data such as the theme and emotions, and transmitting it to the server.

[0257] Step 2:

[0258] The server analyzes the received prompt message and processes the information using a generation AI model in the creation unit. This process takes the prompt message as input to generate presentation materials and demonstration videos optimized for the user. The generation AI model's calculations ensure that content matching the user's needs is output.

[0259] Step 3:

[0260] The generated materials and videos are provided to the user's terminal via the server's distribution section. Optimized content is sent from the server, and actions are taken for the user to access and use that content.

[0261] Step 4:

[0262] Users perform exercises based on the provided materials, recording the exercises using their devices. Through this concrete action of recording, the system generates video input data of the user's presentation.

[0263] Step 5:

[0264] The recorded practice video is transmitted from the terminal to the server's video receiving unit. The received video data is supplied to the analysis unit, where visual and audio data are analyzed. This generates performance feedback. The video received as input undergoes analysis to produce output data in the form of feedback.

[0265] Step 6:

[0266] The generated feedback is then converted into educational videos in the Education Department on the server. These videos are provided to the user as output, supporting them in viewing the feedback and using it to make improvements.

[0267] Step 7:

[0268] The emotion analysis unit analyzes the user's emotions in real time and immediately provides feedback to the user using a generative model. The input is the user's emotional data, and the system provides feedback to the user based on that data as output data.

[0269] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0270] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0271] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0272] [Second Embodiment]

[0273] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0274] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0275] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0276] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0277] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0278] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0279] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0280] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0281] The specific processing program 56 is an example of the "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by operating as the specific processing unit 290 according to the specific processing program 56 executed by the processor 28 on the RAM 30.

[0282] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the specific processing unit 290.

[0283] In the smart glasses 214, the reception and output processing is performed by the processor 46. The storage 50 stores a reception and output program 60. The processor 46 reads the reception and output program 60 from the storage 50 and executes the read reception and output program 60 on the RAM 48. The reception and output processing is realized by operating as the control unit 46A according to the reception and output program 60 executed by the processor 46 on the RAM 48.

[0284] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0285] This invention is a system that automatically generates presentation materials, speech manuscripts, and demonstration videos based on a user's prompt input and provides feedback through the analysis of practice videos taken by the user. This system is mainly composed of three elements: a server, a terminal, and a user.

[0286] First, the user uses the terminal to input a prompt that includes the theme and purpose of the presentation. The prompt plays an important role in providing specific information based on the user's needs. Next, the terminal sends this prompt to the server. Based on the received prompt, the server uses a generation model to generate presentation materials, speech manuscripts, and demo videos. As a result, the user can quickly receive content that meets their needs.

[0287] When the user practices based on the generated materials and manuscripts, the user shoots their own presentation with the terminal. This practice video is important data for contributing to the improvement of the user's performance. The terminal uploads the shot practice video to the server.

[0288] The uploaded video is analyzed by the analysis section in the server for items such as volume, speech speed, eye line, and posture. Based on this analysis result, the server generates feedback. This feedback includes specific points and advice for improving the user's presentation skills.

[0289] Finally, the server creates a guidance video based on the generated feedback and provides it to the user via the terminal. The guidance video includes advice that can be expected to improve the user's performance, and the user can utilize this to improve the accuracy of the next presentation.

[0290] As a specific example, consider preparing a speech for a wedding. The user inputs a prompt such as "Wedding speech, best man" and the server generates a speech manuscript and slide materials based on that information. When the user practices the speech and uploads the video shot with the terminal to the server, feedback regarding the volume of the voice and attitude is generated. The user can utilize this feedback to conduct further practice and improvement.

[0291] This system allows users to effectively prepare and improve their presentations themselves, eliminating the need to rely on expensive professional guidance.

[0292] The following describes the processing flow.

[0293] Step 1:

[0294] The user will use their device to enter detailed information about the purpose and theme of the presentation as prompts.

[0295] Step 2:

[0296] The terminal receives the entered prompt, formats it, and prepares to send it to the server.

[0297] Step 3:

[0298] The terminal sends the prepared prompt data to the server.

[0299] Step 4:

[0300] The server analyzes the received prompt data and passes it to the generative model as input data.

[0301] Step 5:

[0302] The server automatically generates presentation materials, speech scripts, and demo videos based on prompts using a generative model.

[0303] Step 6:

[0304] The server organizes the generated content and sends it to the terminal.

[0305] Step 7:

[0306] The terminal provides the user with content received from the server and displays it so that the user can review the content.

[0307] Step 8:

[0308] The user checks the generated materials and manuscripts and practices the presentation based on them.

[0309] Step 9:

[0310] The user shoots a video of the practice process using the terminal.

[0311] Step 10:

[0312] The user selects the best one from the shot practice videos and uploads this video to the server through the terminal.

[0313] Step 11:

[0314] The server receives the uploaded practice video and starts analyzing the content using the video analysis unit.

[0315] Step 12:

[0316] The server identifies items such as the volume of the voice, the speaking speed, the line of sight, and the posture from the video, and generates feedback based on the analysis results.

[0317] Step 13:

[0318] The server incorporates the generated feedback into a guidance video and transmits it to the terminal in a form that includes the user's improvement points.

[0319] Step 14:

[0320] The terminal provides the guidance video received from the server to the user so that the user can check the improvement points and utilize them in the next practice.

[0321] (Example 1)

[0322] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0323] When preparing presentations and speeches, there is a need for automated content generation tailored to individual user needs and circumstances, as well as appropriate feedback during practice. However, a comprehensive system to efficiently achieve this has not existed until now. Furthermore, it has been difficult to provide users with specific guidance methods to improve their performance without expert instruction.

[0324] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0325] In this invention, the server includes an interface unit means for receiving input prompts, a generation unit means for automatically generating materials, manuscripts, and videos using a generation model based on the received prompts, and a provision unit means for providing the generated content to the user. This allows the user to efficiently prepare their presentations, receive feedback based on their practice results, and improve their skills.

[0326] The "interface unit that receives input prompts" is a component that receives text and instructions from the user, and it is the starting point for initiating processing based on that information.

[0327] The "generative unit that automatically generates materials, manuscripts, and videos using a generative model" refers to the part that has the function of generating presentation materials, speech manuscripts, and video content using machine learning algorithms based on received prompts.

[0328] The "content delivery unit that provides generated content to the user" is a component that has the role of sending and presenting the content created by the generation unit to the user.

[0329] The "video receiving unit that receives videos shot by the user" is a component that receives video files created by the user and acquires data for subsequent analysis processing.

[0330] The "analysis unit that analyzes received videos and generates performance feedback" is a part that analyzes video data and evaluates user performance to create feedback that includes areas for improvement and advice.

[0331] The "instructional unit that generates and provides instructional videos including feedback" is a component that creates instructional materials based on analysis results and provides users with the information they need to make improvements.

[0332] A "terminal device with a user interface" is an interactive device that allows users to input prompts or videos, and has the functionality to send and receive that data within the system.

[0333] This invention is an information processing system that enables users to efficiently prepare and practice presentations. The system consists of three elements: a server, a terminal, and the user.

[0334] The user uses the terminal to enter prompts indicating the theme and purpose of the presentation. An example of a prompt might be, "Create slides and a speech script for a new product launch." These prompts are sent to the server through the input interface.

[0335] The server uses a generative AI model based on received prompts to generate presentation materials, speech scripts, and demo videos. Specifically, the server utilizes a generative model based on a machine learning platform, for example. This generative model automatically analyzes prompts and generates related information, enabling the creation of high-quality content in a short amount of time.

[0336] The generated content is sent to the terminal via the service provider and displayed for the user to visually review. The user can then use this to practice their actual presentations.

[0337] During practice, users record their performance using the device's camera. This recorded video is uploaded to the server via a video receiving unit.

[0338] The server processes uploaded videos in its analysis unit. The analysis unit meticulously evaluates performance aspects such as voice, eye gaze, speaking speed, and posture. The data obtained from this analysis is used to generate feedback, which includes specific advice and suggestions for improving the user's skills.

[0339] Ultimately, the server creates instructional videos based on the feedback and provides them to users via their terminals through the instructional team. These instructional videos visually demonstrate areas for improvement and recommended techniques, allowing users to enhance the quality of their next presentations.

[0340] In this way, users can prepare and improve their presentations on their own, without the need for expert assistance.

[0341] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0342] Step 1:

[0343] The user enters prompts related to presentations and speeches through the terminal. These entered prompts are processed by the terminal's interface and sent to the server as text data. At this point, a specific prompt might be something like "Create slides and speech script for the new product launch."

[0344] Step 2:

[0345] The server calls a generative AI model based on prompts received from the terminal and performs data analysis. This analysis generates presentation materials, speech scripts, and demo videos that correspond to the content of the prompts. The generative AI model used by the server utilizes machine learning algorithms, for example, to produce highly reliable output. As a result of the analysis, content tailored to the user's requests is generated.

[0346] Step 3:

[0347] The generated content is sent to the terminal via the server's delivery unit. The terminal visually presents this received data to the user, who then reviews and downloads the content. At this time, presentation slides or speech scripts are displayed on the terminal, and the user can use them as practice material.

[0348] Step 4:

[0349] Users practice their presentations using their devices and record the process as a video. The recorded videos are treated as important data for analyzing the user's performance. In this step, actions and the flow of the speech during practice are recorded in detail.

[0350] Step 5:

[0351] The device uploads the recorded practice videos to the server via a video receiver. This video data is converted to an appropriate format and prepared for analysis on the server.

[0352] Step 6:

[0353] The server processes the received video in its analysis unit, performing a detailed analysis of attributes such as volume, speed, gaze, and posture. The analysis results are used to create feedback to improve the user's presentation skills. This process reveals the user's specific strengths and areas for improvement.

[0354] Step 7:

[0355] Based on the data obtained from the analysis, the server generates feedback and creates instructional videos. These instructional videos include specific points and advice to help users improve their skills.

[0356] Step 8:

[0357] The server sends the generated feedback and instructional videos to the device. The device receives this and provides it to the user so that they can use it to improve their presentation. As a result, the user has the material to take effective measures for their next presentation.

[0358] (Application Example 1)

[0359] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0360] In today's busy society, there is a need for efficient ways for individuals to prepare for presentations and speeches. While specialized and personalized instruction is desired, its high cost necessitates more accessible and effective support. Furthermore, there is a lack of technology-driven practice environments and difficulty in providing feedback tailored to individual skills and needs.

[0361] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0362] In this invention, the server includes communication means for receiving prompts, generation means for automatically generating materials, manuscripts, and videos using generation technology based on the received prompts, and speech recognition means for recognizing voice input through a robot and engaging in dialogue based on the individual's practice content. This makes it possible for individuals to easily practice, receive feedback, and improve their skills.

[0363] A "communication method" is an interface that exchanges data between a terminal and a server in order to send a prompt to the server.

[0364] "Generation means" refers to a function that automatically creates materials, manuscripts, and videos using generation technology based on received prompts.

[0365] "Means of delivery" refers to an interface for presenting generated content to an individual.

[0366] "Video receiving means" refers to a function that allows a server to receive practice videos filmed by individuals.

[0367] The "analysis means" is a function that analyzes the received practice video and generates feedback based on audio and visual information.

[0368] "Instructional tools" refer to a function that creates instructional videos based on feedback and provides them to individuals.

[0369] "Voice recognition means" refers to a function that recognizes voice input through a robot and engages in dialogue based on the individual's practice content.

[0370] "Adjustment means" refers to a function that adjusts the generated content and feedback according to voice instructions.

[0371] A "personal terminal device" is a device that enables the transmission of personal prompts and practice videos.

[0372] "Target audience" refers to the collective term for the audience and stakeholders who will receive the generated content.

[0373] "Characteristics" refer to the distinctive features and properties of the target audience or event, and are an important element in content creation.

[0374] To implement this invention, the user first uses a terminal device to input prompts that include a theme and purpose. These prompts serve as information to convey the user's intentions in detail. The terminal transmits these prompts to a server via a communication means. Based on the received prompts, the server uses generation technology to generate presentation materials, speech scripts, and demo videos.

[0375] The generated content is delivered to the user via a distribution method. The user practices using the generated materials and manuscripts, and uploads the practice videos to the server via a video receiving method.

[0376] The server analyzes practice videos using analytical tools and generates feedback based on speaking speed, visual information, and audio information. This feedback is provided to the individual as an instructional video using instructional tools. Furthermore, it has speech recognition capabilities that recognize voice input through the robot, allowing individuals to interact directly with the robot during practice.

[0377] A concrete example is when a user enters a prompt such as "wedding speech, groom's friend," and the server generates a relevant speech script and slides, providing feedback on their content. By receiving this feedback in audio format, the user can improve their practice.

[0378] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0379] Step 1:

[0380] The user enters prompts using a terminal device. The input consists of text data related to the presentation's theme and purpose. The terminal formats this data into a prompt message and transmits it to the server via a communication method.

[0381] Step 2:

[0382] The server receives the received prompt message as input and processes the data using a generation AI model. Specifically, it analyzes the prompt message and automatically generates presentation materials, speech scripts, and demo videos based on the theme. The output of this step is the generated content files.

[0383] Step 3:

[0384] The server compiles the generated content and transmits it to the terminal via a delivery method. The terminal displays these content files to the user, allowing the user to view the details.

[0385] Step 4:

[0386] The user practices based on this content. The device records the user's video and audio during practice and saves the data as a video file. This video file serves as input for the next step.

[0387] Step 5:

[0388] The terminal uploads saved practice videos to the server via a video receiving device. The server receives the video file as input and performs data calculations using an analysis device. It analyzes audio volume, speaking speed, visual information, etc., and generates feedback information. The output of this step is specific feedback information regarding the user's performance.

[0389] Step 6:

[0390] The server creates instructional videos based on the generated feedback information. Using instructional tools, it generates content that visually and audibly showcases areas for improvement for the user and sends it to the device. Users can then refer to these instructional videos and make improvements for their next presentation.

[0391] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0392] This invention is a system that recognizes user emotions and prepares and improves presentations based on those emotions. The system can analyze user emotions through prompts and practice videos and reflect this in the content of the presentation. It mainly consists of three elements: a server, a terminal, and a user, and is characterized by including an emotion analysis unit equipped with an emotion engine.

[0393] First, the user uses a terminal to input prompts related to the presentation's theme and purpose. These prompts are crucial as foundational data for reflecting the user's emotions. The terminal sends the entered prompts to the server. The server receives the prompts and recognizes the user's emotions in its emotion analysis unit. The emotion engine uses natural language processing technology to analyze the user's emotions from their input.

[0394] Next, the server uses a generative model to generate presentation materials, speech scripts, and demo videos based on the acquired sentiment data. The generated content takes the user's emotions into account. This process allows users to receive content that effectively reflects their own feelings.

[0395] The user practices their presentation using the generated materials and records a video of the practice session on their device. The recorded video is uploaded to the server via the device. The server analyzes the practice video, identifying changes in emotion in addition to audio and visual information. Based on this analysis, appropriate feedback is generated for the user.

[0396] This feedback includes specific advice to improve the user's presentation skills and is provided as instructional videos. These instructional videos, which include sentiment analysis results, help users gain a deeper understanding of the relationship between their emotions and their performance.

[0397] Specifically, when a user is preparing a wedding speech and enters "wedding speech groom's friend nervous" into the prompt, the server generates content that takes nervousness into account. Analysis of the practice video includes suggestions regarding speaking speed and volume to alleviate nervousness, providing emotionally responsive feedback. In this way, emotionally responsive support maximizes the effectiveness of the presentation.

[0398] The following describes the processing flow.

[0399] Step 1:

[0400] The user enters prompts on their device that include information about the presentation's theme and purpose. For example, they might type "wedding speech groom's friend nervous".

[0401] Step 2:

[0402] The terminal sends the entered prompt to the server. This prompt becomes important data for emotion recognition.

[0403] Step 3:

[0404] The server passes the received prompt to the emotion analysis unit, which uses natural language processing technology to recognize the user's emotions from the prompt. This is where emotions like "tension" are extracted.

[0405] Step 4:

[0406] Based on the recognized emotions, the server uses a generative model to generate presentation materials, speech scripts, and demo videos. This content is then adjusted to reflect the user's emotions.

[0407] Step 5:

[0408] The server sends the generated content to the terminal, which then displays it to the user. The user reviews the content and prepares to use it in the next step.

[0409] Step 6:

[0410] Users practice their speeches using materials and scripts displayed on their devices, and record themselves doing so on video. These practice videos contain important data, including the user's emotional state.

[0411] Step 7:

[0412] Users upload practice videos they've recorded to the server via their device. These videos are necessary to evaluate the user's practical presentation skills.

[0413] Step 8:

[0414] The server activates the video analysis unit to analyze the practice video, detecting changes in the user's emotions in addition to volume, speaking speed, gaze, and posture.

[0415] Step 9:

[0416] Based on the analysis results, the server generates feedback to improve the user's presentation skills. This feedback also includes emotionally responsive advice.

[0417] Step 10:

[0418] The server creates instructional videos and incorporates the generated feedback. These videos help users understand the connection between their emotions and their presentation skills.

[0419] Step 11:

[0420] The device provides users with instructional videos received from the server, enabling them to improve their next presentation through feedback. Users then use this as a reference for further practice and adjustments.

[0421] (Example 2)

[0422] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0423] In recent years, presentations have become an important element in a variety of situations, but many people find it difficult to effectively convey their emotions. Furthermore, emotions such as nervousness and anxiety can also affect the quality of a presentation. For this reason, there is a need for a system that prepares and improves effective presentations that reflect the user's emotions.

[0424] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0425] In this invention, the server includes an information input unit means for receiving information input, an analysis engine means for analyzing the input information and recognizing the emotions of the person who entered it, and a generation device means for automatically creating presentation materials, presentation documents, and demonstration videos using generation technology based on the analysis engine. This makes it possible to automatically support the preparation and improvement of presentations while reflecting the user's emotions.

[0426] The "information input section" is where users can input information related to the presentation's theme and purpose.

[0427] An "analysis engine" is a device that processes received information and uses natural language processing techniques to generate emotion data in order to identify emotions.

[0428] The "generation device" is the part that has the function of creating presentation materials, presentation documents, and demonstration videos using emotional data obtained from the analysis engine.

[0429] The "communication section" is the part that provides the generated content to the user and makes it available for use.

[0430] The "video receiving unit" is an element used to transmit practice videos recorded by the user to the server.

[0431] The "analysis device" is the part that analyzes the received practice video, identifies changes in audio and visual information, and creates feedback.

[0432] The "instructional device" is the part that generates and provides instructional videos incorporating feedback to the user.

[0433] A "personal information device" is a device that allows a user to send prompts and video data to a server.

[0434] This invention is a system that incorporates user emotions to assist in the preparation and improvement of presentations, and mainly consists of three elements: a server, a terminal, and the user. It utilizes a generative AI model to analyze the user's emotions through prompt text and reflect the results in the presentation.

[0435] First, the user uses their device to input prompts related to the presentation's theme and purpose. At this point, they input prompt text via a text editor or dedicated application on their device, such as "wedding speech groom's friend nervous." This forms the basis of the emotion recognition process described later.

[0436] The terminal sends the entered prompt to the server, which uses an emotion analysis engine to analyze the user's emotions from the prompt. This analysis uses software that implements natural language processing technology. Specifically, a Python library running on the server extracts emotions from text data and provides that data as input to a generative model.

[0437] Next, the server uses a generative AI model to generate presentation materials, speech scripts, and demo videos that reflect the user's emotions. Here, for example, natural language models and image generation models are used to generate information that matches the user's input and emotional data. The generated content is then provided to the user again via the terminal, allowing the user to practice their presentation.

[0438] After the user records a practice video on their device, the device uploads the video to a server. The server analyzes this practice video, using speech recognition and visual analysis technologies to thoroughly check the audio, visual information, and emotional changes in the presentation. The feedback obtained from this analysis is provided to the user as specific advice.

[0439] For example, when a user is preparing a wedding speech, if they enter the prompt "wedding speech groom's friend nervous," the server will create content tailored to that emotion and generate advice on tone and speaking speed to alleviate nervousness based on the analysis of a practice video. This system enables users to prepare presentations that reflect their emotions, maximizing their effectiveness.

[0440] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0441] Step 1:

[0442] The user uses a terminal to enter prompts about the presentation's theme and purpose.

[0443] In terms of specific actions, the user inputs information in text format into a designated application on the terminal using the keyboard and sends a prompt message such as "wedding speech groom's friend nervous".

[0444] Input: Text input of a prompt message by the user.

[0445] Output: Prompt message data sent from the terminal to the server.

[0446] Step 2:

[0447] The terminal sends a prompt message to the server.

[0448] The terminal sends the prompt text entered by the user as a request to the server via its network communication function.

[0449] Input: Prompt message data held by the terminal.

[0450] Output: Prompt message data reached the server.

[0451] Step 3:

[0452] The server passes the received prompt message to the sentiment analysis engine, which then analyzes the user's emotions.

[0453] The server uses natural language processing libraries such as Python to analyze the prompt text and extract sentiment data.

[0454] Input: Prompt statement data.

[0455] Output: User sentiment data.

[0456] Step 4:

[0457] The server generates presentation materials, speech scripts, and demo videos using an AI model based on emotional data.

[0458] The generative AI model outputs content that reflects the user's emotions, creating a presentation tailored to the user.

[0459] Input: User sentiment data.

[0460] Output: Generated presentation materials, speech transcript, and demo video data.

[0461] Step 5:

[0462] The server provides the generated content to the user through the terminal.

[0463] Transfer the generated results to the user's device and make them available for download or streaming.

[0464] Input: Generated presentation and video data.

[0465] Output: Content data provided to the user's device.

[0466] Step 6:

[0467] Users practice their presentations on their devices and record practice videos.

[0468] Users practice their presentations using their device's camera function and record themselves doing so.

[0469] Input: Actual presentation performance by users.

[0470] Output: Recorded practice video data.

[0471] Step 7:

[0472] The device uploads the recorded practice video to the server.

[0473] The user operates the device to send the video file to the server.

[0474] Input: Practice video data saved on the device.

[0475] Output: Practice video data received by the server.

[0476] Step 8:

[0477] The server analyzes the received practice videos, analyzing audio and visual information to generate feedback.

[0478] The server utilizes speech recognition and video analysis technologies to identify areas for improvement in the presentation.

[0479] Input: Practice video data.

[0480] Output: Feedback data based on analysis results.

[0481] Step 9:

[0482] The server generates instructional videos, including feedback, and provides them to the user.

[0483] The server generates instructional videos incorporating the advice and provides them to the user's device.

[0484] Input: Feedback data.

[0485] Output: Video data for instruction.

[0486] (Application Example 2)

[0487] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0488] Existing presentation preparation systems often lack sufficient feedback that adapts to user emotions, making it difficult to address situations where users feel nervous or anxious. Furthermore, real-time emotion analysis and feedback are rarely seamless, making it difficult to immediately obtain specific advice to improve presentation quality.

[0489] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0490] In this invention, the server includes an information exchange unit means for receiving prompts, a creation unit means for automatically generating presentation materials, speaking patterns, and demonstration videos using a generation model based on the received prompts, and an emotion analysis unit means for analyzing the user's emotions in real time and providing immediate feedback. This enables the preparation of presentations that are tailored to the user's emotions and realizes the provision of appropriate feedback through real-time emotion analysis.

[0491] The "information exchange unit" is an interface that receives prompts from the user and exchanges information with other system components.

[0492] The "creation unit" is a component that uses a generative model based on received prompts to automatically generate presentation materials and demonstration videos suitable for the user.

[0493] The "distribution section" is the means by which generated information is provided to users, and it is the part that plays the role of disseminating information in a form that is easily accessible to users.

[0494] The "video receiving unit" is a system component that receives the training video shot by the user and sends it to the next analysis step.

[0495] The "analysis unit" is the part that analyzes the received practice video and provides functions to generate feedback that encourages users to improve their audio and visual performance.

[0496] The "Education Department" is a section that promotes education by creating educational videos based on improvement suggestions provided by the Analysis Department and making them available to users.

[0497] The "emotional analysis unit" is an analysis component that analyzes the user's emotions in real time and provides immediate feedback based on the results.

[0498] To implement this invention, a terminal device equipped with a user interface is used to receive prompts from the user. The user first inputs the theme and intention of their presentation as a prompt. This prompt is sent to the server via the information exchange unit. The server analyzes the received prompt and uses the creation unit to utilize a generation AI model to automatically generate presentation materials and demonstration videos suitable for the user.

[0499] The generated materials are provided to the user through the distribution unit. The user then uses the provided materials to perform exercises and records the exercises via a terminal device. The recorded video is then transmitted to the server from the video receiving unit.

[0500] The server analyzes the received video in its analysis unit and generates feedback on audio and visual performance. This feedback information is then processed by the education department into educational videos and distributed to users.

[0501] Furthermore, an emotion analysis unit is provided, which analyzes the user's emotions in real time and provides immediate feedback based on the analysis results. This entire process utilizes natural language processing technologies such as the Google Cloud Natural Language API, and GPT-based AI models are used as the generative AI models.

[0502] As a concrete example, when a user is preparing a presentation to introduce a new product, they can enter "new product presentation audience engaging" into the prompt. Based on this prompt, the generated materials and videos will be tailored to the user's objectives, and feedback based on emotional analysis will provide appropriate information, contributing to the improvement of the user's presentation skills.

[0503] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0504] Step 1:

[0505] The user inputs information about the presentation's theme and intent as prompt messages into the terminal. These prompt messages are sent to the server via the information exchange unit. The input process involves receiving text data such as the theme and emotions, and transmitting it to the server.

[0506] Step 2:

[0507] The server analyzes the received prompt message and processes the information using a generation AI model in the creation unit. This process takes the prompt message as input to generate presentation materials and demonstration videos optimized for the user. The generation AI model's calculations ensure that content matching the user's needs is output.

[0508] Step 3:

[0509] The generated materials and videos are provided to the user's terminal via the server's distribution section. Optimized content is sent from the server, and actions are taken for the user to access and use that content.

[0510] Step 4:

[0511] Users perform exercises based on the provided materials, recording the exercises using their devices. Through this concrete action of recording, the system generates video input data of the user's presentation.

[0512] Step 5:

[0513] The recorded practice video is transmitted from the terminal to the server's video receiving unit. The received video data is supplied to the analysis unit, where visual and audio data are analyzed. This generates performance feedback. The video received as input undergoes analysis to produce output data in the form of feedback.

[0514] Step 6:

[0515] The generated feedback is then converted into educational videos in the Education Department on the server. These videos are provided to the user as output, supporting them in viewing the feedback and using it to make improvements.

[0516] Step 7:

[0517] The emotion analysis unit analyzes the user's emotions in real time and immediately provides feedback to the user using a generative model. The input is the user's emotional data, and the system provides feedback to the user based on that data as output data.

[0518] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0519] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0520] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0521] [Third Embodiment]

[0522] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0523] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0524] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0525] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0526] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0527] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0528] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0529] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0530] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0531] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0532] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0533] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0534] This invention is a system that automatically generates presentation materials, speech scripts, and demo videos based on user prompt input, and provides feedback through analysis of practice videos recorded by the user. This system mainly consists of three elements: a server, a terminal, and the user.

[0535] First, the user uses a terminal to enter prompts that include the presentation's theme and objectives. These prompts play a crucial role in providing specific information based on the user's needs. Next, the terminal sends these prompts to the server. Based on the received prompts, the server uses a generative model to generate presentation materials, a speech script, and a demo video. This allows the user to quickly receive content tailored to their needs.

[0536] When users practice using generated materials and scripts, they record their presentations with their devices. These practice videos are important data for improving user performance. The devices upload the recorded practice videos to the server.

[0537] Uploaded videos are analyzed by an analysis unit on the server, which examines items such as volume, speaking speed, gaze, and posture. Based on this analysis, the server generates feedback. This feedback includes specific suggestions and advice to improve the user's presentation skills.

[0538] Finally, the server creates instructional videos based on the generated feedback and provides them to the user via the terminal. These instructional videos contain advice that can help improve the user's performance, allowing them to enhance the quality of their next presentation.

[0539] As a concrete example, consider preparing a speech for a wedding. The user enters the prompt "wedding speech groom's friend," and the server generates a speech script and slides based on that information. When the user practices the speech and uploads a video recorded on their device to the server, feedback is generated regarding the volume of their voice and their demeanor. The user can then use this feedback to practice further and make improvements.

[0540] This system allows users to effectively prepare and improve their presentations themselves, eliminating the need to rely on expensive professional guidance.

[0541] The following describes the processing flow.

[0542] Step 1:

[0543] The user will use their device to enter detailed information about the purpose and theme of the presentation as prompts.

[0544] Step 2:

[0545] The terminal receives the entered prompt, formats it, and prepares to send it to the server.

[0546] Step 3:

[0547] The terminal sends the prepared prompt data to the server.

[0548] Step 4:

[0549] The server analyzes the received prompt data and passes it to the generative model as input data.

[0550] Step 5:

[0551] The server automatically generates presentation materials, speech scripts, and demo videos based on prompts using a generative model.

[0552] Step 6:

[0553] The server organizes the generated content and sends it to the terminal.

[0554] Step 7:

[0555] The terminal provides the user with content received from the server and displays it so that the user can review the content.

[0556] Step 8:

[0557] Users review the generated materials and drafts and practice their presentations based on them.

[0558] Step 9:

[0559] Users record the practice process on their devices.

[0560] Step 10:

[0561] The user selects the best practice video they've recorded and uploads it to the server via their device.

[0562] Step 11:

[0563] The server receives the uploaded practice video and begins analyzing its content using the video analysis unit.

[0564] Step 12:

[0565] The server identifies items such as audio volume, speaking speed, gaze, and posture from the video and generates feedback based on the analysis results.

[0566] Step 13:

[0567] The server incorporates the generated feedback into the instructional video and sends it to the user's device, including areas for improvement.

[0568] Step 14:

[0569] The terminal provides users with instructional videos received from the server, allowing them to review areas for improvement and use that information in their next practice session.

[0570] (Example 1)

[0571] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0572] When preparing presentations and speeches, there is a need for automated content generation tailored to individual user needs and circumstances, as well as appropriate feedback during practice. However, a comprehensive system to efficiently achieve this has not existed until now. Furthermore, it has been difficult to provide users with specific guidance methods to improve their performance without expert instruction.

[0573] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0574] In this invention, the server includes an interface unit means for receiving input prompts, a generation unit means for automatically generating materials, manuscripts, and videos using a generation model based on the received prompts, and a provision unit means for providing the generated content to the user. This allows the user to efficiently prepare their presentations, receive feedback based on their practice results, and improve their skills.

[0575] The "interface unit that receives input prompts" is a component that receives text and instructions from the user, and it is the starting point for initiating processing based on that information.

[0576] The "generative unit that automatically generates materials, manuscripts, and videos using a generative model" refers to the part that has the function of generating presentation materials, speech manuscripts, and video content using machine learning algorithms based on received prompts.

[0577] The "content delivery unit that provides generated content to the user" is a component that has the role of sending and presenting the content created by the generation unit to the user.

[0578] The "video receiving unit that receives videos shot by the user" is a component that receives video files created by the user and acquires data for subsequent analysis processing.

[0579] The "analysis unit that analyzes received videos and generates performance feedback" is a part that analyzes video data and evaluates user performance to create feedback that includes areas for improvement and advice.

[0580] The "instructional unit that generates and provides instructional videos including feedback" is a component that creates instructional materials based on analysis results and provides users with the information they need to make improvements.

[0581] A "terminal device with a user interface" is an interactive device that allows users to input prompts or videos, and has the functionality to send and receive that data within the system.

[0582] This invention is an information processing system that enables users to efficiently prepare and practice presentations. The system consists of three elements: a server, a terminal, and the user.

[0583] The user uses the terminal to enter prompts indicating the theme and purpose of the presentation. An example of a prompt might be, "Create slides and a speech script for a new product launch." These prompts are sent to the server through the input interface.

[0584] The server uses a generative AI model based on received prompts to generate presentation materials, speech scripts, and demo videos. Specifically, the server utilizes a generative model based on a machine learning platform, for example. This generative model automatically analyzes prompts and generates related information, enabling the creation of high-quality content in a short amount of time.

[0585] The generated content is sent to the terminal via the service provider and displayed for the user to visually review. The user can then use this to practice their actual presentations.

[0586] During practice, users record their performance using the device's camera. This recorded video is uploaded to the server via a video receiving unit.

[0587] The server processes uploaded videos in its analysis unit. The analysis unit meticulously evaluates performance aspects such as voice, eye gaze, speaking speed, and posture. The data obtained from this analysis is used to generate feedback, which includes specific advice and suggestions for improving the user's skills.

[0588] Ultimately, the server creates instructional videos based on the feedback and provides them to users via their terminals through the instructional team. These instructional videos visually demonstrate areas for improvement and recommended techniques, allowing users to enhance the quality of their next presentations.

[0589] In this way, users can prepare and improve their presentations on their own, without the need for expert assistance.

[0590] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0591] Step 1:

[0592] The user enters prompts related to presentations and speeches through the terminal. These entered prompts are processed by the terminal's interface and sent to the server as text data. At this point, a specific prompt might be something like "Create slides and speech script for the new product launch."

[0593] Step 2:

[0594] The server calls a generative AI model based on prompts received from the terminal and performs data analysis. This analysis generates presentation materials, speech scripts, and demo videos that correspond to the content of the prompts. The generative AI model used by the server utilizes machine learning algorithms, for example, to produce highly reliable output. As a result of the analysis, content tailored to the user's requests is generated.

[0595] Step 3:

[0596] The generated content is sent to the terminal via the server's delivery unit. The terminal visually presents this received data to the user, who then reviews and downloads the content. At this time, presentation slides or speech scripts are displayed on the terminal, and the user can use them as practice material.

[0597] Step 4:

[0598] Users practice their presentations using their devices and record the process as a video. The recorded videos are treated as important data for analyzing the user's performance. In this step, actions and the flow of the speech during practice are recorded in detail.

[0599] Step 5:

[0600] The device uploads the recorded practice videos to the server via a video receiver. This video data is converted to an appropriate format and prepared for analysis on the server.

[0601] Step 6:

[0602] The server processes the received video in its analysis unit, performing a detailed analysis of attributes such as volume, speed, gaze, and posture. The analysis results are used to create feedback to improve the user's presentation skills. This process reveals the user's specific strengths and areas for improvement.

[0603] Step 7:

[0604] Based on the data obtained from the analysis, the server generates feedback and creates instructional videos. These instructional videos include specific points and advice to help users improve their skills.

[0605] Step 8:

[0606] The server sends the generated feedback and instructional videos to the device. The device receives this and provides it to the user so that they can use it to improve their presentation. As a result, the user has the material to take effective measures for their next presentation.

[0607] (Application Example 1)

[0608] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0609] In today's busy society, there is a need for efficient ways for individuals to prepare for presentations and speeches. While specialized and personalized instruction is desired, its high cost necessitates more accessible and effective support. Furthermore, there is a lack of technology-driven practice environments and difficulty in providing feedback tailored to individual skills and needs.

[0610] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0611] In this invention, the server includes communication means for receiving prompts, generation means for automatically generating materials, manuscripts, and videos using generation technology based on the received prompts, and speech recognition means for recognizing voice input through a robot and engaging in dialogue based on the individual's practice content. This makes it possible for individuals to easily practice, receive feedback, and improve their skills.

[0612] A "communication method" is an interface that exchanges data between a terminal and a server in order to send a prompt to the server.

[0613] "Generation means" refers to a function that automatically creates materials, manuscripts, and videos using generation technology based on received prompts.

[0614] "Means of delivery" refers to an interface for presenting generated content to an individual.

[0615] "Video receiving means" refers to a function that allows a server to receive practice videos filmed by individuals.

[0616] The "analysis means" is a function that analyzes the received practice video and generates feedback based on audio and visual information.

[0617] "Instructional tools" refer to a function that creates instructional videos based on feedback and provides them to individuals.

[0618] "Voice recognition means" refers to a function that recognizes voice input through a robot and engages in dialogue based on the individual's practice content.

[0619] "Adjustment means" refers to a function that adjusts the generated content and feedback according to voice instructions.

[0620] A "personal terminal device" is a device that enables the transmission of personal prompts and practice videos.

[0621] "Target audience" refers to the collective term for the audience and stakeholders who will receive the generated content.

[0622] "Characteristics" refer to the distinctive features and properties of the target audience or event, and are an important element in content creation.

[0623] To implement this invention, the user first uses a terminal device to input prompts that include a theme and purpose. These prompts serve as information to convey the user's intentions in detail. The terminal transmits these prompts to a server via a communication means. Based on the received prompts, the server uses generation technology to generate presentation materials, speech scripts, and demo videos.

[0624] The generated content is delivered to the user via a distribution method. The user practices using the generated materials and manuscripts, and uploads the practice videos to the server via a video receiving method.

[0625] The server analyzes practice videos using analytical tools and generates feedback based on speaking speed, visual information, and audio information. This feedback is provided to the individual as an instructional video using instructional tools. Furthermore, it has speech recognition capabilities that recognize voice input through the robot, allowing individuals to interact directly with the robot during practice.

[0626] A concrete example is when a user enters a prompt such as "wedding speech, groom's friend," and the server generates a relevant speech script and slides, providing feedback on their content. By receiving this feedback in audio format, the user can improve their practice.

[0627] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0628] Step 1:

[0629] The user enters prompts using a terminal device. The input consists of text data related to the presentation's theme and purpose. The terminal formats this data into a prompt message and transmits it to the server via a communication method.

[0630] Step 2:

[0631] The server receives the received prompt message as input and processes the data using a generation AI model. Specifically, it analyzes the prompt message and automatically generates presentation materials, speech scripts, and demo videos based on the theme. The output of this step is the generated content files.

[0632] Step 3:

[0633] The server compiles the generated content and transmits it to the terminal via a delivery method. The terminal displays these content files to the user, allowing the user to view the details.

[0634] Step 4:

[0635] The user practices based on this content. The device records the user's video and audio during practice and saves the data as a video file. This video file serves as input for the next step.

[0636] Step 5:

[0637] The terminal uploads saved practice videos to the server via a video receiving device. The server receives the video file as input and performs data calculations using an analysis device. It analyzes audio volume, speaking speed, visual information, etc., and generates feedback information. The output of this step is specific feedback information regarding the user's performance.

[0638] Step 6:

[0639] The server creates instructional videos based on the generated feedback information. Using instructional tools, it generates content that visually and audibly showcases areas for improvement for the user and sends it to the device. Users can then refer to these instructional videos and make improvements for their next presentation.

[0640] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0641] This invention is a system that recognizes user emotions and prepares and improves presentations based on those emotions. The system can analyze user emotions through prompts and practice videos and reflect this in the content of the presentation. It mainly consists of three elements: a server, a terminal, and a user, and is characterized by including an emotion analysis unit equipped with an emotion engine.

[0642] First, the user uses a terminal to input prompts related to the presentation's theme and purpose. These prompts are crucial as foundational data for reflecting the user's emotions. The terminal sends the entered prompts to the server. The server receives the prompts and recognizes the user's emotions in its emotion analysis unit. The emotion engine uses natural language processing technology to analyze the user's emotions from their input.

[0643] Next, the server uses a generative model to generate presentation materials, speech scripts, and demo videos based on the acquired sentiment data. The generated content takes the user's emotions into account. This process allows users to receive content that effectively reflects their own feelings.

[0644] The user practices their presentation using the generated materials and records a video of the practice session on their device. The recorded video is uploaded to the server via the device. The server analyzes the practice video, identifying changes in emotion in addition to audio and visual information. Based on this analysis, appropriate feedback is generated for the user.

[0645] This feedback includes specific advice to improve the user's presentation skills and is provided as instructional videos. These instructional videos, which include sentiment analysis results, help users gain a deeper understanding of the relationship between their emotions and their performance.

[0646] Specifically, when a user is preparing a wedding speech and enters "wedding speech groom's friend nervous" into the prompt, the server generates content that takes nervousness into account. Analysis of the practice video includes suggestions regarding speaking speed and volume to alleviate nervousness, providing emotionally responsive feedback. In this way, emotionally responsive support maximizes the effectiveness of the presentation.

[0647] The following describes the processing flow.

[0648] Step 1:

[0649] The user enters prompts on their device that include information about the presentation's theme and purpose. For example, they might type "wedding speech groom's friend nervous".

[0650] Step 2:

[0651] The terminal sends the entered prompt to the server. This prompt becomes important data for emotion recognition.

[0652] Step 3:

[0653] The server passes the received prompt to the emotion analysis unit, which uses natural language processing technology to recognize the user's emotions from the prompt. This is where emotions like "tension" are extracted.

[0654] Step 4:

[0655] Based on the recognized emotions, the server uses a generative model to generate presentation materials, speech scripts, and demo videos. This content is then adjusted to reflect the user's emotions.

[0656] Step 5:

[0657] The server sends the generated content to the terminal, which then displays it to the user. The user reviews the content and prepares to use it in the next step.

[0658] Step 6:

[0659] Users practice their speeches using materials and scripts displayed on their devices, and record themselves doing so on video. These practice videos contain important data, including the user's emotional state.

[0660] Step 7:

[0661] Users upload practice videos they've recorded to the server via their device. These videos are necessary to evaluate the user's practical presentation skills.

[0662] Step 8:

[0663] The server activates the video analysis unit to analyze the practice video, detecting changes in the user's emotions in addition to volume, speaking speed, gaze, and posture.

[0664] Step 9:

[0665] Based on the analysis results, the server generates feedback to improve the user's presentation skills. This feedback also includes emotionally responsive advice.

[0666] Step 10:

[0667] The server creates instructional videos and incorporates the generated feedback. These videos help users understand the connection between their emotions and their presentation skills.

[0668] Step 11:

[0669] The device provides users with instructional videos received from the server, enabling them to improve their next presentation through feedback. Users then use this as a reference for further practice and adjustments.

[0670] (Example 2)

[0671] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0672] In recent years, presentations have become an important element in a variety of situations, but many people find it difficult to effectively convey their emotions. Furthermore, emotions such as nervousness and anxiety can also affect the quality of a presentation. For this reason, there is a need for a system that prepares and improves effective presentations that reflect the user's emotions.

[0673] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0674] In this invention, the server includes an information input unit means for receiving information input, an analysis engine means for analyzing the input information and recognizing the emotions of the person who entered it, and a generation device means for automatically creating presentation materials, presentation documents, and demonstration videos using generation technology based on the analysis engine. This makes it possible to automatically support the preparation and improvement of presentations while reflecting the user's emotions.

[0675] The "information input section" is where users can input information related to the presentation's theme and purpose.

[0676] An "analysis engine" is a device that processes received information and uses natural language processing techniques to generate emotion data in order to identify emotions.

[0677] The "generation device" is the part that has the function of creating presentation materials, presentation documents, and demonstration videos using emotional data obtained from the analysis engine.

[0678] The "communication section" is the part that provides the generated content to the user and makes it available for use.

[0679] The "video receiving unit" is an element used to transmit practice videos recorded by the user to the server.

[0680] The "analysis device" is the part that analyzes the received practice video, identifies changes in audio and visual information, and creates feedback.

[0681] The "instructional device" is the part that generates and provides instructional videos incorporating feedback to the user.

[0682] A "personal information device" is a device that allows a user to send prompts and video data to a server.

[0683] This invention is a system that incorporates user emotions to assist in the preparation and improvement of presentations, and mainly consists of three elements: a server, a terminal, and the user. It utilizes a generative AI model to analyze the user's emotions through prompt text and reflect the results in the presentation.

[0684] First, the user uses their device to input prompts related to the presentation's theme and purpose. At this point, they input prompt text via a text editor or dedicated application on their device, such as "wedding speech groom's friend nervous." This forms the basis of the emotion recognition process described later.

[0685] The terminal sends the entered prompt to the server, which uses an emotion analysis engine to analyze the user's emotions from the prompt. This analysis uses software that implements natural language processing technology. Specifically, a Python library running on the server extracts emotions from text data and provides that data as input to a generative model.

[0686] Next, the server uses a generative AI model to generate presentation materials, speech scripts, and demo videos that reflect the user's emotions. Here, for example, natural language models and image generation models are used to generate information that matches the user's input and emotional data. The generated content is then provided to the user again via the terminal, allowing the user to practice their presentation.

[0687] After the user records a practice video on their device, the device uploads the video to a server. The server analyzes this practice video, using speech recognition and visual analysis technologies to thoroughly check the audio, visual information, and emotional changes in the presentation. The feedback obtained from this analysis is provided to the user as specific advice.

[0688] For example, when a user is preparing a wedding speech, if they enter the prompt "wedding speech groom's friend nervous," the server will create content tailored to that emotion and generate advice on tone and speaking speed to alleviate nervousness based on the analysis of a practice video. This system enables users to prepare presentations that reflect their emotions, maximizing their effectiveness.

[0689] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0690] Step 1:

[0691] The user uses a terminal to enter prompts about the presentation's theme and purpose.

[0692] In terms of specific actions, the user inputs information in text format into a designated application on the terminal using the keyboard and sends a prompt message such as "wedding speech groom's friend nervous".

[0693] Input: Text input of a prompt message by the user.

[0694] Output: Prompt message data sent from the terminal to the server.

[0695] Step 2:

[0696] The terminal sends a prompt message to the server.

[0697] The terminal sends the prompt text entered by the user as a request to the server via its network communication function.

[0698] Input: Prompt message data held by the terminal.

[0699] Output: Prompt message data reached the server.

[0700] Step 3:

[0701] The server passes the received prompt message to the sentiment analysis engine, which then analyzes the user's emotions.

[0702] The server uses natural language processing libraries such as Python to analyze the prompt text and extract sentiment data.

[0703] Input: Prompt statement data.

[0704] Output: User sentiment data.

[0705] Step 4:

[0706] The server generates presentation materials, speech scripts, and demo videos using an AI model based on emotional data.

[0707] The generative AI model outputs content that reflects the user's emotions, creating a presentation tailored to the user.

[0708] Input: User sentiment data.

[0709] Output: Generated presentation materials, speech transcript, and demo video data.

[0710] Step 5:

[0711] The server provides the generated content to the user through the terminal.

[0712] Transfer the generated results to the user's device and make them available for download or streaming.

[0713] Input: Generated presentation and video data.

[0714] Output: Content data provided to the user's device.

[0715] Step 6:

[0716] Users practice their presentations on their devices and record practice videos.

[0717] Users practice their presentations using their device's camera function and record themselves doing so.

[0718] Input: Actual presentation performance by users.

[0719] Output: Recorded practice video data.

[0720] Step 7:

[0721] The device uploads the recorded practice video to the server.

[0722] The user operates the device to send the video file to the server.

[0723] Input: Practice video data saved on the device.

[0724] Output: Practice video data received by the server.

[0725] Step 8:

[0726] The server analyzes the received practice videos, analyzing audio and visual information to generate feedback.

[0727] The server utilizes speech recognition and video analysis technologies to identify areas for improvement in the presentation.

[0728] Input: Practice video data.

[0729] Output: Feedback data based on analysis results.

[0730] Step 9:

[0731] The server generates instructional videos, including feedback, and provides them to the user.

[0732] The server generates instructional videos incorporating the advice and provides them to the user's device.

[0733] Input: Feedback data.

[0734] Output: Video data for instruction.

[0735] (Application Example 2)

[0736] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0737] Existing presentation preparation systems often lack sufficient feedback that adapts to user emotions, making it difficult to address situations where users feel nervous or anxious. Furthermore, real-time emotion analysis and feedback are rarely seamless, making it difficult to immediately obtain specific advice to improve presentation quality.

[0738] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0739] In this invention, the server includes an information exchange unit means for receiving prompts, a creation unit means for automatically generating presentation materials, speaking patterns, and demonstration videos using a generation model based on the received prompts, and an emotion analysis unit means for analyzing the user's emotions in real time and providing immediate feedback. This enables the preparation of presentations that are tailored to the user's emotions and realizes the provision of appropriate feedback through real-time emotion analysis.

[0740] The "information exchange unit" is an interface that receives prompts from the user and exchanges information with other system components.

[0741] The "creation unit" is a component that uses a generative model based on received prompts to automatically generate presentation materials and demonstration videos suitable for the user.

[0742] The "distribution section" is the means by which generated information is provided to users, and it is the part that plays the role of disseminating information in a form that is easily accessible to users.

[0743] The "video receiving unit" is a system component that receives the training video shot by the user and sends it to the next analysis step.

[0744] The "analysis unit" is the part that analyzes the received practice video and provides functions to generate feedback that encourages users to improve their audio and visual performance.

[0745] The "Education Department" is a section that promotes education by creating educational videos based on improvement suggestions provided by the Analysis Department and making them available to users.

[0746] The "emotional analysis unit" is an analysis component that analyzes the user's emotions in real time and provides immediate feedback based on the results.

[0747] To implement this invention, a terminal device equipped with a user interface is used to receive prompts from the user. The user first inputs the theme and intention of their presentation as a prompt. This prompt is sent to the server via the information exchange unit. The server analyzes the received prompt and uses the creation unit to utilize a generation AI model to automatically generate presentation materials and demonstration videos suitable for the user.

[0748] The generated materials are provided to the user through the distribution unit. The user then uses the provided materials to perform exercises and records the exercises via a terminal device. The recorded video is then transmitted to the server from the video receiving unit.

[0749] The server analyzes the received video in its analysis unit and generates feedback on audio and visual performance. This feedback information is then processed by the education department into educational videos and distributed to users.

[0750] Furthermore, an emotion analysis unit is provided, which analyzes the user's emotions in real time and provides immediate feedback based on the analysis results. This entire process utilizes natural language processing technologies such as the Google Cloud Natural Language API, and GPT-based AI models are used as the generative AI models.

[0751] As a concrete example, when a user is preparing a presentation to introduce a new product, they can enter "new product presentation audience engaging" into the prompt. Based on this prompt, the generated materials and videos will be tailored to the user's objectives, and feedback based on emotional analysis will provide appropriate information, contributing to the improvement of the user's presentation skills.

[0752] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0753] Step 1:

[0754] The user inputs information about the presentation's theme and intent as prompt messages into the terminal. These prompt messages are sent to the server via the information exchange unit. The input process involves receiving text data such as the theme and emotions, and transmitting it to the server.

[0755] Step 2:

[0756] The server analyzes the received prompt message and processes the information using a generation AI model in the creation unit. This process takes the prompt message as input to generate presentation materials and demonstration videos optimized for the user. The generation AI model's calculations ensure that content matching the user's needs is output.

[0757] Step 3:

[0758] The generated materials and videos are provided to the user's terminal via the server's distribution section. Optimized content is sent from the server, and actions are taken for the user to access and use that content.

[0759] Step 4:

[0760] Users perform exercises based on the provided materials, recording the exercises using their devices. Through this concrete action of recording, the system generates video input data of the user's presentation.

[0761] Step 5:

[0762] The recorded practice video is transmitted from the terminal to the server's video receiving unit. The received video data is supplied to the analysis unit, where visual and audio data are analyzed. This generates performance feedback. The video received as input undergoes analysis to produce output data in the form of feedback.

[0763] Step 6:

[0764] The generated feedback is then converted into educational videos in the Education Department on the server. These videos are provided to the user as output, supporting them in viewing the feedback and using it to make improvements.

[0765] Step 7:

[0766] The emotion analysis unit analyzes the user's emotions in real time and immediately provides feedback to the user using a generative model. The input is the user's emotional data, and the system provides feedback to the user based on that data as output data.

[0767] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0768] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0769] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0770] [Fourth Embodiment]

[0771] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0772] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0773] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0774] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0775] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0776] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0777] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0778] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0779] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0780] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0781] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0782] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0783] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0784] This invention is a system that automatically generates presentation materials, speech scripts, and demo videos based on user prompt input, and provides feedback through analysis of practice videos recorded by the user. This system mainly consists of three elements: a server, a terminal, and the user.

[0785] First, the user uses a terminal to enter prompts that include the presentation's theme and objectives. These prompts play a crucial role in providing specific information based on the user's needs. Next, the terminal sends these prompts to the server. Based on the received prompts, the server uses a generative model to generate presentation materials, a speech script, and a demo video. This allows the user to quickly receive content tailored to their needs.

[0786] When users practice using generated materials and scripts, they record their presentations with their devices. These practice videos are important data for improving user performance. The devices upload the recorded practice videos to the server.

[0787] Uploaded videos are analyzed by an analysis unit on the server, which examines items such as volume, speaking speed, gaze, and posture. Based on this analysis, the server generates feedback. This feedback includes specific suggestions and advice to improve the user's presentation skills.

[0788] Finally, the server creates instructional videos based on the generated feedback and provides them to the user via the terminal. These instructional videos contain advice that can help improve the user's performance, allowing them to enhance the quality of their next presentation.

[0789] As a concrete example, consider preparing a speech for a wedding. The user enters the prompt "wedding speech groom's friend," and the server generates a speech script and slides based on that information. When the user practices the speech and uploads a video recorded on their device to the server, feedback is generated regarding the volume of their voice and their demeanor. The user can then use this feedback to practice further and make improvements.

[0790] This system allows users to effectively prepare and improve their presentations themselves, eliminating the need to rely on expensive professional guidance.

[0791] The following describes the processing flow.

[0792] Step 1:

[0793] The user will use their device to enter detailed information about the purpose and theme of the presentation as prompts.

[0794] Step 2:

[0795] The terminal receives the entered prompt, formats it, and prepares to send it to the server.

[0796] Step 3:

[0797] The terminal sends the prepared prompt data to the server.

[0798] Step 4:

[0799] The server analyzes the received prompt data and passes it to the generative model as input data.

[0800] Step 5:

[0801] The server automatically generates presentation materials, speech scripts, and demo videos based on prompts using a generative model.

[0802] Step 6:

[0803] The server organizes the generated content and sends it to the terminal.

[0804] Step 7:

[0805] The terminal provides the user with content received from the server and displays it so that the user can review the content.

[0806] Step 8:

[0807] Users review the generated materials and drafts and practice their presentations based on them.

[0808] Step 9:

[0809] Users record the practice process on their devices.

[0810] Step 10:

[0811] The user selects the best practice video they've recorded and uploads it to the server via their device.

[0812] Step 11:

[0813] The server receives the uploaded practice video and begins analyzing its content using the video analysis unit.

[0814] Step 12:

[0815] The server identifies items such as audio volume, speaking speed, gaze, and posture from the video and generates feedback based on the analysis results.

[0816] Step 13:

[0817] The server incorporates the generated feedback into the instructional video and sends it to the user's device, including areas for improvement.

[0818] Step 14:

[0819] The terminal provides users with instructional videos received from the server, allowing them to review areas for improvement and use that information in their next practice session.

[0820] (Example 1)

[0821] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0822] When preparing presentations and speeches, there is a need for automated content generation tailored to individual user needs and circumstances, as well as appropriate feedback during practice. However, a comprehensive system to efficiently achieve this has not existed until now. Furthermore, it has been difficult to provide users with specific guidance methods to improve their performance without expert instruction.

[0823] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0824] In this invention, the server includes an interface unit means for receiving input prompts, a generation unit means for automatically generating materials, manuscripts, and videos using a generation model based on the received prompts, and a provision unit means for providing the generated content to the user. This allows the user to efficiently prepare their presentations, receive feedback based on their practice results, and improve their skills.

[0825] The "interface unit that receives input prompts" is a component that receives text and instructions from the user, and it is the starting point for initiating processing based on that information.

[0826] The "generative unit that automatically generates materials, manuscripts, and videos using a generative model" refers to the part that has the function of generating presentation materials, speech manuscripts, and video content using machine learning algorithms based on received prompts.

[0827] The "content delivery unit that provides generated content to the user" is a component that has the role of sending and presenting the content created by the generation unit to the user.

[0828] The "video receiving unit that receives videos shot by the user" is a component that receives video files created by the user and acquires data for subsequent analysis processing.

[0829] The "analysis unit that analyzes received videos and generates performance feedback" is a part that analyzes video data and evaluates user performance to create feedback that includes areas for improvement and advice.

[0830] The "instructional unit that generates and provides instructional videos including feedback" is a component that creates instructional materials based on analysis results and provides users with the information they need to make improvements.

[0831] A "terminal device with a user interface" is an interactive device that allows users to input prompts or videos, and has the functionality to send and receive that data within the system.

[0832] This invention is an information processing system that enables users to efficiently prepare and practice presentations. The system consists of three elements: a server, a terminal, and the user.

[0833] The user uses the terminal to enter prompts indicating the theme and purpose of the presentation. An example of a prompt might be, "Create slides and a speech script for a new product launch." These prompts are sent to the server through the input interface.

[0834] The server uses a generative AI model based on received prompts to generate presentation materials, speech scripts, and demo videos. Specifically, the server utilizes a generative model based on a machine learning platform, for example. This generative model automatically analyzes prompts and generates related information, enabling the creation of high-quality content in a short amount of time.

[0835] The generated content is sent to the terminal via the service provider and displayed for the user to visually review. The user can then use this to practice their actual presentations.

[0836] During practice, users record their performance using the device's camera. This recorded video is uploaded to the server via a video receiving unit.

[0837] The server processes uploaded videos in its analysis unit. The analysis unit meticulously evaluates performance aspects such as voice, eye gaze, speaking speed, and posture. The data obtained from this analysis is used to generate feedback, which includes specific advice and suggestions for improving the user's skills.

[0838] Ultimately, the server creates instructional videos based on the feedback and provides them to users via their terminals through the instructional team. These instructional videos visually demonstrate areas for improvement and recommended techniques, allowing users to enhance the quality of their next presentations.

[0839] In this way, users can prepare and improve their presentations on their own, without the need for expert assistance.

[0840] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0841] Step 1:

[0842] The user enters prompts related to presentations and speeches through the terminal. These entered prompts are processed by the terminal's interface and sent to the server as text data. At this point, a specific prompt might be something like "Create slides and speech script for the new product launch."

[0843] Step 2:

[0844] The server calls a generative AI model based on prompts received from the terminal and performs data analysis. This analysis generates presentation materials, speech scripts, and demo videos that correspond to the content of the prompts. The generative AI model used by the server utilizes machine learning algorithms, for example, to produce highly reliable output. As a result of the analysis, content tailored to the user's requests is generated.

[0845] Step 3:

[0846] The generated content is sent to the terminal via the server's delivery unit. The terminal visually presents this received data to the user, who then reviews and downloads the content. At this time, presentation slides or speech scripts are displayed on the terminal, and the user can use them as practice material.

[0847] Step 4:

[0848] Users practice their presentations using their devices and record the process as a video. The recorded videos are treated as important data for analyzing the user's performance. In this step, actions and the flow of the speech during practice are recorded in detail.

[0849] Step 5:

[0850] The device uploads the recorded practice videos to the server via a video receiver. This video data is converted to an appropriate format and prepared for analysis on the server.

[0851] Step 6:

[0852] The server processes the received video in its analysis unit, performing a detailed analysis of attributes such as volume, speed, gaze, and posture. The analysis results are used to create feedback to improve the user's presentation skills. This process reveals the user's specific strengths and areas for improvement.

[0853] Step 7:

[0854] Based on the data obtained from the analysis, the server generates feedback and creates instructional videos. These instructional videos include specific points and advice to help users improve their skills.

[0855] Step 8:

[0856] The server sends the generated feedback and instructional videos to the device. The device receives this and provides it to the user so that they can use it to improve their presentation. As a result, the user has the material to take effective measures for their next presentation.

[0857] (Application Example 1)

[0858] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0859] In today's busy society, there is a need for efficient ways for individuals to prepare for presentations and speeches. While specialized and personalized instruction is desired, its high cost necessitates more accessible and effective support. Furthermore, there is a lack of technology-driven practice environments and difficulty in providing feedback tailored to individual skills and needs.

[0860] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0861] In this invention, the server includes communication means for receiving prompts, generation means for automatically generating materials, manuscripts, and videos using generation technology based on the received prompts, and speech recognition means for recognizing voice input through a robot and engaging in dialogue based on the individual's practice content. This makes it possible for individuals to easily practice, receive feedback, and improve their skills.

[0862] A "communication method" is an interface that exchanges data between a terminal and a server in order to send a prompt to the server.

[0863] "Generation means" refers to a function that automatically creates materials, manuscripts, and videos using generation technology based on received prompts.

[0864] "Means of delivery" refers to an interface for presenting generated content to an individual.

[0865] "Video receiving means" refers to a function that allows a server to receive practice videos filmed by individuals.

[0866] The "analysis means" is a function that analyzes the received practice video and generates feedback based on audio and visual information.

[0867] "Instructional tools" refer to a function that creates instructional videos based on feedback and provides them to individuals.

[0868] "Voice recognition means" refers to a function that recognizes voice input through a robot and engages in dialogue based on the individual's practice content.

[0869] "Adjustment means" refers to a function that adjusts the generated content and feedback according to voice instructions.

[0870] A "personal terminal device" is a device that enables the transmission of personal prompts and practice videos.

[0871] "Target audience" refers to the collective term for the audience and stakeholders who will receive the generated content.

[0872] "Characteristics" refer to the distinctive features and properties of the target audience or event, and are an important element in content creation.

[0873] To implement this invention, the user first uses a terminal device to input prompts that include a theme and purpose. These prompts serve as information to convey the user's intentions in detail. The terminal transmits these prompts to a server via a communication means. Based on the received prompts, the server uses generation technology to generate presentation materials, speech scripts, and demo videos.

[0874] The generated content is delivered to the user via a distribution method. The user practices using the generated materials and manuscripts, and uploads the practice videos to the server via a video receiving method.

[0875] The server analyzes practice videos using analytical tools and generates feedback based on speaking speed, visual information, and audio information. This feedback is provided to the individual as an instructional video using instructional tools. Furthermore, it has speech recognition capabilities that recognize voice input through the robot, allowing individuals to interact directly with the robot during practice.

[0876] A concrete example is when a user enters a prompt such as "wedding speech, groom's friend," and the server generates a relevant speech script and slides, providing feedback on their content. By receiving this feedback in audio format, the user can improve their practice.

[0877] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0878] Step 1:

[0879] The user enters prompts using a terminal device. The input consists of text data related to the presentation's theme and purpose. The terminal formats this data into a prompt message and transmits it to the server via a communication method.

[0880] Step 2:

[0881] The server receives the received prompt message as input and processes the data using a generation AI model. Specifically, it analyzes the prompt message and automatically generates presentation materials, speech scripts, and demo videos based on the theme. The output of this step is the generated content files.

[0882] Step 3:

[0883] The server compiles the generated content and transmits it to the terminal via a delivery method. The terminal displays these content files to the user, allowing the user to view the details.

[0884] Step 4:

[0885] The user practices based on this content. The device records the user's video and audio during practice and saves the data as a video file. This video file serves as input for the next step.

[0886] Step 5:

[0887] The terminal uploads saved practice videos to the server via a video receiving device. The server receives the video file as input and performs data calculations using an analysis device. It analyzes audio volume, speaking speed, visual information, etc., and generates feedback information. The output of this step is specific feedback information regarding the user's performance.

[0888] Step 6:

[0889] The server creates instructional videos based on the generated feedback information. Using instructional tools, it generates content that visually and audibly showcases areas for improvement for the user and sends it to the device. The user can then refer to these instructional videos and make improvements for their next presentation.

[0890] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0891] This invention is a system that recognizes user emotions and prepares and improves presentations based on those emotions. The system can analyze user emotions through prompts and practice videos and reflect this in the content of the presentation. It mainly consists of three elements: a server, a terminal, and a user, and is characterized by including an emotion analysis unit equipped with an emotion engine.

[0892] First, the user uses a terminal to input prompts related to the presentation's theme and purpose. These prompts are crucial as foundational data for reflecting the user's emotions. The terminal sends the entered prompts to the server. The server receives the prompts and recognizes the user's emotions in its emotion analysis unit. The emotion engine uses natural language processing technology to analyze the user's emotions from their input.

[0893] Next, the server uses a generative model to generate presentation materials, speech scripts, and demo videos based on the acquired sentiment data. The generated content takes the user's emotions into account. This process allows users to receive content that effectively reflects their own feelings.

[0894] The user practices their presentation using the generated materials and records a video of the practice session on their device. The recorded video is uploaded to the server via the device. The server analyzes the practice video, identifying changes in emotion in addition to audio and visual information. Based on this analysis, appropriate feedback is generated for the user.

[0895] This feedback includes specific advice to improve the user's presentation skills and is provided as instructional videos. These instructional videos, which include sentiment analysis results, help users gain a deeper understanding of the relationship between their emotions and their performance.

[0896] Specifically, when a user is preparing a wedding speech and enters "wedding speech groom's friend nervous" into the prompt, the server generates content that takes nervousness into account. Analysis of the practice video includes suggestions regarding speaking speed and volume to alleviate nervousness, providing emotionally responsive feedback. In this way, emotionally responsive support maximizes the effectiveness of the presentation.

[0897] The following describes the processing flow.

[0898] Step 1:

[0899] The user enters prompts on their device that include information about the presentation's theme and purpose. For example, they might type "wedding speech groom's friend nervous".

[0900] Step 2:

[0901] The terminal sends the entered prompt to the server. This prompt becomes important data for emotion recognition.

[0902] Step 3:

[0903] The server passes the received prompt to the emotion analysis unit, which uses natural language processing technology to recognize the user's emotions from the prompt. This is where emotions like "tension" are extracted.

[0904] Step 4:

[0905] Based on the recognized emotions, the server uses a generative model to generate presentation materials, speech scripts, and demo videos. This content is then adjusted to reflect the user's emotions.

[0906] Step 5:

[0907] The server sends the generated content to the terminal, which then displays it to the user. The user reviews the content and prepares to use it in the next step.

[0908] Step 6:

[0909] Users practice their speeches using materials and scripts displayed on their devices, and record themselves doing so on video. These practice videos contain important data, including the user's emotional state.

[0910] Step 7:

[0911] Users upload practice videos they've recorded to the server via their device. These videos are necessary to evaluate the user's practical presentation skills.

[0912] Step 8:

[0913] The server activates the video analysis unit to analyze the practice video, detecting changes in the user's emotions in addition to volume, speaking speed, gaze, and posture.

[0914] Step 9:

[0915] Based on the analysis results, the server generates feedback to improve the user's presentation skills. This feedback also includes emotionally responsive advice.

[0916] Step 10:

[0917] The server creates instructional videos and incorporates the generated feedback. These videos help users understand the connection between their emotions and their presentation skills.

[0918] Step 11:

[0919] The device provides users with instructional videos received from the server, enabling them to improve their next presentation through feedback. Users then use this as a reference for further practice and adjustments.

[0920] (Example 2)

[0921] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0922] In recent years, presentations have become an important element in a variety of situations, but many people find it difficult to effectively convey their emotions. Furthermore, emotions such as nervousness and anxiety can also affect the quality of a presentation. For this reason, there is a need for a system that prepares and improves effective presentations that reflect the user's emotions.

[0923] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0924] In this invention, the server includes an information input unit means for receiving information input, an analysis engine means for analyzing the input information and recognizing the emotions of the person who entered it, and a generation device means for automatically creating presentation materials, presentation documents, and demonstration videos using generation technology based on the analysis engine. This makes it possible to automatically support the preparation and improvement of presentations while reflecting the user's emotions.

[0925] The "information input section" is where users can input information related to the presentation's theme and purpose.

[0926] An "analysis engine" is a device that processes received information and uses natural language processing techniques to generate emotion data in order to identify emotions.

[0927] The "generation device" is the part that has the function of creating presentation materials, presentation documents, and demonstration videos using emotional data obtained from the analysis engine.

[0928] The "communication section" is the part that provides the generated content to the user and makes it available for use.

[0929] The "video receiving unit" is an element used to transmit practice videos recorded by the user to the server.

[0930] The "analysis device" is the part that analyzes the received practice video, identifies changes in audio and visual information, and creates feedback.

[0931] The "instructional device" is the part that generates and provides instructional videos incorporating feedback to the user.

[0932] A "personal information device" is a device that allows a user to send prompts and video data to a server.

[0933] This invention is a system that incorporates user emotions to assist in the preparation and improvement of presentations, and mainly consists of three elements: a server, a terminal, and the user. It utilizes a generative AI model to analyze the user's emotions through prompt text and reflect the results in the presentation.

[0934] First, the user uses their device to input prompts related to the presentation's theme and purpose. At this point, they input prompt text via a text editor or dedicated application on their device, such as "wedding speech groom's friend nervous." This forms the basis of the emotion recognition process described later.

[0935] The terminal sends the entered prompt to the server, which uses an emotion analysis engine to analyze the user's emotions from the prompt. This analysis uses software that implements natural language processing technology. Specifically, a Python library running on the server extracts emotions from text data and provides that data as input to a generative model.

[0936] Next, the server uses a generative AI model to generate presentation materials, speech scripts, and demo videos that reflect the user's emotions. Here, for example, natural language models and image generation models are used to generate information that matches the user's input and emotional data. The generated content is then provided to the user again via the terminal, allowing the user to practice their presentation.

[0937] After the user records a practice video on their device, the device uploads the video to a server. The server analyzes this practice video, using speech recognition and visual analysis technologies to thoroughly check the audio, visual information, and emotional changes in the presentation. The feedback obtained from this analysis is provided to the user as specific advice.

[0938] For example, when a user is preparing a wedding speech, if they enter the prompt "wedding speech groom's friend nervous," the server will create content tailored to that emotion and generate advice on tone and speaking speed to alleviate nervousness based on the analysis of a practice video. This system enables users to prepare presentations that reflect their emotions, maximizing their effectiveness.

[0939] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0940] Step 1:

[0941] The user uses a terminal to enter prompts about the presentation's theme and purpose.

[0942] In terms of specific actions, the user inputs information in text format into a designated application on the terminal using the keyboard and sends a prompt message such as "wedding speech groom's friend nervous".

[0943] Input: Text input of a prompt message by the user.

[0944] Output: Prompt message data sent from the terminal to the server.

[0945] Step 2:

[0946] The terminal sends a prompt message to the server.

[0947] The terminal sends the prompt text entered by the user as a request to the server via its network communication function.

[0948] Input: Prompt message data held by the terminal.

[0949] Output: Prompt message data reached the server.

[0950] Step 3:

[0951] The server passes the received prompt message to the sentiment analysis engine, which then analyzes the user's emotions.

[0952] The server uses natural language processing libraries such as Python to analyze the prompt text and extract sentiment data.

[0953] Input: Prompt statement data.

[0954] Output: User sentiment data.

[0955] Step 4:

[0956] The server generates presentation materials, speech scripts, and demo videos using an AI model based on emotional data.

[0957] The generative AI model outputs content that reflects the user's emotions, creating a presentation tailored to the user.

[0958] Input: User sentiment data.

[0959] Output: Generated presentation materials, speech transcript, and demo video data.

[0960] Step 5:

[0961] The server provides the generated content to the user through the terminal.

[0962] Transfer the generated results to the user's device and make them available for download or streaming.

[0963] Input: Generated presentation and video data.

[0964] Output: Content data provided to the user's device.

[0965] Step 6:

[0966] Users practice their presentations on their devices and record practice videos.

[0967] Users practice their presentations using their device's camera function and record themselves doing so.

[0968] Input: Actual presentation performance by users.

[0969] Output: Recorded practice video data.

[0970] Step 7:

[0971] The device uploads the recorded practice video to the server.

[0972] The user operates the device to send the video file to the server.

[0973] Input: Practice video data saved on the device.

[0974] Output: Practice video data received by the server.

[0975] Step 8:

[0976] The server analyzes the received practice videos, analyzing audio and visual information to generate feedback.

[0977] The server utilizes speech recognition and video analysis technologies to identify areas for improvement in the presentation.

[0978] Input: Practice video data.

[0979] Output: Feedback data based on analysis results.

[0980] Step 9:

[0981] The server generates instructional videos, including feedback, and provides them to the user.

[0982] The server generates instructional videos incorporating the advice and provides them to the user's device.

[0983] Input: Feedback data.

[0984] Output: Video data for instruction.

[0985] (Application Example 2)

[0986] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0987] Existing presentation preparation systems often lack sufficient feedback that adapts to user emotions, making it difficult to address situations where users feel nervous or anxious. Furthermore, real-time emotion analysis and feedback are rarely seamless, making it difficult to immediately obtain specific advice to improve presentation quality.

[0988] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0989] In this invention, the server includes an information exchange unit means for receiving prompts, a creation unit means for automatically generating presentation materials, speaking patterns, and demonstration videos using a generation model based on the received prompts, and an emotion analysis unit means for analyzing the user's emotions in real time and providing immediate feedback. This enables the preparation of presentations that are tailored to the user's emotions and realizes the provision of appropriate feedback through real-time emotion analysis.

[0990] The "information exchange unit" is an interface that receives prompts from the user and exchanges information with other system components.

[0991] The "creation unit" is a component that uses a generative model based on received prompts to automatically generate presentation materials and demonstration videos suitable for the user.

[0992] The "distribution section" is the means by which generated information is provided to users, and it is the part that plays the role of disseminating information in a form that is easily accessible to users.

[0993] The "video receiving unit" is a system component that receives the training video shot by the user and sends it to the next analysis step.

[0994] The "analysis unit" is the part that analyzes the received practice video and provides functions to generate feedback that encourages users to improve their audio and visual performance.

[0995] The "Education Department" is a section that promotes education by creating educational videos based on improvement suggestions provided by the Analysis Department and making them available to users.

[0996] The "emotional analysis unit" is an analysis component that analyzes the user's emotions in real time and provides immediate feedback based on the results.

[0997] To implement this invention, a terminal device equipped with a user interface is used to receive prompts from the user. The user first inputs the theme and intention of their presentation as a prompt. This prompt is sent to the server via the information exchange unit. The server analyzes the received prompt and uses the creation unit to utilize a generation AI model to automatically generate presentation materials and demonstration videos suitable for the user.

[0998] The generated materials are provided to the user through the distribution unit. The user then uses the provided materials to perform exercises and records the exercises via a terminal device. The recorded video is then transmitted to the server from the video receiving unit.

[0999] The server analyzes the received video in its analysis unit and generates feedback on audio and visual performance. This feedback information is then processed by the education department into educational videos and distributed to users.

[1000] Furthermore, an emotion analysis unit is provided, which analyzes the user's emotions in real time and provides immediate feedback based on the analysis results. This entire process utilizes natural language processing technologies such as the Google Cloud Natural Language API, and GPT-based AI models are used as the generative AI models.

[1001] As a concrete example, when a user is preparing a presentation to introduce a new product, they can enter "new product presentation audience engaging" into the prompt. Based on this prompt, the generated materials and videos will be tailored to the user's objectives, and feedback based on emotional analysis will provide appropriate information, contributing to the improvement of the user's presentation skills.

[1002] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1003] Step 1:

[1004] The user inputs information about the presentation's theme and intent as prompt messages into the terminal. These prompt messages are sent to the server via the information exchange unit. The input process involves receiving text data such as the theme and emotions, and transmitting it to the server.

[1005] Step 2:

[1006] The server analyzes the received prompt message and processes the information using a generation AI model in the creation unit. This process takes the prompt message as input to generate presentation materials and demonstration videos optimized for the user. The generation AI model's calculations ensure that content matching the user's needs is output.

[1007] Step 3:

[1008] The generated materials and videos are provided to the user's terminal via the server's distribution section. Optimized content is sent from the server, and actions are taken for the user to access and use that content.

[1009] Step 4:

[1010] Users perform exercises based on the provided materials, recording the exercises using their devices. Through this concrete action of recording, the system generates video input data of the user's presentation.

[1011] Step 5:

[1012] The recorded practice video is transmitted from the terminal to the server's video receiving unit. The received video data is supplied to the analysis unit, where visual and audio data are analyzed. This generates performance feedback. The video received as input undergoes analysis to produce output data in the form of feedback.

[1013] Step 6:

[1014] The generated feedback is then converted into educational videos in the Education Department on the server. These videos are provided to the user as output, supporting them in viewing the feedback and using it to make improvements.

[1015] Step 7:

[1016] The emotion analysis unit analyzes the user's emotions in real time and immediately provides feedback to the user using a generative model. The input is the user's emotional data, and the system provides feedback to the user based on that data as output data.

[1017] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1018] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1019] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1020] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1021] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1022] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1023] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1024] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1025] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1026] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1027] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1028] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1029] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1030] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1031] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1032] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1033] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1034] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1035] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1036] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1037] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1038] The following is further disclosed regarding the embodiments described above.

[1039] (Claim 1)

[1040] Interface means for receiving prompts,

[1041] A generation unit means that automatically generates presentation materials, speech scripts, and demo videos using a generation model based on received prompts,

[1042] A provisioning unit means that provides generated content to the user,

[1043] A video receiving unit means for receiving practice videos taken by the user,

[1044] An analysis unit means analyzes the received practice video and generates feedback on audio and visual performance,

[1045] A training system that generates instructional videos including feedback and provides them to users,

[1046] A system that includes this.

[1047] (Claim 2)

[1048] A terminal device having a user interface that enables the transmission of user prompts and practice videos, according to claim 1.

[1049] (Claim 3)

[1050] The system according to claim 1, characterized in that the generation unit creates content suitable for the user's target audience and the nature of the event based on the analysis of the prompt.

[1051] "Example 1"

[1052] (Claim 1)

[1053] An interface unit means for receiving input prompts,

[1054] A generation unit means that automatically generates materials, manuscripts, and videos using a generation model based on received prompts,

[1055] A provisioning unit means for providing the generated content to the user,

[1056] A video receiving unit means for receiving videos taken by the user,

[1057] An analysis unit means that analyzes the received video and generates performance-related feedback,

[1058] A training system that generates instructional videos including feedback and provides them to users,

[1059] An information processing system that includes this.

[1060] (Claim 2)

[1061] The information processing system according to claim 1, comprising a terminal device with a user interface that enables the transmission of user prompts and practice videos.

[1062] (Claim 3)

[1063] The information processing system according to claim 1, characterized in that the generation unit creates content suitable for the user's target audience and the characteristics of the event by analyzing the prompt.

[1064] "Application Example 1"

[1065] (Claim 1)

[1066] A means of communication for receiving prompts,

[1067] A generation means that automatically generates materials, manuscripts, and videos using generation technology based on received prompts,

[1068] A means of providing content generated for individuals,

[1069] A video receiving method for receiving practice videos filmed by individuals,

[1070] An analysis means that analyzes received practice videos and generates feedback regarding audio and visual information,

[1071] A teaching method that generates and provides individual instructional videos including feedback,

[1072] A voice recognition means that recognizes voice input through a robot and engages in dialogue based on the individual's practice content,

[1073] A means for adjusting the content and feedback generated in response to instructions given by voice,

[1074] A system that includes this.

[1075] (Claim 2)

[1076] The system according to claim 1, comprising a personal terminal device that enables the transmission of personal prompts and practice videos.

[1077] (Claim 3)

[1078] The system according to claim 1, characterized in that the generation means creates content suitable for the characteristics of the target person or event based on the analysis of the prompt, and the speech recognition means generates feedback based on the individual's practice.

[1079] "Example 2 of combining an emotion engine"

[1080] (Claim 1)

[1081] Information input unit means for receiving information input,

[1082] An analysis engine means that analyzes the input information and recognizes the emotions of the person who entered it,

[1083] A generation device means that automatically creates presentation materials, presentation documents, and demonstration videos using generation technology based on an analysis engine,

[1084] A transmission unit means for transmitting the created information to an individual,

[1085] A video receiving unit means for receiving practice videos recorded by an individual,

[1086] An analysis device means that analyzes the received practice video, identifies changes in acoustic and visual information, and creates feedback,

[1087] A teaching device that creates instructional videos incorporating feedback and transmits them to individuals,

[1088] A system that includes this.

[1089] (Claim 2)

[1090] The system according to claim 1, comprising a personal information device that enables an individual to transmit information and images.

[1091] (Claim 3)

[1092] The system according to claim 1, characterized in that the information creation device generates content tailored to the characteristics of an individual's target audience or event based on the analyzed information.

[1093] "Application example 2 when combining with an emotional engine"

[1094] (Claim 1)

[1095] Information exchange unit means for receiving prompts,

[1096] A creation unit means that automatically generates presentation materials, speaking patterns, and demonstration videos using a generative model based on received prompts,

[1097] A distribution unit means that provides generated information to the user,

[1098] A video receiving unit means for receiving practice videos taken by the user,

[1099] An analysis unit means that analyzes the received training video and generates improvement suggestions regarding audio and visual performance,

[1100] An educational means for generating and providing educational videos containing improvement suggestions to users,

[1101] An emotion analysis unit means that analyzes the user's emotions in real time and provides immediate feedback,

[1102] A system that includes this.

[1103] (Claim 2)

[1104] The system according to claim 1, comprising a terminal device that enables the transmission of user prompts and practice videos.

[1105] (Claim 3)

[1106] The system according to claim 1, characterized in that the generation unit creates information suitable for the user's expression style and target audience based on emotional analysis. [Explanation of symbols]

[1107] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Interface means for receiving prompts, A generation unit means that automatically generates presentation materials, speech scripts, and demo videos using a generation model based on received prompts, A provisioning unit means that provides generated content to the user, A video receiving unit means for receiving practice videos taken by the user, An analysis unit means analyzes the received practice video and generates feedback on audio and visual performance, A training system that generates instructional videos including feedback and provides them to users, A system that includes this.

2. The system according to claim 1, comprising a terminal device with a user interface that enables the transmission of user prompts and practice videos.

3. The system according to claim 1, characterized in that the generation unit creates content suitable for the user's target audience and the nature of the event based on the analysis of the prompt.

Images