system

Abstract

We provide the system. [Solution] A means of receiving and storing user information regarding registered readings, A method for analyzing text data from a book being read to identify difficult sections, A means of generating and providing explanatory information for identified difficult parts in response to user requests, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Modern readers are in an environment where they can access a lot of information, but many problems remain in the reading experience. First, because it is difficult to understand the content, many people often give up reading halfway. Also, there is the annoyance of not being able to remember the names and relationships of the characters and having to check every time. Furthermore, although one can enjoy reading more deeply if one knows the original source and background information, there are times when one cannot access such information. There is a need for an effective system to solve such problems and improve the reading experience of readers.

Means for Solving the Problems

[0005] To solve these problems, the present invention provides the following means. First, it provides a system that can register user information and provide reading support according to the user's preferences and level of requirements. Specifically, it includes means for receiving text data of the book being read from the user, identifying difficult parts using a natural language processing algorithm, and generating and providing explanatory information. Furthermore, it includes means for analyzing information on characters, interactively displaying a list of characters, and providing detailed information on each character. In addition, it also includes means for analyzing source material and background information related to the book being read, and generating and providing that information. As a result, readers can easily understand the content, remember the characters, and gain a richer reading experience by understanding the source material and background.

[0006] A "user" is an individual who provides information and receives support to improve their reading experience using this system.

[0007] "User information about reading" refers to information that users provide regarding reading, such as their preferences, reading level, and reading frequency.

[0008] "Text data" refers to digital text information of books that a user wants to read or is currently reading.

[0009] "Analysis" refers to the data processing and evaluation process performed by a system based on text data and user information.

[0010] A "natural language processing algorithm" is a program that analyzes text data and converts complex parts and meanings into a format that a computer can understand.

[0011] "Difficult sections" refer to sections or paragraphs within the book's text data that are considered difficult to understand.

[0012] "Explanatory information" refers to supplementary explanations generated to clarify difficult parts and aid understanding.

[0013] A "character list" is a list containing the names of all the characters that appear in a particular book.

[0014] "Interactive display" refers to a display format that shows information in a way that allows the user to interact with it, and to view detailed information as needed.

[0015] "Detailed information" refers to additional information related to each character included in the list, such as their characteristics, relationships, and roles.

[0016] "Source material" refers to the underlying information, such as past works, events, or cultural backgrounds, that influenced the content or themes of a book.

[0017] "Background information" refers to all historical, cultural, and social information that helps to deepen one's understanding of the book's content and themes. [Brief explanation of the drawing]

[0018] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] Shows an emotion map to which a plurality of emotions are mapped. [Figure 10] Shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be described.

[0021] In the following embodiments, a processor with a reference numeral (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of a plurality of arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of a plurality of types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] Patent Specification

[0040] This invention is an AI assistant system for improving the reading experience, which analyzes and provides various information about the book the user is reading. The program of this system and its specific processing are described below.

[0041] 1. Registration and Initial Setup

[0042] User:

[0043] Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[0044] After launching the app, tap the "New Registration" button, enter the required information such as your name, email address, and password, and set your preferred genre, difficulty level, reading frequency, etc.

[0045] Terminal:

[0046] It receives information entered by the user and sends it to the server.

[0047] server:

[0048] The received user information is saved to the database, a registration completion message is generated, and sent to the terminal.

[0049] 2. Text analysis and information provision

[0050] User:

[0051] Enter the title, author, and specific chapter or page number of the book you are reading into the app. If necessary, specify the parts you are having difficulty understanding.

[0052] Terminal:

[0053] The information entered by the user is sent to the server.

[0054] server:

[0055] The text data of a specified book is retrieved from a database and analyzed using a natural language processing algorithm. Particularly difficult sections are identified, and explanatory information is generated.

[0056] The system checks for spoilers, generates appropriate explanatory information, and sends it to the device.

[0057] Terminal:

[0058] The server sends explanatory information to the user. Graphics and icons are also displayed in an interactive format.

[0059] Specific example:

[0060] If a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. For example, if "The Use of Symbolism" is identified as a difficult section, an explanation of its background and intent will be provided, such as "Symbolism is used in this section, and certain symbols have a deeper meaning than they do in reality."

[0061] 3. Generation and provision of the character list

[0062] User:

[0063] If you can't remember the names of the characters, enter that into the app.

[0064] Terminal:

[0065] Send the request to the server.

[0066] server:

[0067] This program analyzes and generates a character list from the text data of a specified book. It then adds detailed information about each character (e.g., characteristics, relationships, appearances, etc.).

[0068] Terminal:

[0069] The system interactively displays a list of characters sent from the server, allowing users to view detailed information through actions such as clicking.

[0070] Specific example:

[0071] If a user requests a list of characters from "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, who is knowledgeable and always offers advice."

[0072] 4. Providing source material and background information

[0073] User:

[0074] Specify the parts of the book for which you want to know the source material or background information.

[0075] Terminal:

[0076] Send the request to the server.

[0077] server:

[0078] The system collects source material and background information for the specified portion from a database, analyzes it, and generates appropriate information.

[0079] Terminal:

[0080] The server provides users with source material and background information. Care is taken to avoid excessive spoilers.

[0081] Specific example:

[0082] If a user inputs a request to learn about Chapter 3 of "Book C," the server analyzes the source material and background information for that section and provides information such as, "This chapter is based on medieval legends and strongly reflects elements of XXX in particular."

[0083] Thus, the present invention specifically realizes a system that provides information to enrich the user's reading experience and deepen their understanding.

[0084] The following describes the processing flow.

[0085] Program processing flow

[0086] 1. Registration and Initial Setup

[0087] Step 1:

[0088] User: Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[0089] Step 2:

[0090] User: Tap the "Register" button and enter the required information such as your name, email address, and password.

[0091] Step 3:

[0092] User: Sets preferences such as genre, difficulty level, and reading frequency.

[0093] Step 4:

[0094] Terminal: Converts user-entered information into JSON format and sends it to the server using the API.

[0095] Step 5:

[0096] Server: Stores received user information in the database.

[0097] Step 6:

[0098] Server: Generates a registration completion message and sends it to the terminal.

[0099] Step 7:

[0100] Terminal: Displays a registration completion message to the user.

[0101] 2. Text analysis and information provision

[0102] Step 1:

[0103] User: Enter the title, author, and specific chapter or page number of the book you are reading into the app. If there are parts you find difficult to understand, enter them as comments.

[0104] Step 2:

[0105] Terminal: Converts user input information into JSON format and sends it to the server using the API.

[0106] Step 3:

[0107] Server: Retrieves text data for the specified book from the database.

[0108] Step 4:

[0109] Server: Analyzes text data using natural language processing (NLP) algorithms to identify difficult parts.

[0110] Step 5:

[0111] Server: Generates explanatory information for identified difficult sections. Selects appropriate explanations based on whether or not spoilers are included.

[0112] Step 6:

[0113] Server: Converts explanatory information into JSON format and sends it to the terminal.

[0114] Step 7:

[0115] Terminal: Interactively displays and provides explanatory information received from the server to the user.

[0116] 3. Generation and provision of the character list

[0117] Step 1:

[0118] User: If you can't remember the names of the characters, enter that into the app.

[0119] Step 2:

[0120] Terminal: Converts user requests into JSON format and sends them to the server.

[0121] Step 3:

[0122] Server: Analyzes and generates a list of characters from the text data of the specified book.

[0123] Step 4:

[0124] Server: Adds detailed information about each character (e.g., characteristics, relationships, locations of appearance, etc.).

[0125] Step 5:

[0126] Server: Converts the generated character list into JSON format and sends it to the terminal.

[0127] Step 6:

[0128] Terminal: Interactively displays the character list received from the server, allowing users to view detailed information through actions such as clicking.

[0129] 4. Providing source material and background information

[0130] Step 1:

[0131] User: Specify to the app the parts of a book for which you want to know the source material or background information.

[0132] Step 2:

[0133] Terminal: Converts user requests into JSON format and sends them to the server.

[0134] Step 3:

[0135] Server: Collects source material and background information for the specified section from the database.

[0136] Step 4:

[0137] Server: Analyzes collected source material and background information to generate appropriate information.

[0138] Step 5:

[0139] Server: Converts the generated source material and background information into JSON format and sends it to the terminal.

[0140] Step 6:

[0141] Terminal: Provides users with source material and background information received from the server. Care is taken to avoid excessive spoilers.

[0142] In this way, the program enriches the user's reading experience and provides information to deepen their understanding through each processing step.

[0143] (Example 1)

[0144] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0145] Traditional reading systems have a problem in that they cannot provide users with appropriate explanations or background information for difficult or hard-to-understand parts of a book. Furthermore, it is difficult for users to remember character names and detailed information, leading to frequent interruptions in the reading experience. In addition, the source material and background information are not provided in an appropriate format, making it difficult to achieve a deep understanding. There is a need for a system that solves these problems and improves the reading experience.

[0146] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0147] In this invention, the server includes means for receiving and storing user information related to registered readings, means for analyzing text data of the medium being read and identifying difficult sections, and means for generating and providing explanatory information for the identified difficult sections in response to user requests, using a generative artificial intelligence model. This makes it possible to immediately provide appropriate explanatory information for difficult sections that the user encounters while reading.

[0148] The server further includes means for analyzing information about characters appearing in the reading medium and generating a character list, and means for interactively displaying the generated character list and providing detailed information about each character. This allows users to easily check detailed information about characters, dramatically improving the reading experience.

[0149] Furthermore, the server includes means for analyzing source material and background information related to the medium being read and generating that information, and means for providing the generated source material and background information to the user. This allows the user to gain a deeper understanding of the background and source material of the medium being read, thereby enriching the reading experience.

[0150] "Registered user information related to reading" refers to personal information entered by the user into the system, such as name, email address, password, preferred genres, difficulty level, and reading frequency.

[0151] "Text data in a medium" refers to electronic text data such as books, magazines, and academic papers that a user is reading.

[0152] "Analysis" refers to the process of analyzing the content of text data using natural language processing algorithms and other methods to extract specific patterns or meanings.

[0153] "Difficult parts" refer to sections of text that users find hard to understand.

[0154] "Generative artificial intelligence models" refer to machine learning models and deep learning models used for natural language processing and data analysis. Specifically, this includes advanced text generation models such as GPT-3®.

[0155] "Explanatory information" refers to additional explanations or background information for parts that users found difficult to understand.

[0156] A "character list" refers to a list of characters that appear in the medium being read. This list includes each character's name, role, and characteristics.

[0157] "Source material and background information" refers to information about the historical and cultural background and original sources that influenced the content of the medium being read.

[0158] "Interactive display" refers to a display method that allows users to view additional details or explanations by performing actions such as tapping or clicking.

[0159] An "HTTP request" refers to a form of data communication based on a protocol used when sending data from a terminal to a server.

[0160] This invention is an AI assistant system designed to enhance the user's reading experience. It utilizes natural language processing and generative AI models to analyze and provide various information about the medium. The system analyzes the text the user is reading in detail and interactively provides lists of difficult passages and characters, as well as source material and background information.

[0161] Registration and initial setup

[0162] User: Download and install the application on your smartphone or tablet. After installation, launch the application and tap the "New Registration" button. Complete the initial setup by entering your name, email address, password, and preferred genres, difficulty level, reading frequency, etc.

[0163] Terminal: Receives information entered by the user and sends it to the server. Specifically, it converts the user's information into JSON format and sends it to the server as an HTTP request.

[0164] Server: Stores received user information in the database. The database used is an RDBMS (Relational Database Management System) such as MySQL® or PostgreSQL. After saving is complete, a registration completion message is generated and sent to the terminal.

[0165] Text analysis and information provision

[0166] User: Enter the title, author, and specific chapter or page number of the book you are reading into the app. If necessary, specify the parts you are having difficulty understanding.

[0167] Terminal: Sends user-entered information to the server. During transmission, text data is converted to JSON format and sent via the API.

[0168] Server: Retrieves text data from the specified medium from the database. Next, it uses a natural language processing algorithm to analyze the data and identify the difficult parts pointed out by the user. It then generates explanatory information for the difficult parts using a generation AI model (e.g., GPT-3). During this process, it checks for spoilers and generates explanatory information within an appropriate scope.

[0169] For example, if a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. If, for instance, "The Use of Symbolism" is identified as a difficult section, the server provides an explanation of its background and intent, such as, "Symbolism is used in this section, and certain symbols have a deeper meaning than they do in reality."

[0170] Terminal: Displays explanatory information sent from the server to the user. The explanatory information is displayed in an interactive format, including graphics and icons.

[0171] Generating and providing a list of characters

[0172] User: If you can't remember the names of the characters, enter that into the app.

[0173] Terminal: Sends requests to the server. Specifically, it converts the input request into JSON format and makes an API call.

[0174] Server: Analyzes and generates a character list from the text data of the specified medium. Adds detailed information about each character (e.g., characteristics, relationships, appearances, etc.). Uses natural language processing algorithms for analysis and retrieves necessary information from the database.

[0175] For example, if a user requests a list of characters from "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, who is knowledgeable and always offers advice."

[0176] Terminal: Interactively displays the character list sent from the server. Users can view detailed information by tapping on names in the list.

[0177] Providing source material and background information

[0178] User: Specify the parts of the book for which you want to know the source material or background information.

[0179] For example, if a user inputs that they want to know about Chapter 3 of "Book C," the server will analyze the source material and background information for that section and provide information such as, "This chapter is based on medieval legends and particularly strongly reflects elements of XXX."

[0180] Terminal: Sends the request to the server.

[0181] Server: Collects source material and background information for the specified section from the database, analyzes it, and generates appropriate information.

[0182] Terminal: Provides users with source material and background information sent from the server. Displays information while taking care to avoid excessive spoilers.

[0183] Example of a prompt

[0184] Prompt 1: "I am reading Chapter 5 of Book A, but I don't understand the use of symbolism. Please provide a detailed explanation."

[0185] Prompt 2: "Please provide a list of characters in 'Book B' and their respective characteristics."

[0186] Prompt 3: "What source material or background information is found in Chapter 3 of 'Book C'?"

[0187] As described above, the present invention specifically realizes the provision of information that enriches the user's reading experience and deepens their understanding.

[0188] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0189] Step 1:

[0190] The user downloads and installs the application on their smartphone or tablet. This provides the basic platform for accessing the system. Next, they launch the application and enter their name, email address, password, preferred genres, etc., on the new user registration screen. This collects the user's basic information.

[0191] Input: Username, email address, password, preferred genre, etc.

[0192] Output: User information entered into the terminal

[0193] Specific steps: The user searches for the app on the App Store or Google Play (registered trademark) and taps the download button. After the download is complete, the installation starts automatically and, once finished, an icon appears on the device's home screen. After launching the app, the user enters the required information on the registration screen that appears.

[0194] Step 2:

[0195] The terminal receives the information entered by the user and sends it to the server. The user's information is converted to JSON format and sent to the server as an HTTP request.

[0196] Input: User information entered on the terminal

[0197] Output: HTTP request sent to the server

[0198] Specific operation: When the user taps the "Send" button, the application on the device converts the input data into JSON format and sends the data to the server via an API call.

[0199] Step 3:

[0200] The server saves the received user information to a database. An RDBMS such as MySQL or PostgreSQL is used for the database. After saving is complete, a registration completion message is generated and sent to the terminal.

[0201] Input: User information received as an HTTP request

[0202] Output: Registration complete message

[0203] Specific operation: The server interprets the data in JSON format and inserts a new record into the corresponding database. If the insertion is successful, it generates a confirmation message and sends it to the terminal as an HTTP response.

[0204] Step 4:

[0205] The user enters the title, author, and specific chapter or page number of the book they are reading into the app and requests an analysis. They can also enter any parts they find difficult to understand, if necessary.

[0206] Input: Book title, author name, specific chapter or page number, difficult-to-understand sections

[0207] Output: Parsing request entered into the terminal

[0208] Specific steps: Enter the book information into the form on the app screen and tap the "Request Analysis" button.

[0209] Step 5:

[0210] The device sends the user's analysis request to the server. During transmission, the necessary information is converted to JSON format and sent to the server as an HTTP request.

[0211] Input: Analysis request entered on the terminal

[0212] Output: HTTP request sent to the server

[0213] Specific operation: When the user taps the "Request Analysis" button, the application on the device converts the input data into JSON format and sends the data to the server via an API call.

[0214] Step 6:

[0215] The server retrieves text data from a database for a specified book and analyzes it using a natural language processing algorithm. A generative AI model (e.g., GPT-3) is then used to generate explanatory information for specific, difficult sections.

[0216] Input: Parsing request received as an HTTP request

[0217] Output: Generated explanatory information

[0218] Specific operation: The server queries the database to retrieve the relevant book data and analyzes the data using a natural language processing algorithm. Based on the analysis results, it generates explanatory information using a generative AI model such as GPT-3.

[0219] Step 7:

[0220] The server sends the generated explanatory information to the terminal. The explanatory information is sent as an HTTP response in JSON format.

[0221] Input: Generated explanatory information

[0222] Output: HTTP response sent to the terminal

[0223] Specific action: The generated explanatory information is sent to the terminal as an HTTP response.

[0224] Step 8:

[0225] The terminal displays explanatory information sent from the server to the user. The information is presented in an interactive format, incorporating graphics and icons to facilitate user understanding.

[0226] Input: Explanatory information received from the server

[0227] Output: Explanatory information displayed to the user

[0228] Specific actions: Display text explanations on the app screen and insert related graphics and icons. Allow users to tap to display detailed information.

[0229] Step 9:

[0230] If the user requests the generation of a character list, this request is entered into the app.

[0231] Input: Request to generate character list

[0232] Output: Request entered into the terminal

[0233] Specific action: Enter "I need a character list" into the app's form and tap the "Submit" button.

[0234] Step 10:

[0235] The terminal sends a request for the character list to the server. During transmission, the input information is converted to JSON format and sent to the server as an HTTP request.

[0236] Input: Request entered on the terminal

[0237] Output: HTTP request sent to the server

[0238] Specific operation: Convert the user's request into JSON format and send it to the server as an HTTP request.

[0239] Step 11:

[0240] The server analyzes and generates a character list from the text data of the specified book. Detailed information about each character (e.g., characteristics, relationships, appearances, etc.) is added.

[0241] Input: Request received as an HTTP request

[0242] Output: Generated character list

[0243] Specific operation: The server retrieves book data from the database and uses a natural language processing algorithm to extract the names of the characters and their detailed information.

[0244] Step 12:

[0245] The server sends a character list it has generated to the terminal. The list is sent as an HTTP response in JSON format.

[0246] Input: Generated character list

[0247] Output: HTTP response sent to the terminal

[0248] Specific action: The generated character list is sent to the terminal as an HTTP response.

[0249] Step 13:

[0250] The device interactively displays a list of characters received from the server. Users can view detailed information by tapping on each character's name.

[0251] Input: Character list received from the server

[0252] Output: List of characters displayed to the user

[0253] Specific operation: The app screen displays a list of characters, and when the user taps a character, detailed information is displayed.

[0254] Step 14:

[0255] The user specifies the parts of the book for which they want to know the source material or background information.

[0256] Input: Specify source material or background information.

[0257] Output: Specified information entered into the terminal

[0258] Specific steps: Enter the information you want to know about the source material or background into the app's form and tap the "Submit" button.

[0259] Step 15:

[0260] The device sends a request for source material and background information to the server. During transmission, the input information is converted to JSON format and sent to the server as an HTTP request.

[0261] Input: Request entered on the terminal

[0262] Output: HTTP request sent to the server

[0263] Specific operation: Convert the user's request into JSON format and send it to the server as an HTTP request.

[0264] Step 16:

[0265] The server collects source material and background information for the specified section from the database, analyzes it, and generates appropriate information.

[0266] Input: Request received as an HTTP request

[0267] Output: Generated source material and background information

[0268] Specific operation: The server queries the database to retrieve relevant source material and background information, and then uses a generative AI model to analyze and generate information.

[0269] Step 17:

[0270] The server sends the source material and background information it generates to the terminal. The information is sent as an HTTP response in JSON format.

[0271] Input: Generated source material and background information

[0272] Output: HTTP response sent to the terminal

[0273] Specific action: The generated information is sent to the terminal as an HTTP response.

[0274] Step 18:

[0275] The terminal provides the user with source material and background information received from the server. The information is displayed with care to avoid excessive spoilers.

[0276] Input: Source material and background information received from the server

[0277] Output: Source material and background information provided to the user

[0278] Specific actions: The app screen displays source material and background information, and interactive elements (e.g., links or taps to display details) are added as needed.

[0279] (Application Example 1)

[0280] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0281] Conventional reading support systems primarily focus on assisting with the reading comprehension of books, and lack sufficient support for factory work procedures and manuals. Therefore, there was a need to resolve the difficulties workers face in understanding these procedures and manuals. This invention aims to solve these problems and provide a system that improves work efficiency and safety in factories.

[0282] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0283] In this invention, the server includes means for receiving and storing registered reading information, means for analyzing text data of the material being read and identifying difficult sections, means for generating and providing explanatory information for the identified difficult sections in response to a user's request, and means for analyzing the explanations and supplementary information of work procedure manuals and providing visual explanations to the user. This enables factory workers to efficiently understand procedure manuals and manuals and perform their work appropriately.

[0284] The means of "receiving and storing information related to registered reading" is a mechanism that has the function of receiving data such as personal information, preferred genres, and reading history provided by the user and storing them in the server.

[0285] The means of "analyzing the text data of materials during reading and identifying difficult parts" is a mechanism that has the function of analyzing the content of materials by natural language processing technology and identifying particularly difficult-to-understand parts in the text.

[0286] The means of "generating and providing explanatory information for the identified difficult parts in response to the user's request" is a mechanism that has the function of generating a detailed explanation of the difficult parts and providing that information to the user in response to the questions or requests sent by the user.

[0287] The means of "analyzing the explanations and supplementary information in the work procedure manual and providing visual explanations to the user" is a mechanism that has the function of analyzing work procedure manuals and manuals used in the factory, generating explanations and supplementary information, and providing them in a way that the user can visually understand.

[0288] The means of "analyzing the person information in the text data and generating a person list" is a mechanism that has the function of analyzing the names and characteristics of the people appearing in the materials and summarizing them as a list.

[0289] The means of "interactively displaying the generated person list and providing detailed information about each person" is a mechanism that has the function of displaying the generated person list in a form that the user can operate and allowing the user to view detailed information about each person.

[0290] The means of "analyzing the original source and background information related to the materials and generating that information" is a mechanism that has the function of analyzing the original source and background information related to the content of the materials and generating detailed explanations and supplementary information based on them.

[0291] "A means of analyzing voice-inputted work instructions and page numbers of procedure manuals, and generating and providing explanations about the actions" refers to a system that analyzes the page numbers of work procedure manuals and specific actions instructed by the user via voice input, and generates and provides explanations and supplementary information.

[0292] This invention develops an AI assistant system to enhance the reading experience and implements a system to improve work efficiency and safety in factories. This system includes functions such as managing registered information, analyzing procedure manuals, generating and providing explanatory information, and analyzing voice input.

[0293] 1. System Program

[0294] The following describes the specific processing details of each program in the system.

[0295] Hardware to use

[0296] Smart glasses (e.g., Microsoft HoloLens®)

[0297] Software to use

[0298] NLP libraries (e.g., spaCy, NLTK)

[0299] AR frameworks (e.g., Unity, Vuforia)

[0300] Backend frameworks (e.g., Node.js, Flask)

[0301] Generative AI models (e.g., OpenAI®, GPT-4®)

[0302] Program Processing Description

[0303] 1. Management of registered information

[0304] The server receives data such as personal information, preferred genres, and reading histories provided by the user through an application and stores it in a database. The user inputs this information using terminals such as smartphones, tablets, and smart glasses.

[0305] 2. Analysis of the Procedure Manual

[0306] The server uses an NLP library (e.g., spaCy) to analyze the text data of the materials the user is reading and identify difficult parts. At this stage, an algorithm for evaluating the difficulty of words and phrases in the text is used.

[0307] 3. Generation and Provision of Explanation Information

[0308] The server uses a generative AI model (e.g., GPT-4) to generate explanation information about the identified difficult parts. The generated explanation information is provided to the user through smart glasses or other terminals. For example, for a prompt like "Please tell me the points to note in Chapter 3 of the operation manual", the server generates a detailed explanation and provides information such as "In Chapter 3, special attention is required for the part regarding the safe operation of the machine".

[0309] 4. Voice Input Analysis

[0310] When the user makes a voice input through smart glasses, the voice data is converted into text and sent to the server. The server analyzes voice instructions and page numbers and generates corresponding explanations and supplementary information. For example, for a prompt like "Please explain in detail the procedure for using this machine", detailed procedures regarding the operation of the machine are provided.

[0311] Specific Example

[0312] If a user is having trouble understanding a particular procedure manual while working in a factory, they can voice-input through smart glasses, "Please tell me the important points to pay attention to in Chapter 3 of the work procedure manual." The server analyzes this information, generates an explanation of the parts of Chapter 3 that require special attention, and displays it in augmented reality (AR) in the user's field of view.

[0313] Example of a prompt:

[0314] 1. "Please tell me the key points to pay attention to in Chapter 3 of the work procedure manual."

[0315] 2. "Please explain in detail how to use this machine."

[0316] This system allows users to better understand the complex parts of work procedures and perform tasks efficiently and safely.

[0317] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0318] Step 1:

[0319] Users enter personal information and reading history using an application installed on their device (e.g., smart glasses, smartphone). This information includes name, email address, reading preferences, and reading frequency. The device sends this information to a server. The entered data is stored in a user information database.

[0320] Step 2:

[0321] The user inputs information about the document they are reading or the factory work instructions. Specifically, they input the title of the document, chapter, page number, and any parts they find difficult to understand. The terminal sends this information to the server. The server retrieves the received document information from a text database and analyzes the text data using natural language processing (NLP) technology.

[0322] Step 3:

[0323] The server uses an NLP library (e.g., spaCy) to analyze the text data of the document. The analysis understands grammatical structure and word meanings, and identifies particularly difficult sections. Information about these identified difficult sections is then input as prompts into a generative AI model (e.g., GPT-4). These prompts might include phrases like, "Please tell me the points to pay attention to in Chapter 3 of the work procedure manual."

[0324] Step 4:

[0325] The generative AI model (e.g., GPT-4) generates explanatory information for complex parts based on the prompt text received from the server. The generated explanatory information is written in detail to make it easy for the user to understand. The server sends the generated explanatory information to the terminal. This output data includes specific explanatory text.

[0326] Step 5:

[0327] The device provides the user with explanatory information received from the server. Specifically, the explanatory information is displayed as an augmented reality (AR) display on the smart glasses' screen. Audio guidance is also available. Based on the provided explanatory information, the user can understand the work procedure and perform it appropriately.

[0328] Step 6:

[0329] When a user provides voice input through smart glasses, the device converts the voice data into text data and sends it to the server. For example, a voice command such as "Please explain the instructions for using this machine in detail" might be input. The server analyzes the received voice command and page number text data to generate explanatory information about the operation. This explanatory information is then sent back to the device.

[0330] Step 7:

[0331] The terminal then provides the user with explanatory information about the operation received from the server. This explanatory information is displayed in the user's field of view using augmented reality (AR), and includes audio guidance along with detailed explanations of the operation. This allows the user to accurately understand and safely perform complex tasks.

[0332] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0333] Patent Specification

[0334] This invention combines an AI assistant system with an emotion engine to enhance the reading experience. It analyzes the user's emotional state and provides appropriate feedback and customized information based on that analysis. The program and specific processing of this system are described below.

[0335] 1. Registration and Initial Setup

[0336] User:

[0337] Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[0338] After launching the application, tap the "New Registration" button and enter the required information such as your name, email address, and password.

[0339] Set your preferences, such as genre, difficulty level, and reading frequency.

[0340] Terminal:

[0341] It receives information entered by the user and sends it to the server.

[0342] server:

[0343] The received user information is saved to the database, a registration completion message is generated, and sent to the terminal.

[0344] Terminal:

[0345] Display a registration completion message to the user.

[0346] 2. Initial setup of the emotion engine

[0347] User:

[0348] Enable emotion recognition within the app.

[0349] Terminal:

[0350] It uses a camera and microphone to capture the user's facial expressions and voice.

[0351] server:

[0352] The system analyzes the received emotional data and generates an initial emotional profile.

[0353] 3. Text analysis and information provision

[0354] User:

[0355] Enter the title, author, and specific chapter or page number of the book you are reading into the app. If there are parts you find difficult to understand, enter them as comments.

[0356] Terminal:

[0357] Send the user's input information to the server.

[0358] server:

[0359] Retrieve the text data of the specified book from the database.

[0360] We analyze text data using natural language processing algorithms to identify difficult parts.

[0361] Generate explanatory information for identified difficult sections. Select the appropriate explanation based on whether or not spoilers are included.

[0362] Customize explanatory information based on the user's emotional state.

[0363] Terminal:

[0364] Customized explanatory information received from the server is displayed interactively and provided to the user.

[0365] Specific example:

[0366] If a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. For example, if "The Use of Symbolism" is identified as a difficult section, an explanation is provided regarding its background and intent, such as "Symbolism is used in this section, and certain symbols have a deeper meaning than they actually do." Furthermore, if the user's emotional state is recognized as "confused," additional advice is customized and displayed, such as "Read slowly and think about what the symbols mean."

[0367] 4. Generation and provision of the character list

[0368] User:

[0369] If you can't remember the names of the characters, enter that into the app.

[0370] Terminal:

[0371] Send the request to the server.

[0372] server:

[0373] Analyze and generate a list of characters from the text data of a specified book.

[0374] Add detailed information about each character (e.g., characteristics, relationships, locations of appearance, etc.).

[0375] Customize character descriptions based on the user's emotional state.

[0376] Terminal:

[0377] The system interactively displays a list of characters sent from the server, allowing users to view detailed information through actions such as clicking.

[0378] Specific example:

[0379] If a user requests a character list for "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, knowledgeable and always offering advice." If the user's emotional state is "excited," an additional comment will be displayed, such as, "Pay attention to how this character will influence the story going forward."

[0380] 5. Providing source material and background information.

[0381] User:

[0382] You can specify in the app which parts of a book you want to know about, such as the source material or background information.

[0383] Terminal:

[0384] Send the request to the server.

[0385] server:

[0386] The system collects source material and background information for the specified portion from a database.

[0387] It analyzes collected source material and background information to generate appropriate information.

[0388] Customize information based on the user's emotional state.

[0389] Terminal:

[0390] The system provides users with source material and background information received from the server. Care is taken to avoid excessive spoilers.

[0391] Specific example:

[0392] If a user inputs that they want to know about Chapter 3 of "Book C," the server analyzes the source material and background information for that section and provides information such as, "This chapter is based on a medieval legend and strongly reflects elements of XXX in particular." If the user's emotional state indicates "surprise," additional advice such as, "Try to predict what will happen next based on this information," will be displayed.

[0393] Thus, the present invention is a system that recognizes the user's emotional state and provides various information based on that state, thereby enriching the reading experience and deepening understanding.

[0394] The following describes the processing flow.

[0395] Forms for carrying out the invention - A system combining an emotion engine

[0396] 1. Registration and Initial Setup

[0397] Step 1:

[0398] User: Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[0399] Step 2:

[0400] User: Tap the "Register" button and enter the required information such as your name, email address, and password.

[0401] Step 3:

[0402] User: Sets preferences such as genre, difficulty level, and reading frequency.

[0403] Step 4:

[0404] Terminal: Converts user-entered information into JSON format and sends it to the server using the API.

[0405] Step 5:

[0406] Server: Stores the received user information in the database, generates a registration completion message, and sends it to the terminal.

[0407] Step 6:

[0408] Terminal: Displays a registration completion message to the user.

[0409] 2. Initial setup of the emotion engine

[0410] Step 1:

[0411] User: Enable emotion recognition within the app.

[0412] Step 2:

[0413] Device: Captures the user's facial expressions and voice using the camera and microphone.

[0414] Step 3:

[0415] Server: Analyzes received emotion data and generates an initial emotion profile.

[0416] 3. Text analysis and information provision

[0417] Step 1:

[0418] User: Enter the title, author, and specific chapter or page number of the book you are reading into the app. If there are parts you find difficult to understand, enter them as comments.

[0419] Step 2:

[0420] Terminal: Converts user input information into JSON format and sends it to the server using the API.

[0421] Step 3:

[0422] Server: Retrieves text data for the specified book from the database.

[0423] Step 4:

[0424] Server: Analyzes text data using natural language processing (NLP) algorithms to identify difficult parts.

[0425] Step 5:

[0426] Server: Generates explanatory information for identified difficult sections. Selects appropriate explanations based on whether or not spoilers are included.

[0427] Step 6:

[0428] Server: Analyzes the user's emotional state and customizes the explanatory information.

[0429] Step 7:

[0430] Server: Converts explanatory information into JSON format and sends it to the terminal.

[0431] Step 8:

[0432] Terminal: Interactively displays and provides customized explanatory information received from the server to the user.

[0433] Specific example:

[0434] If a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. For example, if "The Use of Symbolism" is identified as a difficult section, an explanation is provided regarding its background and intent, such as "Symbolism is used in this section, and certain symbols have a deeper meaning than they actually do." If the emotion recognition determines that the user is "confused," advice such as "Try reading the next chapter to find out what this symbol means" is added.

[0435] 4. Generation and provision of the character list

[0436] Step 1:

[0437] User: If you can't remember the names of the characters, enter that into the app.

[0438] Step 2:

[0439] Terminal: Converts the request to JSON format and sends it to the server.

[0440] Step 3:

[0441] Server: Analyzes and generates a list of characters from the text data of the specified book.

[0442] Step 4:

[0443] Server: Adds detailed information about each character (e.g., characteristics, relationships, locations of appearance, etc.).

[0444] Step 5:

[0445] Server: Analyzes the user's emotional state and customizes the character descriptions.

[0446] Step 6:

[0447] Server: Converts the generated character list into JSON format and sends it to the terminal.

[0448] Step 7:

[0449] Terminal: Interactively displays the character list received from the server, allowing the user to view detailed information through actions such as clicking.

[0450] Specific example:

[0451] If a user requests a list of characters from "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, knowledgeable and always offering advice." If emotion recognition determines that the user is "excited," an additional comment will be displayed, such as, "Pay attention to how this character will influence the story going forward."

[0452] 5. Providing source material and background information.

[0453] Step 1:

[0454] User: Specify to the app the parts of a book for which you want to know the source material or background information.

[0455] Step 2:

[0456] Terminal: Converts the request to JSON format and sends it to the server.

[0457] Step 3:

[0458] Server: Collects source material and background information for the specified section from the database.

[0459] Step 4:

[0460] Server: Analyzes collected source material and background information to generate appropriate information.

[0461] Step 5:

[0462] Server: Analyzes the user's emotional state and customizes the information.

[0463] Step 6:

[0464] Server: Converts the generated source material and background information into JSON format and sends it to the terminal.

[0465] Step 7:

[0466] Terminal: Provides users with source material and background information received from the server. Care is taken to avoid excessive spoilers.

[0467] Specific example:

[0468] If a user inputs that they want to know about Chapter 3 of "Book C," the server analyzes the source material and background information for that section and provides information such as, "This chapter is based on a medieval legend and strongly reflects elements of XXX in particular." If the user's emotion recognition detects "surprise," advice such as, "Try to predict what will happen next based on this information," will be displayed.

[0469] In this way, the program can enrich the user's reading experience and deepen their understanding by recognizing the user's emotional state at each processing step and customizing and providing information based on that state.

[0470] (Example 2)

[0471] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0472] Existing reading support systems provide uniform information without considering the user's emotional state, resulting in reading experiences that do not meet individual needs. Furthermore, information about characters and source material is often insufficient, hindering comprehension during reading. This leads to difficulties for users during reading and a lack of information consistency, posing a significant challenge.

[0473] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving and storing information of registered users, means for capturing and analyzing the user's emotional state using a camera and microphone, means for analyzing the text data of the book being read and identifying difficult parts, and means for generating and providing customized explanatory information for the identified difficult parts in accordance with the user's emotional state and requests. This makes it possible to provide personalized information based on the user's emotional state, thereby providing a richer and more symmetrical reading experience.

[0474] "Registered user information" refers to personal information and settings information that users provide when registering with the system, such as their name, email address, password, and preferred genres.

[0475] "User emotional state" refers to the psychological state of a user obtained by analyzing data of the user's facial expressions and voice captured by cameras and microphones.

[0476] "Means of capturing and analyzing using cameras and microphones" refers to means including software and hardware for collecting and analyzing a user's facial expressions and voice using cameras and microphones installed on a device.

[0477] "A means of analyzing text data and identifying difficult parts" refers to a program that uses natural language processing algorithms to analyze the text data of a book and automatically identify parts that users find difficult to understand.

[0478] "Customized explanatory information" refers to individually optimized explanatory information generated based on the user's emotional state and requests for specific, difficult-to-understand parts.

[0479] "A means of generating a character list" refers to a program that analyzes the text data of a book and lists the names of the main characters that appear in the book.

[0480] A "customized character list" is a character list generated based on the user's emotional state, with descriptions that take into account the characteristics and relationships of each character.

[0481] A "means for generating source material and background information" refers to a program that collects additional information, such as source material and historical background, for specific parts of a book, analyzes that information, and provides it.

[0482] "Customized information that avoids excessive spoilers" refers to background information and source material provided with consideration for the user's emotional state, without compromising the entertainment value.

[0483] This invention combines an AI assistant system with an emotion engine to enhance the reading experience, analyzing the user's emotional state and providing appropriate feedback and customized information based on that analysis.

[0484] First, the user downloads the application to their device (smartphone, tablet, etc.), installs it, and then launches it. The user launches the application, taps the "New Registration" button, and enters necessary information such as their name, email address, and password. Next, the user sets their preferred genre, reading difficulty level, and reading frequency. The device then receives the user's input information and sends it to the server. The server saves the information to its database, generates a registration completion message, and sends it to the device. The device then displays this message to the user.

[0485] Next, the user configures the app to enable emotion recognition. The device uses its camera and microphone to capture the user's facial expressions and voice, and sends this data to the server. The server analyzes the emotion data, generates an initial emotion profile using an AI model (e.g., an emotion recognition model), and stores it in a database.

[0486] The user enters the title, author, and specific chapter or page number of the book they are reading into the app, and adds comments on parts they find difficult to understand. The device sends this information to the server. The server retrieves the text data of the specified book from a database, analyzes it using natural language processing algorithms (e.g., BERT, GPT-3), and identifies the difficult parts. The server also generates customized explanatory information, taking into account the user's emotional state, and sends it to the device. The device then interactively displays and provides the customized explanatory information to the user.

[0487] For example, if a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. If the use of symbolism is identified as a difficult part, an explanation is provided regarding its background and intent, such as, "Symbolism is used in this section, and certain symbols have a deeper meaning than they actually do." Additionally, if the user's emotional state is recognized as "confused," extra advice is displayed, such as, "Read slowly and think about what the symbols mean."

[0488] Furthermore, if the user cannot remember the names of the characters, they can input this into the app. The device sends this request to the server. The server parses the text data and generates a list of characters. The list includes detailed information about each character (characteristics, relationships, appearances, etc.) and is customized based on their emotional state. The customized list is sent to the device, allowing the user to interactively view the detailed information.

[0489] For example, if a user requests a character list for "Book B," the server will provide detailed information such as "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, knowledgeable and always offering advice." If the user's emotional state is "excited," an additional comment will be displayed such as, "Pay attention to how this character will influence the story going forward."

[0490] Finally, the user specifies to the app which parts of the book they want to know about, such as the source material and background information. The device sends the request to the server. The server retrieves the source material and background information from its database, analyzes it, and provides the generated information. Customized information is provided to the user while avoiding excessive spoilers.

[0491] For example, if a user inputs that they want to know about Chapter 3 of "Book C," the server analyzes the source material and background information for that section and provides information such as, "This chapter is based on a medieval legend and strongly reflects elements of XXX in particular." If the user's emotional state indicates "surprise," additional advice such as, "Try to predict what happens next based on this information," will be displayed.

[0492] Example of a prompt:

[0493] Could you please explain the meaning of "Muggle" in Chapter 1 of the book "Harry Potter and the Sorcerer's Stone"?

[0494] Please display the list of characters from the book "The Lord of the Rings".

[0495] Thus, the present invention is a system that recognizes the emotional state of the user and provides various information based on that state, thereby enriching the reading experience and deepening understanding.

[0496] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0497] Step 1: Install and launch the application

[0498] User:

[0499] The user downloads and installs the application on their device (such as a smartphone or tablet).

[0500] Then, launch the application.

[0501] Terminal:

[0502] The device launches the installed application and displays the initial screen.

[0503] input:

[0504] Downloaded applications

[0505] output:

[0506] Displaying the application's initial screen.

[0507] Step 2: New Registration

[0508] User:

[0509] After launching the application, the user taps the "Register" button and enters the required information such as their name, email address, and password.

[0510] Terminal:

[0511] The terminal temporarily saves the entered information and displays a button to proceed to the next step.

[0512] input:

[0513] User information such as name, email address, and password

[0514] output:

[0515] A button to proceed to the next step is displayed.

[0516] Step 3: Customize your settings

[0517] User:

[0518] Users can set their preferred genre, reading difficulty level, and reading frequency.

[0519] Terminal:

[0520] The device temporarily stores the configured information and displays a confirmation screen to send all the information to the server.

[0521] input:

[0522] Settings such as preferred genre, difficulty level, and reading frequency.

[0523] output:

[0524] Confirmation screen displayed

[0525] Step 4: Sending and saving information

[0526] Terminal:

[0527] The terminal sends the information entered and configured by the user to the server.

[0528] server:

[0529] The server stores the received information in a database.

[0530] Generate a registration completion message and send it to the device.

[0531] input:

[0532] Sending user information and settings information

[0533] output:

[0534] Storing information in a database

[0535] Generate and send registration completion message

[0536] Step 5: Display of registration completion message

[0537] Terminal:

[0538] The device displays a registration completion message to the user.

[0539] input:

[0540] Registration complete message received

[0541] output:

[0542] Display of registration completion message

[0543] Step 6: Setting up emotion recognition function

[0544] User:

[0545] Users configure settings within the app to enable the emotion recognition feature.

[0546] Terminal:

[0547] The device requests permission for the camera and microphone and verifies that the settings are enabled.

[0548] input:

[0549] Emotion recognition function settings

[0550] output:

[0551] Checking camera and microphone usage permissions

[0552] Step 7: Capture and send emotional data

[0553] User:

[0554] The user faces the device's camera and microphone, providing standard facial expressions and voice.

[0555] Terminal:

[0556] The device uses its camera and microphone to capture the user's facial expressions and voice, and temporarily stores this as emotion data.

[0557] The captured emotion data is sent to the server.

[0558] input:

[0559] User's facial expressions and voice

[0560] output:

[0561] Capture and transmit emotional data

[0562] Step 8: Generating an emotional profile

[0563] server:

[0564] The server analyzes the received emotional data using an AI model and generates an initial emotional profile.

[0565] The generated emotion profile is saved to the database.

[0566] input:

[0567] Sent emotional data

[0568] output:

[0569] Generating and saving initial emotion profiles

[0570] Step 9: Enter reading information

[0571] User:

[0572] Users enter the title, author, and specific chapter or page number of the book they are reading into the app. They can also add comments to sections they find difficult to understand.

[0573] Terminal:

[0574] The terminal temporarily stores the entered information and sends it to the server.

[0575] input:

[0576] Book title, author name, specific chapter or page number, comments

[0577] output:

[0578] Sending information to the server

[0579] Step 10: Acquisition and analysis of text data

[0580] server:

[0581] The server retrieves the text data of the specified book from the database.

[0582] We analyze text data using natural language processing algorithms to identify difficult parts.

[0583] input:

[0584] Book title, author name, specific chapter or page number

[0585] output:

[0586] Identifying the difficult parts

[0587] Step 11: Generating customized explanatory information

[0588] server:

[0589] The server generates explanatory information for the identified difficult parts.

[0590] It generates customized explanatory information that takes into account the user's emotional state.

[0591] input:

[0592] User emotion profiles, text analysis data

[0593] output:

[0594] Generating customized explanatory information

[0595] Step 12: Send and display customized explanatory information

[0596] server:

[0597] The server sends the generated customized explanation information to the terminal.

[0598] Terminal:

[0599] The device interactively displays and provides customized explanatory information to the user.

[0600] input:

[0601] Customized explanatory information

[0602] output:

[0603] Display of explanatory information

[0604] Step 13: Requesting and generating the character list

[0605] User:

[0606] If a user cannot remember the names of the characters, they can enter that information into the app.

[0607] Terminal:

[0608] The terminal sends the request to the server.

[0609] server:

[0610] The server generates a list of characters from the text data of the book.

[0611] input:

[0612] Request for character list

[0613] output:

[0614] Generating a list of characters

[0615] Step 14: Provide a customized character list

[0616] server:

[0617] The server adds detailed information about each character and customizes the list based on the user's emotional state.

[0618] Terminal:

[0619] The terminal interactively displays a list of characters sent from the server, and the user can view detailed information by clicking or performing other actions.

[0620] input:

[0621] Character List

[0622] output:

[0623] Providing a customized character list

[0624] Step 15: Requesting and providing source material and background information.

[0625] User:

[0626] Users specify to the app which parts of a book they want to know about, such as the source material or background information.

[0627] Terminal:

[0628] The terminal sends the request to the server.

[0629] server:

[0630] The server collects source material and background information for the specified section from the database and generates appropriate information.

[0631] input:

[0632] Request for source material and background information

[0633] output:

[0634] Generated source material and background information

[0635] Terminal:

[0636] The system interactively displays source material and background information received from the server and provides it to the user. Filtering is performed to avoid excessive spoilers.

[0637] input:

[0638] Source material and background information

[0639] output:

[0640] Providing customized background information

[0641] (Application Example 2)

[0642] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0643] Traditional reading-related information systems often provided uniform information without considering the user's emotional state. As a result, support for individual reading experiences was insufficient, and customization tailored to the user's specific emotional state was needed, particularly in understanding difficult passages, grasping characters, and providing source material and background information. Furthermore, a system capable of providing real-time feedback on the user's interests and questions during reading was required.

[0644] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0645] In this invention, the server includes means for receiving and storing user information related to registered reading; means for analyzing text data of the book being read and identifying difficult sections; means for generating and providing explanatory information for the identified difficult sections in response to user requests; means for analyzing the user's emotional state using a camera and microphone; and means for customizing the explanatory information based on the analyzed emotional state. This enables personalized feedback and explanations based on the user's emotional state, making it possible to provide information optimized for each user.

[0646] "Registered user information regarding reading" refers to personal information provided by the user regarding their reading, such as their name, preferred genres, reading frequency, and difficulty level.

[0647] "Book text data" refers to data that represents the content of a book a user is reading in digital format.

[0648] "Difficult sections" refer to parts of a book that are identified as being hard to understand.

[0649] "Explanatory information" refers to explanations and supplementary information provided to make difficult parts easier for users to understand.

[0650] "Means of analyzing a user's emotional state using cameras and microphones" refers to devices and software that capture a user's facial expressions and voice and identify their emotions from them.

[0651] "Means for customizing explanatory information based on analyzed emotional states" refers to methods and software for adjusting and individually adapting explanatory information based on the user's emotional state.

[0652] "Character information" refers to data such as the names, characteristics, and relationships of all characters that appear in the book.

[0653] "Means for generating a character list" refers to a method and software for extracting character names and related information from text data and listing them.

[0654] "Character details" refers to more detailed explanations and background information about the characters.

[0655] "Source material and background information" refers to additional information such as the source, references, and historical context of specific parts of a book.

[0656] "Means for providing users with generated source material and background information" refers to methods and software for displaying or communicating analyzed source material and background information to users.

[0657] This invention utilizes a user terminal, a server, and a network environment connecting them. Specifically, the invention can be implemented in the following forms.

[0658] The user downloads and installs the "Smart Book Reader" app on their smartphone. Upon launching the app, the user registration screen is displayed first. The user enters the necessary information, such as their name, email address, and password, and the device sends this information to the server. The server stores the received information in its database and sends a registration completion message to the device, which then displays the message to the user.

[0659] Next, an option to enable emotion recognition within the app will appear. If the user enables it, the device will use the camera and microphone to capture the user's facial expressions and voice in real time. This data is sent to a server, which analyzes it using an emotion engine to generate an initial emotion profile.

[0660] When a user begins reading, they enter the title, author, and specific chapter or page number of the book they are reading into the app. They can also add comments if they find certain sections difficult to understand. This information is sent from the device to a server, which retrieves the book's text data from a database and analyzes it using natural language processing algorithms. The server then identifies difficult sections and generates explanatory information. Furthermore, the explanatory information is customized based on the user's emotional state. This explanatory information is sent to the device and displayed interactively to the user.

[0661] For example, if a user is reading Chapter 5 of "Book A" and is confused about the "use of symbolism," the server will analyze that section and provide an explanation such as, "Symbolism is used in this section, and certain symbols have a deeper meaning than they actually do." Furthermore, if the server recognizes that the user's emotional state is "confused," it will display customized additional advice such as, "Read slowly and think about what the symbols mean."

[0662] In generating the character list, when a user enters a character's name they can't remember into the app, the device sends a request to the server. The server analyzes the text data from the book and generates the character list. It then adds detailed information about each character and customizes it according to the user's emotional state. For example, if the user is "excited," an additional comment such as "Pay attention to how this character will influence the story going forward" will be displayed.

[0663] Regarding source material and background information, when a user inputs the part they want to know about, the server analyzes the source material and background information for the specified part and generates appropriate information. For example, if a user wants to know about Chapter 3 of "Book C," the server will provide information such as, "This chapter is based on a medieval legend and strongly reflects elements of XXX in particular." If the user indicates "surprise," additional advice such as, "Try to predict what will happen next based on this information," will be displayed.

[0664] The implementation of this system involves using smartphones as user terminals, servers for analysis and data storage, and software such as TENSORFLOW® and OpenCV for emotion recognition, spaCy for natural language processing, and Flask for interface construction.

[0665] An example of a prompt sentence would be, "I would like to understand the concept of symbolism in Chapter 5 of Book A. I am confused."

[0666] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0667] Step 1:

[0668] The user downloads and installs the "Smart Book Reader" app on their smartphone.

[0669] Input: An app installed on a smartphone.

[0670] Output: The app is launched.

[0671] Specific operation: After the user downloads the app from the Google® Play Store or Apple App Store and the installation is complete, they tap the icon on the home screen to launch the app.

[0672] Step 2:

[0673] The user enters the required information on the app's new registration screen and registers.

[0674] Input: Name, email address, password, preferred genre, difficulty level, reading frequency.

[0675] Output: Registration information is sent to the server and stored in the database.

[0676] Specific operation: The user enters the required information on the new registration screen and taps the "Register" button. The device sends the entered information to the server, which receives it and saves it to the database. Once saving is complete, the server sends a registration completion message back to the device, which then displays it to the user.

[0677] Step 3:

[0678] The user enables the emotion recognition feature within the app.

[0679] Input: Enabling / enabling the emotion recognition function.

[0680] Output: The device uses its camera and microphone to capture the user's facial expressions and voice, and sends them to the server.

[0681] Specific operation: When the user enables the emotion recognition feature in the app's settings menu, the device activates the camera and microphone, capturing the user's facial expressions and voice and sending them to the server in real time.

[0682] Step 4:

[0683] The server analyzes the received emotion data and generates an initial emotion profile.

[0684] Input: User's facial expression data and voice data.

[0685] Output: Initial emotion profile.

[0686] Specific operation: The server uses TensorFlow and OpenCV to analyze facial expression data and voice data, generates an initial emotion profile of the user based on that analysis, and saves it to a database.

[0687] Step 5:

[0688] The user enters the title of the book they are reading, the author's name, a specific chapter or page number, and any parts they find difficult to understand into the app.

[0689] Input: Book title, author name, chapter and page numbers, and comments.

[0690] Output: This information is sent to the server.

[0691] Specific operation: The user enters the required information on an input screen within the app and taps the "Submit" button. The device then sends this information to the server.

[0692] Step 6:

[0693] The server retrieves the text data of the specified book from the database and analyzes it using a natural language processing algorithm.

[0694] Input: Book title, chapter or page number.

[0695] Output: Identification of difficult sections and explanatory information for those sections.

[0696] Specific operation: The server retrieves text data of a specified book from the database and analyzes it using spaCy. Based on the analysis results, it identifies difficult sections and generates appropriate explanatory information.

[0697] Step 7:

[0698] The server customizes the explanatory information based on the user's emotional state.

[0699] Input: User's emotion profile, generated explanatory information.

[0700] Output: Customized explanatory information.

[0701] Specific operation: The server references the user's emotional profile and customizes the generated explanatory information each time. For example, for a confused user, it adds more detailed explanations and words of encouragement.

[0702] Step 8:

[0703] Customized explanatory information is sent from the server to the terminal and provided to the user.

[0704] Input: Customized explanatory information.

[0705] Output: Interactively displayed explanatory information.

[0706] Specific operation: The server sends customized explanatory information to the device, which then displays it interactively. The user can tap on the information to view more details.

[0707] An example of a prompt sentence would be, "I would like to understand the concept of symbolism in Chapter 5 of Book A. I am confused."

[0708] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0709] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0710] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0711] [Second Embodiment]

[0712] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0713] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0714] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0715] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0716] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0717] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0718] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0719] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0720] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0721] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0722] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0723] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0724] Patent Specification

[0725] This invention is an AI assistant system for improving the reading experience, which analyzes and provides various information about the book the user is reading. The program of this system and its specific processing are described below.

[0726] 1. Registration and Initial Setup

[0727] User:

[0728] Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[0729] After launching the app, tap the "New Registration" button, enter the required information such as your name, email address, and password, and set your preferred genre, difficulty level, reading frequency, etc.

[0730] Terminal:

[0731] It receives information entered by the user and sends it to the server.

[0732] server:

[0733] The received user information is saved to the database, a registration completion message is generated, and sent to the terminal.

[0734] 2. Text analysis and information provision

[0735] User:

[0736] Enter the title, author, and specific chapter or page number of the book you are reading into the app. If necessary, specify the parts you are having difficulty understanding.

[0737] Terminal:

[0738] The information entered by the user is sent to the server.

[0739] server:

[0740] The text data of a specified book is retrieved from a database and analyzed using a natural language processing algorithm. Particularly difficult sections are identified, and explanatory information is generated.

[0741] The system checks for spoilers, generates appropriate explanatory information, and sends it to the device.

[0742] Terminal:

[0743] The server sends explanatory information to the user. Graphics and icons are also displayed in an interactive format.

[0744] Specific example:

[0745] If a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. For example, if "The Use of Symbolism" is identified as a difficult section, an explanation of its background and intent will be provided, such as "Symbolism is used in this section, and certain symbols have a deeper meaning than they do in reality."

[0746] 3. Generation and provision of the character list

[0747] User:

[0748] If you can't remember the names of the characters, enter that into the app.

[0749] Terminal:

[0750] Send the request to the server.

[0751] server:

[0752] This program analyzes and generates a character list from the text data of a specified book. It then adds detailed information about each character (e.g., characteristics, relationships, appearances, etc.).

[0753] Terminal:

[0754] The system interactively displays a list of characters sent from the server, allowing users to view detailed information through actions such as clicking.

[0755] Specific example:

[0756] If a user requests a list of characters from "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, who is knowledgeable and always offers advice."

[0757] 4. Providing source material and background information

[0758] User:

[0759] Specify the parts of the book for which you want to know the source material or background information.

[0760] Terminal:

[0761] Send the request to the server.

[0762] server:

[0763] The system collects source material and background information for the specified portion from a database, analyzes it, and generates appropriate information.

[0764] Terminal:

[0765] The server provides users with source material and background information. Care is taken to avoid excessive spoilers.

[0766] Specific example:

[0767] If a user inputs a request to learn about Chapter 3 of "Book C," the server analyzes the source material and background information for that section and provides information such as, "This chapter is based on medieval legends and strongly reflects elements of XXX in particular."

[0768] Thus, the present invention specifically realizes a system that provides information to enrich the user's reading experience and deepen their understanding.

[0769] The following describes the processing flow.

[0770] Program processing flow

[0771] 1. Registration and Initial Setup

[0772] Step 1:

[0773] User: Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[0774] Step 2:

[0775] User: Tap the "Register" button and enter the required information such as your name, email address, and password.

[0776] Step 3:

[0777] User: Sets preferences such as genre, difficulty level, and reading frequency.

[0778] Step 4:

[0779] Terminal: Converts user-entered information into JSON format and sends it to the server using the API.

[0780] Step 5:

[0781] Server: Stores received user information in the database.

[0782] Step 6:

[0783] Server: Generates a registration completion message and sends it to the terminal.

[0784] Step 7:

[0785] Terminal: Displays a registration completion message to the user.

[0786] 2. Text analysis and information provision

[0787] Step 1:

[0788] User: Enter the title, author, and specific chapter or page number of the book you are reading into the app. If there are parts you find difficult to understand, enter them as comments.

[0789] Step 2:

[0790] Terminal: Converts user input information into JSON format and sends it to the server using the API.

[0791] Step 3:

[0792] Server: Retrieves text data for the specified book from the database.

[0793] Step 4:

[0794] Server: Analyzes text data using natural language processing (NLP) algorithms to identify difficult parts.

[0795] Step 5:

[0796] Server: Generates explanatory information for identified difficult sections. Selects appropriate explanations based on whether or not spoilers are included.

[0797] Step 6:

[0798] Server: Converts explanatory information into JSON format and sends it to the terminal.

[0799] Step 7:

[0800] Terminal: Interactively displays and provides explanatory information received from the server to the user.

[0801] 3. Generation and provision of the character list

[0802] Step 1:

[0803] User: If you can't remember the names of the characters, enter that into the app.

[0804] Step 2:

[0805] Terminal: Converts user requests into JSON format and sends them to the server.

[0806] Step 3:

[0807] Server: Analyzes and generates a list of characters from the text data of the specified book.

[0808] Step 4:

[0809] Server: Adds detailed information about each character (e.g., characteristics, relationships, locations of appearance, etc.).

[0810] Step 5:

[0811] Server: Converts the generated character list into JSON format and sends it to the terminal.

[0812] Step 6:

[0813] Terminal: Interactively displays the character list received from the server, allowing users to view detailed information through actions such as clicking.

[0814] 4. Providing source material and background information

[0815] Step 1:

[0816] User: Specify to the app the parts of a book for which you want to know the source material or background information.

[0817] Step 2:

[0818] Terminal: Converts user requests into JSON format and sends them to the server.

[0819] Step 3:

[0820] Server: Collects source material and background information for the specified section from the database.

[0821] Step 4:

[0822] Server: Analyzes collected source material and background information to generate appropriate information.

[0823] Step 5:

[0824] Server: Converts the generated source material and background information into JSON format and sends it to the terminal.

[0825] Step 6:

[0826] Terminal: Provides users with source material and background information received from the server. Care is taken to avoid excessive spoilers.

[0827] In this way, the program enriches the user's reading experience and provides information to deepen their understanding through each processing step.

[0828] (Example 1)

[0829] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0830] Traditional reading systems have a problem in that they cannot provide users with appropriate explanations or background information for difficult or hard-to-understand parts of a book. Furthermore, it is difficult for users to remember character names and detailed information, leading to frequent interruptions in the reading experience. In addition, the source material and background information are not provided in an appropriate format, making it difficult to achieve a deep understanding. There is a need for a system that solves these problems and improves the reading experience.

[0831] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0832] In this invention, the server includes means for receiving and storing user information related to registered readings, means for analyzing text data of the medium being read and identifying difficult sections, and means for generating and providing explanatory information for the identified difficult sections in response to user requests, using a generative artificial intelligence model. This makes it possible to immediately provide appropriate explanatory information for difficult sections that the user encounters while reading.

[0833] The server further includes means for analyzing information about characters appearing in the reading medium and generating a character list, and means for interactively displaying the generated character list and providing detailed information about each character. This allows users to easily check detailed information about characters, dramatically improving the reading experience.

[0834] Furthermore, the server includes means for analyzing source material and background information related to the medium being read and generating that information, and means for providing the generated source material and background information to the user. This allows the user to gain a deeper understanding of the background and source material of the medium being read, thereby enriching the reading experience.

[0835] "Registered user information related to reading" refers to personal information entered by the user into the system, such as name, email address, password, preferred genres, difficulty level, and reading frequency.

[0836] "Text data in a medium" refers to electronic text data such as books, magazines, and academic papers that a user is reading.

[0837] "Analysis" refers to the process of analyzing the content of text data using natural language processing algorithms and other methods to extract specific patterns or meanings.

[0838] "Difficult parts" refer to sections of text that users find hard to understand.

[0839] "Generative artificial intelligence models" refer to machine learning models and deep learning models used for natural language processing and data analysis. Specifically, this includes advanced text generation models such as GPT-3.

[0840] "Explanatory information" refers to additional explanations or background information for parts that users found difficult to understand.

[0841] A "character list" refers to a list of characters that appear in the medium being read. This list includes each character's name, role, and characteristics.

[0842] "Source material and background information" refers to information about the historical and cultural background and original sources that influenced the content of the medium being read.

[0843] "Interactive display" refers to a display method that allows users to view additional details or explanations by performing actions such as tapping or clicking.

[0844] An "HTTP request" refers to a form of data communication based on a protocol used when sending data from a terminal to a server.

[0845] This invention is an AI assistant system designed to enhance the user's reading experience. It utilizes natural language processing and generative AI models to analyze and provide various information about the medium. The system analyzes the text the user is reading in detail and interactively provides lists of difficult passages and characters, as well as source material and background information.

[0846] Registration and initial setup

[0847] User: Download and install the application on your smartphone or tablet. After installation, launch the application and tap the "New Registration" button. Complete the initial setup by entering your name, email address, password, and preferred genres, difficulty level, reading frequency, etc.

[0848] Terminal: Receives information entered by the user and sends it to the server. Specifically, it converts the user's information into JSON format and sends it to the server as an HTTP request.

[0849] Server: Saves received user information to the database. The database used is an RDBMS (Relational Database Management System) such as MySQL or PostgreSQL. After saving is complete, a registration completion message is generated and sent to the terminal.

[0850] Text analysis and information provision

[0851] User: Enter the title, author, and specific chapter or page number of the book you are reading into the app. If necessary, specify the parts you are having difficulty understanding.

[0852] Terminal: Sends user-entered information to the server. During transmission, text data is converted to JSON format and sent via the API.

[0853] Server: Retrieves text data from the specified medium from the database. Next, it uses a natural language processing algorithm to analyze the data and identify the difficult parts pointed out by the user. It then generates explanatory information for the difficult parts using a generation AI model (e.g., GPT-3). During this process, it checks for spoilers and generates explanatory information within an appropriate scope.

[0854] For example, if a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. If, for instance, "The Use of Symbolism" is identified as a difficult section, the server provides an explanation of its background and intent, such as, "Symbolism is used in this section, and certain symbols have a deeper meaning than they do in reality."

[0855] Terminal: Displays explanatory information sent from the server to the user. The explanatory information is displayed in an interactive format, including graphics and icons.

[0856] Generating and providing a list of characters

[0857] User: If you can't remember the names of the characters, enter that into the app.

[0858] Terminal: Sends requests to the server. Specifically, it converts the input request into JSON format and makes an API call.

[0859] Server: Analyzes and generates a character list from the text data of the specified medium. Adds detailed information about each character (e.g., characteristics, relationships, appearances, etc.). Uses natural language processing algorithms for analysis and retrieves necessary information from the database.

[0860] For example, if a user requests a list of characters from "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, who is knowledgeable and always offers advice."

[0861] Terminal: Interactively displays the character list sent from the server. Users can view detailed information by tapping on names in the list.

[0862] Providing source material and background information

[0863] User: Specify the parts of the book for which you want to know the source material or background information.

[0864] For example, if a user inputs that they want to know about Chapter 3 of "Book C," the server will analyze the source material and background information for that section and provide information such as, "This chapter is based on medieval legends and particularly strongly reflects elements of XXX."

[0865] Terminal: Sends the request to the server.

[0866] Server: Collects source material and background information for the specified section from the database, analyzes it, and generates appropriate information.

[0867] Terminal: Provides users with source material and background information sent from the server. Displays information while taking care to avoid excessive spoilers.

[0868] Example of a prompt

[0869] Prompt 1: "I am reading Chapter 5 of Book A, but I don't understand the use of symbolism. Please provide a detailed explanation."

[0870] Prompt 2: "Please provide a list of characters in 'Book B' and their respective characteristics."

[0871] Prompt 3: "What source material or background information is found in Chapter 3 of 'Book C'?"

[0872] As described above, the present invention specifically realizes the provision of information that enriches the user's reading experience and deepens their understanding.

[0873] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0874] Step 1:

[0875] The user downloads and installs the application on their smartphone or tablet. This provides the basic platform for accessing the system. Next, they launch the application and enter their name, email address, password, preferred genres, etc., on the new user registration screen. This collects the user's basic information.

[0876] Input: Username, email address, password, preferred genre, etc.

[0877] Output: User information entered into the terminal

[0878] Specific steps: The user searches for the app on the App Store or Google Play and taps the download button. After the download is complete, the installation starts automatically, and once finished, an icon appears on the device's home screen. After launching the app, the user enters the required information on the registration screen that appears.

[0879] Step 2:

[0880] The terminal receives the information entered by the user and sends it to the server. The user's information is converted to JSON format and sent to the server as an HTTP request.

[0881] Input: User information entered on the terminal

[0882] Output: HTTP request sent to the server

[0883] Specific operation: When the user taps the "Send" button, the application on the device converts the input data into JSON format and sends the data to the server via an API call.

[0884] Step 3:

[0885] The server saves the received user information to a database. An RDBMS such as MySQL or PostgreSQL is used for the database. After saving is complete, a registration completion message is generated and sent to the terminal.

[0886] Input: User information received as an HTTP request

[0887] Output: Registration complete message

[0888] Specific operation: The server interprets the data in JSON format and inserts a new record into the corresponding database. If the insertion is successful, it generates a confirmation message and sends it to the terminal as an HTTP response.

[0889] Step 4:

[0890] The user enters the title, author, and specific chapter or page number of the book they are reading into the app and requests an analysis. They can also enter any parts they find difficult to understand, if necessary.

[0891] Input: Book title, author name, specific chapter or page number, difficult-to-understand sections

[0892] Output: Parsing request entered into the terminal

[0893] Specific steps: Enter the book information into the form on the app screen and tap the "Request Analysis" button.

[0894] Step 5:

[0895] The device sends the user's analysis request to the server. During transmission, the necessary information is converted to JSON format and sent to the server as an HTTP request.

[0896] Input: Analysis request entered on the terminal

[0897] Output: HTTP request sent to the server

[0898] Specific operation: When the user taps the "Request Analysis" button, the application on the device converts the input data into JSON format and sends the data to the server via an API call.

[0899] Step 6:

[0900] The server retrieves text data from a database for a specified book and analyzes it using a natural language processing algorithm. A generative AI model (e.g., GPT-3) is then used to generate explanatory information for specific, difficult sections.

[0901] Input: Parsing request received as an HTTP request

[0902] Output: Generated explanatory information

[0903] Specific operation: The server queries the database to retrieve the relevant book data and analyzes the data using a natural language processing algorithm. Based on the analysis results, it generates explanatory information using a generative AI model such as GPT-3.

[0904] Step 7:

[0905] The server sends the generated explanatory information to the terminal. The explanatory information is sent as an HTTP response in JSON format.

[0906] Input: Generated explanatory information

[0907] Output: HTTP response sent to the terminal

[0908] Specific action: The generated explanatory information is sent to the terminal as an HTTP response.

[0909] Step 8:

[0910] The terminal displays explanatory information sent from the server to the user. The information is presented in an interactive format, incorporating graphics and icons to facilitate user understanding.

[0911] Input: Explanatory information received from the server

[0912] Output: Explanatory information displayed to the user

[0913] Specific actions: Display text explanations on the app screen and insert related graphics and icons. Allow users to tap to display detailed information.

[0914] Step 9:

[0915] If the user requests the generation of a character list, this request is entered into the app.

[0916] Input: Request to generate character list

[0917] Output: Request entered into the terminal

[0918] Specific action: Enter "I need a character list" into the app's form and tap the "Submit" button.

[0919] Step 10:

[0920] The terminal sends a request for the character list to the server. During transmission, the input information is converted to JSON format and sent to the server as an HTTP request.

[0921] Input: Request entered on the terminal

[0922] Output: HTTP request sent to the server

[0923] Specific operation: Convert the user's request into JSON format and send it to the server as an HTTP request.

[0924] Step 11:

[0925] The server analyzes and generates a character list from the text data of the specified book. Detailed information about each character (e.g., characteristics, relationships, appearances, etc.) is added.

[0926] Input: Request received as an HTTP request

[0927] Output: Generated character list

[0928] Specific operation: The server retrieves book data from the database and uses a natural language processing algorithm to extract the names of the characters and their detailed information.

[0929] Step 12:

[0930] The server sends a character list it has generated to the terminal. The list is sent as an HTTP response in JSON format.

[0931] Input: Generated character list

[0932] Output: HTTP response sent to the terminal

[0933] Specific action: The generated character list is sent to the terminal as an HTTP response.

[0934] Step 13:

[0935] The device interactively displays a list of characters received from the server. Users can view detailed information by tapping on each character's name.

[0936] Input: Character list received from the server

[0937] Output: List of characters displayed to the user

[0938] Specific operation: The app screen displays a list of characters, and when the user taps a character, detailed information is displayed.

[0939] Step 14:

[0940] The user specifies the parts of the book for which they want to know the source material or background information.

[0941] Input: Specify source material or background information.

[0942] Output: Specified information entered into the terminal

[0943] Specific steps: Enter the information you want to know about the source material or background into the app's form and tap the "Submit" button.

[0944] Step 15:

[0945] The device sends a request for source material and background information to the server. During transmission, the input information is converted to JSON format and sent to the server as an HTTP request.

[0946] Input: Request entered on the terminal

[0947] Output: HTTP request sent to the server

[0948] Specific operation: Convert the user's request into JSON format and send it to the server as an HTTP request.

[0949] Step 16:

[0950] The server collects source material and background information for the specified section from the database, analyzes it, and generates appropriate information.

[0951] Input: Request received as an HTTP request

[0952] Output: Generated source material and background information

[0953] Specific operation: The server queries the database to retrieve relevant source material and background information, and then uses a generative AI model to analyze and generate information.

[0954] Step 17:

[0955] The server sends the source material and background information it generates to the terminal. The information is sent as an HTTP response in JSON format.

[0956] Input: Generated source material and background information

[0957] Output: HTTP response sent to the terminal

[0958] Specific action: The generated information is sent to the terminal as an HTTP response.

[0959] Step 18:

[0960] The terminal provides the user with source material and background information received from the server. The information is displayed with care to avoid excessive spoilers.

[0961] Input: Source material and background information received from the server

[0962] Output: Source material and background information provided to the user

[0963] Specific actions: The app screen displays source material and background information, and interactive elements (e.g., links or taps to display details) are added as needed.

[0964] (Application Example 1)

[0965] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0966] Conventional reading support systems primarily focus on assisting with the reading comprehension of books, and lack sufficient support for factory work procedures and manuals. Therefore, there was a need to resolve the difficulties workers face in understanding these procedures and manuals. This invention aims to solve these problems and provide a system that improves work efficiency and safety in factories.

[0967] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0968] In this invention, the server includes means for receiving and storing registered reading information, means for analyzing text data of the material being read and identifying difficult sections, means for generating and providing explanatory information for the identified difficult sections in response to a user's request, and means for analyzing the explanations and supplementary information of work procedure manuals and providing visual explanations to the user. This enables factory workers to efficiently understand procedure manuals and manuals and perform their work appropriately.

[0969] "Means for receiving and storing registered reading information" refers to a system that has the function of receiving data such as personal information, preferred genres, and reading history provided by users, and storing it on a server.

[0970] "A means of analyzing text data of materials being read and identifying difficult parts" refers to a system that uses natural language processing technology to analyze the content of materials and has the function of identifying parts of the text that are particularly difficult to understand.

[0971] "A means of generating and providing explanatory information on specific difficult parts in response to user requests" refers to a mechanism that has the function of generating detailed explanations of difficult parts in response to questions and requests submitted by users and providing that information to the users.

[0972] "A means of analyzing work procedure manuals and supplementary information to provide users with visual explanations" refers to a system that analyzes work procedure manuals and other manuals used in factories, generates explanations and supplementary information, and provides them to users in a way that they can understand visually.

[0973] "A means of analyzing person information in text data and generating a list of people" refers to a system that has the function of analyzing the names and characteristics of people appearing in a document and compiling them into a list.

[0974] "A means of interactively displaying a generated list of people and providing detailed information about each person" refers to a mechanism that displays the generated list of people in a user-operable format and has the functionality to view detailed information about each person.

[0975] "Means for analyzing source material and background information related to documents and generating that information" refers to a system that analyzes source material and background information related to the content of documents and generates detailed explanations and supplementary information based on that analysis.

[0976] "A means of analyzing voice-inputted work instructions and page numbers of procedure manuals, and generating and providing explanations about the actions" refers to a system that analyzes the page numbers of work procedure manuals and specific actions instructed by the user via voice input, and generates and provides explanations and supplementary information.

[0977] This invention develops an AI assistant system to enhance the reading experience and implements a system to improve work efficiency and safety in factories. This system includes functions such as managing registered information, analyzing procedure manuals, generating and providing explanatory information, and analyzing voice input.

[0978] 1. System Program

[0979] The following describes the specific processing details of each program in the system.

[0980] Hardware to use

[0981] Smart glasses (e.g., Microsoft HoloLens)

[0982] Software to use

[0983] NLP libraries (e.g., spaCy, NLTK)

[0984] AR frameworks (e.g., Unity, Vuforia)

[0985] Backend frameworks (e.g., Node.js, Flask)

[0986] Generative AI models (e.g., OpenAI GPT-4)

[0987] Program Processing Description

[0988] 1. Management of registered information

[0989] The server receives data such as personal information, preferred genres, and reading history provided by users through the application and stores it in a database. Users input this information using devices such as smartphones, tablets, and smart glasses.

[0990] 2. Analysis of Procedure Manuals

[0991] The server uses an NLP library (e.g., spaCy) to analyze the text data of the material the user is reading and identify difficult sections. At this stage, an algorithm is used to evaluate the difficulty of words and phrases within the text.

[0992] 3. Generation and provision of explanatory information

[0993] The server uses a generative AI model (e.g., GPT-4) to generate explanatory information about the identified difficult sections. This generated explanatory information is provided to the user via smart glasses or other devices. For example, in response to a prompt such as, "Please tell me the points to pay attention to in Chapter 3 of the work procedure manual," the server generates a detailed explanation, providing information such as, "In Chapter 3, you should pay particular attention to the section concerning the safe operation of the machine."

[0994] 4. Voice input analysis

[0995] When a user provides voice input through smart glasses, the voice data is converted into text and sent to a server. The server analyzes the voice instructions and page numbers and generates corresponding explanations and supplementary information. For example, in response to a prompt such as "Please explain the instructions for using this machine in detail," detailed instructions on how to operate the machine are provided.

[0996] Specific example

[0997] If a user is having trouble understanding a particular procedure manual while working in a factory, they can voice-input through smart glasses, "Please tell me the important points to pay attention to in Chapter 3 of the work procedure manual." The server analyzes this information, generates an explanation of the parts of Chapter 3 that require special attention, and displays it in augmented reality (AR) in the user's field of view.

[0998] Example of a prompt:

[0999] 1. "Please tell me the key points to pay attention to in Chapter 3 of the work procedure manual."

[1000] 2. "Please explain in detail how to use this machine."

[1001] This system allows users to better understand the complex parts of work procedures and perform tasks efficiently and safely.

[1002] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1003] Step 1:

[1004] Users enter personal information and reading history using an application installed on their device (e.g., smart glasses, smartphone). This information includes name, email address, reading preferences, and reading frequency. The device sends this information to a server. The entered data is stored in a user information database.

[1005] Step 2:

[1006] The user inputs information about the document they are reading or the factory work instructions. Specifically, they input the title of the document, chapter, page number, and any parts they find difficult to understand. The terminal sends this information to the server. The server retrieves the received document information from a text database and analyzes the text data using natural language processing (NLP) technology.

[1007] Step 3:

[1008] The server uses an NLP library (e.g., spaCy) to analyze the text data of the document. The analysis understands grammatical structure and word meanings, and identifies particularly difficult sections. Information about these identified difficult sections is then input as prompts into a generative AI model (e.g., GPT-4). These prompts might include phrases like, "Please tell me the points to pay attention to in Chapter 3 of the work procedure manual."

[1009] Step 4:

[1010] The generative AI model (e.g., GPT-4) generates explanatory information for complex parts based on the prompt text received from the server. The generated explanatory information is written in detail to make it easy for the user to understand. The server sends the generated explanatory information to the terminal. This output data includes specific explanatory text.

[1011] Step 5:

[1012] The device provides the user with explanatory information received from the server. Specifically, the explanatory information is displayed as an augmented reality (AR) display on the smart glasses' screen. Audio guidance is also available. Based on the provided explanatory information, the user can understand the work procedure and perform it appropriately.

[1013] Step 6:

[1014] When a user provides voice input through smart glasses, the device converts the voice data into text data and sends it to the server. For example, a voice command such as "Please explain the instructions for using this machine in detail" might be input. The server analyzes the received voice command and page number text data to generate explanatory information about the operation. This explanatory information is then sent back to the device.

[1015] Step 7:

[1016] The terminal then provides the user with explanatory information about the operation received from the server. This explanatory information is displayed in the user's field of view using augmented reality (AR), and includes audio guidance along with detailed explanations of the operation. This allows the user to accurately understand and safely perform complex tasks.

[1017] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1018] Patent Specification

[1019] This invention combines an AI assistant system with an emotion engine to enhance the reading experience. It analyzes the user's emotional state and provides appropriate feedback and customized information based on that analysis. The program and specific processing of this system are described below.

[1020] 1. Registration and Initial Setup

[1021] User:

[1022] Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[1023] After launching the application, tap the "New Registration" button and enter the required information such as your name, email address, and password.

[1024] Set your preferences, such as genre, difficulty level, and reading frequency.

[1025] Terminal:

[1026] It receives information entered by the user and sends it to the server.

[1027] server:

[1028] The received user information is saved to the database, a registration completion message is generated, and sent to the terminal.

[1029] Terminal:

[1030] Display a registration completion message to the user.

[1031] 2. Initial setup of the emotion engine

[1032] User:

[1033] Enable emotion recognition within the app.

[1034] Terminal:

[1035] It uses a camera and microphone to capture the user's facial expressions and voice.

[1036] server:

[1037] The system analyzes the received emotional data and generates an initial emotional profile.

[1038] 3. Text analysis and information provision

[1039] User:

[1040] Enter the title, author, and specific chapter or page number of the book you are reading into the app. If there are parts you find difficult to understand, enter them as comments.

[1041] Terminal:

[1042] Send the user's input information to the server.

[1043] server:

[1044] Retrieve the text data of the specified book from the database.

[1045] We analyze text data using natural language processing algorithms to identify difficult parts.

[1046] Generate explanatory information for identified difficult sections. Select the appropriate explanation based on whether or not spoilers are included.

[1047] Customize explanatory information based on the user's emotional state.

[1048] Terminal:

[1049] Customized explanatory information received from the server is displayed interactively and provided to the user.

[1050] Specific example:

[1051] If a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. For example, if "The Use of Symbolism" is identified as a difficult section, an explanation is provided regarding its background and intent, such as "Symbolism is used in this section, and certain symbols have a deeper meaning than they actually do." Furthermore, if the user's emotional state is recognized as "confused," additional advice is customized and displayed, such as "Read slowly and think about what the symbols mean."

[1052] 4. Generation and provision of the character list

[1053] User:

[1054] If you can't remember the names of the characters, enter that into the app.

[1055] Terminal:

[1056] Send the request to the server.

[1057] server:

[1058] Analyze and generate a list of characters from the text data of a specified book.

[1059] Add detailed information about each character (e.g., characteristics, relationships, locations of appearance, etc.).

[1060] Customize character descriptions based on the user's emotional state.

[1061] Terminal:

[1062] The system interactively displays a list of characters sent from the server, allowing users to view detailed information through actions such as clicking.

[1063] Specific example:

[1064] If a user requests a character list for "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, knowledgeable and always offering advice." If the user's emotional state is "excited," an additional comment will be displayed, such as, "Pay attention to how this character will influence the story going forward."

[1065] 5. Providing source material and background information.

[1066] User:

[1067] You can specify in the app which parts of a book you want to know about, such as the source material or background information.

[1068] Terminal:

[1069] Send the request to the server.

[1070] server:

[1071] The system collects source material and background information for the specified portion from a database.

[1072] It analyzes collected source material and background information to generate appropriate information.

[1073] Customize information based on the user's emotional state.

[1074] Terminal:

[1075] The system provides users with source material and background information received from the server. Care is taken to avoid excessive spoilers.

[1076] Specific example:

[1077] If a user inputs that they want to know about Chapter 3 of "Book C," the server analyzes the source material and background information for that section and provides information such as, "This chapter is based on a medieval legend and strongly reflects elements of XXX in particular." If the user's emotional state indicates "surprise," additional advice such as, "Try to predict what will happen next based on this information," will be displayed.

[1078] Thus, the present invention is a system that recognizes the user's emotional state and provides various information based on that state, thereby enriching the reading experience and deepening understanding.

[1079] The following describes the processing flow.

[1080] Forms for carrying out the invention - A system combining an emotion engine

[1081] 1. Registration and Initial Setup

[1082] Step 1:

[1083] User: Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[1084] Step 2:

[1085] User: Tap the "Register" button and enter the required information such as your name, email address, and password.

[1086] Step 3:

[1087] User: Sets preferences such as genre, difficulty level, and reading frequency.

[1088] Step 4:

[1089] Terminal: Converts user-entered information into JSON format and sends it to the server using the API.

[1090] Step 5:

[1091] Server: Stores the received user information in the database, generates a registration completion message, and sends it to the terminal.

[1092] Step 6:

[1093] Terminal: Displays a registration completion message to the user.

[1094] 2. Initial setup of the emotion engine

[1095] Step 1:

[1096] User: Enable emotion recognition within the app.

[1097] Step 2:

[1098] Device: Captures the user's facial expressions and voice using the camera and microphone.

[1099] Step 3:

[1100] Server: Analyzes received emotion data and generates an initial emotion profile.

[1101] 3. Text analysis and information provision

[1102] Step 1:

[1103] User: Enter the title, author, and specific chapter or page number of the book you are reading into the app. If there are parts you find difficult to understand, enter them as comments.

[1104] Step 2:

[1105] Terminal: Converts user input information into JSON format and sends it to the server using the API.

[1106] Step 3:

[1107] Server: Retrieves text data for the specified book from the database.

[1108] Step 4:

[1109] Server: Analyzes text data using natural language processing (NLP) algorithms to identify difficult parts.

[1110] Step 5:

[1111] Server: Generates explanatory information for identified difficult sections. Selects appropriate explanations based on whether or not spoilers are included.

[1112] Step 6:

[1113] Server: Analyzes the user's emotional state and customizes the explanatory information.

[1114] Step 7:

[1115] Server: Converts explanatory information into JSON format and sends it to the terminal.

[1116] Step 8:

[1117] Terminal: Interactively displays and provides customized explanatory information received from the server to the user.

[1118] Specific example:

[1119] If a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. For example, if "The Use of Symbolism" is identified as a difficult section, an explanation is provided regarding its background and intent, such as "Symbolism is used in this section, and certain symbols have a deeper meaning than they actually do." If the emotion recognition determines that the user is "confused," advice such as "Try reading the next chapter to find out what this symbol means" is added.

[1120] 4. Generation and provision of the character list

[1121] Step 1:

[1122] User: If you can't remember the names of the characters, enter that into the app.

[1123] Step 2:

[1124] Terminal: Converts the request to JSON format and sends it to the server.

[1125] Step 3:

[1126] Server: Analyzes and generates a list of characters from the text data of the specified book.

[1127] Step 4:

[1128] Server: Adds detailed information about each character (e.g., characteristics, relationships, locations of appearance, etc.).

[1129] Step 5:

[1130] Server: Analyzes the user's emotional state and customizes the character descriptions.

[1131] Step 6:

[1132] Server: Converts the generated character list into JSON format and sends it to the terminal.

[1133] Step 7:

[1134] Terminal: Interactively displays the character list received from the server, allowing the user to view detailed information through actions such as clicking.

[1135] Specific example:

[1136] If a user requests a list of characters from "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, knowledgeable and always offering advice." If emotion recognition determines that the user is "excited," an additional comment will be displayed, such as, "Pay attention to how this character will influence the story going forward."

[1137] 5. Providing source material and background information.

[1138] Step 1:

[1139] User: Specify to the app the parts of a book for which you want to know the source material or background information.

[1140] Step 2:

[1141] Terminal: Converts the request to JSON format and sends it to the server.

[1142] Step 3:

[1143] Server: Collects source material and background information for the specified section from the database.

[1144] Step 4:

[1145] Server: Analyzes collected source material and background information to generate appropriate information.

[1146] Step 5:

[1147] Server: Analyzes the user's emotional state and customizes the information.

[1148] Step 6:

[1149] Server: Converts the generated source material and background information into JSON format and sends it to the terminal.

[1150] Step 7:

[1151] Terminal: Provides users with source material and background information received from the server. Care is taken to avoid excessive spoilers.

[1152] Specific example:

[1153] If a user inputs that they want to know about Chapter 3 of "Book C," the server analyzes the source material and background information for that section and provides information such as, "This chapter is based on a medieval legend and strongly reflects elements of XXX in particular." If the user's emotion recognition detects "surprise," advice such as, "Try to predict what will happen next based on this information," will be displayed.

[1154] In this way, the program can enrich the user's reading experience and deepen their understanding by recognizing the user's emotional state at each processing step and customizing and providing information based on that state.

[1155] (Example 2)

[1156] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[1157] Existing reading support systems provide uniform information without considering the user's emotional state, resulting in reading experiences that do not meet individual needs. Furthermore, information about characters and source material is often insufficient, hindering comprehension during reading. This leads to difficulties for users during reading and a lack of information consistency, posing a significant challenge.

[1158] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving and storing information of registered users, means for capturing and analyzing the user's emotional state using a camera and microphone, means for analyzing the text data of the book being read and identifying difficult parts, and means for generating and providing customized explanatory information for the identified difficult parts in accordance with the user's emotional state and requests. This makes it possible to provide personalized information based on the user's emotional state, thereby providing a richer and more symmetrical reading experience.

[1159] "Registered user information" refers to personal information and settings information that users provide when registering with the system, such as their name, email address, password, and preferred genres.

[1160] "User emotional state" refers to the psychological state of a user obtained by analyzing data of the user's facial expressions and voice captured by cameras and microphones.

[1161] "Means of capturing and analyzing using cameras and microphones" refers to means including software and hardware for collecting and analyzing a user's facial expressions and voice using cameras and microphones installed on a device.

[1162] "A means of analyzing text data and identifying difficult parts" refers to a program that uses natural language processing algorithms to analyze the text data of a book and automatically identify parts that users find difficult to understand.

[1163] "Customized explanatory information" refers to individually optimized explanatory information generated based on the user's emotional state and requests for specific, difficult-to-understand parts.

[1164] "A means of generating a character list" refers to a program that analyzes the text data of a book and lists the names of the main characters that appear in the book.

[1165] A "customized character list" is a character list generated based on the user's emotional state, with descriptions that take into account the characteristics and relationships of each character.

[1166] A "means for generating source material and background information" refers to a program that collects additional information, such as source material and historical background, for specific parts of a book, analyzes that information, and provides it.

[1167] "Customized information that avoids excessive spoilers" refers to background information and source material provided with consideration for the user's emotional state, without compromising the entertainment value.

[1168] This invention combines an AI assistant system with an emotion engine to enhance the reading experience, analyzing the user's emotional state and providing appropriate feedback and customized information based on that analysis.

[1169] First, the user downloads the application to their device (smartphone, tablet, etc.), installs it, and then launches it. The user launches the application, taps the "New Registration" button, and enters necessary information such as their name, email address, and password. Next, the user sets their preferred genre, reading difficulty level, and reading frequency. The device then receives the user's input information and sends it to the server. The server saves the information to its database, generates a registration completion message, and sends it to the device. The device then displays this message to the user.

[1170] Next, the user configures the app to enable emotion recognition. The device uses its camera and microphone to capture the user's facial expressions and voice, and sends this data to the server. The server analyzes the emotion data, generates an initial emotion profile using an AI model (e.g., an emotion recognition model), and stores it in a database.

[1171] The user enters the title, author, and specific chapter or page number of the book they are reading into the app, and adds comments on parts they find difficult to understand. The device sends this information to the server. The server retrieves the text data of the specified book from a database, analyzes it using natural language processing algorithms (e.g., BERT, GPT-3), and identifies the difficult parts. The server also generates customized explanatory information, taking into account the user's emotional state, and sends it to the device. The device then interactively displays and provides the customized explanatory information to the user.

[1172] For example, if a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. If the use of symbolism is identified as a difficult part, an explanation is provided regarding its background and intent, such as, "Symbolism is used in this section, and certain symbols have a deeper meaning than they actually do." Additionally, if the user's emotional state is recognized as "confused," extra advice is displayed, such as, "Read slowly and think about what the symbols mean."

[1173] Furthermore, if the user cannot remember the names of the characters, they can input this into the app. The device sends this request to the server. The server parses the text data and generates a list of characters. The list includes detailed information about each character (characteristics, relationships, appearances, etc.) and is customized based on their emotional state. The customized list is sent to the device, allowing the user to interactively view the detailed information.

[1174] For example, if a user requests a character list for "Book B," the server will provide detailed information such as "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, knowledgeable and always offering advice." If the user's emotional state is "excited," an additional comment will be displayed such as, "Pay attention to how this character will influence the story going forward."

[1175] Finally, the user specifies to the app which parts of the book they want to know about, such as the source material and background information. The device sends the request to the server. The server retrieves the source material and background information from its database, analyzes it, and provides the generated information. Customized information is provided to the user while avoiding excessive spoilers.

[1176] For example, if a user inputs that they want to know about Chapter 3 of "Book C," the server analyzes the source material and background information for that section and provides information such as, "This chapter is based on a medieval legend and strongly reflects elements of XXX in particular." If the user's emotional state indicates "surprise," additional advice such as, "Try to predict what happens next based on this information," will be displayed.

[1177] Example of a prompt:

[1178] Could you please explain the meaning of "Muggle" in Chapter 1 of the book "Harry Potter and the Sorcerer's Stone"?

[1179] Please display the list of characters from the book "The Lord of the Rings".

[1180] Thus, the present invention is a system that recognizes the emotional state of the user and provides various information based on that state, thereby enriching the reading experience and deepening understanding.

[1181] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1182] Step 1: Install and launch the application

[1183] User:

[1184] The user downloads and installs the application on their device (such as a smartphone or tablet).

[1185] Then, launch the application.

[1186] Terminal:

[1187] The device launches the installed application and displays the initial screen.

[1188] input:

[1189] Downloaded applications

[1190] output:

[1191] Displaying the application's initial screen.

[1192] Step 2: New Registration

[1193] User:

[1194] After launching the application, the user taps the "Register" button and enters the required information such as their name, email address, and password.

[1195] Terminal:

[1196] The terminal temporarily saves the entered information and displays a button to proceed to the next step.

[1197] input:

[1198] User information such as name, email address, and password

[1199] output:

[1200] A button to proceed to the next step is displayed.

[1201] Step 3: Customize your settings

[1202] User:

[1203] Users can set their preferred genre, reading difficulty level, and reading frequency.

[1204] Terminal:

[1205] The device temporarily stores the configured information and displays a confirmation screen to send all the information to the server.

[1206] input:

[1207] Settings such as preferred genre, difficulty level, and reading frequency.

[1208] output:

[1209] Confirmation screen displayed

[1210] Step 4: Sending and saving information

[1211] Terminal:

[1212] The terminal sends the information entered and configured by the user to the server.

[1213] server:

[1214] The server stores the received information in a database.

[1215] Generate a registration completion message and send it to the device.

[1216] input:

[1217] Sending user information and settings information

[1218] output:

[1219] Storing information in a database

[1220] Generate and send registration completion message

[1221] Step 5: Display of registration completion message

[1222] Terminal:

[1223] The device displays a registration completion message to the user.

[1224] input:

[1225] Registration complete message received

[1226] output:

[1227] Display of registration completion message

[1228] Step 6: Setting up emotion recognition function

[1229] User:

[1230] Users configure settings within the app to enable the emotion recognition feature.

[1231] Terminal:

[1232] The device requests permission for the camera and microphone and verifies that the settings are enabled.

[1233] input:

[1234] Emotion recognition function settings

[1235] output:

[1236] Checking camera and microphone usage permissions

[1237] Step 7: Capture and send emotional data

[1238] User:

[1239] The user faces the device's camera and microphone, providing standard facial expressions and voice.

[1240] Terminal:

[1241] The device uses its camera and microphone to capture the user's facial expressions and voice, and temporarily stores this as emotion data.

[1242] The captured emotion data is sent to the server.

[1243] input:

[1244] User's facial expressions and voice

[1245] output:

[1246] Capture and transmit emotional data

[1247] Step 8: Generating an emotional profile

[1248] server:

[1249] The server analyzes the received emotional data using an AI model and generates an initial emotional profile.

[1250] The generated emotion profile is saved to the database.

[1251] input:

[1252] Sent emotional data

[1253] output:

[1254] Generating and saving initial emotion profiles

[1255] Step 9: Enter reading information

[1256] User:

[1257] Users enter the title, author, and specific chapter or page number of the book they are reading into the app. They can also add comments to sections they find difficult to understand.

[1258] Terminal:

[1259] The terminal temporarily stores the entered information and sends it to the server.

[1260] input:

[1261] Book title, author name, specific chapter or page number, comments

[1262] output:

[1263] Sending information to the server

[1264] Step 10: Acquisition and analysis of text data

[1265] server:

[1266] The server retrieves the text data of the specified book from the database.

[1267] We analyze text data using natural language processing algorithms to identify difficult parts.

[1268] input:

[1269] Book title, author name, specific chapter or page number

[1270] output:

[1271] Identifying the difficult parts

[1272] Step 11: Generating customized explanatory information

[1273] server:

[1274] The server generates explanatory information for the identified difficult parts.

[1275] It generates customized explanatory information that takes into account the user's emotional state.

[1276] input:

[1277] User emotion profiles, text analysis data

[1278] output:

[1279] Generating customized explanatory information

[1280] Step 12: Send and display customized explanatory information

[1281] server:

[1282] The server sends the generated customized explanation information to the terminal.

[1283] Terminal:

[1284] The device interactively displays and provides customized explanatory information to the user.

[1285] input:

[1286] Customized explanatory information

[1287] output:

[1288] Display of explanatory information

[1289] Step 13: Requesting and generating the character list

[1290] User:

[1291] If a user cannot remember the names of the characters, they can enter that information into the app.

[1292] Terminal:

[1293] The terminal sends the request to the server.

[1294] server:

[1295] The server generates a list of characters from the text data of the book.

[1296] input:

[1297] Request for character list

[1298] output:

[1299] Generating a list of characters

[1300] Step 14: Provide a customized character list

[1301] server:

[1302] The server adds detailed information about each character and customizes the list based on the user's emotional state.

[1303] Terminal:

[1304] The terminal interactively displays a list of characters sent from the server, and the user can view detailed information by clicking or performing other actions.

[1305] input:

[1306] Character List

[1307] output:

[1308] Providing a customized character list

[1309] Step 15: Requesting and providing source material and background information.

[1310] User:

[1311] Users specify to the app which parts of a book they want to know about, such as the source material or background information.

[1312] Terminal:

[1313] The terminal sends the request to the server.

[1314] server:

[1315] The server collects source material and background information for the specified section from the database and generates appropriate information.

[1316] input:

[1317] Request for source material and background information

[1318] output:

[1319] Generated source material and background information

[1320] Terminal:

[1321] The system interactively displays source material and background information received from the server and provides it to the user. Filtering is performed to avoid excessive spoilers.

[1322] input:

[1323] Source material and background information

[1324] output:

[1325] Providing customized background information

[1326] (Application Example 2)

[1327] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[1328] Traditional reading-related information systems often provided uniform information without considering the user's emotional state. As a result, support for individual reading experiences was insufficient, and customization tailored to the user's specific emotional state was needed, particularly in understanding difficult passages, grasping characters, and providing source material and background information. Furthermore, a system capable of providing real-time feedback on the user's interests and questions during reading was required.

[1329] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1330] In this invention, the server includes means for receiving and storing user information related to registered reading; means for analyzing text data of the book being read and identifying difficult sections; means for generating and providing explanatory information for the identified difficult sections in response to user requests; means for analyzing the user's emotional state using a camera and microphone; and means for customizing the explanatory information based on the analyzed emotional state. This enables personalized feedback and explanations based on the user's emotional state, making it possible to provide information optimized for each user.

[1331] "Registered user information regarding reading" refers to personal information provided by the user regarding their reading, such as their name, preferred genres, reading frequency, and difficulty level.

[1332] "Book text data" refers to data that represents the content of a book a user is reading in digital format.

[1333] "Difficult sections" refer to parts of a book that are identified as being hard to understand.

[1334] "Explanatory information" refers to explanations and supplementary information provided to make difficult parts easier for users to understand.

[1335] "Means of analyzing a user's emotional state using cameras and microphones" refers to devices and software that capture a user's facial expressions and voice and identify their emotions from them.

[1336] "Means for customizing explanatory information based on analyzed emotional states" refers to methods and software for adjusting and individually adapting explanatory information based on the user's emotional state.

[1337] "Character information" refers to data such as the names, characteristics, and relationships of all characters that appear in the book.

[1338] "Means for generating a character list" refers to a method and software for extracting character names and related information from text data and listing them.

[1339] "Character details" refers to more detailed explanations and background information about the characters.

[1340] "Source material and background information" refers to additional information such as the source, references, and historical context of specific parts of a book.

[1341] "Means for providing users with generated source material and background information" refers to methods and software for displaying or communicating analyzed source material and background information to users.

[1342] This invention utilizes a user terminal, a server, and a network environment connecting them. Specifically, the invention can be implemented in the following forms.

[1343] The user downloads and installs the "Smart Book Reader" app on their smartphone. Upon launching the app, the user registration screen is displayed first. The user enters the necessary information, such as their name, email address, and password, and the device sends this information to the server. The server stores the received information in its database and sends a registration completion message to the device, which then displays the message to the user.

[1344] Next, an option to enable emotion recognition within the app will appear. If the user enables it, the device will use the camera and microphone to capture the user's facial expressions and voice in real time. This data is sent to a server, which analyzes it using an emotion engine to generate an initial emotion profile.

[1345] When a user begins reading, they enter the title, author, and specific chapter or page number of the book they are reading into the app. They can also add comments if they find certain sections difficult to understand. This information is sent from the device to a server, which retrieves the book's text data from a database and analyzes it using natural language processing algorithms. The server then identifies difficult sections and generates explanatory information. Furthermore, the explanatory information is customized based on the user's emotional state. This explanatory information is sent to the device and displayed interactively to the user.

[1346] For example, if a user is reading Chapter 5 of "Book A" and is confused about the "use of symbolism," the server will analyze that section and provide an explanation such as, "Symbolism is used in this section, and certain symbols have a deeper meaning than they actually do." Furthermore, if the server recognizes that the user's emotional state is "confused," it will display customized additional advice such as, "Read slowly and think about what the symbols mean."

[1347] In generating the character list, when a user enters a character's name they can't remember into the app, the device sends a request to the server. The server analyzes the text data from the book and generates the character list. It then adds detailed information about each character and customizes it according to the user's emotional state. For example, if the user is "excited," an additional comment such as "Pay attention to how this character will influence the story going forward" will be displayed.

[1348] Regarding source material and background information, when a user inputs the part they want to know about, the server analyzes the source material and background information for the specified part and generates appropriate information. For example, if a user wants to know about Chapter 3 of "Book C," the server will provide information such as, "This chapter is based on a medieval legend and strongly reflects elements of XXX in particular." If the user indicates "surprise," additional advice such as, "Try to predict what will happen next based on this information," will be displayed.

[1349] The implementation of this system involves using smartphones as user terminals, servers for analysis and data storage, and software such as TensorFlow and OpenCV for emotion recognition, spaCy for natural language processing, and Flask for interface construction.

[1350] An example of a prompt sentence would be, "I would like to understand the concept of symbolism in Chapter 5 of Book A. I am confused."

[1351] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1352] Step 1:

[1353] The user downloads and installs the "Smart Book Reader" app on their smartphone.

[1354] Input: An app installed on a smartphone.

[1355] Output: The app is launched.

[1356] Specific operation: After the user downloads the app from the Google Play Store or Apple App Store and the installation is complete, they tap the icon on the home screen to launch the app.

[1357] Step 2:

[1358] The user enters the required information on the app's new registration screen and registers.

[1359] Input: Name, email address, password, preferred genre, difficulty level, reading frequency.

[1360] Output: Registration information is sent to the server and stored in the database.

[1361] Specific operation: The user enters the required information on the new registration screen and taps the "Register" button. The device sends the entered information to the server, which receives it and saves it to the database. Once saving is complete, the server sends a registration completion message back to the device, which then displays it to the user.

[1362] Step 3:

[1363] The user enables the emotion recognition feature within the app.

[1364] Input: Enabling / enabling the emotion recognition function.

[1365] Output: The device uses its camera and microphone to capture the user's facial expressions and voice, and sends them to the server.

[1366] Specific operation: When the user enables the emotion recognition feature in the app's settings menu, the device activates the camera and microphone, capturing the user's facial expressions and voice and sending them to the server in real time.

[1367] Step 4:

[1368] The server analyzes the received emotion data and generates an initial emotion profile.

[1369] Input: User's facial expression data and voice data.

[1370] Output: Initial emotion profile.

[1371] Specific operation: The server uses TensorFlow and OpenCV to analyze facial expression data and voice data, generates an initial emotion profile of the user based on that analysis, and saves it to a database.

[1372] Step 5:

[1373] The user enters the title of the book they are reading, the author's name, a specific chapter or page number, and any parts they find difficult to understand into the app.

[1374] Input: Book title, author name, chapter and page numbers, and comments.

[1375] Output: This information is sent to the server.

[1376] Specific operation: The user enters the required information on an input screen within the app and taps the "Submit" button. The device then sends this information to the server.

[1377] Step 6:

[1378] The server retrieves the text data of the specified book from the database and analyzes it using a natural language processing algorithm.

[1379] Input: Book title, chapter or page number.

[1380] Output: Identification of difficult sections and explanatory information for those sections.

[1381] Specific operation: The server retrieves text data of a specified book from the database and analyzes it using spaCy. Based on the analysis results, it identifies difficult sections and generates appropriate explanatory information.

[1382] Step 7:

[1383] The server customizes the explanatory information based on the user's emotional state.

[1384] Input: User's emotion profile, generated explanatory information.

[1385] Output: Customized explanatory information.

[1386] Specific operation: The server references the user's emotional profile and customizes the generated explanatory information each time. For example, for a confused user, it adds more detailed explanations and words of encouragement.

[1387] Step 8:

[1388] Customized explanatory information is sent from the server to the terminal and provided to the user.

[1389] Input: Customized explanatory information.

[1390] Output: Interactively displayed explanatory information.

[1391] Specific operation: The server sends customized explanatory information to the device, which then displays it interactively. The user can tap on the information to view more details.

[1392] An example of a prompt sentence would be, "I would like to understand the concept of symbolism in Chapter 5 of Book A. I am confused."

[1393] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1394] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1395] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[1396] [Third Embodiment]

[1397] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[1398] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1399] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1400] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[1401] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1402] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1403] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1404] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1405] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1406] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1407] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1408] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[1409] Patent Specification

[1410] This invention is an AI assistant system for improving the reading experience, which analyzes and provides various information about the book the user is reading. The program of this system and its specific processing are described below.

[1411] 1. Registration and Initial Setup

[1412] User:

[1413] Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[1414] After launching the app, tap the "New Registration" button, enter the required information such as your name, email address, and password, and set your preferred genre, difficulty level, reading frequency, etc.

[1415] Terminal:

[1416] It receives information entered by the user and sends it to the server.

[1417] server:

[1418] The received user information is saved to the database, a registration completion message is generated, and sent to the terminal.

[1419] 2. Text analysis and information provision

[1420] User:

[1421] Enter the title, author, and specific chapter or page number of the book you are reading into the app. If necessary, specify the parts you are having difficulty understanding.

[1422] Terminal:

[1423] The information entered by the user is sent to the server.

[1424] server:

[1425] The text data of a specified book is retrieved from a database and analyzed using a natural language processing algorithm. Particularly difficult sections are identified, and explanatory information is generated.

[1426] The system checks for spoilers, generates appropriate explanatory information, and sends it to the device.

[1427] Terminal:

[1428] The server sends explanatory information to the user. Graphics and icons are also displayed in an interactive format.

[1429] Specific example:

[1430] If a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. For example, if "The Use of Symbolism" is identified as a difficult section, an explanation of its background and intent will be provided, such as "Symbolism is used in this section, and certain symbols have a deeper meaning than they do in reality."

[1431] 3. Generation and provision of the character list

[1432] User:

[1433] If you can't remember the names of the characters, enter that into the app.

[1434] Terminal:

[1435] Send the request to the server.

[1436] server:

[1437] This program analyzes and generates a character list from the text data of a specified book. It then adds detailed information about each character (e.g., characteristics, relationships, appearances, etc.).

[1438] Terminal:

[1439] The system interactively displays a list of characters sent from the server, allowing users to view detailed information through actions such as clicking.

[1440] Specific example:

[1441] If a user requests a list of characters from "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, who is knowledgeable and always offers advice."

[1442] 4. Providing source material and background information

[1443] User:

[1444] Specify the parts of the book for which you want to know the source material or background information.

[1445] Terminal:

[1446] Send the request to the server.

[1447] server:

[1448] The system collects source material and background information for the specified portion from a database, analyzes it, and generates appropriate information.

[1449] Terminal:

[1450] The server provides users with source material and background information. Care is taken to avoid excessive spoilers.

[1451] Specific example:

[1452] If a user inputs a request to learn about Chapter 3 of "Book C," the server analyzes the source material and background information for that section and provides information such as, "This chapter is based on medieval legends and strongly reflects elements of XXX in particular."

[1453] Thus, the present invention specifically realizes a system that provides information to enrich the user's reading experience and deepen their understanding.

[1454] The following describes the processing flow.

[1455] Program processing flow

[1456] 1. Registration and Initial Setup

[1457] Step 1:

[1458] User: Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[1459] Step 2:

[1460] User: Tap the "Register" button and enter the required information such as your name, email address, and password.

[1461] Step 3:

[1462] User: Sets preferences such as genre, difficulty level, and reading frequency.

[1463] Step 4:

[1464] Terminal: Converts user-entered information into JSON format and sends it to the server using the API.

[1465] Step 5:

[1466] Server: Stores received user information in the database.

[1467] Step 6:

[1468] Server: Generates a registration completion message and sends it to the terminal.

[1469] Step 7:

[1470] Terminal: Displays a registration completion message to the user.

[1471] 2. Text analysis and information provision

[1472] Step 1:

[1473] User: Enter the title, author, and specific chapter or page number of the book you are reading into the app. If there are parts you find difficult to understand, enter them as comments.

[1474] Step 2:

[1475] Terminal: Converts user input information into JSON format and sends it to the server using the API.

[1476] Step 3:

[1477] Server: Retrieves text data for the specified book from the database.

[1478] Step 4:

[1479] Server: Analyzes text data using natural language processing (NLP) algorithms to identify difficult parts.

[1480] Step 5:

[1481] Server: Generates explanatory information for identified difficult sections. Selects appropriate explanations based on whether or not spoilers are included.

[1482] Step 6:

[1483] Server: Converts explanatory information into JSON format and sends it to the terminal.

[1484] Step 7:

[1485] Terminal: Interactively displays and provides explanatory information received from the server to the user.

[1486] 3. Generation and provision of the character list

[1487] Step 1:

[1488] User: If you can't remember the names of the characters, enter that into the app.

[1489] Step 2:

[1490] Terminal: Converts user requests into JSON format and sends them to the server.

[1491] Step 3:

[1492] Server: Analyzes and generates a list of characters from the text data of the specified book.

[1493] Step 4:

[1494] Server: Adds detailed information about each character (e.g., characteristics, relationships, locations of appearance, etc.).

[1495] Step 5:

[1496] Server: Converts the generated character list into JSON format and sends it to the terminal.

[1497] Step 6:

[1498] Terminal: Interactively displays the character list received from the server, allowing users to view detailed information through actions such as clicking.

[1499] 4. Providing source material and background information

[1500] Step 1:

[1501] User: Specify to the app the parts of a book for which you want to know the source material or background information.

[1502] Step 2:

[1503] Terminal: Converts user requests into JSON format and sends them to the server.

[1504] Step 3:

[1505] Server: Collects source material and background information for the specified section from the database.

[1506] Step 4:

[1507] Server: Analyzes collected source material and background information to generate appropriate information.

[1508] Step 5:

[1509] Server: Converts the generated source material and background information into JSON format and sends it to the terminal.

[1510] Step 6:

[1511] Terminal: Provides users with source material and background information received from the server. Care is taken to avoid excessive spoilers.

[1512] In this way, the program enriches the user's reading experience and provides information to deepen their understanding through each processing step.

[1513] (Example 1)

[1514] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1515] Traditional reading systems have a problem in that they cannot provide users with appropriate explanations or background information for difficult or hard-to-understand parts of a book. Furthermore, it is difficult for users to remember character names and detailed information, leading to frequent interruptions in the reading experience. In addition, the source material and background information are not provided in an appropriate format, making it difficult to achieve a deep understanding. There is a need for a system that solves these problems and improves the reading experience.

[1516] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1517] In this invention, the server includes means for receiving and storing user information related to registered readings, means for analyzing text data of the medium being read and identifying difficult sections, and means for generating and providing explanatory information for the identified difficult sections in response to user requests, using a generative artificial intelligence model. This makes it possible to immediately provide appropriate explanatory information for difficult sections that the user encounters while reading.

[1518] The server further includes means for analyzing information about characters appearing in the reading medium and generating a character list, and means for interactively displaying the generated character list and providing detailed information about each character. This allows users to easily check detailed information about characters, dramatically improving the reading experience.

[1519] Furthermore, the server includes means for analyzing source material and background information related to the medium being read and generating that information, and means for providing the generated source material and background information to the user. This allows the user to gain a deeper understanding of the background and source material of the medium being read, thereby enriching the reading experience.

[1520] "Registered user information related to reading" refers to personal information entered by the user into the system, such as name, email address, password, preferred genres, difficulty level, and reading frequency.

[1521] "Text data in a medium" refers to electronic text data such as books, magazines, and academic papers that a user is reading.

[1522] "Analysis" refers to the process of analyzing the content of text data using natural language processing algorithms and other methods to extract specific patterns or meanings.

[1523] "Difficult parts" refer to sections of text that users find hard to understand.

[1524] "Generative artificial intelligence models" refer to machine learning models and deep learning models used for natural language processing and data analysis. Specifically, this includes advanced text generation models such as GPT-3.

[1525] "Explanatory information" refers to additional explanations or background information for parts that users found difficult to understand.

[1526] A "character list" refers to a list of characters that appear in the medium being read. This list includes each character's name, role, and characteristics.

[1527] "Source material and background information" refers to information about the historical and cultural background and original sources that influenced the content of the medium being read.

[1528] "Interactive display" refers to a display method that allows users to view additional details or explanations by performing actions such as tapping or clicking.

[1529] An "HTTP request" refers to a form of data communication based on a protocol used when sending data from a terminal to a server.

[1530] This invention is an AI assistant system designed to enhance the user's reading experience. It utilizes natural language processing and generative AI models to analyze and provide various information about the medium. The system analyzes the text the user is reading in detail and interactively provides lists of difficult passages and characters, as well as source material and background information.

[1531] Registration and initial setup

[1532] User: Download and install the application on your smartphone or tablet. After installation, launch the application and tap the "New Registration" button. Complete the initial setup by entering your name, email address, password, and preferred genres, difficulty level, reading frequency, etc.

[1533] Terminal: Receives information entered by the user and sends it to the server. Specifically, it converts the user's information into JSON format and sends it to the server as an HTTP request.

[1534] Server: Saves received user information to the database. The database used is an RDBMS (Relational Database Management System) such as MySQL or PostgreSQL. After saving is complete, a registration completion message is generated and sent to the terminal.

[1535] Text analysis and information provision

[1536] User: Enter the title, author, and specific chapter or page number of the book you are reading into the app. If necessary, specify the parts you are having difficulty understanding.

[1537] Terminal: Sends user-entered information to the server. During transmission, text data is converted to JSON format and sent via the API.

[1538] Server: Retrieves text data from the specified medium from the database. Next, it uses a natural language processing algorithm to analyze the data and identify the difficult parts pointed out by the user. It then generates explanatory information for the difficult parts using a generation AI model (e.g., GPT-3). During this process, it checks for spoilers and generates explanatory information within an appropriate scope.

[1539] For example, if a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. If, for instance, "The Use of Symbolism" is identified as a difficult section, the server provides an explanation of its background and intent, such as, "Symbolism is used in this section, and certain symbols have a deeper meaning than they do in reality."

[1540] Terminal: Displays explanatory information sent from the server to the user. The explanatory information is displayed in an interactive format, including graphics and icons.

[1541] Generating and providing a list of characters

[1542] User: If you can't remember the names of the characters, enter that into the app.

[1543] Terminal: Sends requests to the server. Specifically, it converts the input request into JSON format and makes an API call.

[1544] Server: Analyzes and generates a character list from the text data of the specified medium. Adds detailed information about each character (e.g., characteristics, relationships, appearances, etc.). Uses natural language processing algorithms for analysis and retrieves necessary information from the database.

[1545] For example, if a user requests a list of characters from "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, who is knowledgeable and always offers advice."

[1546] Terminal: Interactively displays the character list sent from the server. Users can view detailed information by tapping on names in the list.

[1547] Providing source material and background information

[1548] User: Specify the parts of the book for which you want to know the source material or background information.

[1549] For example, if a user inputs that they want to know about Chapter 3 of "Book C," the server will analyze the source material and background information for that section and provide information such as, "This chapter is based on medieval legends and particularly strongly reflects elements of XXX."

[1550] Terminal: Sends the request to the server.

[1551] Server: Collects source material and background information for the specified section from the database, analyzes it, and generates appropriate information.

[1552] Terminal: Provides users with source material and background information sent from the server. Displays information while taking care to avoid excessive spoilers.

[1553] Example of a prompt

[1554] Prompt 1: "I am reading Chapter 5 of Book A, but I don't understand the use of symbolism. Please provide a detailed explanation."

[1555] Prompt 2: "Please provide a list of characters in 'Book B' and their respective characteristics."

[1556] Prompt 3: "What source material or background information is found in Chapter 3 of 'Book C'?"

[1557] As described above, the present invention specifically realizes the provision of information that enriches the user's reading experience and deepens their understanding.

[1558] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1559] Step 1:

[1560] The user downloads and installs the application on their smartphone or tablet. This provides the basic platform for accessing the system. Next, they launch the application and enter their name, email address, password, preferred genres, etc., on the new user registration screen. This collects the user's basic information.

[1561] Input: Username, email address, password, preferred genre, etc.

[1562] Output: User information entered into the terminal

[1563] Specific steps: The user searches for the app on the App Store or Google Play and taps the download button. After the download is complete, the installation starts automatically, and once finished, an icon appears on the device's home screen. After launching the app, the user enters the required information on the registration screen that appears.

[1564] Step 2:

[1565] The terminal receives the information entered by the user and sends it to the server. The user's information is converted to JSON format and sent to the server as an HTTP request.

[1566] Input: User information entered on the terminal

[1567] Output: HTTP request sent to the server

[1568] Specific operation: When the user taps the "Send" button, the application on the device converts the input data into JSON format and sends the data to the server via an API call.

[1569] Step 3:

[1570] The server saves the received user information to a database. An RDBMS such as MySQL or PostgreSQL is used for the database. After saving is complete, a registration completion message is generated and sent to the terminal.

[1571] Input: User information received as an HTTP request

[1572] Output: Registration complete message

[1573] Specific operation: The server interprets the data in JSON format and inserts a new record into the corresponding database. If the insertion is successful, it generates a confirmation message and sends it to the terminal as an HTTP response.

[1574] Step 4:

[1575] The user enters the title, author, and specific chapter or page number of the book they are reading into the app and requests an analysis. They can also enter any parts they find difficult to understand, if necessary.

[1576] Input: Book title, author name, specific chapter or page number, difficult-to-understand sections

[1577] Output: Parsing request entered into the terminal

[1578] Specific steps: Enter the book information into the form on the app screen and tap the "Request Analysis" button.

[1579] Step 5:

[1580] The device sends the user's analysis request to the server. During transmission, the necessary information is converted to JSON format and sent to the server as an HTTP request.

[1581] Input: Analysis request entered on the terminal

[1582] Output: HTTP request sent to the server

[1583] Specific operation: When the user taps the "Request Analysis" button, the application on the device converts the input data into JSON format and sends the data to the server via an API call.

[1584] Step 6:

[1585] The server retrieves text data from a database for a specified book and analyzes it using a natural language processing algorithm. A generative AI model (e.g., GPT-3) is then used to generate explanatory information for specific, difficult sections.

[1586] Input: Parsing request received as an HTTP request

[1587] Output: Generated explanatory information

[1588] Specific operation: The server queries the database to retrieve the relevant book data and analyzes the data using a natural language processing algorithm. Based on the analysis results, it generates explanatory information using a generative AI model such as GPT-3.

[1589] Step 7:

[1590] The server sends the generated explanatory information to the terminal. The explanatory information is sent as an HTTP response in JSON format.

[1591] Input: Generated explanatory information

[1592] Output: HTTP response sent to the terminal

[1593] Specific action: The generated explanatory information is sent to the terminal as an HTTP response.

[1594] Step 8:

[1595] The terminal displays explanatory information sent from the server to the user. The information is presented in an interactive format, incorporating graphics and icons to facilitate user understanding.

[1596] Input: Explanatory information received from the server

[1597] Output: Explanatory information displayed to the user

[1598] Specific actions: Display text explanations on the app screen and insert related graphics and icons. Allow users to tap to display detailed information.

[1599] Step 9:

[1600] If the user requests the generation of a character list, this request is entered into the app.

[1601] Input: Request to generate character list

[1602] Output: Request entered into the terminal

[1603] Specific action: Enter "I need a character list" into the app's form and tap the "Submit" button.

[1604] Step 10:

[1605] The terminal sends a request for the character list to the server. During transmission, the input information is converted to JSON format and sent to the server as an HTTP request.

[1606] Input: Request entered on the terminal

[1607] Output: HTTP request sent to the server

[1608] Specific operation: Convert the user's request into JSON format and send it to the server as an HTTP request.

[1609] Step 11:

[1610] The server analyzes and generates a character list from the text data of the specified book. Detailed information about each character (e.g., characteristics, relationships, appearances, etc.) is added.

[1611] Input: Request received as an HTTP request

[1612] Output: Generated character list

[1613] Specific operation: The server retrieves book data from the database and uses a natural language processing algorithm to extract the names of the characters and their detailed information.

[1614] Step 12:

[1615] The server sends a character list it has generated to the terminal. The list is sent as an HTTP response in JSON format.

[1616] Input: Generated character list

[1617] Output: HTTP response sent to the terminal

[1618] Specific action: The generated character list is sent to the terminal as an HTTP response.

[1619] Step 13:

[1620] The device interactively displays a list of characters received from the server. Users can view detailed information by tapping on each character's name.

[1621] Input: Character list received from the server

[1622] Output: List of characters displayed to the user

[1623] Specific operation: The app screen displays a list of characters, and when the user taps a character, detailed information is displayed.

[1624] Step 14:

[1625] The user specifies the parts of the book for which they want to know the source material or background information.

[1626] Input: Specify source material or background information.

[1627] Output: Specified information entered into the terminal

[1628] Specific steps: Enter the information you want to know about the source material or background into the app's form and tap the "Submit" button.

[1629] Step 15:

[1630] The device sends a request for source material and background information to the server. During transmission, the input information is converted to JSON format and sent to the server as an HTTP request.

[1631] Input: Request entered on the terminal

[1632] Output: HTTP request sent to the server

[1633] Specific operation: Convert the user's request into JSON format and send it to the server as an HTTP request.

[1634] Step 16:

[1635] The server collects source material and background information for the specified section from the database, analyzes it, and generates appropriate information.

[1636] Input: Request received as an HTTP request

[1637] Output: Generated source material and background information

[1638] Specific operation: The server queries the database to retrieve relevant source material and background information, and then uses a generative AI model to analyze and generate information.

[1639] Step 17:

[1640] The server sends the source material and background information it generates to the terminal. The information is sent as an HTTP response in JSON format.

[1641] Input: Generated source material and background information

[1642] Output: HTTP response sent to the terminal

[1643] Specific action: The generated information is sent to the terminal as an HTTP response.

[1644] Step 18:

[1645] The terminal provides the user with source material and background information received from the server. The information is displayed with care to avoid excessive spoilers.

[1646] Input: Source material and background information received from the server

[1647] Output: Source material and background information provided to the user

[1648] Specific actions: The app screen displays source material and background information, and interactive elements (e.g., links or taps to display details) are added as needed.

[1649] (Application Example 1)

[1650] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1651] Conventional reading support systems primarily focus on assisting with the reading comprehension of books, and lack sufficient support for factory work procedures and manuals. Therefore, there was a need to resolve the difficulties workers face in understanding these procedures and manuals. This invention aims to solve these problems and provide a system that improves work efficiency and safety in factories.

[1652] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1653] In this invention, the server includes means for receiving and storing registered reading information, means for analyzing text data of the material being read and identifying difficult sections, means for generating and providing explanatory information for the identified difficult sections in response to a user's request, and means for analyzing the explanations and supplementary information of work procedure manuals and providing visual explanations to the user. This enables factory workers to efficiently understand procedure manuals and manuals and perform their work appropriately.

[1654] "Means for receiving and storing registered reading information" refers to a system that has the function of receiving data such as personal information, preferred genres, and reading history provided by users, and storing it on a server.

[1655] "A means of analyzing text data of materials being read and identifying difficult parts" refers to a system that uses natural language processing technology to analyze the content of materials and has the function of identifying parts of the text that are particularly difficult to understand.

[1656] "A means of generating and providing explanatory information on specific difficult parts in response to user requests" refers to a mechanism that has the function of generating detailed explanations of difficult parts in response to questions and requests submitted by users and providing that information to the users.

[1657] "A means of analyzing work procedure manuals and supplementary information to provide users with visual explanations" refers to a system that analyzes work procedure manuals and other manuals used in factories, generates explanations and supplementary information, and provides them to users in a way that they can understand visually.

[1658] "A means of analyzing person information in text data and generating a list of people" refers to a system that has the function of analyzing the names and characteristics of people appearing in a document and compiling them into a list.

[1659] "A means of interactively displaying a generated list of people and providing detailed information about each person" refers to a mechanism that displays the generated list of people in a user-operable format and has the functionality to view detailed information about each person.

[1660] "Means for analyzing source material and background information related to documents and generating that information" refers to a system that analyzes source material and background information related to the content of documents and generates detailed explanations and supplementary information based on that analysis.

[1661] "A means of analyzing voice-inputted work instructions and page numbers of procedure manuals, and generating and providing explanations about the actions" refers to a system that analyzes the page numbers of work procedure manuals and specific actions instructed by the user via voice input, and generates and provides explanations and supplementary information.

[1662] This invention develops an AI assistant system to enhance the reading experience and implements a system to improve work efficiency and safety in factories. This system includes functions such as managing registered information, analyzing procedure manuals, generating and providing explanatory information, and analyzing voice input.

[1663] 1. System Program

[1664] The following describes the specific processing details of each program in the system.

[1665] Hardware to use

[1666] Smart glasses (e.g., Microsoft HoloLens)

[1667] Software to use

[1668] NLP libraries (e.g., spaCy, NLTK)

[1669] AR frameworks (e.g., Unity, Vuforia)

[1670] Backend frameworks (e.g., Node.js, Flask)

[1671] Generative AI models (e.g., OpenAI GPT-4)

[1672] Program Processing Description

[1673] 1. Management of registered information

[1674] The server receives data such as personal information, preferred genres, and reading history provided by users through the application and stores it in a database. Users input this information using devices such as smartphones, tablets, and smart glasses.

[1675] 2. Analysis of Procedure Manuals

[1676] The server uses an NLP library (e.g., spaCy) to analyze the text data of the material the user is reading and identify difficult sections. At this stage, an algorithm is used to evaluate the difficulty of words and phrases within the text.

[1677] 3. Generation and provision of explanatory information

[1678] The server uses a generative AI model (e.g., GPT-4) to generate explanatory information about the identified difficult sections. This generated explanatory information is provided to the user via smart glasses or other devices. For example, in response to a prompt such as, "Please tell me the points to pay attention to in Chapter 3 of the work procedure manual," the server generates a detailed explanation, providing information such as, "In Chapter 3, you should pay particular attention to the section concerning the safe operation of the machine."

[1679] 4. Voice input analysis

[1680] When a user provides voice input through smart glasses, the voice data is converted into text and sent to a server. The server analyzes the voice instructions and page numbers and generates corresponding explanations and supplementary information. For example, in response to a prompt such as "Please explain the instructions for using this machine in detail," detailed instructions on how to operate the machine are provided.

[1681] Specific example

[1682] If a user is having trouble understanding a particular procedure manual while working in a factory, they can voice-input through smart glasses, "Please tell me the important points to pay attention to in Chapter 3 of the work procedure manual." The server analyzes this information, generates an explanation of the parts of Chapter 3 that require special attention, and displays it in augmented reality (AR) in the user's field of view.

[1683] Example of a prompt:

[1684] 1. "Please tell me the key points to pay attention to in Chapter 3 of the work procedure manual."

[1685] 2. "Please explain in detail how to use this machine."

[1686] This system allows users to better understand the complex parts of work procedures and perform tasks efficiently and safely.

[1687] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1688] Step 1:

[1689] Users enter personal information and reading history using an application installed on their device (e.g., smart glasses, smartphone). This information includes name, email address, reading preferences, and reading frequency. The device sends this information to a server. The entered data is stored in a user information database.

[1690] Step 2:

[1691] The user inputs information about the document they are reading or the factory work instructions. Specifically, they input the title of the document, chapter, page number, and any parts they find difficult to understand. The terminal sends this information to the server. The server retrieves the received document information from a text database and analyzes the text data using natural language processing (NLP) technology.

[1692] Step 3:

[1693] The server uses an NLP library (e.g., spaCy) to analyze the text data of the document. The analysis understands grammatical structure and word meanings, and identifies particularly difficult sections. Information about these identified difficult sections is then input as prompts into a generative AI model (e.g., GPT-4). These prompts might include phrases like, "Please tell me the points to pay attention to in Chapter 3 of the work procedure manual."

[1694] Step 4:

[1695] The generative AI model (e.g., GPT-4) generates explanatory information for complex parts based on the prompt text received from the server. The generated explanatory information is written in detail to make it easy for the user to understand. The server sends the generated explanatory information to the terminal. This output data includes specific explanatory text.

[1696] Step 5:

[1697] The device provides the user with explanatory information received from the server. Specifically, the explanatory information is displayed as an augmented reality (AR) display on the smart glasses' screen. Audio guidance is also available. Based on the provided explanatory information, the user can understand the work procedure and perform it appropriately.

[1698] Step 6:

[1699] When a user provides voice input through smart glasses, the device converts the voice data into text data and sends it to the server. For example, a voice command such as "Please explain the instructions for using this machine in detail" might be input. The server analyzes the received voice command and page number text data to generate explanatory information about the operation. This explanatory information is then sent back to the device.

[1700] Step 7:

[1701] The terminal then provides the user with explanatory information about the operation received from the server. This explanatory information is displayed in the user's field of view using augmented reality (AR), and includes audio guidance along with detailed explanations of the operation. This allows the user to accurately understand and safely perform complex tasks.

[1702] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1703] Patent Specification

[1704] This invention combines an AI assistant system with an emotion engine to enhance the reading experience. It analyzes the user's emotional state and provides appropriate feedback and customized information based on that analysis. The program and specific processing of this system are described below.

[1705] 1. Registration and Initial Setup

[1706] User:

[1707] Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[1708] After launching the application, tap the "New Registration" button and enter the required information such as your name, email address, and password.

[1709] Set your preferences, such as genre, difficulty level, and reading frequency.

[1710] Terminal:

[1711] It receives information entered by the user and sends it to the server.

[1712] server:

[1713] The received user information is saved to the database, a registration completion message is generated, and sent to the terminal.

[1714] Terminal:

[1715] Display a registration completion message to the user.

[1716] 2. Initial setup of the emotion engine

[1717] User:

[1718] Enable emotion recognition within the app.

[1719] Terminal:

[1720] It uses a camera and microphone to capture the user's facial expressions and voice.

[1721] server:

[1722] The system analyzes the received emotional data and generates an initial emotional profile.

[1723] 3. Text analysis and information provision

[1724] User:

[1725] Enter the title, author, and specific chapter or page number of the book you are reading into the app. If there are parts you find difficult to understand, enter them as comments.

[1726] Terminal:

[1727] Send the user's input information to the server.

[1728] server:

[1729] Retrieve the text data of the specified book from the database.

[1730] We analyze text data using natural language processing algorithms to identify difficult parts.

[1731] Generate explanatory information for identified difficult sections. Select the appropriate explanation based on whether or not spoilers are included.

[1732] Customize explanatory information based on the user's emotional state.

[1733] Terminal:

[1734] Customized explanatory information received from the server is displayed interactively and provided to the user.

[1735] Specific example:

[1736] If a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. For example, if "The Use of Symbolism" is identified as a difficult section, an explanation is provided regarding its background and intent, such as "Symbolism is used in this section, and certain symbols have a deeper meaning than they actually do." Furthermore, if the user's emotional state is recognized as "confused," additional advice is customized and displayed, such as "Read slowly and think about what the symbols mean."

[1737] 4. Generation and provision of the character list

[1738] User:

[1739] If you can't remember the names of the characters, enter that into the app.

[1740] Terminal:

[1741] Send the request to the server.

[1742] server:

[1743] Analyze and generate a list of characters from the text data of a specified book.

[1744] Add detailed information about each character (e.g., characteristics, relationships, locations of appearance, etc.).

[1745] Customize character descriptions based on the user's emotional state.

[1746] Terminal:

[1747] The system interactively displays a list of characters sent from the server, allowing users to view detailed information through actions such as clicking.

[1748] Specific example:

[1749] If a user requests a character list for "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, knowledgeable and always offering advice." If the user's emotional state is "excited," an additional comment will be displayed, such as, "Pay attention to how this character will influence the story going forward."

[1750] 5. Providing source material and background information.

[1751] User:

[1752] You can specify in the app which parts of a book you want to know about, such as the source material or background information.

[1753] Terminal:

[1754] Send the request to the server.

[1755] server:

[1756] The system collects source material and background information for the specified portion from a database.

[1757] It analyzes collected source material and background information to generate appropriate information.

[1758] Customize information based on the user's emotional state.

[1759] Terminal:

[1760] The system provides users with source material and background information received from the server. Care is taken to avoid excessive spoilers.

[1761] Specific example:

[1762] If a user inputs that they want to know about Chapter 3 of "Book C," the server analyzes the source material and background information for that section and provides information such as, "This chapter is based on a medieval legend and strongly reflects elements of XXX in particular." If the user's emotional state indicates "surprise," additional advice such as, "Try to predict what will happen next based on this information," will be displayed.

[1763] Thus, the present invention is a system that recognizes the user's emotional state and provides various information based on that state, thereby enriching the reading experience and deepening understanding.

[1764] The following describes the processing flow.

[1765] Forms for carrying out the invention - A system combining an emotion engine

[1766] 1. Registration and Initial Setup

[1767] Step 1:

[1768] User: Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[1769] Step 2:

[1770] User: Tap the "Register" button and enter the required information such as your name, email address, and password.

[1771] Step 3:

[1772] User: Sets preferences such as genre, difficulty level, and reading frequency.

[1773] Step 4:

[1774] Terminal: Converts user-entered information into JSON format and sends it to the server using the API.

[1775] Step 5:

[1776] Server: Stores the received user information in the database, generates a registration completion message, and sends it to the terminal.

[1777] Step 6:

[1778] Terminal: Displays a registration completion message to the user.

[1779] 2. Initial setup of the emotion engine

[1780] Step 1:

[1781] User: Enable emotion recognition within the app.

[1782] Step 2:

[1783] Device: Captures the user's facial expressions and voice using the camera and microphone.

[1784] Step 3:

[1785] Server: Analyzes received emotion data and generates an initial emotion profile.

[1786] 3. Text analysis and information provision

[1787] Step 1:

[1788] User: Enter the title, author, and specific chapter or page number of the book you are reading into the app. If there are parts you find difficult to understand, enter them as comments.

[1789] Step 2:

[1790] Terminal: Converts user input information into JSON format and sends it to the server using the API.

[1791] Step 3:

[1792] Server: Retrieves text data for the specified book from the database.

[1793] Step 4:

[1794] Server: Analyzes text data using natural language processing (NLP) algorithms to identify difficult parts.

[1795] Step 5:

[1796] Server: Generates explanatory information for identified difficult sections. Selects appropriate explanations based on whether or not spoilers are included.

[1797] Step 6:

[1798] Server: Analyzes the user's emotional state and customizes the explanatory information.

[1799] Step 7:

[1800] Server: Converts explanatory information into JSON format and sends it to the terminal.

[1801] Step 8:

[1802] Terminal: Interactively displays and provides customized explanatory information received from the server to the user.

[1803] Specific example:

[1804] If a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. For example, if "The Use of Symbolism" is identified as a difficult section, an explanation is provided regarding its background and intent, such as "Symbolism is used in this section, and certain symbols have a deeper meaning than they actually do." If the emotion recognition determines that the user is "confused," advice such as "Try reading the next chapter to find out what this symbol means" is added.

[1805] 4. Generation and provision of the character list

[1806] Step 1:

[1807] User: If you can't remember the names of the characters, enter that into the app.

[1808] Step 2:

[1809] Terminal: Converts the request to JSON format and sends it to the server.

[1810] Step 3:

[1811] Server: Analyzes and generates a list of characters from the text data of the specified book.

[1812] Step 4:

[1813] Server: Adds detailed information about each character (e.g., characteristics, relationships, locations of appearance, etc.).

[1814] Step 5:

[1815] Server: Analyzes the user's emotional state and customizes the character descriptions.

[1816] Step 6:

[1817] Server: Converts the generated character list into JSON format and sends it to the terminal.

[1818] Step 7:

[1819] Terminal: Interactively displays the character list received from the server, allowing the user to view detailed information through actions such as clicking.

[1820] Specific example:

[1821] If a user requests a list of characters from "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, knowledgeable and always offering advice." If emotion recognition determines that the user is "excited," an additional comment will be displayed, such as, "Pay attention to how this character will influence the story going forward."

[1822] 5. Providing source material and background information.

[1823] Step 1:

[1824] User: Specify to the app the parts of a book for which you want to know the source material or background information.

[1825] Step 2:

[1826] Terminal: Converts the request to JSON format and sends it to the server.

[1827] Step 3:

[1828] Server: Collects source material and background information for the specified section from the database.

[1829] Step 4:

[1830] Server: Analyzes collected source material and background information to generate appropriate information.

[1831] Step 5:

[1832] Server: Analyzes the user's emotional state and customizes the information.

[1833] Step 6:

[1834] Server: Converts the generated source material and background information into JSON format and sends it to the terminal.

[1835] Step 7:

[1836] Terminal: Provides users with source material and background information received from the server. Care is taken to avoid excessive spoilers.

[1837] Specific example:

[1838] If a user inputs that they want to know about Chapter 3 of "Book C," the server analyzes the source material and background information for that section and provides information such as, "This chapter is based on a medieval legend and strongly reflects elements of XXX in particular." If the user's emotion recognition detects "surprise," advice such as, "Try to predict what will happen next based on this information," will be displayed.

[1839] In this way, the program can enrich the user's reading experience and deepen their understanding by recognizing the user's emotional state at each processing step and customizing and providing information based on that state.

[1840] (Example 2)

[1841] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1842] Existing reading support systems provide uniform information without considering the user's emotional state, resulting in reading experiences that do not meet individual needs. Furthermore, information about characters and source material is often insufficient, hindering comprehension during reading. This leads to difficulties for users during reading and a lack of information consistency, posing a significant challenge.

[1843] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving and storing information of registered users, means for capturing and analyzing the user's emotional state using a camera and microphone, means for analyzing the text data of the book being read and identifying difficult parts, and means for generating and providing customized explanatory information for the identified difficult parts in accordance with the user's emotional state and requests. This makes it possible to provide personalized information based on the user's emotional state, thereby providing a richer and more symmetrical reading experience.

[1844] "Registered user information" refers to personal information and settings information that users provide when registering with the system, such as their name, email address, password, and preferred genres.

[1845] "User emotional state" refers to the psychological state of a user obtained by analyzing data of the user's facial expressions and voice captured by cameras and microphones.

[1846] "Means of capturing and analyzing using cameras and microphones" refers to means including software and hardware for collecting and analyzing a user's facial expressions and voice using cameras and microphones installed on a device.

[1847] "A means of analyzing text data and identifying difficult parts" refers to a program that uses natural language processing algorithms to analyze the text data of a book and automatically identify parts that users find difficult to understand.

[1848] "Customized explanatory information" refers to individually optimized explanatory information generated based on the user's emotional state and requests for specific, difficult-to-understand parts.

[1849] "A means of generating a character list" refers to a program that analyzes the text data of a book and lists the names of the main characters that appear in the book.

[1850] A "customized character list" is a character list generated based on the user's emotional state, with descriptions that take into account the characteristics and relationships of each character.

[1851] A "means for generating source material and background information" refers to a program that collects additional information, such as source material and historical background, for specific parts of a book, analyzes that information, and provides it.

[1852] "Customized information that avoids excessive spoilers" refers to background information and source material provided with consideration for the user's emotional state, without compromising the entertainment value.

[1853] This invention combines an AI assistant system with an emotion engine to enhance the reading experience, analyzing the user's emotional state and providing appropriate feedback and customized information based on that analysis.

[1854] First, the user downloads the application to their device (smartphone, tablet, etc.), installs it, and then launches it. The user launches the application, taps the "New Registration" button, and enters necessary information such as their name, email address, and password. Next, the user sets their preferred genre, reading difficulty level, and reading frequency. The device then receives the user's input information and sends it to the server. The server saves the information to its database, generates a registration completion message, and sends it to the device. The device then displays this message to the user.

[1855] Next, the user configures the app to enable emotion recognition. The device uses its camera and microphone to capture the user's facial expressions and voice, and sends this data to the server. The server analyzes the emotion data, generates an initial emotion profile using an AI model (e.g., an emotion recognition model), and stores it in a database.

[1856] The user enters the title, author, and specific chapter or page number of the book they are reading into the app, and adds comments on parts they find difficult to understand. The device sends this information to the server. The server retrieves the text data of the specified book from a database, analyzes it using natural language processing algorithms (e.g., BERT, GPT-3), and identifies the difficult parts. The server also generates customized explanatory information, taking into account the user's emotional state, and sends it to the device. The device then interactively displays and provides the customized explanatory information to the user.

[1857] For example, if a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. If the use of symbolism is identified as a difficult part, an explanation is provided regarding its background and intent, such as, "Symbolism is used in this section, and certain symbols have a deeper meaning than they actually do." Additionally, if the user's emotional state is recognized as "confused," extra advice is displayed, such as, "Read slowly and think about what the symbols mean."

[1858] Furthermore, if the user cannot remember the names of the characters, they can input this into the app. The device sends this request to the server. The server parses the text data and generates a list of characters. The list includes detailed information about each character (characteristics, relationships, appearances, etc.) and is customized based on their emotional state. The customized list is sent to the device, allowing the user to interactively view the detailed information.

[1859] For example, if a user requests a character list for "Book B," the server will provide detailed information such as "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, knowledgeable and always offering advice." If the user's emotional state is "excited," an additional comment will be displayed such as, "Pay attention to how this character will influence the story going forward."

[1860] Finally, the user specifies to the app which parts of the book they want to know about, such as the source material and background information. The device sends the request to the server. The server retrieves the source material and background information from its database, analyzes it, and provides the generated information. Customized information is provided to the user while avoiding excessive spoilers.

[1861] For example, if a user inputs that they want to know about Chapter 3 of "Book C," the server analyzes the source material and background information for that section and provides information such as, "This chapter is based on a medieval legend and strongly reflects elements of XXX in particular." If the user's emotional state indicates "surprise," additional advice such as, "Try to predict what happens next based on this information," will be displayed.

[1862] Example of a prompt:

[1863] Could you please explain the meaning of "Muggle" in Chapter 1 of the book "Harry Potter and the Sorcerer's Stone"?

[1864] Please display the list of characters from the book "The Lord of the Rings".

[1865] Thus, the present invention is a system that recognizes the emotional state of the user and provides various information based on that state, thereby enriching the reading experience and deepening understanding.

[1866] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1867] Step 1: Install and launch the application

[1868] User:

[1869] The user downloads and installs the application on their device (such as a smartphone or tablet).

[1870] Then, launch the application.

[1871] Terminal:

[1872] The device launches the installed application and displays the initial screen.

[1873] input:

[1874] Downloaded applications

[1875] output:

[1876] Displaying the application's initial screen.

[1877] Step 2: New Registration

[1878] User:

[1879] After launching the application, the user taps the "Register" button and enters the required information such as their name, email address, and password.

[1880] Terminal:

[1881] The terminal temporarily saves the entered information and displays a button to proceed to the next step.

[1882] input:

[1883] User information such as name, email address, and password

[1884] output:

[1885] A button to proceed to the next step is displayed.

[1886] Step 3: Customize your settings

[1887] User:

[1888] Users can set their preferred genre, reading difficulty level, and reading frequency.

[1889] Terminal:

[1890] The device temporarily stores the configured information and displays a confirmation screen to send all the information to the server.

[1891] input:

[1892] Settings such as preferred genre, difficulty level, and reading frequency.

[1893] output:

[1894] Confirmation screen displayed

[1895] Step 4: Sending and saving information

[1896] Terminal:

[1897] The terminal sends the information entered and configured by the user to the server.

[1898] server:

[1899] The server stores the received information in a database.

[1900] Generate a registration completion message and send it to the device.

[1901] input:

[1902] Sending user information and settings information

[1903] output:

[1904] Storing information in a database

[1905] Generate and send registration completion message

[1906] Step 5: Display of registration completion message

[1907] Terminal:

[1908] The device displays a registration completion message to the user.

[1909] input:

[1910] Registration complete message received

[1911] output:

[1912] Display of registration completion message

[1913] Step 6: Setting up emotion recognition function

[1914] User:

[1915] Users configure settings within the app to enable the emotion recognition feature.

[1916] Terminal:

[1917] The device requests permission for the camera and microphone and verifies that the settings are enabled.

[1918] input:

[1919] Emotion recognition function settings

[1920] output:

[1921] Checking camera and microphone usage permissions

[1922] Step 7: Capture and send emotional data

[1923] User:

[1924] The user faces the device's camera and microphone, providing standard facial expressions and voice.

[1925] Terminal:

[1926] The device uses its camera and microphone to capture the user's facial expressions and voice, and temporarily stores this as emotion data.

[1927] The captured emotion data is sent to the server.

[1928] input:

[1929] User's facial expressions and voice

[1930] output:

[1931] Capture and transmit emotional data

[1932] Step 8: Generating an emotional profile

[1933] server:

[1934] The server analyzes the received emotional data using an AI model and generates an initial emotional profile.

[1935] The generated emotion profile is saved to the database.

[1936] input:

[1937] Sent emotional data

[1938] output:

[1939] Generating and saving initial emotion profiles

[1940] Step 9: Enter reading information

[1941] User:

[1942] Users enter the title, author, and specific chapter or page number of the book they are reading into the app. They can also add comments to sections they find difficult to understand.

[1943] Terminal:

[1944] The terminal temporarily stores the entered information and sends it to the server.

[1945] input:

[1946] Book title, author name, specific chapter or page number, comments

[1947] output:

[1948] Sending information to the server

[1949] Step 10: Acquisition and analysis of text data

[1950] server:

[1951] The server retrieves the text data of the specified book from the database.

[1952] We analyze text data using natural language processing algorithms to identify difficult parts.

[1953] input:

[1954] Book title, author name, specific chapter or page number

[1955] output:

[1956] Identifying the difficult parts

[1957] Step 11: Generating customized explanatory information

[1958] server:

[1959] The server generates explanatory information for the identified difficult parts.

[1960] It generates customized explanatory information that takes into account the user's emotional state.

[1961] input:

[1962] User emotion profiles, text analysis data

[1963] output:

[1964] Generating customized explanatory information

[1965] Step 12: Send and display customized explanatory information

[1966] server:

[1967] The server sends the generated customized explanation information to the terminal.

[1968] Terminal:

[1969] The device interactively displays and provides customized explanatory information to the user.

[1970] input:

[1971] Customized explanatory information

[1972] output:

[1973] Display of explanatory information

[1974] Step 13: Requesting and generating the character list

[1975] User:

[1976] If a user cannot remember the names of the characters, they can enter that information into the app.

[1977] Terminal:

[1978] The terminal sends the request to the server.

[1979] server:

[1980] The server generates a list of characters from the text data of the book.

[1981] input:

[1982] Request for character list

[1983] output:

[1984] Generating a list of characters

[1985] Step 14: Provide a customized character list

[1986] server:

[1987] The server adds detailed information about each character and customizes the list based on the user's emotional state.

[1988] Terminal:

[1989] The terminal interactively displays a list of characters sent from the server, and the user can view detailed information by clicking or performing other actions.

[1990] input:

[1991] Character List

[1992] output:

[1993] Providing a customized character list

[1994] Step 15: Requesting and providing source material and background information.

[1995] User:

[1996] Users specify to the app which parts of a book they want to know about, such as the source material or background information.

[1997] Terminal:

[1998] The terminal sends the request to the server.

[1999] server:

[2000] The server collects source material and background information for the specified section from the database and generates appropriate information.

[2001] input:

[2002] Request for source material and background information

[2003] output:

[2004] Generated source material and background information

[2005] Terminal:

[2006] The system interactively displays source material and background information received from the server and provides it to the user. Filtering is performed to avoid excessive spoilers.

[2007] input:

[2008] Source material and background information

[2009] output:

[2010] Providing customized background information

[2011] (Application Example 2)

[2012] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[2013] Traditional reading-related information systems often provided uniform information without considering the user's emotional state. As a result, support for individual reading experiences was insufficient, and customization tailored to the user's specific emotional state was needed, particularly in understanding difficult passages, grasping characters, and providing source material and background information. Furthermore, a system capable of providing real-time feedback on the user's interests and questions during reading was required.

[2014] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[2015] In this invention, the server includes means for receiving and storing user information related to registered reading; means for analyzing text data of the book being read and identifying difficult sections; means for generating and providing explanatory information for the identified difficult sections in response to user requests; means for analyzing the user's emotional state using a camera and microphone; and means for customizing the explanatory information based on the analyzed emotional state. This enables personalized feedback and explanations based on the user's emotional state, making it possible to provide information optimized for each user.

[2016] "Registered user information regarding reading" refers to personal information provided by the user regarding their reading, such as their name, preferred genres, reading frequency, and difficulty level.

[2017] "Book text data" refers to data that represents the content of a book a user is reading in digital format.

[2018] "Difficult sections" refer to parts of a book that are identified as being hard to understand.

[2019] "Explanatory information" refers to explanations and supplementary information provided to make difficult parts easier for users to understand.

[2020] "Means of analyzing a user's emotional state using cameras and microphones" refers to devices and software that capture a user's facial expressions and voice and identify their emotions from them.

[2021] "Means for customizing explanatory information based on analyzed emotional states" refers to methods and software for adjusting and individually adapting explanatory information based on the user's emotional state.

[2022] "Character information" refers to data such as the names, characteristics, and relationships of all characters that appear in the book.

[2023] "Means for generating a character list" refers to a method and software for extracting character names and related information from text data and listing them.

[2024] "Character details" refers to more detailed explanations and background information about the characters.

[2025] "Source material and background information" refers to additional information such as the source, references, and historical context of specific parts of a book.

[2026] "Means for providing users with generated source material and background information" refers to methods and software for displaying or communicating analyzed source material and background information to users.

[2027] This invention utilizes a user terminal, a server, and a network environment connecting them. Specifically, the invention can be implemented in the following forms.

[2028] The user downloads and installs the "Smart Book Reader" app on their smartphone. Upon launching the app, the user registration screen is displayed first. The user enters the necessary information, such as their name, email address, and password, and the device sends this information to the server. The server stores the received information in its database and sends a registration completion message to the device, which then displays the message to the user.

[2029] Next, an option to enable emotion recognition within the app will appear. If the user enables it, the device will use the camera and microphone to capture the user's facial expressions and voice in real time. This data is sent to a server, which analyzes it using an emotion engine to generate an initial emotion profile.

[2030] When a user begins reading, they enter the title, author, and specific chapter or page number of the book they are reading into the app. They can also add comments if they find certain sections difficult to understand. This information is sent from the device to a server, which retrieves the book's text data from a database and analyzes it using natural language processing algorithms. The server then identifies difficult sections and generates explanatory information. Furthermore, the explanatory information is customized based on the user's emotional state. This explanatory information is sent to the device and displayed interactively to the user.

[2031] For example, if a user is reading Chapter 5 of "Book A" and is confused about the "use of symbolism," the server will analyze that section and provide an explanation such as, "Symbolism is used in this section, and certain symbols have a deeper meaning than they actually do." Furthermore, if the server recognizes that the user's emotional state is "confused," it will display customized additional advice such as, "Read slowly and think about what the symbols mean."

[2032] In generating the character list, when a user enters a character's name they can't remember into the app, the device sends a request to the server. The server analyzes the text data from the book and generates the character list. It then adds detailed information about each character and customizes it according to the user's emotional state. For example, if the user is "excited," an additional comment such as "Pay attention to how this character will influence the story going forward" will be displayed.

[2033] Regarding source material and background information, when a user inputs the part they want to know about, the server analyzes the source material and background information for the specified part and generates appropriate information. For example, if a user wants to know about Chapter 3 of "Book C," the server will provide information such as, "This chapter is based on a medieval legend and strongly reflects elements of XXX in particular." If the user indicates "surprise," additional advice such as, "Try to predict what will happen next based on this information," will be displayed.

[2034] The implementation of this system involves using smartphones as user terminals, servers for analysis and data storage, and software such as TensorFlow and OpenCV for emotion recognition, spaCy for natural language processing, and Flask for interface construction.

[2035] An example of a prompt sentence would be, "I would like to understand the concept of symbolism in Chapter 5 of Book A. I am confused."

[2036] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[2037] Step 1:

[2038] The user downloads and installs the "Smart Book Reader" app on their smartphone.

[2039] Input: An app installed on a smartphone.

[2040] Output: The app is launched.

[2041] Specific operation: After the user downloads the app from the Google Play Store or Apple App Store and the installation is complete, they tap the icon on the home screen to launch the app.

[2042] Step 2:

[2043] The user enters the required information on the app's new registration screen and registers.

[2044] Input: Name, email address, password, preferred genre, difficulty level, reading frequency.

[2045] Output: Registration information is sent to the server and stored in the database.

[2046] Specific operation: The user enters the required information on the new registration screen and taps the "Register" button. The device sends the entered information to the server, which receives it and saves it to the database. Once saving is complete, the server sends a registration completion message back to the device, which then displays it to the user.

[2047] Step 3:

[2048] The user enables the emotion recognition feature within the app.

[2049] Input: Enabling / enabling the emotion recognition function.

[2050] Output: The device uses its camera and microphone to capture the user's facial expressions and voice, and sends them to the server.

[2051] Specific operation: When the user enables the emotion recognition feature in the app's settings menu, the device activates the camera and microphone, capturing the user's facial expressions and voice and sending them to the server in real time.

[2052] Step 4:

[2053] The server analyzes the received emotion data and generates an initial emotion profile.

[2054] Input: User's facial expression data and voice data.

[2055] Output: Initial emotion profile.

[2056] Specific operation: The server uses TensorFlow and OpenCV to analyze facial expression data and voice data, generates an initial emotion profile of the user based on that analysis, and saves it to a database.

[2057] Step 5:

[2058] The user enters the title of the book they are reading, the author's name, a specific chapter or page number, and any parts they find difficult to understand into the app.

[2059] Input: Book title, author name, chapter and page numbers, and comments.

[2060] Output: This information is sent to the server.

[2061] Specific operation: The user enters the required information on an input screen within the app and taps the "Submit" button. The device then sends this information to the server.

[2062] Step 6:

[2063] The server retrieves the text data of the specified book from the database and analyzes it using a natural language processing algorithm.

[2064] Input: Book title, chapter or page number.

[2065] Output: Identification of difficult sections and explanatory information for those sections.

[2066] Specific operation: The server retrieves text data of a specified book from the database and analyzes it using spaCy. Based on the analysis results, it identifies difficult sections and generates appropriate explanatory information.

[2067] Step 7:

[2068] The server customizes the explanatory information based on the user's emotional state.

[2069] Input: User's emotion profile, generated explanatory information.

[2070] Output: Customized explanatory information.

[2071] Specific operation: The server references the user's emotional profile and customizes the generated explanatory information each time. For example, for a confused user, it adds more detailed explanations and words of encouragement.

[2072] Step 8:

[2073] Customized explanatory information is sent from the server to the terminal and provided to the user.

[2074] Input: Customized explanatory information.

[2075] Output: Interactively displayed explanatory information.

[2076] Specific operation: The server sends customized explanatory information to the device, which then displays it interactively. The user can tap on the information to view more details.

[2077] An example of a prompt sentence would be, "I would like to understand the concept of symbolism in Chapter 5 of Book A. I am confused."

[2078] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[2079] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include those described above. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions shown by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2080] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[2081] [Fourth Embodiment]

[2082] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[2083] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2084] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2085] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[2086] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[2087] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[2088] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[2089] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[2090] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[2091] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2092] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2093] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[2094] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2095] Patent Specification

[2096] This invention is an AI assistant system for improving the reading experience, which analyzes and provides various information about the book the user is reading. The program of this system and its specific processing are described below.

[2097] 1. Registration and Initial Setup

[2098] User:

[2099] Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[2100] After launching the app, tap the "New Registration" button, enter the required information such as your name, email address, and password, and set your preferred genre, difficulty level, reading frequency, etc.

[2101] Terminal:

[2102] It receives information entered by the user and sends it to the server.

[2103] server:

[2104] The received user information is saved to the database, a registration completion message is generated, and sent to the terminal.

[2105] 2. Text analysis and information provision

[2106] User:

[2107] Enter the title, author, and specific chapter or page number of the book you are reading into the app. If necessary, specify the parts you are having difficulty understanding.

[2108] Terminal:

[2109] The information entered by the user is sent to the server.

[2110] server:

[2111] The text data of a specified book is retrieved from a database and analyzed using a natural language processing algorithm. Particularly difficult sections are identified, and explanatory information is generated.

[2112] The system checks for spoilers, generates appropriate explanatory information, and sends it to the device.

[2113] Terminal:

[2114] The server sends explanatory information to the user. Graphics and icons are also displayed in an interactive format.

[2115] Specific example:

[2116] If a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. For example, if "The Use of Symbolism" is identified as a difficult section, an explanation of its background and intent will be provided, such as "Symbolism is used in this section, and certain symbols have a deeper meaning than they do in reality."

[2117] 3. Generation and provision of the character list

[2118] User:

[2119] If you can't remember the names of the characters, enter that into the app.

[2120] Terminal:

[2121] Send the request to the server.

[2122] server:

[2123] This program analyzes and generates a character list from the text data of a specified book. It then adds detailed information about each character (e.g., characteristics, relationships, appearances, etc.).

[2124] Terminal:

[2125] The system interactively displays a list of characters sent from the server, allowing users to view detailed information through actions such as clicking.

[2126] Specific example:

[2127] If a user requests a list of characters from "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, who is knowledgeable and always offers advice."

[2128] 4. Providing source material and background information

[2129] User:

[2130] Specify the parts of the book for which you want to know the source material or background information.

[2131] Terminal:

[2132] Send the request to the server.

[2133] server:

[2134] The system collects source material and background information for the specified portion from a database, analyzes it, and generates appropriate information.

[2135] Terminal:

[2136] The server provides users with source material and background information. Care is taken to avoid excessive spoilers.

[2137] Specific example:

[2138] If a user inputs a request to learn about Chapter 3 of "Book C," the server analyzes the source material and background information for that section and provides information such as, "This chapter is based on medieval legends and strongly reflects elements of XXX in particular."

[2139] Thus, the present invention specifically realizes a system that provides information to enrich the user's reading experience and deepen their understanding.

[2140] The following describes the processing flow.

[2141] Program processing flow

[2142] 1. Registration and Initial Setup

[2143] Step 1:

[2144] User: Download the application to your device (smartphone, tablet, etc.), install it, and then launch it.

[2145] Step 2:

[2146] User: Tap the "Register" button and enter the required information such as your name, email address, and password.

[2147] Step 3:

[2148] User: Sets preferences such as genre, difficulty level, and reading frequency.

[2149] Step 4:

[2150] Terminal: Converts user-entered information into JSON format and sends it to the server using the API.

[2151] Step 5:

[2152] Server: Stores received user information in the database.

[2153] Step 6:

[2154] Server: Generates a registration completion message and sends it to the terminal.

[2155] Step 7:

[2156] Terminal: Displays a registration completion message to the user.

[2157] 2. Text analysis and information provision

[2158] Step 1:

[2159] User: Enter the title, author, and specific chapter or page number of the book you are reading into the app. If there are parts you find difficult to understand, enter them as comments.

[2160] Step 2:

[2161] Terminal: Converts user input information into JSON format and sends it to the server using the API.

[2162] Step 3:

[2163] Server: Retrieves text data for the specified book from the database.

[2164] Step 4:

[2165] Server: Analyzes text data using natural language processing (NLP) algorithms to identify difficult parts.

[2166] Step 5:

[2167] Server: Generates explanatory information for identified difficult sections. Selects appropriate explanations based on whether or not spoilers are included.

[2168] Step 6:

[2169] Server: Converts explanatory information into JSON format and sends it to the terminal.

[2170] Step 7:

[2171] Terminal: Interactively displays and provides explanatory information received from the server to the user.

[2172] 3. Generation and provision of the character list

[2173] Step 1:

[2174] User: If you can't remember the names of the characters, enter that into the app.

[2175] Step 2:

[2176] Terminal: Converts user requests into JSON format and sends them to the server.

[2177] Step 3:

[2178] Server: Analyzes and generates a list of characters from the text data of the specified book.

[2179] Step 4:

[2180] Server: Adds detailed information about each character (e.g., characteristics, relationships, locations of appearance, etc.).

[2181] Step 5:

[2182] Server: Converts the generated character list into JSON format and sends it to the terminal.

[2183] Step 6:

[2184] Terminal: Interactively displays the character list received from the server, allowing users to view detailed information through actions such as clicking.

[2185] 4. Providing source material and background information

[2186] Step 1:

[2187] User: Specify to the app the parts of a book for which you want to know the source material or background information.

[2188] Step 2:

[2189] Terminal: Converts user requests into JSON format and sends them to the server.

[2190] Step 3:

[2191] Server: Collects source material and background information for the specified section from the database.

[2192] Step 4:

[2193] Server: Analyzes collected source material and background information to generate appropriate information.

[2194] Step 5:

[2195] Server: Converts the generated source material and background information into JSON format and sends it to the terminal.

[2196] Step 6:

[2197] Terminal: Provides users with source material and background information received from the server. Care is taken to avoid excessive spoilers.

[2198] In this way, the program enriches the user's reading experience and provides information to deepen their understanding through each processing step.

[2199] (Example 1)

[2200] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2201] Traditional reading systems have a problem in that they cannot provide users with appropriate explanations or background information for difficult or hard-to-understand parts of a book. Furthermore, it is difficult for users to remember character names and detailed information, leading to frequent interruptions in the reading experience. In addition, the source material and background information are not provided in an appropriate format, making it difficult to achieve a deep understanding. There is a need for a system that solves these problems and improves the reading experience.

[2202] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[2203] In this invention, the server includes means for receiving and storing user information related to registered readings, means for analyzing text data of the medium being read and identifying difficult sections, and means for generating and providing explanatory information for the identified difficult sections in response to user requests, using a generative artificial intelligence model. This makes it possible to immediately provide appropriate explanatory information for difficult sections that the user encounters while reading.

[2204] The server further includes means for analyzing information about characters appearing in the reading medium and generating a character list, and means for interactively displaying the generated character list and providing detailed information about each character. This allows users to easily check detailed information about characters, dramatically improving the reading experience.

[2205] Furthermore, the server includes means for analyzing source material and background information related to the medium being read and generating that information, and means for providing the generated source material and background information to the user. This allows the user to gain a deeper understanding of the background and source material of the medium being read, thereby enriching the reading experience.

[2206] "Registered user information related to reading" refers to personal information entered by the user into the system, such as name, email address, password, preferred genres, difficulty level, and reading frequency.

[2207] "Text data in a medium" refers to electronic text data such as books, magazines, and academic papers that a user is reading.

[2208] "Analysis" refers to the process of analyzing the content of text data using natural language processing algorithms and other methods to extract specific patterns or meanings.

[2209] "Difficult parts" refer to sections of text that users find hard to understand.

[2210] "Generative artificial intelligence models" refer to machine learning models and deep learning models used for natural language processing and data analysis. Specifically, this includes advanced text generation models such as GPT-3.

[2211] "Explanatory information" refers to additional explanations or background information for parts that users found difficult to understand.

[2212] A "character list" refers to a list of characters that appear in the medium being read. This list includes each character's name, role, and characteristics.

[2213] "Source material and background information" refers to information about the historical and cultural background and original sources that influenced the content of the medium being read.

[2214] "Interactive display" refers to a display method that allows users to view additional details or explanations by performing actions such as tapping or clicking.

[2215] An "HTTP request" refers to a form of data communication based on a protocol used when sending data from a terminal to a server.

[2216] This invention is an AI assistant system designed to enhance the user's reading experience. It utilizes natural language processing and generative AI models to analyze and provide various information about the medium. The system analyzes the text the user is reading in detail and interactively provides lists of difficult passages and characters, as well as source material and background information.

[2217] Registration and initial setup

[2218] User: Download and install the application on your smartphone or tablet. After installation, launch the application and tap the "New Registration" button. Complete the initial setup by entering your name, email address, password, and preferred genres, difficulty level, reading frequency, etc.

[2219] Terminal: Receives information entered by the user and sends it to the server. Specifically, it converts the user's information into JSON format and sends it to the server as an HTTP request.

[2220] Server: Saves received user information to the database. The database used is an RDBMS (Relational Database Management System) such as MySQL or PostgreSQL. After saving is complete, a registration completion message is generated and sent to the terminal.

[2221] Text analysis and information provision

[2222] User: Enter the title, author, and specific chapter or page number of the book you are reading into the app. If necessary, specify the parts you are having difficulty understanding.

[2223] Terminal: Sends user-entered information to the server. During transmission, text data is converted to JSON format and sent via the API.

[2224] Server: Retrieves text data from the specified medium from the database. Next, it uses a natural language processing algorithm to analyze the data and identify the difficult parts pointed out by the user. It then generates explanatory information for the difficult parts using a generation AI model (e.g., GPT-3). During this process, it checks for spoilers and generates explanatory information within an appropriate scope.

[2225] For example, if a user inputs that they want to understand Chapter 5 of "Book A," the server retrieves and analyzes the text data for that section. If, for instance, "The Use of Symbolism" is identified as a difficult section, the server provides an explanation of its background and intent, such as, "Symbolism is used in this section, and certain symbols have a deeper meaning than they do in reality."

[2226] Terminal: Displays explanatory information sent from the server to the user. The explanatory information is displayed in an interactive format, including graphics and icons.

[2227] Generating and providing a list of characters

[2228] User: If you can't remember the names of the characters, enter that into the app.

[2229] Terminal: Sends requests to the server. Specifically, it converts the input request into JSON format and makes an API call.

[2230] Server: Analyzes and generates a character list from the text data of the specified medium. Adds detailed information about each character (e.g., characteristics, relationships, appearances, etc.). Uses natural language processing algorithms for analysis and retrieves necessary information from the database.

[2231] For example, if a user requests a list of characters from "Book B," the server will analyze the list and provide detailed information such as, "Character 1: The protagonist, who embarks on a great adventure at a young age. Character 2: The protagonist's friend, who is knowledgeable and always offers advice."

[2232] Terminal: Interactively displays the character list sent from the server. Users can view detailed information by tapping on names in the list.

[2233] Providing source material and background information

[2234] User: Specify the parts of the book for which you want to know the source material or background information.

[2235] For example, if a user inputs that they want to know about Chapter 3 of "Book C," the server will analyze the source material and background information for that section and provide information such as, "This chapter is based on medieval legends and particularly strongly reflects elements of XXX."

[2236] Terminal: Sends the request to the server.

[2237] Server: Collects source material and background information for the specified section from the database, analyzes it, and generates appropriate information.

[2238] Terminal: Provides users with source material and background information sent from the server. Displays information while taking care to avoid excessive spoilers.

[2239] Example of a prompt

[2240] Prompt 1: "I am reading Chapter 5 of Book A, but I don't understand the use of symbolism. Please provide a detailed explanation."

[2241] Prompt 2: "Please provide a list of characters in 'Book B' and their respective characteristics."

[2242] Prompt 3: "What source material or background information is found in Chapter 3 of 'Book C'?"

[2243] As described above, the present invention specifically realizes the provision of information that enriches the user's reading experience and deepens their understanding.

[2244] The flow of the specific processing in Example 1 will be explained using Figure 11.

[2245] Step 1:

[2246] The user downloads and installs the application on their smartphone or tablet. This provides the basic platform for accessing the system. Next, they launch the application and enter their name, email address, password, preferred genres, etc., on the new user registration screen. This collects the user's basic information.

[2247] Input: Username, email address, password, preferred genre, etc.

[2248] Output: User information entered into the terminal

[2249] Specific steps: The user searches for the app on the App Store or Google Play and taps the download button. After the download is complete, the installation starts automatically, and once finished, an icon appears on the device's home screen. After launching the app, the user enters the required information on the registration screen that appears.

[2250] Step 2:

[2251] The terminal receives the information entered by the user and sends it to the server. The user's information is converted to JSON format and sent to the server as an HTTP request.

[2252] Input: User information entered on the terminal

[2253] Output: HTTP request sent to the server

[2254] Specific operation: When the user taps the "Send" button, the application on the device converts the input data into JSON format and sends the data to the server via an API call.

[2255] Step 3:

[2256] The server saves the received user information to a database. An RDBMS such as MySQL or PostgreSQL is used for the database. After saving is complete, a registration completion message is generated and sent to the terminal.

[2257] Input: User information received as an HTTP request

[2258] Output: Registration complete message

[2259] Specific operation: The server interprets the data in JSON format and inserts a new record into the corresponding database. If the insertion is successful, it generates a confirmation message and sends it to the terminal as an HTTP response.

[2260] Step 4:

[2261] The user enters the title, author, and specific chapter or page number of the book they are reading into the app and requests an analysis. They can also enter any parts they find difficult to understand, if necessary.

[2262] Input: Book title, author name, specific chapter or page number, difficult-to-understand sections

[2263] Output: Parsing request entered into the terminal

[2264] Specific steps: Enter the book information into the form on the app screen and tap the "Request Analysis" button.

[2265] Step 5:

[2266] The device sends the user's analysis request to the server. During transmission, the necessary information is converted to JSON format and sent to the server as an HTTP request.

[2267] Input: Analysis request entered on the terminal

[2268] Output: HTTP request sent to the server

[2269] Specific operation: When the user taps the "Request Analysis" button, the application on the device converts the input data into JSON format and sends the data to the server via an API call.

[2270] Step 6:

[2271] The server retrieves text data from a database for a specified book and analyzes it using a natural language processing algorithm. A generative AI model (e.g., GPT-3) is then used to generate explanatory information for specific, difficult sections.

[2272] Input: Parsing request received as an HTTP request

[2273] Output: Generated explanatory information

[2274] Specific operation: The server queries the database to retrieve the relevant book data and analyzes the data using a natural language processing algorithm. Based on the analysis results, it generates explanatory information using a generative AI model such as GPT-3.

[2275] Step 7:

[2276] The server sends the generated explanatory information to the terminal. The explanatory information is sent as an HTTP response in JSON format.

[2277] Input: Generated explanatory information

[2278] Output: HTTP response sent to the terminal

[2279] Specific action: The generated explanatory information is sent to the terminal as an HTTP response.

[2280] Step 8:

[2281] The terminal displays explanatory information sent from the server to the user. The information is presented in an interactive format, incorporating graphics and icons to facilitate user understanding.

[2282] Input: Explanatory information received from the server

[2283] Output: Explanatory information displayed to the user

[2284] Specific actions: Display text explanations on the app screen and insert related graphics and icons. Allow users to tap to display detailed information.

[2285] Step 9:

[2286] If the user requests the generation of a character list, this request is entered into the app.

[2287] Input: Request to generate character list

[2288] Output: Request entered into the terminal

[2289] Specific action: Enter "I need a character list" into the app's form and tap the "Submit" button.

[2290] Step 10:

[2291] The terminal sends a request for the character list to the server. During transmission, the input information is converted to JSON format and sent to the server as an HTTP request.

[2292] Input: Request entered on the terminal

[2293] Output: HTTP request sent to the server

[2294] Specific operation: Convert the user's request into JSON format and send it to the server as an HTTP request.

[2295] Step 11:

[2296] The server analyzes and generates a character list from the text data of the specified book. Detailed information about each character (e.g., characteristics, relationships, appearances, etc.) is added.

[2297] Input: Request received as an HTTP request

[2298] Output: Generated character list

[2299] Specific operation: The server retrieves book data from the database and uses a natural language processing algorithm to extract the names of the characters and their detailed information.

[2300] Step 12:

[2301] The server sends a character list it has generated to the terminal. The list is sent as an HTTP response in JSON format.

[2302] Input: Generated character list

[2303] Output: HTTP response sent to the terminal

[2304] Specific action: The generated character list is sent to the terminal as an HTTP response.

[2305] Step 13:

[2306] The device interactively displays a list of characters received from the server. Users can view detailed information by tapping on each character's name.

[2307] Input: Character list received from the server

[2308] Output: List of characters displayed to the user

[2309] Specific operation: The app screen displays a list of characters, and when the user taps a character, detailed information is displayed.

[2310] Step 14:

[2311] The user specifies the parts of the book for which they want to know the source material or background information.

[2312] Input: Specify source material or background information.

[2313] Output: Specified information entered into the terminal

[2314] Specific steps: Enter the information you want to know about the source material or background into the app's form and tap the "Submit" button.

[2315] Step 15:

[2316] The device sends a request for source material and background information to the server. During transmission, the input information is converted to JSON format and sent to the server as an HTTP request.

[2317] Input: Request entered on the terminal

[2318] Output: HTTP request sent to the server

[2319] Specific operation: Convert the user's request into JSON format and send it to the server as an HTTP request.

[2320] Step 16:

[2321] The server collects source material and background information for the specified section from the database, analyzes it, and generates appropriate information.

[2322] Input: Request received as an HTTP request

[2323] Output: Generated source material and background information

[2324] Specific operation: The server queries the database to retrieve relevant source material and background information, and then uses a generative AI model to analyze and generate information.

[2325] Step 17:

[2326] The server sends the source material and background information it generates to the terminal. The information is sent as an HTTP response in JSON format.

[2327] Input: Generated source material and background information

[2328] Output: HTTP response sent to the terminal

[2329] Specific action: The generated information is sent to the terminal as an HTTP response.

[2330] Step 18:

[2331] The terminal provides the user with source material and background information received from the server. The information is displayed with care to avoid excessive spoilers.

[2332] Input: Source material and background information received from the server

[2333] Output: Source material and background information provided to the user

[2334] Specific actions: The app screen displays source material and background information, and interactive elements (e.g., links or taps to display details) are added as needed.

[2335] (Application Example 1)

[2336] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[2337] Conventional reading support systems primarily focus on assisting with the reading comprehension of books, and lack sufficient support for factory work procedures and manuals. Therefore, there was a need to resolve the difficulties workers face in understanding these procedures and manuals. This invention aims to solve these problems and provide a system that improves work efficiency and safety in factories.

[2338] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[2339] In this invention, the server includes means for receiving and storing registered reading information, means for analyzing text data of the material being read and identifying difficult sections, means for generating and providing explanatory information for the identified difficult sections in response to a user's request, and means for analyzing the explanations and supplementary information of work procedure manuals and providing visual explanations to the user. This enables factory workers to efficiently understand procedure manuals and manuals and perform their work appropriately.

[2340] "Means for receiving and storing registered reading information" refers to a system that has the function of receiving data such as personal information, preferred genres, and reading history provided by users, and storing it on a server.

[2341] "A means of analyzing text data of materials being read and identifying difficult parts" refers to a system that uses natural language processing technology to analyze the content of materials and has the function of identifying parts of the text that are particularly difficult to understand.

[2342] "A means of generating and providing explanatory information on specific difficult parts in response to user requests" refers to a mechanism that has the function of generating detailed explanations of difficult parts in response to questions and requests submitted by users and providing that information to the users.

[2343] "A means of analyzing work procedure manuals and supplementary information to provide users with visual explanations" refers to a system that analyzes work procedure manuals and other manuals used in factories, generates explanations and supplementary information, and provides them to users in a way that they can understand visually.

[2344] "A means of analyzing person information in text data and generating a list of people" refers to a system that has the function of analyzing the names and characteristics of people appearing in a document and compiling them into a list.

[2345] "A means of interactively displaying a generated list of people and providing detailed information about each person" refers to a mechanism that displays the generated list of people in a user-operable format and has the functionality to view detailed information about each person.

[2346] "Means for analyzing source material and background information related to documents and generating that information" refers to a system that analyzes source material and background information related to the content of documents and generates detailed explanations and supplementary information based on that analysis.

[2347] "A means of analyzing voice-inputted work instructions and page numbers of procedure manuals, and generating and providing explanations about the actions" refers to a system that analyzes the page numbers of work procedure manuals and specific actions instructed by the user via voice input, and generates and provides explanations and supplementary information.

[2348] This invention d...

Claims

1. A means of receiving and storing user information regarding registered readings, A method for analyzing text data from a book being read to identify difficult sections, A means of generating and providing explanatory information for identified difficult parts in response to user requests, A system that includes this.

2. A means of analyzing information about the characters and generating a character list, A means to interactively display the generated character list and provide detailed information about each character, The system according to claim 1, further comprising:

3. A means of analyzing the source material and background information related to a book being read, and generating that information, A means of providing users with the generated source material and background information, The system according to claim 1, further comprising:

Images