system

Abstract

Provide a system. A system for acquiring operation methods and troubleshooting information for home appliances via an information terminal owned by a user, comprising: means for receiving information about home appliances input by a user; means for analyzing a user's question using natural language processing and generating corresponding operating procedures and troubleshooting information; A method for displaying operating procedures and guidelines for home appliances through the camera of an information terminal using augmented reality technology; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Modern home appliances are becoming increasingly multifunctional, making their operation and troubleshooting complex. As a result, users must refer to thick instruction manuals to understand how to operate them, which is particularly inconvenient for elderly users and those unfamiliar with technology. Furthermore, if the instruction manual itself is lost or missing, or if a sudden problem occurs, it can be difficult to respond quickly. In addition, traditional customer support is often conducted via phone or email, which can be time-consuming. [Means for solving the problem]

[0005] The present invention provides a system for obtaining operating instructions and troubleshooting information for home appliances via a user's information terminal. Specifically, these problems are solved by building a system that includes a means for receiving information about home appliances entered by the user, a means for analyzing the user's questions using natural language processing and generating relevant operating procedures and troubleshooting information, and a means for displaying operating procedures and guidelines for the home appliance through the information terminal's camera using augmented reality technology. The system also includes an AI support chatbot that provides instant and appropriate answers to user questions, thereby reducing wait times and achieving more efficient support than traditional customer support. Furthermore, the system allows users to quickly obtain product information by scanning the QR code (registered trademark) on home appliances, improving the user experience.

[0006] "Information terminal" refers to a smartphone, tablet, or other computer device used by a user, and is a device used to send, receive, and operate information.

[0007] "Home appliances" refer to electrical appliances used in the home, such as microwave ovens, washing machines, air conditioners, and refrigerators, and are mechanical devices that make daily life more convenient.

[0008] "Natural language processing" refers to technology that analyzes natural language text entered by a user, understands its meaning, and generates an appropriate response.

[0009] "Augmented reality technology" is a technology that overlays digital information onto images of the real world, and has the function of displaying virtual guidelines and operating procedures for the real world through a camera.

[0010] An "operation procedure" refers to a series of steps required to use a specific function of a home appliance, and is a procedure that a user can follow to achieve the desired function.

[0011] "Troubleshooting information" refers to information on methods and procedures for identifying the cause and resolving problems when an abnormality or error occurs in a home appliance.

[0012] An "AI support chatbot" is an automatic response system that uses artificial intelligence technology to provide instant and appropriate answers to questions from users.

[0013] A "QR code" is a two-dimensional barcode that stores product information and can be scanned to quickly obtain digital information. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention provides a system that allows users to easily obtain operating instructions and troubleshooting information for home appliances via their own information terminals. Specific embodiments of this system will be described below.

[0036] Program processing flow (natural language explanation)

[0037] 1. Registering Home Appliances

[0038] Step 1.1: Install and register the app

[0039] User: Install the dedicated application on your smartphone and create an account the first time you start it. When creating the account, you enter basic user information.

[0040] Step 1.2: Enter your appliance information

[0041] User: Scans the QR code on the appliance they purchased with their smartphone camera or manually enters the product's model number and serial number into the app.

[0042] Terminal: Parses the QR code or manually entered product information and sends it to the server.

[0043] Step 1.3: Save product information

[0044] Server: Stores the received information about the home appliances in a database. Also, prepares and stores the relevant product manuals and operating procedures on the server.

[0045] 2. Obtaining operation instructions

[0046] Step 2.1: Enter your question

[0047] Users: Select their registered appliances within the app and enter their specific operating instructions or troubleshooting questions.

[0048] Terminal: Sends the entered question to the server.

[0049] Step 2.2: Parse the question

[0050] Server: Analyzes the question using a natural language processing (NLP) engine and understands the user's intent. For example, in response to the question "Where is the start button on the microwave?", it searches for information about the location of the start button.

[0051] Step 2.3: Generate an answer

[0052] Server: Generates an appropriate answer based on the information retrieved from the database and sends the generated answer to the device.

[0053] Step 2.4: View your answers

[0054] Device: Display the generated answer on a screen within the app.

[0055] User: By checking the displayed information, the user understands how to operate the home appliance and operates it appropriately.

[0056] 3. Use AI support chatbots

[0057] Step 3.1: Chat Interface

[0058] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[0059] Terminal: Sends a question to the server.

[0060] Step 3.2: Processing Questions

[0061] Server: The chatbot AI analyzes the question and searches the database for the appropriate answer.

[0062] Step 3.3: Chatbot response

[0063] Server: Sends the answers generated by the chatbot AI to the device.

[0064] On your device: Show the response in the chat screen.

[0065] User: Review the answers provided and take action as needed.

[0066] 4. Providing operation guidance using AR

[0067] Step 4.1: Using the Camera

[0068] User: Uses smartphone camera to scan home appliances.

[0069] Terminal: Analyzes scanned images and recognizes specific parts of home appliances.

[0070] Step 4.2: Identify the operation location

[0071] Terminal: Based on the recognized operation locations, the terminal uses AR technology to generate virtual operation procedures and guidelines, and requests additional information from the server as needed.

[0072] Step 4.3: AR display

[0073] Server: Generates additional information and sends it to the device.

[0074] Device: Operating procedures and guidelines are displayed as AR on the smartphone screen.

[0075] User: Follow the AR guidelines displayed through the camera and perform the appropriate operation.

[0076] Specific examples

[0077] Example 1: Getting instructions for operating a microwave

[0078] 1. User: Scans the QR code to register the microwave's information in the app.

[0079] 2. User: In the app, type "Where is the start button on the microwave?"

[0080] 3. Server: Analyzes the question and searches for the location of the start button.

[0081] 4. Server: Sends text and an image to the device explaining the location of the start button.

[0082] 5. Terminal: Display the answer.

[0083] 6. User: Check your answers and find and operate the start button on the microwave.

[0084] Example 2: Dealing with washing machine error codes

[0085] 1. User: Scan the QR code on the washing machine to register the product information.

[0086] 2. User: Asks in the app, "What does the E03 error code mean?"

[0087] 3. Server: Parse the question and find troubleshooting information for the E03 error.

[0088] 4. Server: Sends the meaning of the error code and how to deal with it to the device.

[0089] 5. Terminal: Display the answer.

[0090] 6. User: Follow the solutions to resolve the error.

[0091] These specific examples allow users to easily access the instruction manuals for home appliances, enabling them to quickly understand how to operate them and troubleshoot problems. This system is particularly useful for products that require complex operation and for elderly users.

[0092] The processing flow will be explained below.

[0093] Specific processing flow for the home appliance instruction manual platform

[0094] 1. Registering Home Appliances

[0095] Step 1: Install and register the app

[0096] Users: Install the dedicated application on their smartphone and create an account by entering personal information on the account creation screen when first starting the application.

[0097] Step 2: Enter your appliance information

[0098] User: Opens the app and scans the QR code of the purchased appliance with their smartphone camera or manually enters the product's model number or serial number.

[0099] Terminal: Scans the QR code, analyzes the product information (or manually entered information), and sends it to the server.

[0100] Step 3: Save your product information

[0101] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions in the database.

[0102] 2. Obtaining operation instructions

[0103] Step 1: Enter your question

[0104] User: Selects a registered appliance within the app and enters a specific instruction or troubleshooting question in the text box.

[0105] Terminal: Sends the entered question to the server.

[0106] Step 2: Parsing the Question

[0107] Server: Analyzes the question using a natural language processing (NLP) engine to understand the intent of the user's question. For example, it analyzes the question "Where is the start button on the microwave?" and identifies the location of the start button.

[0108] Step 3: Generate an answer

[0109] Server: Generates an appropriate response based on the information retrieved from the database and sends it to the device.

[0110] Step 4: View your answers

[0111] Device: The received answer is displayed on the screen within the app.

[0112] User: Check the displayed information, understand how to operate the appliance, and operate it correctly.

[0113] 3. Use AI support chatbots

[0114] Step 1: Chat Interface

[0115] Users: Use the app's chat interface to type in instructions and troubleshooting questions.

[0116] Terminal: Sends a question to the server.

[0117] Step 2: Processing questions

[0118] Server: The chatbot AI analyzes the question and searches for corresponding information from the database.

[0119] Step 3: Chatbot response

[0120] Server: Sends the answers generated by the chatbot AI to the device.

[0121] On your device: Show the response in the chat screen.

[0122] User: Review the answers provided and take action as needed.

[0123] 4. Providing operation guidance using AR

[0124] Step 1: Using the camera

[0125] User: Uses smartphone camera to scan home appliances.

[0126] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[0127] Step 2: Identify the operation location

[0128] Terminal: Based on the recognized operation points, the device uses AR technology to generate virtual operation procedures and guidelines, and requests additional information from the server as needed.

[0129] Step 3: AR display

[0130] Server: Generates any additional information needed and sends it to the device.

[0131] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen.

[0132] User: Follow the AR guidelines displayed through the camera to operate the home appliance appropriately.

[0133] In this way, through each step, the user can easily obtain the instruction manual and troubleshooting information for the home appliance and perform the operation appropriately.

[0134] Example 1

[0135] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0136] Quickly and easily obtaining operating instructions and troubleshooting information for home appliances is a major challenge for many users. For elderly users and those less familiar with technology, simply reading instruction manuals can be difficult, making access to appropriate information crucial. Furthermore, the increasing diversity of products creates a need for efficient methods to provide specific information for individual products.

[0137] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0138] In this invention, the server includes means for receiving information about a home appliance input by a user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through a camera of the information terminal using augmented reality technology, means for saving information about the home appliance in a database, means for managing user account information, and means for receiving a question from an information terminal owned by the user, analyzing the question by the server to generate an appropriate answer, and transmitting the answer back to the information terminal. This allows the user to easily and quickly access operating methods and troubleshooting information for the home appliance.

[0139] "User" refers to an individual who uses a home appliance and operates an information terminal.

[0140] An "information terminal" refers to a device, such as a smartphone or tablet, that has Internet connectivity and can be operated by the user.

[0141] "Home appliances" refers to electronic devices and electrical appliances used in the home.

[0142] A "server" refers to a computer system that receives requests from users over a network, processes the information, and returns it.

[0143] "Database" refers to an information management system that efficiently manages large amounts of data and enables quick access to it.

[0144] "Natural language processing" refers to the technology that enables computers to understand, analyze, and generate human language.

[0145] "Augmented reality technology" refers to a technology that displays virtual information overlaid on images captured in real time.

[0146] "Operating procedures and guidelines" refers to information that explains the steps and methods required to use a home appliance.

[0147] An "AI support chatbot" refers to a program that uses artificial intelligence technology to automatically provide answers to users' questions.

[0148] The term "means for receiving a question" refers to a system for receiving question data entered by a user.

[0149] "QR Code" refers to a type of two-dimensional barcode, a matrix code used to store information.

[0150] "Means for analysis" refers to a system that processes input data and executes a process to understand its meaning.

[0151] "Answer generation means" refers to techniques and methods for creating appropriate answers to users' questions.

[0152] "AR Guidelines" refers to visual information about how to operate and procedures for home appliances that is displayed using augmented reality technology.

[0153] The present invention provides a system for quickly and easily obtaining operating instructions and troubleshooting information for home appliances via an information terminal owned by a user. Specific embodiments of the present invention will be described below.

[0154] First, users install a dedicated application on their information terminals, such as smartphones or tablets. This application allows users to register their home appliances and obtain information on how to operate them and troubleshooting. When launching the application for the first time, users must create an account. To create an account, users must enter basic information such as their name, email address, and password.

[0155] For home appliances that users want to register, they can scan the product's QR code or manually enter the model number and serial number. The information terminal then sends the QR code information or manually entered information to the server. The server then stores the received product information in a database and prepares related operating instructions and manuals.

[0156] When a user selects a specific appliance within the application and enters a question about how to use it or troubleshooting, the question is sent to the server. The server uses a natural language processing engine to analyze the question and understand the user's intent. For example, in response to the question, "Where is the start button on my microwave?", the server searches for information about the start button and generates an appropriate answer. This answer is sent to the information terminal in text or image format and displayed to the user.

[0157] The present invention also includes an AI support chatbot that can automatically provide appropriate answers when a user inputs a question through the chat interface of the application. For example, in response to the question, "What does the E03 error on my washing machine mean?", the chatbot will search the database and provide the meaning of the error code and how to deal with it.

[0158] Furthermore, it also has a function that uses augmented reality technology to display operating procedures and guidelines for home appliances through the information terminal's camera. When a user scans an appliance using the camera, the information terminal recognizes specific parts of the product and uses AR technology to display virtual operating procedures and guidelines. This allows the user to operate the product appropriately while visually checking the operating procedures.

[0159] Specific examples

[0160] Example 1: Getting instructions for operating a microwave

[0161] The user scans the QR code to register the microwave's information in the app. Then, they enter "Where is the start button on the microwave?" in the app. The server analyzes the question and searches for the location of the start button. The server sends text and an image explaining the location of the start button to the device, which displays it. The user checks the answer, finds the start button on the microwave, and operates it.

[0162] Example 2: Dealing with washing machine error codes

[0163] The user scans the washing machine's QR code to register the product information. Then, within the app, they ask, "What does the E03 error code mean?" The server analyzes the question and searches for troubleshooting information related to the E03 error. The server then sends the meaning of the error code and the solution to the device, which displays it. The user can then follow the solution to resolve the error.

[0164] In this way, the present invention allows users to easily and quickly obtain operating instructions and troubleshooting information for home appliances.

[0165] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0166] Step 1: Install the app and create an account

[0167] User: Install the dedicated application on your smartphone or tablet. Once installation is complete, launch the app for the first time and proceed to the account creation screen. Enter basic information such as your name, email address, and password, and press the "Create Account" button.

[0168] Input: Name, Email Address, Password

[0169] Output: Encrypted account information

[0170] Terminal: Basic information entered by the user is encrypted and sent to the server.

[0171] Input: Encrypted account information

[0172] Output: Data sent to the server

[0173] Server: Receives the encrypted account information, stores it in the database as a new account, and returns a registration completion response to the device.

[0174] Input: Encrypted user information

[0175] Output: Registration complete response

[0176] Step 2: Enter your appliance information

[0177] User: Scans a QR code on a home appliance's manual with a smartphone camera or manually enters the product's model number and serial number into designated fields in the app.

[0178] Input: QR code information or model number, serial number

[0179] Output: Product information submission request

[0180] Terminal: Analyzes the information in the QR code scanned with the camera or the product information entered manually, converts the analysis results into JSON format, and sends them to the server.

[0181] Input: QR code information or manually entered product information

[0182] Output: Product information in JSON format

[0183] Server: Analyzes the received product information and searches for a corresponding database entry for the home appliance. If a matching product is found, it registers the product's details and manual data in the database.

[0184] Input: Product Information

[0185] Output: Database registration results

[0186] Step 3: Enter your operating instructions

[0187] User: Selects a specific appliance within the app and types a how-to or troubleshooting question into the text field. For example, "Where is the start button on my microwave?"

[0188] Input: Question text

[0189] Output: Request to send question data

[0190] Terminal: The entered question is sent to the server as text data.

[0191] Input: Question text

[0192] Output: Query data to the server

[0193] Step 4: Parsing the Question

[0194] Server: The received question text is sent to a natural language processing (NLP) engine for semantic analysis. For example, keywords such as "start button" and "microwave" are extracted.

[0195] Input: Question text

[0196] Output: Parsed keywords

[0197] Step 5: Generate an answer

[0198] Server: Searches the database for appropriate operation methods and troubleshooting information. Based on the analysis results of the NLP engine, generates the most appropriate response and sends it to the device as text and image data.

[0199] Input: Parsed keyword

[0200] Output: Generate text and image data

[0201] Device: Analyzes the data received from the server and displays it in the app in a format that is easy for the user to view.

[0202] Input: Text and image data

[0203] Output: Data displayed in the app

[0204] Step 6: View your answers

[0205] User: Review the in-app instructions and troubleshooting information and use the information provided to operate the home appliance appropriately.

[0206] Input: The information shown

[0207] Output: User understanding and operation of home appliances

[0208] Step 7: Use a chatbot

[0209] User: Open the in-app chat interface and type a how-to or troubleshooting question. For example, "What does error E03 on my washing machine mean?"

[0210] Input: Chat question

[0211] Output: Request to send question data

[0212] Terminal: Sends the question to the server as text data.

[0213] Input: Chat question

[0214] Output: Query data to the server

[0215] Server: The chatbot AI analyzes the received questions and generates appropriate answers, which are then sent to the device in text format.

[0216] Input: Chat question

[0217] Output: Textual response

[0218] On your device: Display the received response on the chat screen.

[0219] Input: Text response

[0220] Output: Chat screen display data

[0221] User: Checks the chatbot's response and operates or troubleshoots the appliance based on it.

[0222] Enter: the displayed answer

[0223] Output: Operate or troubleshoot home appliances

[0224] Step 8: AR operation guidance

[0225] User: Uses the app's AR feature to activate their smartphone camera and scan home appliances.

[0226] Input: Video data from home appliances

[0227] Output: Scan data

[0228] Terminal: Analyzes video of home appliances in real time and recognizes specific parts. Based on the recognition results, it uses AR technology to display operating procedures and guidelines.

[0229] Input: Video data

[0230] Output: AR display data

[0231] Server: Generates any additional information needed and sends it to the device.

[0232] Input: Request information

[0233] output: additional information data

[0234] Device: AR guidelines and operating procedures are superimposed on the image captured by the camera.

[0235] Input: Additional information data

[0236] Output: Display data overlaid on camera image

[0237] User: Check the AR guidelines and follow the instructions to operate the home appliance.

[0238] Input: AR guidelines

[0239] Output: Control of home appliances

[0240] (Application example 1)

[0241] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0242] It is difficult for users to quickly and accurately obtain operating instructions and troubleshooting information for home appliances. Conventional systems can take time to provide operating instructions and troubleshooting, potentially reducing user satisfaction. Furthermore, elderly users and those unfamiliar with technology have difficulty operating products and resolving problems.

[0243] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0244] In this invention, the server includes means for receiving information about the home appliance input by the user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through the camera of the information terminal using augmented reality technology, and means for displaying the generated answers on the information terminal, thereby enabling the user to quickly and accurately obtain operating methods and troubleshooting information for the home appliance.

[0245] (definition statement)

[0246] "Information terminal" is a general term for an electronic device that a user owns and uses to obtain operating instructions and troubleshooting information for home appliances.

[0247] "Natural language processing" is a technology that analyzes user questions, understands their intent, and generates appropriate operating procedures and troubleshooting information.

[0248] "Augmented reality technology" is a technology that uses the camera of an information terminal to display operating procedures and guidelines for home appliances overlaid on real-world images.

[0249] A "generative AI model" is an artificial intelligence technology that generates answers and guidelines to user questions.

[0250] A "prompt" is a question or instruction phrase that is input to a generative AI model.

[0251] A "QR code" is a two-dimensional code that contains information about a home appliance, and is an identification means that can be read by scanning it with the camera of an information terminal.

[0252] "AR display" refers to the use of augmented reality technology to display virtual operating procedures and guidelines on the screen of an information terminal.

[0253] "Home appliances" is a general term for electrical appliances used in ordinary households that may require instruction on operation or troubleshooting.

[0254] A "database" is a system for accumulating information about home appliances, operating procedures, and troubleshooting information, and storing them in a searchable format.

[0255] A "chat interface" is an operation screen that allows users to input questions in natural language and receive answers in an interactive format.

[0256] The present invention provides a system that allows users to easily obtain operating instructions and troubleshooting information for home appliances via their own information terminals. This system begins when the user installs a dedicated application on their smartphone. Specific embodiments of this system are described below.

[0257] System configuration

[0258] The system consists of the following main components:

[0259] 1. Information terminal: Electronic device such as a smartphone.

[0260] 2. Server: A remote computer containing a database and a natural language processing engine.

[0261] 3. Generative AI model: An artificial intelligence technology that generates appropriate answers to user questions.

[0262] Program processing

[0263] 1. Registering Home Appliances

[0264] Users install a dedicated application on their smartphone and create an account when they first start the app.

[0265] Users can either scan the QR code on the home appliance they purchased with their smartphone camera or manually enter the product's model number and serial number into the app.

[0266] The terminal analyzes the QR code or manually entered product information and sends the information to the server.

[0267] The server stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating procedures on the server.

[0268] 2. Obtaining operation instructions

[0269] Users select registered home appliances within the app and enter specific operating instructions and troubleshooting questions.

[0270] The terminal transmits the entered question to the server.

[0271] The server analyzes the question using a natural language processing (NLP) engine to understand the user's intent.

[0272] The server generates an appropriate response based on the information retrieved from the database and sends it to the terminal.

[0273] The device displays the generated answer on the screen within the app.

[0274] 3. Use AI support chatbots

[0275] Users use the app's chat interface to type in their how-to and troubleshooting questions.

[0276] The terminal sends a query to the server.

[0277] The server uses a chatbot AI to analyze the question and search for the appropriate answer from a database.

[0278] The server sends the answers generated by the chatbot AI to the terminal, where they are displayed.

[0279] 4. Providing operation guidance using AR

[0280] Users use their smartphone camera to scan home appliances.

[0281] The device analyzes the scanned image and recognizes specific parts of the home appliance.

[0282] Based on the recognized operation locations, the device uses AR technology to generate virtual operation procedures and guidelines, and requests additional information from the server if necessary.

[0283] The server generates additional information and sends it to the terminal.

[0284] The device displays operating procedures and guidelines as AR displays on the smartphone screen.

[0285] Specific examples

[0286] Example 1: Getting instructions for operating a microwave

[0287] 1. The user scans the QR code to register the microwave's information in the app.

[0288] 2. The user types in the app, "Where is the start button on the microwave?"

[0289] 3. The server analyzes the question and searches for the location of the start button.

[0290] 4. The server sends text and an image to the device explaining the location of the start button.

[0291] 5. The device will display the answer.

[0292] 6. The user checks their answer, finds the start button on the microwave, and operates it.

[0293] Example 2: Dealing with washing machine error codes

[0294] 1. The user scans the QR code on the washing machine to register the product information.

[0295] 2. The user asks within the app, "What does the E03 error code mean?"

[0296] 3. The server parses the question and searches for troubleshooting information about the E03 error.

[0297] 4. The server sends the meaning of the error code and how to deal with it to the terminal.

[0298] 5. The device will display the answer.

[0299] 6. The user follows the instructions to resolve the error.

[0300] Prompt Sentence Examples

[0301] "Please tell me how to use the microwave."

[0302] The system aims to help users operate home appliances quickly and efficiently, and is especially useful for elderly and less technologically savvy users.

[0303] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0304] Step 1:

[0305] Users install a dedicated application on their smartphone and create an account the first time they start it. At this time, the user enters basic information such as an email address and password. The information entered is stored in the application's database and used for future authentication.

[0306] Step 2:

[0307] Users can scan the QR code of a purchased home appliance with their smartphone camera or manually enter the product's model number or serial number into the app. The entered information (QR code or text) is parsed by the device and product information is extracted, allowing the home appliance to be identified.

[0308] Step 3:

[0309] The device then sends the analyzed product information to the server, which then stores the received product information in a database and prepares and stores the corresponding product manuals and operating procedures. During this process, the product information and manuals are linked.

[0310] Step 4:

[0311] Users select a registered home appliance within the app and enter specific operating instructions or troubleshooting questions. The entered questions (text) are sent from the device to the server, where a generative AI model is used to answer them.

[0312] Step 5:

[0313] The server analyzes the question using a natural language processing (NLP) engine to understand the user's intent. The server extracts keywords from the question and uses them to search a database for relevant operating procedures and troubleshooting information. The analysis results are then confirmed by a generative AI model.

[0314] Step 6:

[0315] The server generates an appropriate answer based on the information retrieved from the database, and the answer (text or image) is sent to the device.

[0316] Step 7:

[0317] The device displays the received response on the app screen, where the user can review the displayed information to understand how to operate the appliance and troubleshoot the problem.

[0318] Step 8:

[0319] Users use the app's chat interface to type in any follow-up instructions or troubleshooting questions, which are also sent from the device to the server.

[0320] Step 9:

[0321] The server uses chatbot AI to analyze the question and generate an appropriate answer. A generative AI model is used in this process to ensure the accuracy of the answer. The generated answer is then sent to the device.

[0322] Step 10:

[0323] The device displays the answer on the chat interface, and the user can check the answer and operate the home appliance as needed.

[0324] Step 11:

[0325] Users scan home appliances using their smartphone cameras, and the scanned images (image data) are analyzed by the device to identify specific parts of the appliance.

[0326] Step 12:

[0327] The device uses AR technology to generate virtual operating procedures and guidelines based on the recognized operation points, using data provided by the generative AI model, and requests additional information from the server if necessary.

[0328] Step 13:

[0329] The server generates additional information and sends it to the device. The device displays operation procedures and guidelines as AR on the smartphone screen. The user follows the AR guidelines displayed through the camera to perform the appropriate operation.

[0330] Example prompt: "Tell me how to use a microwave."

[0331] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0332] The present invention combines an emotion engine with a system for obtaining operating instructions and troubleshooting information for home appliances via a user's information terminal. This system has the function of analyzing a user's questions and providing appropriate operating instructions and troubleshooting information, as well as the function of recognizing the user's emotional state and generating a response accordingly. Specific embodiments of this system are described below.

[0333] Program processing flow (natural language explanation)

[0334] 1. Registering Home Appliances

[0335] Step 1.1: Install and register the app

[0336] Users: Install the dedicated application on their smartphone and create an account by entering personal information on the account creation screen when first starting the application.

[0337] Step 1.2: Enter your appliance information

[0338] User: Opens the app and scans the QR code of the purchased appliance with their smartphone camera or manually enters the product's model number or serial number.

[0339] Terminal: Scans the QR code, analyzes the product information (or manually entered information), and sends it to the server.

[0340] Step 1.3: Save product information

[0341] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions.

[0342] 2. Obtaining operation instructions

[0343] Step 1: Enter your question

[0344] User: Selects a registered appliance within the app and enters a specific instruction or troubleshooting question in the text box.

[0345] Terminal: Sends the entered question to the server.

[0346] Step 2: Parse the question

[0347] Server: Analyzes the question using a natural language processing (NLP) engine to understand the intent of the user's question. For example, it analyzes the question "Where is the start button on the microwave?" and identifies the location of the start button.

[0348] Step 3: Emotion recognition with the emotion engine

[0349] Device: Using a camera and microphone, the device analyzes the user's facial expressions and tone of voice, and uses an emotion engine to recognize the user's emotional state.

[0350] Step 4: Generate an answer

[0351] Server: Generates appropriate responses based on information retrieved from the database, and adjusts the tone and content of the responses depending on the emotional state recognized by the emotion engine.

[0352] Server: Sends the generated answer to the device.

[0353] Step 5: View your answers

[0354] Device: The received answer is displayed on the screen within the app.

[0355] User: Check the displayed information, understand how to operate the appliance, and operate it correctly.

[0356] 3. Use AI support chatbots

[0357] Step 1: Chat Interface

[0358] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[0359] Terminal: Sends a question to the server.

[0360] Step 2: Processing questions

[0361] Server: The chatbot AI analyzes the question and searches the database for the appropriate answer. The emotion engine also analyzes the user's emotions.

[0362] Step 3: Chatbot response

[0363] Server: The server adjusts the answers generated by the chatbot AI according to the user's emotional state and sends them to the device.

[0364] On your device: Show the response in the chat screen.

[0365] User: Review the answers provided and take action as needed.

[0366] 4. Providing operation guidance using AR

[0367] Step 1: Use the camera

[0368] User: Scans home appliances using the smartphone camera.

[0369] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[0370] Step 2: Identify the operation location

[0371] Terminal: Using AR technology, it generates virtual operating procedures and guidelines based on the recognized operation locations, and requests additional information from the server as needed.

[0372] Step 3: AR display

[0373] Server: Generates any additional information needed and sends it to the device.

[0374] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen.

[0375] User: Follow the AR guidelines displayed through the camera to operate the home appliance appropriately.

[0376] Specific examples

[0377] Example 1: Getting instructions for operating a microwave

[0378] 1. User: Scans the QR code to register the microwave's information in the app.

[0379] 2. User: In the app, type "Where is the start button on the microwave?"

[0380] 3. Server: Analyzes the question and searches for the location of the start button.

[0381] 4. Device: Uses the camera and microphone to identify the user's emotions and prepares a quick, concise response if it determines, for example, that the user is feeling impatient.

[0382] 5. Server: Sends text and an image to the device explaining the location of the start button.

[0383] 6. Terminal: Display the answer.

[0384] 7. User: Check your answers and find and operate the start button on the microwave.

[0385] Example 2: Dealing with washing machine error codes

[0386] 1. User: Scan the QR code on the washing machine to register the product information.

[0387] 2. User: Asks in the app, "What does the E03 error code mean?"

[0388] 3. Server: Parse the question and find troubleshooting information for the E03 error.

[0389] 4. Device: The emotion engine analyzes the user's emotions and determines whether they are feeling anxious or impatient.

[0390] 5. Server: Depending on the user's emotional state, the server creates a message in a calm tone explaining the meaning of the error code and how to deal with it, and sends it to the device.

[0391] 6. Terminal: Display the answer.

[0392] 7. User: Follow the solutions to resolve the error.

[0393] This system allows users to easily access their appliances' instruction manuals and troubleshooting information, and also responds to their emotional state, improving the user experience and enabling them to operate their appliances more effectively.

[0394] The processing flow will be explained below.

[0395] Specific processing flow for the home appliance instruction manual platform

[0396] 1. Registering Home Appliances

[0397] Step 1: Install and register the app

[0398] Users: Install the dedicated application on their smartphone and create an account by entering personal information on the account creation screen when first starting the application.

[0399] Step 2: Enter your appliance information

[0400] User: Opens the app and scans the QR code of the purchased appliance with their smartphone camera or manually enters the product's model number or serial number.

[0401] Terminal: Scans the QR code, analyzes the product information (or manually entered information), and sends it to the server.

[0402] Step 3: Save your product information

[0403] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions.

[0404] 2. Obtaining operation instructions

[0405] Step 1: Enter your question

[0406] User: Selects a registered appliance within the app and enters a specific instruction or troubleshooting question in the text box.

[0407] Terminal: Sends the entered question to the server.

[0408] Step 2: Parsing the Question

[0409] Server: Analyzes the question using a natural language processing (NLP) engine to understand the intent of the user's question. For example, it analyzes the question "Where is the start button on the microwave?" and identifies the location of the start button.

[0410] Step 3: Emotion recognition by the emotion engine

[0411] On the device: Using a camera and microphone, the device analyzes the user's facial expressions and tone of voice, and uses an emotion engine to recognize the user's emotional state. For example, emotions such as smile, surprise, and anxiety are identified and an index is generated.

[0412] Step 4: Generate an answer

[0413] Server: Generates appropriate answers based on information retrieved from a database, and adjusts the tone and content of the answers based on the user's perceived emotional state, for example, explaining things in a calmer tone to an anxious user.

[0414] Server: Sends the generated answer to the device.

[0415] Step 5: View your answers

[0416] Device: The received answer is displayed on the screen within the app, with an appropriate interface based on the emotion.

[0417] User: Check the displayed information, understand how to operate the appliance, and operate it correctly.

[0418] 3. Use AI support chatbots

[0419] Step 1: Chat Interface

[0420] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[0421] Terminal: Sends a question to the server.

[0422] Step 2: Processing questions

[0423] Server: The chatbot AI analyzes the question and searches for the appropriate answer from the database. The emotion engine also analyzes the user's emotions.

[0424] Step 3: Chatbot response

[0425] Server: The server adjusts the answers generated by the chatbot AI according to the user's emotional state and sends them to the device.

[0426] Device: Responses will be displayed on the chat screen, and responses to specific emotional states may be highlighted with color coding or icons.

[0427] User: Review the answers provided and take action as needed.

[0428] 4. Providing operation guidance using AR

[0429] Step 1: Using the camera

[0430] User: Scans home appliances using the smartphone camera.

[0431] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[0432] Step 2: Identify the operation location

[0433] Terminal: Using AR technology, it generates virtual operating procedures and guidelines based on the recognized operation locations, and requests additional information from the server as needed.

[0434] Step 3: AR display

[0435] Server: Generates any additional information needed and sends it to the device.

[0436] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen.

[0437] User: Follow the AR guidelines displayed through the camera to operate the home appliance appropriately.

[0438] Specific examples

[0439] Example 1: Getting instructions for operating a microwave

[0440] 1. User: Scans the QR code to register the microwave's information in the app.

[0441] 2. User: In the app, type "Where is the start button on the microwave?"

[0442] 3. Server: Analyzes the question and searches for the location of the start button.

[0443] 4. Device: Uses the camera and microphone to identify the user's emotions and prepares a quick, concise response if it determines, for example, that the user is feeling impatient.

[0444] 5. Server: Sends text and an image to the device explaining the location of the start button.

[0445] 6. Terminal: Display the answer.

[0446] 7. User: Check your answers and find and operate the start button on the microwave.

[0447] Example 2: Dealing with washing machine error codes

[0448] 1. User: Scan the QR code on the washing machine to register the product information.

[0449] 2. User: Asks in the app, "What does the E03 error code mean?"

[0450] 3. Server: Parse the question and find troubleshooting information for the E03 error.

[0451] 4. Device: The emotion engine analyzes the user's emotions and determines whether they are feeling anxious or impatient.

[0452] 5. Server: Depending on the user's emotional state, the server creates a message in a calm tone explaining the meaning of the error code and how to deal with it, and sends it to the device.

[0453] 6. Terminal: Display the answer.

[0454] 7. User: Follow the solutions to resolve the error.

[0455] This system allows users to easily access their appliances' instruction manuals and troubleshooting information, and also responds to their emotional state, improving the user experience and enabling them to operate their appliances more effectively.

[0456] Example 2

[0457] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0458] Conventional systems that provide operating instructions and troubleshooting for home appliances often prevent users from properly obtaining the information they need. They also lack the ability to respond to user emotions, making it difficult to improve the user experience. Furthermore, entering specific product information and answering questions can be cumbersome, making intuitive operation difficult.

[0459] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0460] In this invention, the server includes means for receiving information about the home appliance input by the user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through the camera of the information terminal using augmented reality technology, emotion analysis means for recognizing the emotional state of the user, and means for generating a response according to the emotional state recognized by the emotion analysis means. This allows the user to intuitively obtain the information they need and enables responses according to their emotions, which is expected to improve the user experience.

[0461] The "information terminal owned by the user" refers to a portable electronic device that the user personally owns and can operate, such as a smartphone or tablet.

[0462] "Home appliance information" refers to basic information about the home appliance, such as the product name, model number, serial number, and QR code.

[0463] "Natural language processing" is a technology that uses computers to analyze and automatically process human language, including techniques for interpreting the meaning of text and answering questions.

[0464] "Augmented reality technology" is a technology that displays virtual information overlaid on the real environment, and includes, for example, a method of displaying operation guides on the surface of home appliances that can be seen through a camera.

[0465] "Emotion analysis means" refers to technology that analyzes a user's facial expressions and tone of voice to recognize the user's emotional state, and includes, for example, a method of determining a user's emotions in real time using a camera or microphone.

[0466] A "generative AI model" is a model that uses artificial intelligence technology to automatically generate answers to user questions, and includes, for example, large-scale natural language generation models.

[0467] A "database" is a collection of structured data that systematically stores large amounts of data and makes it easy to search and update. This system stores operating procedures and troubleshooting information for home appliances.

[0468] The "means for analyzing a question" refers to a technique for receiving a question from a user and understanding its content, and includes, for example, analyzing the intent of the question using a natural language processing engine.

[0469] The "means for displaying operating procedures and guidelines" refers to a technology that uses augmented reality technology to visually display specific operating procedures and guidelines for home appliances on the screen of an information terminal.

[0470] "Means for generating answers according to the emotional state" refers to technology that generates answers with appropriate tone and content based on the user's emotional state, and includes, for example, a method for adjusting the expression of the answer based on the results of emotional analysis.

[0471] The present invention is a system for obtaining operation methods and troubleshooting information for home appliances via an information terminal owned by a user, and includes the following detailed embodiments.

[0472] This system not only allows users to obtain operating instructions and troubleshooting information for home appliances through their information terminal, but also recognizes the user's emotional state and responds individually. The main hardware used includes smartphones and tablets. The main software used includes a natural language processing engine, augmented reality technology, an emotion analysis engine, and a generative AI model.

[0473] The server first receives information about home appliances entered by the user. The user installs a dedicated application on their smartphone, creates an account when they first start it, and then obtains this information by scanning the QR code of the purchased home appliance or manually entering the product's model number and serial number.

[0474] The device analyzes the received information about the home appliance and sends it to the server. The server stores the received information in a database and prepares and saves the relevant product manuals and operating procedures. This allows users to easily obtain the information they need.

[0475] Next, when the user enters a question about a home appliance within the app, the device sends the question to the server. The server uses a natural language processing engine (such as Google® Cloud Natural Language API or Microsoft® Azure® Text Analytics) to analyze the question and understand the user's intent. For example, the server understands the question, "Where is the start button on the microwave?" and retrieves the corresponding information.

[0476] The device also analyzes the user's facial expressions and tone of voice to recognize the user's emotional state using an emotion analysis engine (e.g., Affectiva's emotion analysis API). For example, if the user is feeling anxious, the emotion engine will detect that the user's voice is high-pitched and fast-paced.

[0477] The server extracts appropriate operating procedures and troubleshooting information from the database and adjusts the tone and content of the response based on the results of sentiment analysis. For example, if the user is impatient, it provides a concise and quick response. Appropriate responses are automatically generated using a generative AI model (e.g., OpenAI's GPT-3). The server sends the generated response to the device, which then displays it on the app screen.

[0478] The AI ​​support chatbot can be used when users input questions about operation methods or troubleshooting. The chatbot analyzes the question using a natural language processing engine, recognizes the user's emotional state using an emotion analysis engine, and responds appropriately. For example, the chatbot generates an appropriate answer for the question, "What should I do if my washing machine displays an E03 error?"

[0479] Furthermore, it also includes a function that uses augmented reality technology to display operating procedures and guidelines for home appliances through the information terminal's camera. When a user scans an appliance with their smartphone's camera, the terminal displays virtual guidelines for operation points based on the recognized information. This makes it easier for users to intuitively understand operating procedures.

[0480] Specific examples

[0481] Example 1: Getting instructions for operating a microwave

[0482] User: Scan the QR code to register the microwave's information in the app.

[0483] User: Within the app, type "Where is the start button on the microwave?"

[0484] Server: Analyzes the question and searches for the location of the start button.

[0485] On the device: Use the camera and microphone to identify the user's emotions and, for example, prepare a quick, concise response if it detects impatience.

[0486] Server: Sends text and an image to the device explaining the location of the start button.

[0487] Terminal: Display the answer.

[0488] User: Check your answers, find the start button on the microwave, and operate it.

[0489] Example 2: Dealing with washing machine error codes

[0490] User: Scan the QR code on the washing machine to register the product information.

[0491] User: Asks within the app, "What does the E03 error code mean?"

[0492] Server: Parse the question and find troubleshooting information for the E03 error.

[0493] Device: The emotion engine analyzes the user's emotions and determines whether they are feeling anxious or impatient.

[0494] Server: Depending on the user's emotional state, the server creates a message in a calm tone explaining the meaning of the error code and how to deal with it, and sends it to the device.

[0495] Terminal: Display the answer.

[0496] User: Follow the solutions to resolve the error.

[0497] The above-described configuration allows users to quickly and intuitively obtain operating instructions and troubleshooting information for home appliances, and also enables responses that correspond to the user's emotional state, significantly improving the user experience.

[0498] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0499] Specific processing flow of the system program

[0500] Step 1:

[0501] Installing the app and creating an account

[0502] User: Install the dedicated application on their smartphone, launch the app, and create an account by entering their first and last name, email address, password, etc. on the account creation screen. Then, click on the confirmation link sent to their email address to confirm their account.

[0503] Input: User's first name, last name, email address, and password

[0504] Output: Created user account information

[0505] Terminal: Receives information entered by the user and sends it to the server.

[0506] Server: Stores the received account information in a database.

[0507] Step 2:

[0508] Entering information about home appliances

[0509] User: Opens the app and scans the QR code of the purchased appliance or manually enters the product's model number and serial number.

[0510] Input: QR code information or product model number, serial number of the home appliance

[0511] Output: Parsed product information

[0512] Terminal: Scans the QR code to obtain product information (or manually entered information), parses it into a string, and sends it to the server.

[0513] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions.

[0514] Step 3:

[0515] Saving home appliance information

[0516] Server: Based on the registered home appliance information, the server references the database to find and store relevant product manuals and operating instructions.

[0517] Input: Registered home appliance information

[0518] Output: Database record of applicable product manuals and operating procedures

[0519] Specific operations: Performs data format conversion and tagging operations in the process of storing product information in a database.

[0520] Step 4:

[0521] User question input

[0522] User: Selects a registered appliance in the app and types a specific how-to or troubleshooting question into the text box, such as "Where is the start button on my microwave?"

[0523] Input: User question text

[0524] Output: The question sent to the server

[0525] Terminal: Sends the entered question to the server.

[0526] Step 5:

[0527] Question Analysis

[0528] Server: Analyzes the received question using a natural language processing (NLP) engine to identify the intent of the question, for example by breaking down the text into tokens and extracting keywords.

[0529] Input: Question text received from the user

[0530] Output: Question intent and keywords

[0531] Specific operation: Uses natural language processing API to perform text analysis and extract the intent of the question.

[0532] Step 6:

[0533] Emotion analysis

[0534] Device: Captures the user's facial expressions and tone of voice in real time and uses an emotion analysis engine to recognize their emotional state.

[0535] Input: User's facial expression data, tone of voice

[0536] Output: User's emotional state (e.g., impatience, anxiety, joy, etc.)

[0537] Specific operation: Using a camera and microphone, the user's facial expressions and voice are input into an emotion analysis engine to analyze their emotional state.

[0538] Step 7:

[0539] Answer generation and coordination

[0540] Server: Generates appropriate responses based on information retrieved from a database and adjusts the tone and content of responses based on sentiment analysis results. For example, prepare a concise and quick response for a user who is impatient. Uses a generative AI model.

[0541] Input: Question intent analyzed by natural language processing, sentiment analysis results, and answer candidates from the database

[0542] Output: Adjusted final answer

[0543] What it does: Uses a generative AI model to generate answers and tailor them appropriately based on sentiment analysis.

[0544] Step 8:

[0545] Submitting and viewing answers

[0546] Server: Sends the generated answer to the device.

[0547] Input: Adjusted final answer

[0548] Output: The answer sent to the user's device

[0549] Device: The received answer is displayed on the screen within the app.

[0550] User: Check the displayed information, understand how to operate the appliance, and operate it appropriately.

[0551] Step 9:

[0552] Use of AI support chatbots

[0553] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[0554] Input: User question text

[0555] Output: The question sent to the server

[0556] Terminal: Sends a question to the server.

[0557] Server: Analyzes the question, identifies the problem, and sends the answer generated by the chatbot AI to the device, analyzing the emotional state accordingly.

[0558] Step 10:

[0559] AR operation guidance

[0560] User: Uses the camera to scan home appliances and display operating instructions guidelines.

[0561] Input: Camera scan of product

[0562] Output: AR guidelines displayed on the smartphone screen

[0563] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[0564] Server: Generates any additional information needed and sends it to the device.

[0565] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen. Users follow the AR guidelines to operate home appliances appropriately.

[0566] These specific actions allow users to intuitively obtain operating procedures and troubleshooting information, and respond in accordance with their emotions.

[0567] (Application example 2)

[0568] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0569] Conventional methods for obtaining operating instructions and troubleshooting information for home appliances require users to refer to instruction manuals, which is time-consuming and inefficient. Furthermore, information is provided without taking into account the user's emotional state, which can amplify negative emotions such as anxiety and impatience. Furthermore, information acquisition is often limited to text-based information, and visual explanations can be difficult. These problems need to be solved.

[0570] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0571] In this invention, the server includes means for receiving information about the home appliance input by the user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through the camera of the information terminal using augmented reality technology, and means for analyzing the user's emotional state using an emotion recognition engine and adjusting responses based on the results. This enables the user to efficiently and appropriately obtain operating methods and troubleshooting information for the home appliance and provides responses that take the user's emotional state into consideration.

[0572] An "information terminal" is a device owned by a user that connects to the Internet and various applications to acquire and process information.

[0573] "Consumer appliances" refers to electrical and electronic devices used in the home that require instruction on how to operate or troubleshoot.

[0574] "Natural language processing" is a language understanding and generation technology that analyzes questions and instructions entered by users and generates appropriate responses based on them.

[0575] "Augmented reality technology" is a technology that displays virtual information overlaid on real-world images through the camera of an information terminal.

[0576] An "emotion recognition engine" is a technology that analyzes a user's facial expressions, tone of voice, etc. to estimate their emotional state.

[0577] An "AI support chatbot" is a conversational interface that uses artificial intelligence to automatically generate appropriate responses to user questions.

[0578] A "QR code" is a type of two-dimensional barcode that can efficiently read data including product information.

[0579] A "server" is a computer system that provides data and performs computations over a network.

[0580] This invention is a system for obtaining operating instructions and troubleshooting information for home appliances via a user's information terminal. The system receives information about the home appliance entered by the user, analyzes the user's question using natural language processing, and generates relevant operating instructions and troubleshooting information. It also uses augmented reality technology to display operating instructions and guidelines for the home appliance through the information terminal's camera, analyzes the user's emotional state using an emotion recognition engine, and adjusts responses based on that.

[0581] The hardware used is an information terminal such as a smartphone or tablet equipped with an internet connection, a camera, and a microphone. The software used includes Flask as a web framework, Hugging Face's Transformers as a natural language processing engine, and Face Recognition and Numpy for facial recognition and emotion analysis.

[0582] The server receives information about home appliances entered by the user and stores it in a database. When the user enters a question, the server analyzes the question using a natural language processing engine and generates relevant information. The generated information is then further adjusted according to the user's emotional state using an emotion recognition engine.

[0583] As a concrete example, consider a case where a user scans a QR code on an electronic product, registers the information in an application, and then enters a question about operation or troubleshooting. The server analyzes the question and generates an appropriate answer. If the user is feeling anxious or impatient, the emotion recognition engine will detect that emotion and provide an answer in a gentler tone accordingly.

[0584] An example of a prompt sentence for the generative AI model is as follows:

[0585] prompt:

[0586] A user asks, "What does washing machine error code E03 mean?" You can see the anxiety in their face. Generate a relaxing answer.

[0587] answer:

[0588] "The E03 error code is a drainage error. This can often be resolved by cleaning the filter. If you have any questions, please let us know. We'll fix it right away."

[0589] This system allows users to obtain information more efficiently than traditional methods of obtaining operating instructions or troubleshooting, and provides a better user experience by receiving responses that take into account their emotional state.

[0590] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0591] Step 1:

[0592] The user installs the dedicated application on their information terminal and starts it. When they first start it, they enter their personal information to create an account. During this process, the user uses their smartphone, and the account information is sent to the server and stored in a database.

[0593] Input: Personal information

[0594] Output: Account information stored on the server

[0595] Step 2:

[0596] A user scans the QR code of a purchased home appliance with the smartphone camera. Product information (model number, serial number, etc.) is obtained from the scanned QR code and sent to the server.

[0597] Input: QR code of home appliance

[0598] Output: Product information stored on the server

[0599] Step 3:

[0600] The server stores the received information about the home appliances in a database and prepares the relevant product manuals and operating procedures. This allows information about the home appliances owned by the user to be registered in the system.

[0601] Input:Product Information

[0602] Output: Product information stored in a database

[0603] Step 4:

[0604] The user uses the information terminal to enter a question about the registered home appliance into a text box within the app, and the question is sent to the server.

[0605] Input: User question

[0606] Output: The query data sent to the server

[0607] Step 5:

[0608] The server receives the question data and analyzes it using a natural language processing (NLP) engine, which identifies the intent of the question and prepares it for answer generation.

[0609] Input: Question data

[0610] Output: Analysis result (question intent)

[0611] Step 6:

[0612] The device uses the smartphone's camera and microphone to capture the user's facial expressions and tone of voice, and then uses an emotion recognition engine to analyze the user's emotional state. The analysis results are then sent to a server.

[0613] Input: User's facial and voice data

[0614] Output: Parsed emotional state data

[0615] Step 7:

[0616] The server searches a database for relevant operating procedures and troubleshooting information, and tailors responses based on the analysis by an emotion-recognition engine. For example, a user who is feeling anxious or impatient will receive a gentler response.

[0617] Input: Analysis results, emotional state data

[0618] Output: Adjusted response data

[0619] Step 8:

[0620] The server sends the generated answer data to the information terminal, which then displays the answer to the user. The user checks the displayed information and operates the home appliance appropriately.

[0621] Input: Adjusted response data

[0622] Output: Answer displayed on terminal

[0623] Step 9:

[0624] The user uses the camera on the information terminal to scan the home appliance, and the operating procedures and guidelines are displayed on the screen using augmented reality technology, allowing the user to visually confirm how to operate the appliance.

[0625] Input: Video data from home appliances

[0626] Output: AR guidelines displayed on the screen

[0627] In this way, users can efficiently and safely learn how to operate and troubleshoot their home appliances. It is designed to enhance the user experience based on step-by-step inputs and outputs.

[0628] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0629] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0630] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0631] [Second embodiment]

[0632] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0633] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0634] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0635] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0636] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0637] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0638] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0639] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0640] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0641] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0642] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0643] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0644] The present invention provides a system that allows users to easily obtain operating instructions and troubleshooting information for home appliances via their own information terminals. Specific embodiments of this system will be described below.

[0645] Program processing flow (natural language explanation)

[0646] 1. Registering Home Appliances

[0647] Step 1.1: Install and register the app

[0648] User: Install the dedicated application on your smartphone and create an account the first time you start it. When creating the account, you enter basic user information.

[0649] Step 1.2: Enter your appliance information

[0650] User: Scans the QR code on the appliance they purchased with their smartphone camera or manually enters the product's model number and serial number into the app.

[0651] Terminal: Parses the QR code or manually entered product information and sends it to the server.

[0652] Step 1.3: Save product information

[0653] Server: Stores the received information about the home appliances in a database. Also, prepares and stores the relevant product manuals and operating procedures on the server.

[0654] 2. Obtaining operation instructions

[0655] Step 2.1: Enter your question

[0656] Users: Select their registered appliances within the app and enter their specific operating instructions or troubleshooting questions.

[0657] Terminal: Sends the entered question to the server.

[0658] Step 2.2: Parse the question

[0659] Server: Analyzes the question using a natural language processing (NLP) engine and understands the user's intent. For example, in response to the question "Where is the start button on the microwave?", it searches for information about the location of the start button.

[0660] Step 2.3: Generate an answer

[0661] Server: Generates an appropriate answer based on the information retrieved from the database and sends the generated answer to the device.

[0662] Step 2.4: View your answers

[0663] Device: Display the generated answer on a screen within the app.

[0664] User: By checking the displayed information, the user understands how to operate the home appliance and operates it appropriately.

[0665] 3. Use AI support chatbots

[0666] Step 3.1: Chat Interface

[0667] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[0668] Terminal: Sends a question to the server.

[0669] Step 3.2: Processing Questions

[0670] Server: The chatbot AI analyzes the question and searches the database for the appropriate answer.

[0671] Step 3.3: Chatbot response

[0672] Server: Sends the answers generated by the chatbot AI to the device.

[0673] On your device: Show the response in the chat screen.

[0674] User: Review the answers provided and take action as needed.

[0675] 4. Providing operation guidance using AR

[0676] Step 4.1: Using the Camera

[0677] User: Uses smartphone camera to scan home appliances.

[0678] Terminal: Analyzes scanned images and recognizes specific parts of home appliances.

[0679] Step 4.2: Identify the operation location

[0680] Terminal: Based on the recognized operation locations, the terminal uses AR technology to generate virtual operation procedures and guidelines, and requests additional information from the server as needed.

[0681] Step 4.3: AR display

[0682] Server: Generates additional information and sends it to the device.

[0683] Device: Operating procedures and guidelines are displayed as AR on the smartphone screen.

[0684] User: Follow the AR guidelines displayed through the camera and perform the appropriate operation.

[0685] Specific examples

[0686] Example 1: Getting instructions for operating a microwave

[0687] 1. User: Scans the QR code to register the microwave's information in the app.

[0688] 2. User: In the app, type "Where is the start button on the microwave?"

[0689] 3. Server: Analyzes the question and searches for the location of the start button.

[0690] 4. Server: Sends text and an image to the device explaining the location of the start button.

[0691] 5. Terminal: Display the answer.

[0692] 6. User: Check your answers and find and operate the start button on the microwave.

[0693] Example 2: Dealing with washing machine error codes

[0694] 1. User: Scan the QR code on the washing machine to register the product information.

[0695] 2. User: Asks in the app, "What does the E03 error code mean?"

[0696] 3. Server: Parse the question and find troubleshooting information for the E03 error.

[0697] 4. Server: Sends the meaning of the error code and how to deal with it to the device.

[0698] 5. Terminal: Display the answer.

[0699] 6. User: Follow the solutions to resolve the error.

[0700] These specific examples allow users to easily access the instruction manuals for home appliances, enabling them to quickly understand how to operate them and troubleshoot problems. This system is particularly useful for products that require complex operation and for elderly users.

[0701] The processing flow will be explained below.

[0702] Specific processing flow for the home appliance instruction manual platform

[0703] 1. Registering Home Appliances

[0704] Step 1: Install and register the app

[0705] Users: Install the dedicated application on their smartphone and create an account by entering personal information on the account creation screen when first starting the application.

[0706] Step 2: Enter your appliance information

[0707] User: Opens the app and scans the QR code of the purchased appliance with their smartphone camera or manually enters the product's model number or serial number.

[0708] Terminal: Scans the QR code, analyzes the product information (or manually entered information), and sends it to the server.

[0709] Step 3: Save your product information

[0710] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions in the database.

[0711] 2. Obtaining operation instructions

[0712] Step 1: Enter your question

[0713] User: Selects a registered appliance within the app and enters a specific instruction or troubleshooting question in the text box.

[0714] Terminal: Sends the entered question to the server.

[0715] Step 2: Parsing the Question

[0716] Server: Analyzes the question using a natural language processing (NLP) engine to understand the intent of the user's question. For example, it analyzes the question "Where is the start button on the microwave?" and identifies the location of the start button.

[0717] Step 3: Generate an answer

[0718] Server: Generates an appropriate response based on the information retrieved from the database and sends it to the device.

[0719] Step 4: View your answers

[0720] Device: The received answer is displayed on the screen within the app.

[0721] User: Check the displayed information, understand how to operate the appliance, and operate it correctly.

[0722] 3. Use AI support chatbots

[0723] Step 1: Chat Interface

[0724] Users: Use the app's chat interface to type in instructions and troubleshooting questions.

[0725] Terminal: Sends a question to the server.

[0726] Step 2: Processing questions

[0727] Server: The chatbot AI analyzes the question and searches for corresponding information from the database.

[0728] Step 3: Chatbot response

[0729] Server: Sends the answers generated by the chatbot AI to the device.

[0730] On your device: Show the response in the chat screen.

[0731] User: Review the answers provided and take action as needed.

[0732] 4. Providing operation guidance using AR

[0733] Step 1: Using the camera

[0734] User: Uses smartphone camera to scan home appliances.

[0735] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[0736] Step 2: Identify the operation location

[0737] Terminal: Based on the recognized operation points, the device uses AR technology to generate virtual operation procedures and guidelines, and requests additional information from the server as needed.

[0738] Step 3: AR display

[0739] Server: Generates any additional information needed and sends it to the device.

[0740] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen.

[0741] User: Follow the AR guidelines displayed through the camera to operate the home appliance appropriately.

[0742] In this way, through each step, the user can easily obtain the instruction manual and troubleshooting information for the home appliance and perform the operation appropriately.

[0743] Example 1

[0744] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0745] Quickly and easily obtaining operating instructions and troubleshooting information for home appliances is a major challenge for many users. For elderly users and those less familiar with technology, simply reading instruction manuals can be difficult, making access to appropriate information crucial. Furthermore, the increasing diversity of products creates a need for efficient methods to provide specific information for individual products.

[0746] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0747] In this invention, the server includes means for receiving information about a home appliance input by a user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through a camera of the information terminal using augmented reality technology, means for saving information about the home appliance in a database, means for managing user account information, and means for receiving a question from an information terminal owned by the user, analyzing the question by the server to generate an appropriate answer, and transmitting the answer back to the information terminal. This allows the user to easily and quickly access operating methods and troubleshooting information for the home appliance.

[0748] "User" refers to an individual who uses a home appliance and operates an information terminal.

[0749] An "information terminal" refers to a device, such as a smartphone or tablet, that has Internet connectivity and can be operated by the user.

[0750] "Home appliances" refers to electronic devices and electrical appliances used in the home.

[0751] A "server" refers to a computer system that receives requests from users over a network, processes the information, and returns it.

[0752] "Database" refers to an information management system that efficiently manages large amounts of data and enables quick access to it.

[0753] "Natural language processing" refers to the technology that enables computers to understand, analyze, and generate human language.

[0754] "Augmented reality technology" refers to a technology that displays virtual information overlaid on images captured in real time.

[0755] "Operating procedures and guidelines" refers to information that explains the steps and methods required to use a home appliance.

[0756] An "AI support chatbot" refers to a program that uses artificial intelligence technology to automatically provide answers to users' questions.

[0757] The term "means for receiving a question" refers to a system for receiving question data entered by a user.

[0758] "QR Code" refers to a type of two-dimensional barcode, a matrix code used to store information.

[0759] "Means for analysis" refers to a system that processes input data and executes a process to understand its meaning.

[0760] "Answer generation means" refers to techniques and methods for creating appropriate answers to users' questions.

[0761] "AR Guidelines" refers to visual information about how to operate and procedures for home appliances that is displayed using augmented reality technology.

[0762] The present invention provides a system for quickly and easily obtaining operating instructions and troubleshooting information for home appliances via an information terminal owned by a user. Specific embodiments of the present invention will be described below.

[0763] First, users install a dedicated application on their information terminals, such as smartphones or tablets. This application allows users to register their home appliances and obtain information on how to operate them and troubleshooting. When launching the application for the first time, users must create an account. To create an account, users must enter basic information such as their name, email address, and password.

[0764] For home appliances that users want to register, they can scan the product's QR code or manually enter the model number and serial number. The information terminal then sends the QR code information or manually entered information to the server. The server then stores the received product information in a database and prepares related operating instructions and manuals.

[0765] When a user selects a specific appliance within the application and enters a question about how to use it or troubleshooting, the question is sent to the server. The server uses a natural language processing engine to analyze the question and understand the user's intent. For example, in response to the question, "Where is the start button on my microwave?", the server searches for information about the start button and generates an appropriate answer. This answer is sent to the information terminal in text or image format and displayed to the user.

[0766] The present invention also includes an AI support chatbot that can automatically provide appropriate answers when a user inputs a question through the chat interface of the application. For example, in response to the question, "What does the E03 error on my washing machine mean?", the chatbot will search the database and provide the meaning of the error code and how to deal with it.

[0767] Furthermore, it also has a function that uses augmented reality technology to display operating procedures and guidelines for home appliances through the information terminal's camera. When a user scans an appliance using the camera, the information terminal recognizes specific parts of the product and uses AR technology to display virtual operating procedures and guidelines. This allows the user to operate the product appropriately while visually checking the operating procedures.

[0768] Specific examples

[0769] Example 1: Getting instructions for operating a microwave

[0770] The user scans the QR code to register the microwave's information in the app. Then, they enter "Where is the start button on the microwave?" in the app. The server analyzes the question and searches for the location of the start button. The server sends text and an image explaining the location of the start button to the device, which displays it. The user checks the answer, finds the start button on the microwave, and operates it.

[0771] Example 2: Dealing with washing machine error codes

[0772] The user scans the washing machine's QR code to register the product information. Then, within the app, they ask, "What does the E03 error code mean?" The server analyzes the question and searches for troubleshooting information related to the E03 error. The server then sends the meaning of the error code and the solution to the device, which displays it. The user can then follow the solution to resolve the error.

[0773] In this way, the present invention allows users to easily and quickly obtain operating instructions and troubleshooting information for home appliances.

[0774] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0775] Step 1: Install the app and create an account

[0776] User: Install the dedicated application on your smartphone or tablet. Once installation is complete, launch the app for the first time and proceed to the account creation screen. Enter basic information such as your name, email address, and password, and press the "Create Account" button.

[0777] Input: Name, Email Address, Password

[0778] Output: Encrypted account information

[0779] Terminal: Basic information entered by the user is encrypted and sent to the server.

[0780] Input: Encrypted account information

[0781] Output: Data sent to the server

[0782] Server: Receives the encrypted account information, stores it in the database as a new account, and returns a registration completion response to the device.

[0783] Input: Encrypted user information

[0784] Output: Registration complete response

[0785] Step 2: Enter your appliance information

[0786] User: Scans a QR code on a home appliance's manual with a smartphone camera or manually enters the product's model number and serial number into designated fields in the app.

[0787] Input: QR code information or model number, serial number

[0788] Output: Product information submission request

[0789] Terminal: Analyzes the information in the QR code scanned with the camera or the product information entered manually, converts the analysis results into JSON format, and sends them to the server.

[0790] Input: QR code information or manually entered product information

[0791] Output: Product information in JSON format

[0792] Server: Analyzes the received product information and searches for a corresponding database entry for the home appliance. If a matching product is found, it registers the product's details and manual data in the database.

[0793] Input: Product Information

[0794] Output: Database registration results

[0795] Step 3: Enter your operating instructions

[0796] User: Selects a specific appliance within the app and types a how-to or troubleshooting question into the text field. For example, "Where is the start button on my microwave?"

[0797] Input: Question text

[0798] Output: Request to send question data

[0799] Terminal: The entered question is sent to the server as text data.

[0800] Input: Question text

[0801] Output: Query data to the server

[0802] Step 4: Parsing the Question

[0803] Server: The received question text is sent to a natural language processing (NLP) engine for semantic analysis. For example, keywords such as "start button" and "microwave" are extracted.

[0804] Input: Question text

[0805] Output: Parsed keywords

[0806] Step 5: Generate an answer

[0807] Server: Searches the database for appropriate operation methods and troubleshooting information. Based on the analysis results of the NLP engine, generates the most appropriate response and sends it to the device as text and image data.

[0808] Input: Parsed keyword

[0809] Output: Generate text and image data

[0810] Device: Analyzes the data received from the server and displays it in the app in a format that is easy for the user to view.

[0811] Input: Text and image data

[0812] Output: Data displayed in the app

[0813] Step 6: View your answers

[0814] User: Review the in-app instructions and troubleshooting information and use the information provided to operate the home appliance appropriately.

[0815] Input: The information shown

[0816] Output: User understanding and operation of home appliances

[0817] Step 7: Use a chatbot

[0818] User: Open the in-app chat interface and type a how-to or troubleshooting question. For example, "What does error E03 on my washing machine mean?"

[0819] Input: Chat question

[0820] Output: Request to send question data

[0821] Terminal: Sends the question to the server as text data.

[0822] Input: Chat question

[0823] Output: Query data to the server

[0824] Server: The chatbot AI analyzes the received questions and generates appropriate answers, which are then sent to the device in text format.

[0825] Input: Chat question

[0826] Output: Textual response

[0827] On your device: Display the received response on the chat screen.

[0828] Input: Text response

[0829] Output: Chat screen display data

[0830] User: Checks the chatbot's response and operates or troubleshoots the appliance based on it.

[0831] Enter: the displayed answer

[0832] Output: Operate or troubleshoot home appliances

[0833] Step 8: AR operation guidance

[0834] User: Uses the app's AR feature to activate their smartphone camera and scan home appliances.

[0835] Input: Video data from home appliances

[0836] Output: Scan data

[0837] Terminal: Analyzes video of home appliances in real time and recognizes specific parts. Based on the recognition results, it uses AR technology to display operating procedures and guidelines.

[0838] Input: Video data

[0839] Output: AR display data

[0840] Server: Generates any additional information needed and sends it to the device.

[0841] Input: Request information

[0842] output: additional information data

[0843] Device: AR guidelines and operating procedures are superimposed on the image captured by the camera.

[0844] Input: Additional information data

[0845] Output: Display data overlaid on camera image

[0846] User: Check the AR guidelines and follow the instructions to operate the home appliance.

[0847] Input: AR guidelines

[0848] Output: Control of home appliances

[0849] (Application example 1)

[0850] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0851] It is difficult for users to quickly and accurately obtain operating instructions and troubleshooting information for home appliances. Conventional systems can take time to provide operating instructions and troubleshooting, potentially reducing user satisfaction. Furthermore, elderly users and those unfamiliar with technology have difficulty operating products and resolving problems.

[0852] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0853] In this invention, the server includes means for receiving information about the home appliance input by the user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through the camera of the information terminal using augmented reality technology, and means for displaying the generated answers on the information terminal, thereby enabling the user to quickly and accurately obtain operating methods and troubleshooting information for the home appliance.

[0854] (definition statement)

[0855] "Information terminal" is a general term for an electronic device that a user owns and uses to obtain operating instructions and troubleshooting information for home appliances.

[0856] "Natural language processing" is a technology that analyzes user questions, understands their intent, and generates appropriate operating procedures and troubleshooting information.

[0857] "Augmented reality technology" is a technology that uses the camera of an information terminal to display operating procedures and guidelines for home appliances overlaid on real-world images.

[0858] A "generative AI model" is an artificial intelligence technology that generates answers and guidelines to user questions.

[0859] A "prompt" is a question or instruction phrase that is input to a generative AI model.

[0860] A "QR code" is a two-dimensional code that contains information about a home appliance, and is an identification means that can be read by scanning it with the camera of an information terminal.

[0861] "AR display" refers to the use of augmented reality technology to display virtual operating procedures and guidelines on the screen of an information terminal.

[0862] "Home appliances" is a general term for electrical appliances used in ordinary households that may require instruction on operation or troubleshooting.

[0863] A "database" is a system for accumulating information about home appliances, operating procedures, and troubleshooting information, and storing them in a searchable format.

[0864] A "chat interface" is an operation screen that allows users to input questions in natural language and receive answers in an interactive format.

[0865] The present invention provides a system that allows users to easily obtain operating instructions and troubleshooting information for home appliances via their own information terminals. This system begins when the user installs a dedicated application on their smartphone. Specific embodiments of this system are described below.

[0866] System configuration

[0867] The system consists of the following main components:

[0868] 1. Information terminal: Electronic device such as a smartphone.

[0869] 2. Server: A remote computer containing a database and a natural language processing engine.

[0870] 3. Generative AI model: An artificial intelligence technology that generates appropriate answers to user questions.

[0871] Program processing

[0872] 1. Registering Home Appliances

[0873] Users install a dedicated application on their smartphone and create an account when they first start the app.

[0874] Users can either scan the QR code on the home appliance they purchased with their smartphone camera or manually enter the product's model number and serial number into the app.

[0875] The terminal analyzes the QR code or manually entered product information and sends the information to the server.

[0876] The server stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating procedures on the server.

[0877] 2. Obtaining operation instructions

[0878] Users select registered home appliances within the app and enter specific operating instructions and troubleshooting questions.

[0879] The terminal transmits the entered question to the server.

[0880] The server analyzes the question using a natural language processing (NLP) engine to understand the user's intent.

[0881] The server generates an appropriate response based on the information retrieved from the database and sends it to the terminal.

[0882] The device displays the generated answer on the screen within the app.

[0883] 3. Use AI support chatbots

[0884] Users use the app's chat interface to type in their how-to and troubleshooting questions.

[0885] The terminal sends a query to the server.

[0886] The server uses a chatbot AI to analyze the question and search for the appropriate answer from a database.

[0887] The server sends the answers generated by the chatbot AI to the terminal, where they are displayed.

[0888] 4. Providing operation guidance using AR

[0889] Users use their smartphone camera to scan home appliances.

[0890] The device analyzes the scanned image and recognizes specific parts of the home appliance.

[0891] Based on the recognized operation locations, the device uses AR technology to generate virtual operation procedures and guidelines, and requests additional information from the server if necessary.

[0892] The server generates additional information and sends it to the terminal.

[0893] The device displays operating procedures and guidelines as AR displays on the smartphone screen.

[0894] Specific examples

[0895] Example 1: Getting instructions for operating a microwave

[0896] 1. The user scans the QR code to register the microwave's information in the app.

[0897] 2. The user types in the app, "Where is the start button on the microwave?"

[0898] 3. The server analyzes the question and searches for the location of the start button.

[0899] 4. The server sends text and an image to the device explaining the location of the start button.

[0900] 5. The device will display the answer.

[0901] 6. The user checks their answer, finds the start button on the microwave, and operates it.

[0902] Example 2: Dealing with washing machine error codes

[0903] 1. The user scans the QR code on the washing machine to register the product information.

[0904] 2. The user asks within the app, "What does the E03 error code mean?"

[0905] 3. The server parses the question and searches for troubleshooting information about the E03 error.

[0906] 4. The server sends the meaning of the error code and how to deal with it to the terminal.

[0907] 5. The device will display the answer.

[0908] 6. The user follows the instructions to resolve the error.

[0909] Prompt Sentence Examples

[0910] "Please tell me how to use the microwave."

[0911] The system aims to help users operate home appliances quickly and efficiently, and is especially useful for elderly and less technologically savvy users.

[0912] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0913] Step 1:

[0914] Users install a dedicated application on their smartphone and create an account the first time they start it. At this time, the user enters basic information such as an email address and password. The information entered is stored in the application's database and used for future authentication.

[0915] Step 2:

[0916] Users can scan the QR code of a purchased home appliance with their smartphone camera or manually enter the product's model number or serial number into the app. The entered information (QR code or text) is parsed by the device and product information is extracted, allowing the home appliance to be identified.

[0917] Step 3:

[0918] The device then sends the analyzed product information to the server, which then stores the received product information in a database and prepares and stores the corresponding product manuals and operating procedures. During this process, the product information and manuals are linked.

[0919] Step 4:

[0920] Users select a registered home appliance within the app and enter specific operating instructions or troubleshooting questions. The entered questions (text) are sent from the device to the server, where a generative AI model is used to answer them.

[0921] Step 5:

[0922] The server analyzes the question using a natural language processing (NLP) engine to understand the user's intent. The server extracts keywords from the question and uses them to search a database for relevant operating procedures and troubleshooting information. The analysis results are then confirmed by a generative AI model.

[0923] Step 6:

[0924] The server generates an appropriate answer based on the information retrieved from the database, and the answer (text or image) is sent to the device.

[0925] Step 7:

[0926] The device displays the received response on the app screen, where the user can review the displayed information to understand how to operate the appliance and troubleshoot the problem.

[0927] Step 8:

[0928] Users use the app's chat interface to type in any follow-up instructions or troubleshooting questions, which are also sent from the device to the server.

[0929] Step 9:

[0930] The server uses chatbot AI to analyze the question and generate an appropriate answer. A generative AI model is used in this process to ensure the accuracy of the answer. The generated answer is then sent to the device.

[0931] Step 10:

[0932] The device displays the answer on the chat interface, and the user can check the answer and operate the home appliance as needed.

[0933] Step 11:

[0934] Users scan home appliances using their smartphone cameras, and the scanned images (image data) are analyzed by the device to identify specific parts of the appliance.

[0935] Step 12:

[0936] The device uses AR technology to generate virtual operating procedures and guidelines based on the recognized operation points, using data provided by the generative AI model, and requests additional information from the server if necessary.

[0937] Step 13:

[0938] The server generates additional information and sends it to the device. The device displays operation procedures and guidelines as AR on the smartphone screen. The user follows the AR guidelines displayed through the camera to perform the appropriate operation.

[0939] Example prompt: "Tell me how to use a microwave."

[0940] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0941] The present invention combines an emotion engine with a system for obtaining operating instructions and troubleshooting information for home appliances via a user's information terminal. This system has the function of analyzing a user's questions and providing appropriate operating instructions and troubleshooting information, as well as the function of recognizing the user's emotional state and generating a response accordingly. Specific embodiments of this system are described below.

[0942] Program processing flow (natural language explanation)

[0943] 1. Registering Home Appliances

[0944] Step 1.1: Install and register the app

[0945] Users: Install the dedicated application on their smartphone and create an account by entering personal information on the account creation screen when first starting the application.

[0946] Step 1.2: Enter your appliance information

[0947] User: Opens the app and scans the QR code of the purchased appliance with their smartphone camera or manually enters the product's model number or serial number.

[0948] Terminal: Scans the QR code, analyzes the product information (or manually entered information), and sends it to the server.

[0949] Step 1.3: Save product information

[0950] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions.

[0951] 2. Obtaining operation instructions

[0952] Step 1: Enter your question

[0953] User: Selects a registered appliance within the app and enters a specific instruction or troubleshooting question in the text box.

[0954] Terminal: Sends the entered question to the server.

[0955] Step 2: Parse the question

[0956] Server: Analyzes the question using a natural language processing (NLP) engine to understand the intent of the user's question. For example, it analyzes the question "Where is the start button on the microwave?" and identifies the location of the start button.

[0957] Step 3: Emotion recognition with the emotion engine

[0958] Device: Using a camera and microphone, the device analyzes the user's facial expressions and tone of voice, and uses an emotion engine to recognize the user's emotional state.

[0959] Step 4: Generate an answer

[0960] Server: Generates appropriate responses based on information retrieved from the database, and adjusts the tone and content of the responses depending on the emotional state recognized by the emotion engine.

[0961] Server: Sends the generated answer to the device.

[0962] Step 5: View your answers

[0963] Device: The received answer is displayed on the screen within the app.

[0964] User: Check the displayed information, understand how to operate the appliance, and operate it correctly.

[0965] 3. Use AI support chatbots

[0966] Step 1: Chat Interface

[0967] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[0968] Terminal: Sends a question to the server.

[0969] Step 2: Processing questions

[0970] Server: The chatbot AI analyzes the question and searches the database for the appropriate answer. The emotion engine also analyzes the user's emotions.

[0971] Step 3: Chatbot response

[0972] Server: The server adjusts the answers generated by the chatbot AI according to the user's emotional state and sends them to the device.

[0973] On your device: Show the response in the chat screen.

[0974] User: Review the answers provided and take action as needed.

[0975] 4. Providing operation guidance using AR

[0976] Step 1: Use the camera

[0977] User: Scans home appliances using the smartphone camera.

[0978] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[0979] Step 2: Identify the operation location

[0980] Terminal: Using AR technology, it generates virtual operating procedures and guidelines based on the recognized operation locations, and requests additional information from the server as needed.

[0981] Step 3: AR display

[0982] Server: Generates any additional information needed and sends it to the device.

[0983] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen.

[0984] User: Follow the AR guidelines displayed through the camera to operate the home appliance appropriately.

[0985] Specific examples

[0986] Example 1: Getting instructions for operating a microwave

[0987] 1. User: Scans the QR code to register the microwave's information in the app.

[0988] 2. User: In the app, type "Where is the start button on the microwave?"

[0989] 3. Server: Analyzes the question and searches for the location of the start button.

[0990] 4. Device: Uses the camera and microphone to identify the user's emotions and prepares a quick, concise response if it determines, for example, that the user is feeling impatient.

[0991] 5. Server: Sends text and an image to the device explaining the location of the start button.

[0992] 6. Terminal: Display the answer.

[0993] 7. User: Check your answers and find and operate the start button on the microwave.

[0994] Example 2: Dealing with washing machine error codes

[0995] 1. User: Scan the QR code on the washing machine to register the product information.

[0996] 2. User: Asks in the app, "What does the E03 error code mean?"

[0997] 3. Server: Parse the question and find troubleshooting information for the E03 error.

[0998] 4. Device: The emotion engine analyzes the user's emotions and determines whether they are feeling anxious or impatient.

[0999] 5. Server: Depending on the user's emotional state, the server creates a message in a calm tone explaining the meaning of the error code and how to deal with it, and sends it to the device.

[1000] 6. Terminal: Display the answer.

[1001] 7. User: Follow the solutions to resolve the error.

[1002] This system allows users to easily access their appliances' instruction manuals and troubleshooting information, and also responds to their emotional state, improving the user experience and enabling them to operate their appliances more effectively.

[1003] The processing flow will be explained below.

[1004] Specific processing flow for the home appliance instruction manual platform

[1005] 1. Registering Home Appliances

[1006] Step 1: Install and register the app

[1007] Users: Install the dedicated application on their smartphone and create an account by entering personal information on the account creation screen when first starting the application.

[1008] Step 2: Enter your appliance information

[1009] User: Opens the app and scans the QR code of the purchased appliance with their smartphone camera or manually enters the product's model number or serial number.

[1010] Terminal: Scans the QR code, analyzes the product information (or manually entered information), and sends it to the server.

[1011] Step 3: Save your product information

[1012] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions.

[1013] 2. Obtaining operation instructions

[1014] Step 1: Enter your question

[1015] User: Selects a registered appliance within the app and enters a specific instruction or troubleshooting question in the text box.

[1016] Terminal: Sends the entered question to the server.

[1017] Step 2: Parsing the Question

[1018] Server: Analyzes the question using a natural language processing (NLP) engine to understand the intent of the user's question. For example, it analyzes the question "Where is the start button on the microwave?" and identifies the location of the start button.

[1019] Step 3: Emotion recognition by the emotion engine

[1020] On the device: Using a camera and microphone, the device analyzes the user's facial expressions and tone of voice, and uses an emotion engine to recognize the user's emotional state. For example, emotions such as smile, surprise, and anxiety are identified and an index is generated.

[1021] Step 4: Generate an answer

[1022] Server: Generates appropriate answers based on information retrieved from a database, and adjusts the tone and content of the answers based on the user's perceived emotional state, for example, explaining things in a calmer tone to an anxious user.

[1023] Server: Sends the generated answer to the device.

[1024] Step 5: View your answers

[1025] Device: The received answer is displayed on the screen within the app, with an appropriate interface based on the emotion.

[1026] User: Check the displayed information, understand how to operate the appliance, and operate it correctly.

[1027] 3. Use AI support chatbots

[1028] Step 1: Chat Interface

[1029] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[1030] Terminal: Sends a question to the server.

[1031] Step 2: Processing questions

[1032] Server: The chatbot AI analyzes the question and searches for the appropriate answer from the database. The emotion engine also analyzes the user's emotions.

[1033] Step 3: Chatbot response

[1034] Server: The server adjusts the answers generated by the chatbot AI according to the user's emotional state and sends them to the device.

[1035] Device: Responses will be displayed on the chat screen, and responses to specific emotional states may be highlighted with color coding or icons.

[1036] User: Review the answers provided and take action as needed.

[1037] 4. Providing operation guidance using AR

[1038] Step 1: Using the camera

[1039] User: Scans home appliances using the smartphone camera.

[1040] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[1041] Step 2: Identify the operation location

[1042] Terminal: Using AR technology, it generates virtual operating procedures and guidelines based on the recognized operation locations, and requests additional information from the server as needed.

[1043] Step 3: AR display

[1044] Server: Generates any additional information needed and sends it to the device.

[1045] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen.

[1046] User: Follow the AR guidelines displayed through the camera to operate the home appliance appropriately.

[1047] Specific examples

[1048] Example 1: Getting instructions for operating a microwave

[1049] 1. User: Scans the QR code to register the microwave's information in the app.

[1050] 2. User: In the app, type "Where is the start button on the microwave?"

[1051] 3. Server: Analyzes the question and searches for the location of the start button.

[1052] 4. Device: Uses the camera and microphone to identify the user's emotions and prepares a quick, concise response if it determines, for example, that the user is feeling impatient.

[1053] 5. Server: Sends text and an image to the device explaining the location of the start button.

[1054] 6. Terminal: Display the answer.

[1055] 7. User: Check your answers and find and operate the start button on the microwave.

[1056] Example 2: Dealing with washing machine error codes

[1057] 1. User: Scan the QR code on the washing machine to register the product information.

[1058] 2. User: Asks in the app, "What does the E03 error code mean?"

[1059] 3. Server: Parse the question and find troubleshooting information for the E03 error.

[1060] 4. Device: The emotion engine analyzes the user's emotions and determines whether they are feeling anxious or impatient.

[1061] 5. Server: Depending on the user's emotional state, the server creates a message in a calm tone explaining the meaning of the error code and how to deal with it, and sends it to the device.

[1062] 6. Terminal: Display the answer.

[1063] 7. User: Follow the solutions to resolve the error.

[1064] This system allows users to easily access their appliances' instruction manuals and troubleshooting information, and also responds to their emotional state, improving the user experience and enabling them to operate their appliances more effectively.

[1065] Example 2

[1066] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1067] Conventional systems that provide operating instructions and troubleshooting for home appliances often prevent users from properly obtaining the information they need. They also lack the ability to respond to user emotions, making it difficult to improve the user experience. Furthermore, entering specific product information and answering questions can be cumbersome, making intuitive operation difficult.

[1068] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1069] In this invention, the server includes means for receiving information about the home appliance input by the user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through the camera of the information terminal using augmented reality technology, emotion analysis means for recognizing the emotional state of the user, and means for generating a response according to the emotional state recognized by the emotion analysis means. This allows the user to intuitively obtain the information they need and enables responses according to their emotions, which is expected to improve the user experience.

[1070] The "information terminal owned by the user" refers to a portable electronic device that the user personally owns and can operate, such as a smartphone or tablet.

[1071] "Home appliance information" refers to basic information about the home appliance, such as the product name, model number, serial number, and QR code.

[1072] "Natural language processing" is a technology that uses computers to analyze and automatically process human language, including techniques for interpreting the meaning of text and answering questions.

[1073] "Augmented reality technology" is a technology that displays virtual information overlaid on the real environment, and includes, for example, a method of displaying operation guides on the surface of home appliances that can be seen through a camera.

[1074] "Emotion analysis means" refers to technology that analyzes a user's facial expressions and tone of voice to recognize the user's emotional state, and includes, for example, a method of determining a user's emotions in real time using a camera or microphone.

[1075] A "generative AI model" is a model that uses artificial intelligence technology to automatically generate answers to user questions, and includes, for example, large-scale natural language generation models.

[1076] A "database" is a collection of structured data that systematically stores large amounts of data and makes it easy to search and update. This system stores operating procedures and troubleshooting information for home appliances.

[1077] The "means for analyzing a question" refers to a technique for receiving a question from a user and understanding its content, and includes, for example, analyzing the intent of the question using a natural language processing engine.

[1078] The "means for displaying operating procedures and guidelines" refers to a technology that uses augmented reality technology to visually display specific operating procedures and guidelines for home appliances on the screen of an information terminal.

[1079] "Means for generating answers according to the emotional state" refers to technology that generates answers with appropriate tone and content based on the user's emotional state, and includes, for example, a method for adjusting the expression of the answer based on the results of emotional analysis.

[1080] The present invention is a system for obtaining operation methods and troubleshooting information for home appliances via an information terminal owned by a user, and includes the following detailed embodiments.

[1081] This system not only allows users to obtain operating instructions and troubleshooting information for home appliances through their information terminal, but also recognizes the user's emotional state and responds individually. The main hardware used includes smartphones and tablets. The main software used includes a natural language processing engine, augmented reality technology, an emotion analysis engine, and a generative AI model.

[1082] The server first receives information about home appliances entered by the user. The user installs a dedicated application on their smartphone, creates an account when they first start it, and then obtains this information by scanning the QR code of the purchased home appliance or manually entering the product's model number and serial number.

[1083] The device analyzes the received information about the home appliance and sends it to the server. The server stores the received information in a database and prepares and saves the relevant product manuals and operating procedures. This allows users to easily obtain the information they need.

[1084] Next, when the user enters a question about a home appliance within the app, the device sends the question to the server. The server then uses a natural language processing engine (such as Google Cloud Natural Language API or Microsoft Azure Text Analytics) to analyze the question and understand the user's intent. For example, the server can understand the question, "Where is the start button on the microwave?" and retrieve the corresponding information.

[1085] The device also analyzes the user's facial expressions and tone of voice to recognize the user's emotional state using an emotion analysis engine (e.g., Affectiva's emotion analysis API). For example, if the user is feeling anxious, the emotion engine will detect that the user's voice is high-pitched and fast-paced.

[1086] The server extracts appropriate operating procedures and troubleshooting information from the database and adjusts the tone and content of the response based on the results of sentiment analysis. For example, if the user is impatient, it will provide a concise and quick response. Appropriate responses are automatically generated using a generative AI model (e.g., OpenAI's GPT-3). The server sends the generated response to the device, which then displays it on the app screen.

[1087] The AI ​​support chatbot can be used when users input questions about operation methods or troubleshooting. The chatbot analyzes the question using a natural language processing engine, recognizes the user's emotional state using an emotion analysis engine, and responds appropriately. For example, the chatbot generates an appropriate answer for the question, "What should I do if my washing machine displays an E03 error?"

[1088] Furthermore, it also includes a function that uses augmented reality technology to display operating procedures and guidelines for home appliances through the information terminal's camera. When a user scans an appliance with their smartphone's camera, the terminal displays virtual guidelines for operation points based on the recognized information. This makes it easier for users to intuitively understand operating procedures.

[1089] Specific examples

[1090] Example 1: Getting instructions for operating a microwave

[1091] User: Scan the QR code to register the microwave's information in the app.

[1092] User: Within the app, type "Where is the start button on the microwave?"

[1093] Server: Analyzes the question and searches for the location of the start button.

[1094] On the device: Use the camera and microphone to identify the user's emotions and, for example, prepare a quick, concise response if it detects impatience.

[1095] Server: Sends text and an image to the device explaining the location of the start button.

[1096] Terminal: Display the answer.

[1097] User: Check your answers, find the start button on the microwave, and operate it.

[1098] Example 2: Dealing with washing machine error codes

[1099] User: Scan the QR code on the washing machine to register the product information.

[1100] User: Asks within the app, "What does the E03 error code mean?"

[1101] Server: Parse the question and find troubleshooting information for the E03 error.

[1102] Device: The emotion engine analyzes the user's emotions and determines whether they are feeling anxious or impatient.

[1103] Server: Depending on the user's emotional state, the server creates a message in a calm tone explaining the meaning of the error code and how to deal with it, and sends it to the device.

[1104] Terminal: Display the answer.

[1105] User: Follow the solutions to resolve the error.

[1106] The above-described configuration allows users to quickly and intuitively obtain operating instructions and troubleshooting information for home appliances, and also enables responses that correspond to the user's emotional state, significantly improving the user experience.

[1107] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1108] Specific processing flow of the system program

[1109] Step 1:

[1110] Installing the app and creating an account

[1111] User: Install the dedicated application on their smartphone, launch the app, and create an account by entering their first and last name, email address, password, etc. on the account creation screen. Then, click on the confirmation link sent to their email address to confirm their account.

[1112] Input: User's first name, last name, email address, and password

[1113] Output: Created user account information

[1114] Terminal: Receives information entered by the user and sends it to the server.

[1115] Server: Stores the received account information in a database.

[1116] Step 2:

[1117] Entering information about home appliances

[1118] User: Opens the app and scans the QR code of the purchased appliance or manually enters the product's model number and serial number.

[1119] Input: QR code information or product model number, serial number of the home appliance

[1120] Output: Parsed product information

[1121] Terminal: Scans the QR code to obtain product information (or manually entered information), parses it into a string, and sends it to the server.

[1122] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions.

[1123] Step 3:

[1124] Saving home appliance information

[1125] Server: Based on the registered home appliance information, the server references the database to find and store relevant product manuals and operating instructions.

[1126] Input: Registered home appliance information

[1127] Output: Database record of applicable product manuals and operating procedures

[1128] Specific operations: Performs data format conversion and tagging operations in the process of storing product information in a database.

[1129] Step 4:

[1130] User question input

[1131] User: Selects a registered appliance in the app and types a specific how-to or troubleshooting question into the text box, such as "Where is the start button on my microwave?"

[1132] Input: User question text

[1133] Output: The question sent to the server

[1134] Terminal: Sends the entered question to the server.

[1135] Step 5:

[1136] Question Analysis

[1137] Server: Analyzes the received question using a natural language processing (NLP) engine to identify the intent of the question, for example by breaking down the text into tokens and extracting keywords.

[1138] Input: Question text received from the user

[1139] Output: Question intent and keywords

[1140] Specific operation: Uses natural language processing API to perform text analysis and extract the intent of the question.

[1141] Step 6:

[1142] Emotion analysis

[1143] Device: Captures the user's facial expressions and tone of voice in real time and uses an emotion analysis engine to recognize their emotional state.

[1144] Input: User's facial expression data, tone of voice

[1145] Output: User's emotional state (e.g., impatience, anxiety, joy, etc.)

[1146] Specific operation: Using a camera and microphone, the user's facial expressions and voice are input into an emotion analysis engine to analyze their emotional state.

[1147] Step 7:

[1148] Answer generation and coordination

[1149] Server: Generates appropriate responses based on information retrieved from a database and adjusts the tone and content of responses based on sentiment analysis results. For example, prepare a concise and quick response for a user who is impatient. Uses a generative AI model.

[1150] Input: Question intent analyzed by natural language processing, sentiment analysis results, and answer candidates from the database

[1151] Output: Adjusted final answer

[1152] What it does: Uses a generative AI model to generate answers and tailor them appropriately based on sentiment analysis.

[1153] Step 8:

[1154] Submitting and viewing answers

[1155] Server: Sends the generated answer to the device.

[1156] Input: Adjusted final answer

[1157] Output: The answer sent to the user's device

[1158] Device: The received answer is displayed on the screen within the app.

[1159] User: Check the displayed information, understand how to operate the appliance, and operate it appropriately.

[1160] Step 9:

[1161] Use of AI support chatbots

[1162] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[1163] Input: User question text

[1164] Output: The question sent to the server

[1165] Terminal: Sends a question to the server.

[1166] Server: Analyzes the question, identifies the problem, and sends the answer generated by the chatbot AI to the device, analyzing the emotional state accordingly.

[1167] Step 10:

[1168] AR operation guidance

[1169] User: Uses the camera to scan home appliances and display operating instructions guidelines.

[1170] Input: Camera scan of product

[1171] Output: AR guidelines displayed on the smartphone screen

[1172] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[1173] Server: Generates any additional information needed and sends it to the device.

[1174] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen. Users follow the AR guidelines to operate home appliances appropriately.

[1175] These specific actions allow users to intuitively obtain operating procedures and troubleshooting information, and respond in accordance with their emotions.

[1176] (Application example 2)

[1177] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1178] Conventional methods for obtaining operating instructions and troubleshooting information for home appliances require users to refer to instruction manuals, which is time-consuming and inefficient. Furthermore, information is provided without taking into account the user's emotional state, which can amplify negative emotions such as anxiety and impatience. Furthermore, information acquisition is often limited to text-based information, and visual explanations can be difficult. These problems need to be solved.

[1179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1180] In this invention, the server includes means for receiving information about the home appliance input by the user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through the camera of the information terminal using augmented reality technology, and means for analyzing the user's emotional state using an emotion recognition engine and adjusting responses based on the results. This enables the user to efficiently and appropriately obtain operating methods and troubleshooting information for the home appliance and provides responses that take the user's emotional state into consideration.

[1181] An "information terminal" is a device owned by a user that connects to the Internet and various applications to acquire and process information.

[1182] "Consumer appliances" refers to electrical and electronic devices used in the home that require instruction on how to operate or troubleshoot.

[1183] "Natural language processing" is a language understanding and generation technology that analyzes questions and instructions entered by users and generates appropriate responses based on them.

[1184] "Augmented reality technology" is a technology that displays virtual information overlaid on real-world images through the camera of an information terminal.

[1185] An "emotion recognition engine" is a technology that analyzes a user's facial expressions, tone of voice, etc. to estimate their emotional state.

[1186] An "AI support chatbot" is a conversational interface that uses artificial intelligence to automatically generate appropriate responses to user questions.

[1187] A "QR code" is a type of two-dimensional barcode that can efficiently read data including product information.

[1188] A "server" is a computer system that provides data and performs computations over a network.

[1189] This invention is a system for obtaining operating instructions and troubleshooting information for home appliances via a user's information terminal. The system receives information about the home appliance entered by the user, analyzes the user's question using natural language processing, and generates relevant operating instructions and troubleshooting information. It also uses augmented reality technology to display operating instructions and guidelines for the home appliance through the information terminal's camera, analyzes the user's emotional state using an emotion recognition engine, and adjusts responses based on that.

[1190] The hardware used is an information terminal such as a smartphone or tablet equipped with an internet connection, a camera, and a microphone. The software used includes Flask as a web framework, Hugging Face's Transformers as a natural language processing engine, and Face Recognition and Numpy for facial recognition and emotion analysis.

[1191] The server receives information about home appliances entered by the user and stores it in a database. When the user enters a question, the server analyzes the question using a natural language processing engine and generates relevant information. The generated information is then further adjusted according to the user's emotional state using an emotion recognition engine.

[1192] As a concrete example, consider a case where a user scans a QR code on an electronic product, registers the information in an application, and then enters a question about operation or troubleshooting. The server analyzes the question and generates an appropriate answer. If the user is feeling anxious or impatient, the emotion recognition engine will detect that emotion and provide an answer in a gentler tone accordingly.

[1193] An example of a prompt sentence for the generative AI model is as follows:

[1194] prompt:

[1195] A user asks, "What does washing machine error code E03 mean?" You can see the anxiety in their face. Generate a relaxing answer.

[1196] answer:

[1197] "The E03 error code is a drainage error. This can often be resolved by cleaning the filter. If you have any questions, please let us know. We'll fix it right away."

[1198] This system allows users to obtain information more efficiently than traditional methods of obtaining operating instructions or troubleshooting, and provides a better user experience by receiving responses that take into account their emotional state.

[1199] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1200] Step 1:

[1201] The user installs the dedicated application on their information terminal and starts it. When they first start it, they enter their personal information to create an account. During this process, the user uses their smartphone, and the account information is sent to the server and stored in a database.

[1202] Input: Personal information

[1203] Output: Account information stored on the server

[1204] Step 2:

[1205] A user scans the QR code of a purchased home appliance with the smartphone camera. Product information (model number, serial number, etc.) is obtained from the scanned QR code and sent to the server.

[1206] Input: QR code of home appliance

[1207] Output: Product information stored on the server

[1208] Step 3:

[1209] The server stores the received information about the home appliances in a database and prepares the relevant product manuals and operating procedures. This allows information about the home appliances owned by the user to be registered in the system.

[1210] Input:Product Information

[1211] Output: Product information stored in a database

[1212] Step 4:

[1213] The user uses the information terminal to enter a question about the registered home appliance into a text box within the app, and the question is sent to the server.

[1214] Input: User question

[1215] Output: The query data sent to the server

[1216] Step 5:

[1217] The server receives the question data and analyzes it using a natural language processing (NLP) engine, which identifies the intent of the question and prepares it for answer generation.

[1218] Input: Question data

[1219] Output: Analysis result (question intent)

[1220] Step 6:

[1221] The device uses the smartphone's camera and microphone to capture the user's facial expressions and tone of voice, and then uses an emotion recognition engine to analyze the user's emotional state. The analysis results are then sent to a server.

[1222] Input: User's facial and voice data

[1223] Output: Parsed emotional state data

[1224] Step 7:

[1225] The server searches a database for relevant operating procedures and troubleshooting information, and tailors responses based on the analysis by an emotion-recognition engine. For example, a user who is feeling anxious or impatient will receive a gentler response.

[1226] Input: Analysis results, emotional state data

[1227] Output: Adjusted response data

[1228] Step 8:

[1229] The server sends the generated answer data to the information terminal, which then displays the answer to the user. The user checks the displayed information and operates the home appliance appropriately.

[1230] Input: Adjusted response data

[1231] Output: Answer displayed on terminal

[1232] Step 9:

[1233] The user uses the camera on the information terminal to scan the home appliance, and the operating procedures and guidelines are displayed on the screen using augmented reality technology, allowing the user to visually confirm how to operate the appliance.

[1234] Input: Video data from home appliances

[1235] Output: AR guidelines displayed on the screen

[1236] In this way, users can efficiently and safely learn how to operate and troubleshoot their home appliances. It is designed to enhance the user experience based on step-by-step inputs and outputs.

[1237] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1238] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1239] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1240] [Third embodiment]

[1241] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1242] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1243] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1244] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1245] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1246] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1247] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1248] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1249] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1250] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1251] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1252] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1253] The present invention provides a system that allows users to easily obtain operating instructions and troubleshooting information for home appliances via their own information terminals. Specific embodiments of this system will be described below.

[1254] Program processing flow (natural language explanation)

[1255] 1. Registering Home Appliances

[1256] Step 1.1: Install and register the app

[1257] User: Install the dedicated application on your smartphone and create an account the first time you start it. When creating the account, you enter basic user information.

[1258] Step 1.2: Enter your appliance information

[1259] User: Scans the QR code on the appliance they purchased with their smartphone camera or manually enters the product's model number and serial number into the app.

[1260] Terminal: Parses the QR code or manually entered product information and sends it to the server.

[1261] Step 1.3: Save product information

[1262] Server: Stores the received information about the home appliances in a database. Also, prepares and stores the relevant product manuals and operating procedures on the server.

[1263] 2. Obtaining operation instructions

[1264] Step 2.1: Enter your question

[1265] Users: Select their registered appliances within the app and enter their specific operating instructions or troubleshooting questions.

[1266] Terminal: Sends the entered question to the server.

[1267] Step 2.2: Parse the question

[1268] Server: Analyzes the question using a natural language processing (NLP) engine and understands the user's intent. For example, in response to the question "Where is the start button on the microwave?", it searches for information about the location of the start button.

[1269] Step 2.3: Generate an answer

[1270] Server: Generates an appropriate answer based on the information retrieved from the database and sends the generated answer to the device.

[1271] Step 2.4: View your answers

[1272] Device: Display the generated answer on a screen within the app.

[1273] User: By checking the displayed information, the user understands how to operate the home appliance and operates it appropriately.

[1274] 3. Use AI support chatbots

[1275] Step 3.1: Chat Interface

[1276] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[1277] Terminal: Sends a question to the server.

[1278] Step 3.2: Processing Questions

[1279] Server: The chatbot AI analyzes the question and searches the database for the appropriate answer.

[1280] Step 3.3: Chatbot response

[1281] Server: Sends the answers generated by the chatbot AI to the device.

[1282] On your device: Show the response in the chat screen.

[1283] User: Review the answers provided and take action as needed.

[1284] 4. Providing operation guidance using AR

[1285] Step 4.1: Using the Camera

[1286] User: Uses smartphone camera to scan home appliances.

[1287] Terminal: Analyzes scanned images and recognizes specific parts of home appliances.

[1288] Step 4.2: Identify the operation location

[1289] Terminal: Based on the recognized operation locations, the terminal uses AR technology to generate virtual operation procedures and guidelines, and requests additional information from the server as needed.

[1290] Step 4.3: AR display

[1291] Server: Generates additional information and sends it to the device.

[1292] Device: Operating procedures and guidelines are displayed as AR on the smartphone screen.

[1293] User: Follow the AR guidelines displayed through the camera and perform the appropriate operation.

[1294] Specific examples

[1295] Example 1: Getting instructions for operating a microwave

[1296] 1. User: Scans the QR code to register the microwave's information in the app.

[1297] 2. User: In the app, type "Where is the start button on the microwave?"

[1298] 3. Server: Analyzes the question and searches for the location of the start button.

[1299] 4. Server: Sends text and an image to the device explaining the location of the start button.

[1300] 5. Terminal: Display the answer.

[1301] 6. User: Check your answers and find and operate the start button on the microwave.

[1302] Example 2: Dealing with washing machine error codes

[1303] 1. User: Scan the QR code on the washing machine to register the product information.

[1304] 2. User: Asks in the app, "What does the E03 error code mean?"

[1305] 3. Server: Parse the question and find troubleshooting information for the E03 error.

[1306] 4. Server: Sends the meaning of the error code and how to deal with it to the device.

[1307] 5. Terminal: Display the answer.

[1308] 6. User: Follow the solutions to resolve the error.

[1309] These specific examples allow users to easily access the instruction manuals for home appliances, enabling them to quickly understand how to operate them and troubleshoot problems. This system is particularly useful for products that require complex operation and for elderly users.

[1310] The processing flow will be explained below.

[1311] Specific processing flow for the home appliance instruction manual platform

[1312] 1. Registering Home Appliances

[1313] Step 1: Install and register the app

[1314] Users: Install the dedicated application on their smartphone and create an account by entering personal information on the account creation screen when first starting the application.

[1315] Step 2: Enter your appliance information

[1316] User: Opens the app and scans the QR code of the purchased appliance with their smartphone camera or manually enters the product's model number or serial number.

[1317] Terminal: Scans the QR code, analyzes the product information (or manually entered information), and sends it to the server.

[1318] Step 3: Save your product information

[1319] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions in the database.

[1320] 2. Obtaining operation instructions

[1321] Step 1: Enter your question

[1322] User: Selects a registered appliance within the app and enters a specific instruction or troubleshooting question in the text box.

[1323] Terminal: Sends the entered question to the server.

[1324] Step 2: Parsing the Question

[1325] Server: Analyzes the question using a natural language processing (NLP) engine to understand the intent of the user's question. For example, it analyzes the question "Where is the start button on the microwave?" and identifies the location of the start button.

[1326] Step 3: Generate an answer

[1327] Server: Generates an appropriate response based on the information retrieved from the database and sends it to the device.

[1328] Step 4: View your answers

[1329] Device: The received answer is displayed on the screen within the app.

[1330] User: Check the displayed information, understand how to operate the appliance, and operate it correctly.

[1331] 3. Use AI support chatbots

[1332] Step 1: Chat Interface

[1333] Users: Use the app's chat interface to type in instructions and troubleshooting questions.

[1334] Terminal: Sends a question to the server.

[1335] Step 2: Processing questions

[1336] Server: The chatbot AI analyzes the question and searches for corresponding information from the database.

[1337] Step 3: Chatbot response

[1338] Server: Sends the answers generated by the chatbot AI to the device.

[1339] On your device: Show the response in the chat screen.

[1340] User: Review the answers provided and take action as needed.

[1341] 4. Providing operation guidance using AR

[1342] Step 1: Using the camera

[1343] User: Uses smartphone camera to scan home appliances.

[1344] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[1345] Step 2: Identify the operation location

[1346] Terminal: Based on the recognized operation points, the device uses AR technology to generate virtual operation procedures and guidelines, and requests additional information from the server as needed.

[1347] Step 3: AR display

[1348] Server: Generates any additional information needed and sends it to the device.

[1349] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen.

[1350] User: Follow the AR guidelines displayed through the camera to operate the home appliance appropriately.

[1351] In this way, through each step, the user can easily obtain the instruction manual and troubleshooting information for the home appliance and perform the operation appropriately.

[1352] Example 1

[1353] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1354] Quickly and easily obtaining operating instructions and troubleshooting information for home appliances is a major challenge for many users. For elderly users and those less familiar with technology, simply reading instruction manuals can be difficult, making access to appropriate information crucial. Furthermore, the increasing diversity of products creates a need for efficient methods to provide specific information for individual products.

[1355] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1356] In this invention, the server includes means for receiving information about a home appliance input by a user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through a camera of the information terminal using augmented reality technology, means for saving information about the home appliance in a database, means for managing user account information, and means for receiving a question from an information terminal owned by the user, analyzing the question by the server to generate an appropriate answer, and transmitting the answer back to the information terminal. This allows the user to easily and quickly access operating methods and troubleshooting information for the home appliance.

[1357] "User" refers to an individual who uses a home appliance and operates an information terminal.

[1358] An "information terminal" refers to a device, such as a smartphone or tablet, that has Internet connectivity and can be operated by the user.

[1359] "Home appliances" refers to electronic devices and electrical appliances used in the home.

[1360] A "server" refers to a computer system that receives requests from users over a network, processes the information, and returns it.

[1361] "Database" refers to an information management system that efficiently manages large amounts of data and enables quick access to it.

[1362] "Natural language processing" refers to the technology that enables computers to understand, analyze, and generate human language.

[1363] "Augmented reality technology" refers to a technology that displays virtual information overlaid on images captured in real time.

[1364] "Operating procedures and guidelines" refers to information that explains the steps and methods required to use a home appliance.

[1365] An "AI support chatbot" refers to a program that uses artificial intelligence technology to automatically provide answers to users' questions.

[1366] The term "means for receiving a question" refers to a system for receiving question data entered by a user.

[1367] "QR Code" refers to a type of two-dimensional barcode, a matrix code used to store information.

[1368] "Means for analysis" refers to a system that processes input data and executes a process to understand its meaning.

[1369] "Answer generation means" refers to techniques and methods for creating appropriate answers to users' questions.

[1370] "AR Guidelines" refers to visual information about how to operate and procedures for home appliances that is displayed using augmented reality technology.

[1371] The present invention provides a system for quickly and easily obtaining operating instructions and troubleshooting information for home appliances via an information terminal owned by a user. Specific embodiments of the present invention will be described below.

[1372] First, users install a dedicated application on their information terminals, such as smartphones or tablets. This application allows users to register their home appliances and obtain information on how to operate them and troubleshooting. When launching the application for the first time, users must create an account. To create an account, users must enter basic information such as their name, email address, and password.

[1373] For home appliances that users want to register, they can scan the product's QR code or manually enter the model number and serial number. The information terminal then sends the QR code information or manually entered information to the server. The server then stores the received product information in a database and prepares related operating instructions and manuals.

[1374] When a user selects a specific appliance within the application and enters a question about how to use it or troubleshooting, the question is sent to the server. The server uses a natural language processing engine to analyze the question and understand the user's intent. For example, in response to the question, "Where is the start button on my microwave?", the server searches for information about the start button and generates an appropriate answer. This answer is sent to the information terminal in text or image format and displayed to the user.

[1375] The present invention also includes an AI support chatbot that can automatically provide appropriate answers when a user inputs a question through the chat interface of the application. For example, in response to the question, "What does the E03 error on my washing machine mean?", the chatbot will search the database and provide the meaning of the error code and how to deal with it.

[1376] Furthermore, it also has a function that uses augmented reality technology to display operating procedures and guidelines for home appliances through the information terminal's camera. When a user scans an appliance using the camera, the information terminal recognizes specific parts of the product and uses AR technology to display virtual operating procedures and guidelines. This allows the user to operate the product appropriately while visually checking the operating procedures.

[1377] Specific examples

[1378] Example 1: Getting instructions for operating a microwave

[1379] The user scans the QR code to register the microwave's information in the app. Then, they enter "Where is the start button on the microwave?" in the app. The server analyzes the question and searches for the location of the start button. The server sends text and an image explaining the location of the start button to the device, which displays it. The user checks the answer, finds the start button on the microwave, and operates it.

[1380] Example 2: Dealing with washing machine error codes

[1381] The user scans the washing machine's QR code to register the product information. Then, within the app, they ask, "What does the E03 error code mean?" The server analyzes the question and searches for troubleshooting information related to the E03 error. The server then sends the meaning of the error code and the solution to the device, which displays it. The user can then follow the solution to resolve the error.

[1382] In this way, the present invention allows users to easily and quickly obtain operating instructions and troubleshooting information for home appliances.

[1383] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1384] Step 1: Install the app and create an account

[1385] User: Install the dedicated application on your smartphone or tablet. Once installation is complete, launch the app for the first time and proceed to the account creation screen. Enter basic information such as your name, email address, and password, and press the "Create Account" button.

[1386] Input: Name, Email Address, Password

[1387] Output: Encrypted account information

[1388] Terminal: Basic information entered by the user is encrypted and sent to the server.

[1389] Input: Encrypted account information

[1390] Output: Data sent to the server

[1391] Server: Receives the encrypted account information, stores it in the database as a new account, and returns a registration completion response to the device.

[1392] Input: Encrypted user information

[1393] Output: Registration complete response

[1394] Step 2: Enter your appliance information

[1395] User: Scans a QR code on a home appliance's manual with a smartphone camera or manually enters the product's model number and serial number into designated fields in the app.

[1396] Input: QR code information or model number, serial number

[1397] Output: Product information submission request

[1398] Terminal: Analyzes the information in the QR code scanned with the camera or the product information entered manually, converts the analysis results into JSON format, and sends them to the server.

[1399] Input: QR code information or manually entered product information

[1400] Output: Product information in JSON format

[1401] Server: Analyzes the received product information and searches for a corresponding database entry for the home appliance. If a matching product is found, it registers the product's details and manual data in the database.

[1402] Input: Product Information

[1403] Output: Database registration results

[1404] Step 3: Enter your operating instructions

[1405] User: Selects a specific appliance within the app and types a how-to or troubleshooting question into the text field. For example, "Where is the start button on my microwave?"

[1406] Input: Question text

[1407] Output: Request to send question data

[1408] Terminal: The entered question is sent to the server as text data.

[1409] Input: Question text

[1410] Output: Query data to the server

[1411] Step 4: Parsing the Question

[1412] Server: The received question text is sent to a natural language processing (NLP) engine for semantic analysis. For example, keywords such as "start button" and "microwave" are extracted.

[1413] Input: Question text

[1414] Output: Parsed keywords

[1415] Step 5: Generate an answer

[1416] Server: Searches the database for appropriate operation methods and troubleshooting information. Based on the analysis results of the NLP engine, generates the most appropriate response and sends it to the device as text and image data.

[1417] Input: Parsed keyword

[1418] Output: Generate text and image data

[1419] Device: Analyzes the data received from the server and displays it in the app in a format that is easy for the user to view.

[1420] Input: Text and image data

[1421] Output: Data displayed in the app

[1422] Step 6: View your answers

[1423] User: Review the in-app instructions and troubleshooting information and use the information provided to operate the home appliance appropriately.

[1424] Input: The information shown

[1425] Output: User understanding and operation of home appliances

[1426] Step 7: Use a chatbot

[1427] User: Open the in-app chat interface and type a how-to or troubleshooting question. For example, "What does error E03 on my washing machine mean?"

[1428] Input: Chat question

[1429] Output: Request to send question data

[1430] Terminal: Sends the question to the server as text data.

[1431] Input: Chat question

[1432] Output: Query data to the server

[1433] Server: The chatbot AI analyzes the received questions and generates appropriate answers, which are then sent to the device in text format.

[1434] Input: Chat question

[1435] Output: Textual response

[1436] On your device: Display the received response on the chat screen.

[1437] Input: Text response

[1438] Output: Chat screen display data

[1439] User: Checks the chatbot's response and operates or troubleshoots the appliance based on it.

[1440] Enter: the displayed answer

[1441] Output: Operate or troubleshoot home appliances

[1442] Step 8: AR operation guidance

[1443] User: Uses the app's AR feature to activate their smartphone camera and scan home appliances.

[1444] Input: Video data from home appliances

[1445] Output: Scan data

[1446] Terminal: Analyzes video of home appliances in real time and recognizes specific parts. Based on the recognition results, it uses AR technology to display operating procedures and guidelines.

[1447] Input: Video data

[1448] Output: AR display data

[1449] Server: Generates any additional information needed and sends it to the device.

[1450] Input: Request information

[1451] output: additional information data

[1452] Device: AR guidelines and operating procedures are superimposed on the image captured by the camera.

[1453] Input: Additional information data

[1454] Output: Display data overlaid on camera image

[1455] User: Check the AR guidelines and follow the instructions to operate the home appliance.

[1456] Input: AR guidelines

[1457] Output: Control of home appliances

[1458] (Application example 1)

[1459] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1460] It is difficult for users to quickly and accurately obtain operating instructions and troubleshooting information for home appliances. Conventional systems can take time to provide operating instructions and troubleshooting, potentially reducing user satisfaction. Furthermore, elderly users and those unfamiliar with technology have difficulty operating products and resolving problems.

[1461] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1462] In this invention, the server includes means for receiving information about the home appliance input by the user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through the camera of the information terminal using augmented reality technology, and means for displaying the generated answers on the information terminal, thereby enabling the user to quickly and accurately obtain operating methods and troubleshooting information for the home appliance.

[1463] (definition statement)

[1464] "Information terminal" is a general term for an electronic device that a user owns and uses to obtain operating instructions and troubleshooting information for home appliances.

[1465] "Natural language processing" is a technology that analyzes user questions, understands their intent, and generates appropriate operating procedures and troubleshooting information.

[1466] "Augmented reality technology" is a technology that uses the camera of an information terminal to display operating procedures and guidelines for home appliances overlaid on real-world images.

[1467] A "generative AI model" is an artificial intelligence technology that generates answers and guidelines to user questions.

[1468] A "prompt" is a question or instruction phrase that is input to a generative AI model.

[1469] A "QR code" is a two-dimensional code that contains information about a home appliance, and is an identification means that can be read by scanning it with the camera of an information terminal.

[1470] "AR display" refers to the use of augmented reality technology to display virtual operating procedures and guidelines on the screen of an information terminal.

[1471] "Home appliances" is a general term for electrical appliances used in ordinary households that may require instruction on operation or troubleshooting.

[1472] A "database" is a system for accumulating information about home appliances, operating procedures, and troubleshooting information, and storing them in a searchable format.

[1473] A "chat interface" is an operation screen that allows users to input questions in natural language and receive answers in an interactive format.

[1474] The present invention provides a system that allows users to easily obtain operating instructions and troubleshooting information for home appliances via their own information terminals. This system begins when the user installs a dedicated application on their smartphone. Specific embodiments of this system are described below.

[1475] System configuration

[1476] The system consists of the following main components:

[1477] 1. Information terminal: Electronic device such as a smartphone.

[1478] 2. Server: A remote computer containing a database and a natural language processing engine.

[1479] 3. Generative AI model: An artificial intelligence technology that generates appropriate answers to user questions.

[1480] Program processing

[1481] 1. Registering Home Appliances

[1482] Users install a dedicated application on their smartphone and create an account when they first start the app.

[1483] Users can either scan the QR code on the home appliance they purchased with their smartphone camera or manually enter the product's model number and serial number into the app.

[1484] The terminal analyzes the QR code or manually entered product information and sends the information to the server.

[1485] The server stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating procedures on the server.

[1486] 2. Obtaining operation instructions

[1487] Users select registered home appliances within the app and enter specific operating instructions and troubleshooting questions.

[1488] The terminal transmits the entered question to the server.

[1489] The server analyzes the question using a natural language processing (NLP) engine to understand the user's intent.

[1490] The server generates an appropriate response based on the information retrieved from the database and sends it to the terminal.

[1491] The device displays the generated answer on the screen within the app.

[1492] 3. Use AI support chatbots

[1493] Users use the app's chat interface to type in their how-to and troubleshooting questions.

[1494] The terminal sends a query to the server.

[1495] The server uses a chatbot AI to analyze the question and search for the appropriate answer from a database.

[1496] The server sends the answers generated by the chatbot AI to the terminal, where they are displayed.

[1497] 4. Providing operation guidance using AR

[1498] Users use their smartphone camera to scan home appliances.

[1499] The device analyzes the scanned image and recognizes specific parts of the home appliance.

[1500] Based on the recognized operation locations, the device uses AR technology to generate virtual operation procedures and guidelines, and requests additional information from the server if necessary.

[1501] The server generates additional information and sends it to the terminal.

[1502] The device displays operating procedures and guidelines as AR displays on the smartphone screen.

[1503] Specific examples

[1504] Example 1: Getting instructions for operating a microwave

[1505] 1. The user scans the QR code to register the microwave's information in the app.

[1506] 2. The user types in the app, "Where is the start button on the microwave?"

[1507] 3. The server analyzes the question and searches for the location of the start button.

[1508] 4. The server sends text and an image to the device explaining the location of the start button.

[1509] 5. The device will display the answer.

[1510] 6. The user checks their answer, finds the start button on the microwave, and operates it.

[1511] Example 2: Dealing with washing machine error codes

[1512] 1. The user scans the QR code on the washing machine to register the product information.

[1513] 2. The user asks within the app, "What does the E03 error code mean?"

[1514] 3. The server parses the question and searches for troubleshooting information about the E03 error.

[1515] 4. The server sends the meaning of the error code and how to deal with it to the terminal.

[1516] 5. The device will display the answer.

[1517] 6. The user follows the instructions to resolve the error.

[1518] Prompt Sentence Examples

[1519] "Please tell me how to use the microwave."

[1520] The system aims to help users operate home appliances quickly and efficiently, and is especially useful for elderly and less technologically savvy users.

[1521] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1522] Step 1:

[1523] Users install a dedicated application on their smartphone and create an account the first time they start it. At this time, the user enters basic information such as an email address and password. The information entered is stored in the application's database and used for future authentication.

[1524] Step 2:

[1525] Users can scan the QR code of a purchased home appliance with their smartphone camera or manually enter the product's model number or serial number into the app. The entered information (QR code or text) is parsed by the device and product information is extracted, allowing the home appliance to be identified.

[1526] Step 3:

[1527] The device then sends the analyzed product information to the server, which then stores the received product information in a database and prepares and stores the corresponding product manuals and operating procedures. During this process, the product information and manuals are linked.

[1528] Step 4:

[1529] Users select a registered home appliance within the app and enter specific operating instructions or troubleshooting questions. The entered questions (text) are sent from the device to the server, where a generative AI model is used to answer them.

[1530] Step 5:

[1531] The server analyzes the question using a natural language processing (NLP) engine to understand the user's intent. The server extracts keywords from the question and uses them to search a database for relevant operating procedures and troubleshooting information. The analysis results are then confirmed by a generative AI model.

[1532] Step 6:

[1533] The server generates an appropriate answer based on the information retrieved from the database, and the answer (text or image) is sent to the device.

[1534] Step 7:

[1535] The device displays the received response on the app screen, where the user can review the displayed information to understand how to operate the appliance and troubleshoot the problem.

[1536] Step 8:

[1537] Users use the app's chat interface to type in any follow-up instructions or troubleshooting questions, which are also sent from the device to the server.

[1538] Step 9:

[1539] The server uses chatbot AI to analyze the question and generate an appropriate answer. A generative AI model is used in this process to ensure the accuracy of the answer. The generated answer is then sent to the device.

[1540] Step 10:

[1541] The device displays the answer on the chat interface, and the user can check the answer and operate the home appliance as needed.

[1542] Step 11:

[1543] Users scan home appliances using their smartphone cameras, and the scanned images (image data) are analyzed by the device to identify specific parts of the appliance.

[1544] Step 12:

[1545] The device uses AR technology to generate virtual operating procedures and guidelines based on the recognized operation points, using data provided by the generative AI model, and requests additional information from the server if necessary.

[1546] Step 13:

[1547] The server generates additional information and sends it to the device. The device displays operation procedures and guidelines as AR on the smartphone screen. The user follows the AR guidelines displayed through the camera to perform the appropriate operation.

[1548] Example prompt: "Tell me how to use a microwave."

[1549] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1550] The present invention combines an emotion engine with a system for obtaining operating instructions and troubleshooting information for home appliances via a user's information terminal. This system has the function of analyzing a user's questions and providing appropriate operating instructions and troubleshooting information, as well as the function of recognizing the user's emotional state and generating a response accordingly. Specific embodiments of this system are described below.

[1551] Program processing flow (natural language explanation)

[1552] 1. Registering Home Appliances

[1553] Step 1.1: Install and register the app

[1554] Users: Install the dedicated application on their smartphone and create an account by entering personal information on the account creation screen when first starting the application.

[1555] Step 1.2: Enter your appliance information

[1556] User: Opens the app and scans the QR code of the purchased appliance with their smartphone camera or manually enters the product's model number or serial number.

[1557] Terminal: Scans the QR code, analyzes the product information (or manually entered information), and sends it to the server.

[1558] Step 1.3: Save product information

[1559] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions.

[1560] 2. Obtaining operation instructions

[1561] Step 1: Enter your question

[1562] User: Selects a registered appliance within the app and enters a specific instruction or troubleshooting question in the text box.

[1563] Terminal: Sends the entered question to the server.

[1564] Step 2: Parse the question

[1565] Server: Analyzes the question using a natural language processing (NLP) engine to understand the intent of the user's question. For example, it analyzes the question "Where is the start button on the microwave?" and identifies the location of the start button.

[1566] Step 3: Emotion recognition with the emotion engine

[1567] Device: Using a camera and microphone, the device analyzes the user's facial expressions and tone of voice, and uses an emotion engine to recognize the user's emotional state.

[1568] Step 4: Generate an answer

[1569] Server: Generates appropriate responses based on information retrieved from the database, and adjusts the tone and content of the responses depending on the emotional state recognized by the emotion engine.

[1570] Server: Sends the generated answer to the device.

[1571] Step 5: View your answers

[1572] Device: The received answer is displayed on the screen within the app.

[1573] User: Check the displayed information, understand how to operate the appliance, and operate it correctly.

[1574] 3. Use AI support chatbots

[1575] Step 1: Chat Interface

[1576] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[1577] Terminal: Sends a question to the server.

[1578] Step 2: Processing questions

[1579] Server: The chatbot AI analyzes the question and searches the database for the appropriate answer. The emotion engine also analyzes the user's emotions.

[1580] Step 3: Chatbot response

[1581] Server: The server adjusts the answers generated by the chatbot AI according to the user's emotional state and sends them to the device.

[1582] On your device: Show the response in the chat screen.

[1583] User: Review the answers provided and take action as needed.

[1584] 4. Providing operation guidance using AR

[1585] Step 1: Use the camera

[1586] User: Scans home appliances using the smartphone camera.

[1587] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[1588] Step 2: Identify the operation location

[1589] Terminal: Using AR technology, it generates virtual operating procedures and guidelines based on the recognized operation locations, and requests additional information from the server as needed.

[1590] Step 3: AR display

[1591] Server: Generates any additional information needed and sends it to the device.

[1592] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen.

[1593] User: Follow the AR guidelines displayed through the camera to operate the home appliance appropriately.

[1594] Specific examples

[1595] Example 1: Getting instructions for operating a microwave

[1596] 1. User: Scans the QR code to register the microwave's information in the app.

[1597] 2. User: In the app, type "Where is the start button on the microwave?"

[1598] 3. Server: Analyzes the question and searches for the location of the start button.

[1599] 4. Device: Uses the camera and microphone to identify the user's emotions and prepares a quick, concise response if it determines, for example, that the user is feeling impatient.

[1600] 5. Server: Sends text and an image to the device explaining the location of the start button.

[1601] 6. Terminal: Display the answer.

[1602] 7. User: Check your answers and find and operate the start button on the microwave.

[1603] Example 2: Dealing with washing machine error codes

[1604] 1. User: Scan the QR code on the washing machine to register the product information.

[1605] 2. User: Asks in the app, "What does the E03 error code mean?"

[1606] 3. Server: Parse the question and find troubleshooting information for the E03 error.

[1607] 4. Device: The emotion engine analyzes the user's emotions and determines whether they are feeling anxious or impatient.

[1608] 5. Server: Depending on the user's emotional state, the server creates a message in a calm tone explaining the meaning of the error code and how to deal with it, and sends it to the device.

[1609] 6. Terminal: Display the answer.

[1610] 7. User: Follow the solutions to resolve the error.

[1611] This system allows users to easily access their appliances' instruction manuals and troubleshooting information, and also responds to their emotional state, improving the user experience and enabling them to operate their appliances more effectively.

[1612] The processing flow will be explained below.

[1613] Specific processing flow for the home appliance instruction manual platform

[1614] 1. Registering Home Appliances

[1615] Step 1: Install and register the app

[1616] Users: Install the dedicated application on their smartphone and create an account by entering personal information on the account creation screen when first starting the application.

[1617] Step 2: Enter your appliance information

[1618] User: Opens the app and scans the QR code of the purchased appliance with their smartphone camera or manually enters the product's model number or serial number.

[1619] Terminal: Scans the QR code, analyzes the product information (or manually entered information), and sends it to the server.

[1620] Step 3: Save your product information

[1621] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions.

[1622] 2. Obtaining operation instructions

[1623] Step 1: Enter your question

[1624] User: Selects a registered appliance within the app and enters a specific instruction or troubleshooting question in the text box.

[1625] Terminal: Sends the entered question to the server.

[1626] Step 2: Parsing the Question

[1627] Server: Analyzes the question using a natural language processing (NLP) engine to understand the intent of the user's question. For example, it analyzes the question "Where is the start button on the microwave?" and identifies the location of the start button.

[1628] Step 3: Emotion recognition by the emotion engine

[1629] On the device: Using a camera and microphone, the device analyzes the user's facial expressions and tone of voice, and uses an emotion engine to recognize the user's emotional state. For example, emotions such as smile, surprise, and anxiety are identified and an index is generated.

[1630] Step 4: Generate an answer

[1631] Server: Generates appropriate answers based on information retrieved from a database, and adjusts the tone and content of the answers based on the user's perceived emotional state, for example, explaining things in a calmer tone to an anxious user.

[1632] Server: Sends the generated answer to the device.

[1633] Step 5: View your answers

[1634] Device: The received answer is displayed on the screen within the app, with an appropriate interface based on the emotion.

[1635] User: Check the displayed information, understand how to operate the appliance, and operate it correctly.

[1636] 3. Use AI support chatbots

[1637] Step 1: Chat Interface

[1638] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[1639] Terminal: Sends a question to the server.

[1640] Step 2: Processing questions

[1641] Server: The chatbot AI analyzes the question and searches for the appropriate answer from the database. The emotion engine also analyzes the user's emotions.

[1642] Step 3: Chatbot response

[1643] Server: The server adjusts the answers generated by the chatbot AI according to the user's emotional state and sends them to the device.

[1644] Device: Responses will be displayed on the chat screen, and responses to specific emotional states may be highlighted with color coding or icons.

[1645] User: Review the answers provided and take action as needed.

[1646] 4. Providing operation guidance using AR

[1647] Step 1: Using the camera

[1648] User: Scans home appliances using the smartphone camera.

[1649] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[1650] Step 2: Identify the operation location

[1651] Terminal: Using AR technology, it generates virtual operating procedures and guidelines based on the recognized operation locations, and requests additional information from the server as needed.

[1652] Step 3: AR display

[1653] Server: Generates any additional information needed and sends it to the device.

[1654] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen.

[1655] User: Follow the AR guidelines displayed through the camera to operate the home appliance appropriately.

[1656] Specific examples

[1657] Example 1: Getting instructions for operating a microwave

[1658] 1. User: Scans the QR code to register the microwave's information in the app.

[1659] 2. User: In the app, type "Where is the start button on the microwave?"

[1660] 3. Server: Analyzes the question and searches for the location of the start button.

[1661] 4. Device: Uses the camera and microphone to identify the user's emotions and prepares a quick, concise response if it determines, for example, that the user is feeling impatient.

[1662] 5. Server: Sends text and an image to the device explaining the location of the start button.

[1663] 6. Terminal: Display the answer.

[1664] 7. User: Check your answers and find and operate the start button on the microwave.

[1665] Example 2: Dealing with washing machine error codes

[1666] 1. User: Scan the QR code on the washing machine to register the product information.

[1667] 2. User: Asks in the app, "What does the E03 error code mean?"

[1668] 3. Server: Parse the question and find troubleshooting information for the E03 error.

[1669] 4. Device: The emotion engine analyzes the user's emotions and determines whether they are feeling anxious or impatient.

[1670] 5. Server: Depending on the user's emotional state, the server creates a message in a calm tone explaining the meaning of the error code and how to deal with it, and sends it to the device.

[1671] 6. Terminal: Display the answer.

[1672] 7. User: Follow the solutions to resolve the error.

[1673] This system allows users to easily access their appliances' instruction manuals and troubleshooting information, and also responds to their emotional state, improving the user experience and enabling them to operate their appliances more effectively.

[1674] Example 2

[1675] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1676] Conventional systems that provide operating instructions and troubleshooting for home appliances often prevent users from properly obtaining the information they need. They also lack the ability to respond to user emotions, making it difficult to improve the user experience. Furthermore, entering specific product information and answering questions can be cumbersome, making intuitive operation difficult.

[1677] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1678] In this invention, the server includes means for receiving information about the home appliance input by the user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through the camera of the information terminal using augmented reality technology, emotion analysis means for recognizing the emotional state of the user, and means for generating a response according to the emotional state recognized by the emotion analysis means. This allows the user to intuitively obtain the information they need and enables responses according to their emotions, which is expected to improve the user experience.

[1679] The "information terminal owned by the user" refers to a portable electronic device that the user personally owns and can operate, such as a smartphone or tablet.

[1680] "Home appliance information" refers to basic information about the home appliance, such as the product name, model number, serial number, and QR code.

[1681] "Natural language processing" is a technology that uses computers to analyze and automatically process human language, including techniques for interpreting the meaning of text and answering questions.

[1682] "Augmented reality technology" is a technology that displays virtual information overlaid on the real environment, and includes, for example, a method of displaying operation guides on the surface of home appliances that can be seen through a camera.

[1683] "Emotion analysis means" refers to technology that analyzes a user's facial expressions and tone of voice to recognize the user's emotional state, and includes, for example, a method of determining a user's emotions in real time using a camera or microphone.

[1684] A "generative AI model" is a model that uses artificial intelligence technology to automatically generate answers to user questions, and includes, for example, large-scale natural language generation models.

[1685] A "database" is a collection of structured data that systematically stores large amounts of data and makes it easy to search and update. This system stores operating procedures and troubleshooting information for home appliances.

[1686] The "means for analyzing a question" refers to a technique for receiving a question from a user and understanding its content, and includes, for example, analyzing the intent of the question using a natural language processing engine.

[1687] The "means for displaying operating procedures and guidelines" refers to a technology that uses augmented reality technology to visually display specific operating procedures and guidelines for home appliances on the screen of an information terminal.

[1688] "Means for generating answers according to the emotional state" refers to technology that generates answers with appropriate tone and content based on the user's emotional state, and includes, for example, a method for adjusting the expression of the answer based on the results of emotional analysis.

[1689] The present invention is a system for obtaining operation methods and troubleshooting information for home appliances via an information terminal owned by a user, and includes the following detailed embodiments.

[1690] This system not only allows users to obtain operating instructions and troubleshooting information for home appliances through their information terminal, but also recognizes the user's emotional state and responds individually. The main hardware used includes smartphones and tablets. The main software used includes a natural language processing engine, augmented reality technology, an emotion analysis engine, and a generative AI model.

[1691] The server first receives information about home appliances entered by the user. The user installs a dedicated application on their smartphone, creates an account when they first start it, and then obtains this information by scanning the QR code of the purchased home appliance or manually entering the product's model number and serial number.

[1692] The device analyzes the received information about the home appliance and sends it to the server. The server stores the received information in a database and prepares and saves the relevant product manuals and operating procedures. This allows users to easily obtain the information they need.

[1693] Next, when the user enters a question about a home appliance within the app, the device sends the question to the server. The server then uses a natural language processing engine (such as Google Cloud Natural Language API or Microsoft Azure Text Analytics) to analyze the question and understand the user's intent. For example, the server can understand the question, "Where is the start button on the microwave?" and retrieve the corresponding information.

[1694] The device also analyzes the user's facial expressions and tone of voice to recognize the user's emotional state using an emotion analysis engine (e.g., Affectiva's emotion analysis API). For example, if the user is feeling anxious, the emotion engine will detect that the user's voice is high-pitched and fast-paced.

[1695] The server extracts appropriate operating procedures and troubleshooting information from the database and adjusts the tone and content of the response based on the results of sentiment analysis. For example, if the user is impatient, it will provide a concise and quick response. Appropriate responses are automatically generated using a generative AI model (e.g., OpenAI's GPT-3). The server sends the generated response to the device, which then displays it on the app screen.

[1696] The AI ​​support chatbot can be used when users input questions about operation methods or troubleshooting. The chatbot analyzes the question using a natural language processing engine, recognizes the user's emotional state using an emotion analysis engine, and responds appropriately. For example, the chatbot generates an appropriate answer for the question, "What should I do if my washing machine displays an E03 error?"

[1697] Furthermore, it also includes a function that uses augmented reality technology to display operating procedures and guidelines for home appliances through the information terminal's camera. When a user scans an appliance with their smartphone's camera, the terminal displays virtual guidelines for operation points based on the recognized information. This makes it easier for users to intuitively understand operating procedures.

[1698] Specific examples

[1699] Example 1: Getting instructions for operating a microwave

[1700] User: Scan the QR code to register the microwave's information in the app.

[1701] User: Within the app, type "Where is the start button on the microwave?"

[1702] Server: Analyzes the question and searches for the location of the start button.

[1703] On the device: Use the camera and microphone to identify the user's emotions and, for example, prepare a quick, concise response if it detects impatience.

[1704] Server: Sends text and an image to the device explaining the location of the start button.

[1705] Terminal: Display the answer.

[1706] User: Check your answers, find the start button on the microwave, and operate it.

[1707] Example 2: Dealing with washing machine error codes

[1708] User: Scan the QR code on the washing machine to register the product information.

[1709] User: Asks within the app, "What does the E03 error code mean?"

[1710] Server: Parse the question and find troubleshooting information for the E03 error.

[1711] Device: The emotion engine analyzes the user's emotions and determines whether they are feeling anxious or impatient.

[1712] Server: Depending on the user's emotional state, the server creates a message in a calm tone explaining the meaning of the error code and how to deal with it, and sends it to the device.

[1713] Terminal: Display the answer.

[1714] User: Follow the solutions to resolve the error.

[1715] The above-described configuration allows users to quickly and intuitively obtain operating instructions and troubleshooting information for home appliances, and also enables responses that correspond to the user's emotional state, significantly improving the user experience.

[1716] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1717] Specific processing flow of the system program

[1718] Step 1:

[1719] Installing the app and creating an account

[1720] User: Install the dedicated application on their smartphone, launch the app, and create an account by entering their first and last name, email address, password, etc. on the account creation screen. Then, click on the confirmation link sent to their email address to confirm their account.

[1721] Input: User's first name, last name, email address, and password

[1722] Output: Created user account information

[1723] Terminal: Receives information entered by the user and sends it to the server.

[1724] Server: Stores the received account information in a database.

[1725] Step 2:

[1726] Entering information about home appliances

[1727] User: Opens the app and scans the QR code of the purchased appliance or manually enters the product's model number and serial number.

[1728] Input: QR code information or product model number, serial number of the home appliance

[1729] Output: Parsed product information

[1730] Terminal: Scans the QR code to obtain product information (or manually entered information), parses it into a string, and sends it to the server.

[1731] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions.

[1732] Step 3:

[1733] Saving home appliance information

[1734] Server: Based on the registered home appliance information, the server references the database to find and store relevant product manuals and operating instructions.

[1735] Input: Registered home appliance information

[1736] Output: Database record of applicable product manuals and operating procedures

[1737] Specific operations: Performs data format conversion and tagging operations in the process of storing product information in a database.

[1738] Step 4:

[1739] User question input

[1740] User: Selects a registered appliance in the app and types a specific how-to or troubleshooting question into the text box, such as "Where is the start button on my microwave?"

[1741] Input: User question text

[1742] Output: The question sent to the server

[1743] Terminal: Sends the entered question to the server.

[1744] Step 5:

[1745] Question Analysis

[1746] Server: Analyzes the received question using a natural language processing (NLP) engine to identify the intent of the question, for example by breaking down the text into tokens and extracting keywords.

[1747] Input: Question text received from the user

[1748] Output: Question intent and keywords

[1749] Specific operation: Uses natural language processing API to perform text analysis and extract the intent of the question.

[1750] Step 6:

[1751] Emotion analysis

[1752] Device: Captures the user's facial expressions and tone of voice in real time and uses an emotion analysis engine to recognize their emotional state.

[1753] Input: User's facial expression data, tone of voice

[1754] Output: User's emotional state (e.g., impatience, anxiety, joy, etc.)

[1755] Specific operation: Using a camera and microphone, the user's facial expressions and voice are input into an emotion analysis engine to analyze their emotional state.

[1756] Step 7:

[1757] Answer generation and coordination

[1758] Server: Generates appropriate responses based on information retrieved from a database and adjusts the tone and content of responses based on sentiment analysis results. For example, prepare a concise and quick response for a user who is impatient. Uses a generative AI model.

[1759] Input: Question intent analyzed by natural language processing, sentiment analysis results, and answer candidates from the database

[1760] Output: Adjusted final answer

[1761] What it does: Uses a generative AI model to generate answers and tailor them appropriately based on sentiment analysis.

[1762] Step 8:

[1763] Submitting and viewing answers

[1764] Server: Sends the generated answer to the device.

[1765] Input: Adjusted final answer

[1766] Output: The answer sent to the user's device

[1767] Device: The received answer is displayed on the screen within the app.

[1768] User: Check the displayed information, understand how to operate the appliance, and operate it appropriately.

[1769] Step 9:

[1770] Use of AI support chatbots

[1771] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[1772] Input: User question text

[1773] Output: The question sent to the server

[1774] Terminal: Sends a question to the server.

[1775] Server: Analyzes the question, identifies the problem, and sends the answer generated by the chatbot AI to the device, analyzing the emotional state accordingly.

[1776] Step 10:

[1777] AR operation guidance

[1778] User: Uses the camera to scan home appliances and display operating instructions guidelines.

[1779] Input: Camera scan of product

[1780] Output: AR guidelines displayed on the smartphone screen

[1781] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[1782] Server: Generates any additional information needed and sends it to the device.

[1783] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen. Users follow the AR guidelines to operate home appliances appropriately.

[1784] These specific actions allow users to intuitively obtain operating procedures and troubleshooting information, and respond in accordance with their emotions.

[1785] (Application example 2)

[1786] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1787] Conventional methods for obtaining operating instructions and troubleshooting information for home appliances require users to refer to instruction manuals, which is time-consuming and inefficient. Furthermore, information is provided without taking into account the user's emotional state, which can amplify negative emotions such as anxiety and impatience. Furthermore, information acquisition is often limited to text-based information, and visual explanations can be difficult. These problems need to be solved.

[1788] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1789] In this invention, the server includes means for receiving information about the home appliance input by the user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through the camera of the information terminal using augmented reality technology, and means for analyzing the user's emotional state using an emotion recognition engine and adjusting responses based on the results. This enables the user to efficiently and appropriately obtain operating methods and troubleshooting information for the home appliance and provides responses that take the user's emotional state into consideration.

[1790] An "information terminal" is a device owned by a user that connects to the Internet and various applications to acquire and process information.

[1791] "Consumer appliances" refers to electrical and electronic devices used in the home that require instruction on how to operate or troubleshoot.

[1792] "Natural language processing" is a language understanding and generation technology that analyzes questions and instructions entered by users and generates appropriate responses based on them.

[1793] "Augmented reality technology" is a technology that displays virtual information overlaid on real-world images through the camera of an information terminal.

[1794] An "emotion recognition engine" is a technology that analyzes a user's facial expressions, tone of voice, etc. to estimate their emotional state.

[1795] An "AI support chatbot" is a conversational interface that uses artificial intelligence to automatically generate appropriate responses to user questions.

[1796] A "QR code" is a type of two-dimensional barcode that can efficiently read data including product information.

[1797] A "server" is a computer system that provides data and performs computations over a network.

[1798] This invention is a system for obtaining operating instructions and troubleshooting information for home appliances via a user's information terminal. The system receives information about the home appliance entered by the user, analyzes the user's question using natural language processing, and generates relevant operating instructions and troubleshooting information. It also uses augmented reality technology to display operating instructions and guidelines for the home appliance through the information terminal's camera, analyzes the user's emotional state using an emotion recognition engine, and adjusts responses based on that.

[1799] The hardware used is an information terminal such as a smartphone or tablet equipped with an internet connection, a camera, and a microphone. The software used includes Flask as a web framework, Hugging Face's Transformers as a natural language processing engine, and Face Recognition and Numpy for facial recognition and emotion analysis.

[1800] The server receives information about home appliances entered by the user and stores it in a database. When the user enters a question, the server analyzes the question using a natural language processing engine and generates relevant information. The generated information is then further adjusted according to the user's emotional state using an emotion recognition engine.

[1801] As a concrete example, consider a case where a user scans a QR code on an electronic product, registers the information in an application, and then enters a question about operation or troubleshooting. The server analyzes the question and generates an appropriate answer. If the user is feeling anxious or impatient, the emotion recognition engine will detect that emotion and provide an answer in a gentler tone accordingly.

[1802] An example of a prompt sentence for the generative AI model is as follows:

[1803] prompt:

[1804] A user asks, "What does washing machine error code E03 mean?" You can see the anxiety in their face. Generate a relaxing answer.

[1805] answer:

[1806] "The E03 error code is a drainage error. This can often be resolved by cleaning the filter. If you have any questions, please let us know. We'll fix it right away."

[1807] This system allows users to obtain information more efficiently than traditional methods of obtaining operating instructions or troubleshooting, and provides a better user experience by receiving responses that take into account their emotional state.

[1808] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1809] Step 1:

[1810] The user installs the dedicated application on their information terminal and starts it. When they first start it, they enter their personal information to create an account. During this process, the user uses their smartphone, and the account information is sent to the server and stored in a database.

[1811] Input: Personal information

[1812] Output: Account information stored on the server

[1813] Step 2:

[1814] A user scans the QR code of a purchased home appliance with the smartphone camera. Product information (model number, serial number, etc.) is obtained from the scanned QR code and sent to the server.

[1815] Input: QR code of home appliance

[1816] Output: Product information stored on the server

[1817] Step 3:

[1818] The server stores the received information about the home appliances in a database and prepares the relevant product manuals and operating procedures. This allows information about the home appliances owned by the user to be registered in the system.

[1819] Input:Product Information

[1820] Output: Product information stored in a database

[1821] Step 4:

[1822] The user uses the information terminal to enter a question about the registered home appliance into a text box within the app, and the question is sent to the server.

[1823] Input: User question

[1824] Output: The query data sent to the server

[1825] Step 5:

[1826] The server receives the question data and analyzes it using a natural language processing (NLP) engine, which identifies the intent of the question and prepares it for answer generation.

[1827] Input: Question data

[1828] Output: Analysis result (question intent)

[1829] Step 6:

[1830] The device uses the smartphone's camera and microphone to capture the user's facial expressions and tone of voice, and then uses an emotion recognition engine to analyze the user's emotional state. The analysis results are then sent to a server.

[1831] Input: User's facial and voice data

[1832] Output: Parsed emotional state data

[1833] Step 7:

[1834] The server searches a database for relevant operating procedures and troubleshooting information, and tailors responses based on the analysis by an emotion-recognition engine. For example, a user who is feeling anxious or impatient will receive a gentler response.

[1835] Input: Analysis results, emotional state data

[1836] Output: Adjusted response data

[1837] Step 8:

[1838] The server sends the generated answer data to the information terminal, which then displays the answer to the user. The user checks the displayed information and operates the home appliance appropriately.

[1839] Input: Adjusted response data

[1840] Output: Answer displayed on terminal

[1841] Step 9:

[1842] The user uses the camera on the information terminal to scan the home appliance, and the operating procedures and guidelines are displayed on the screen using augmented reality technology, allowing the user to visually confirm how to operate the appliance.

[1843] Input: Video data from home appliances

[1844] Output: AR guidelines displayed on the screen

[1845] In this way, users can efficiently and safely learn how to operate and troubleshoot their home appliances. It is designed to enhance the user experience based on step-by-step inputs and outputs.

[1846] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1847] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1848] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1849] [Fourth embodiment]

[1850] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1851] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1852] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1853] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1854] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1855] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1856] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1857] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1858] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1859] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1860] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1861] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1862] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1863] The present invention provides a system that allows users to easily obtain operating instructions and troubleshooting information for home appliances via their own information terminals. Specific embodiments of this system will be described below.

[1864] Program processing flow (natural language explanation)

[1865] 1. Registering Home Appliances

[1866] Step 1.1: Install and register the app

[1867] User: Install the dedicated application on your smartphone and create an account the first time you start it. When creating the account, you enter basic user information.

[1868] Step 1.2: Enter your appliance information

[1869] User: Scans the QR code on the appliance they purchased with their smartphone camera or manually enters the product's model number and serial number into the app.

[1870] Terminal: Parses the QR code or manually entered product information and sends it to the server.

[1871] Step 1.3: Save product information

[1872] Server: Stores the received information about the home appliances in a database. Also, prepares and stores the relevant product manuals and operating procedures on the server.

[1873] 2. Obtaining operation instructions

[1874] Step 2.1: Enter your question

[1875] Users: Select their registered appliances within the app and enter their specific operating instructions or troubleshooting questions.

[1876] Terminal: Sends the entered question to the server.

[1877] Step 2.2: Parse the question

[1878] Server: Analyzes the question using a natural language processing (NLP) engine and understands the user's intent. For example, in response to the question "Where is the start button on the microwave?", it searches for information about the location of the start button.

[1879] Step 2.3: Generate an answer

[1880] Server: Generates an appropriate answer based on the information retrieved from the database and sends the generated answer to the device.

[1881] Step 2.4: View your answers

[1882] Device: Display the generated answer on a screen within the app.

[1883] User: By checking the displayed information, the user understands how to operate the home appliance and operates it appropriately.

[1884] 3. Use AI support chatbots

[1885] Step 3.1: Chat Interface

[1886] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[1887] Terminal: Sends a question to the server.

[1888] Step 3.2: Processing Questions

[1889] Server: The chatbot AI analyzes the question and searches the database for the appropriate answer.

[1890] Step 3.3: Chatbot response

[1891] Server: Sends the answers generated by the chatbot AI to the device.

[1892] On your device: Show the response in the chat screen.

[1893] User: Review the answers provided and take action as needed.

[1894] 4. Providing operation guidance using AR

[1895] Step 4.1: Using the Camera

[1896] User: Uses smartphone camera to scan home appliances.

[1897] Terminal: Analyzes scanned images and recognizes specific parts of home appliances.

[1898] Step 4.2: Identify the operation location

[1899] Terminal: Based on the recognized operation locations, the terminal uses AR technology to generate virtual operation procedures and guidelines, and requests additional information from the server as needed.

[1900] Step 4.3: AR display

[1901] Server: Generates additional information and sends it to the device.

[1902] Device: Operating procedures and guidelines are displayed as AR on the smartphone screen.

[1903] User: Follow the AR guidelines displayed through the camera and perform the appropriate operation.

[1904] Specific examples

[1905] Example 1: Getting instructions for operating a microwave

[1906] 1. User: Scans the QR code to register the microwave's information in the app.

[1907] 2. User: In the app, type "Where is the start button on the microwave?"

[1908] 3. Server: Analyzes the question and searches for the location of the start button.

[1909] 4. Server: Sends text and an image to the device explaining the location of the start button.

[1910] 5. Terminal: Display the answer.

[1911] 6. User: Check your answers and find and operate the start button on the microwave.

[1912] Example 2: Dealing with washing machine error codes

[1913] 1. User: Scan the QR code on the washing machine to register the product information.

[1914] 2. User: Asks in the app, "What does the E03 error code mean?"

[1915] 3. Server: Parse the question and find troubleshooting information for the E03 error.

[1916] 4. Server: Sends the meaning of the error code and how to deal with it to the device.

[1917] 5. Terminal: Display the answer.

[1918] 6. User: Follow the solutions to resolve the error.

[1919] These specific examples allow users to easily access the instruction manuals for home appliances, enabling them to quickly understand how to operate them and troubleshoot problems. This system is particularly useful for products that require complex operation and for elderly users.

[1920] The processing flow will be explained below.

[1921] Specific processing flow for the home appliance instruction manual platform

[1922] 1. Registering Home Appliances

[1923] Step 1: Install and register the app

[1924] Users: Install the dedicated application on their smartphone and create an account by entering personal information on the account creation screen when first starting the application.

[1925] Step 2: Enter your appliance information

[1926] User: Opens the app and scans the QR code of the purchased appliance with their smartphone camera or manually enters the product's model number or serial number.

[1927] Terminal: Scans the QR code, analyzes the product information (or manually entered information), and sends it to the server.

[1928] Step 3: Save your product information

[1929] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions in the database.

[1930] 2. Obtaining operation instructions

[1931] Step 1: Enter your question

[1932] User: Selects a registered appliance within the app and enters a specific instruction or troubleshooting question in the text box.

[1933] Terminal: Sends the entered question to the server.

[1934] Step 2: Parsing the Question

[1935] Server: Analyzes the question using a natural language processing (NLP) engine to understand the intent of the user's question. For example, it analyzes the question "Where is the start button on the microwave?" and identifies the location of the start button.

[1936] Step 3: Generate an answer

[1937] Server: Generates an appropriate response based on the information retrieved from the database and sends it to the device.

[1938] Step 4: View your answers

[1939] Device: The received answer is displayed on the screen within the app.

[1940] User: Check the displayed information, understand how to operate the appliance, and operate it correctly.

[1941] 3. Use AI support chatbots

[1942] Step 1: Chat Interface

[1943] Users: Use the app's chat interface to type in instructions and troubleshooting questions.

[1944] Terminal: Sends a question to the server.

[1945] Step 2: Processing questions

[1946] Server: The chatbot AI analyzes the question and searches for corresponding information from the database.

[1947] Step 3: Chatbot response

[1948] Server: Sends the answers generated by the chatbot AI to the device.

[1949] On your device: Show the response in the chat screen.

[1950] User: Review the answers provided and take action as needed.

[1951] 4. Providing operation guidance using AR

[1952] Step 1: Using the camera

[1953] User: Uses smartphone camera to scan home appliances.

[1954] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[1955] Step 2: Identify the operation location

[1956] Terminal: Based on the recognized operation points, the device uses AR technology to generate virtual operation procedures and guidelines, and requests additional information from the server as needed.

[1957] Step 3: AR display

[1958] Server: Generates any additional information needed and sends it to the device.

[1959] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen.

[1960] User: Follow the AR guidelines displayed through the camera to operate the home appliance appropriately.

[1961] In this way, through each step, the user can easily obtain the instruction manual and troubleshooting information for the home appliance and perform the operation appropriately.

[1962] Example 1

[1963] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1964] Quickly and easily obtaining operating instructions and troubleshooting information for home appliances is a major challenge for many users. For elderly users and those less familiar with technology, simply reading instruction manuals can be difficult, making access to appropriate information crucial. Furthermore, the increasing diversity of products creates a need for efficient methods to provide specific information for individual products.

[1965] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1966] In this invention, the server includes means for receiving information about a home appliance input by a user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through a camera of the information terminal using augmented reality technology, means for saving information about the home appliance in a database, means for managing user account information, and means for receiving a question from an information terminal owned by the user, analyzing the question by the server to generate an appropriate answer, and transmitting the answer back to the information terminal. This allows the user to easily and quickly access operating methods and troubleshooting information for the home appliance.

[1967] "User" refers to an individual who uses a home appliance and operates an information terminal.

[1968] An "information terminal" refers to a device, such as a smartphone or tablet, that has Internet connectivity and can be operated by the user.

[1969] "Home appliances" refers to electronic devices and electrical appliances used in the home.

[1970] A "server" refers to a computer system that receives requests from users over a network, processes the information, and returns it.

[1971] "Database" refers to an information management system that efficiently manages large amounts of data and enables quick access to it.

[1972] "Natural language processing" refers to the technology that enables computers to understand, analyze, and generate human language.

[1973] "Augmented reality technology" refers to a technology that displays virtual information overlaid on images captured in real time.

[1974] "Operating procedures and guidelines" refers to information that explains the steps and methods required to use a home appliance.

[1975] An "AI support chatbot" refers to a program that uses artificial intelligence technology to automatically provide answers to users' questions.

[1976] The term "means for receiving a question" refers to a system for receiving question data entered by a user.

[1977] "QR Code" refers to a type of two-dimensional barcode, a matrix code used to store information.

[1978] "Means for analysis" refers to a system that processes input data and executes a process to understand its meaning.

[1979] "Answer generation means" refers to techniques and methods for creating appropriate answers to users' questions.

[1980] "AR Guidelines" refers to visual information about how to operate and procedures for home appliances that is displayed using augmented reality technology.

[1981] The present invention provides a system for quickly and easily obtaining operating instructions and troubleshooting information for home appliances via an information terminal owned by a user. Specific embodiments of the present invention will be described below.

[1982] First, users install a dedicated application on their information terminals, such as smartphones or tablets. This application allows users to register their home appliances and obtain information on how to operate them and troubleshooting. When launching the application for the first time, users must create an account. To create an account, users must enter basic information such as their name, email address, and password.

[1983] For home appliances that users want to register, they can scan the product's QR code or manually enter the model number and serial number. The information terminal then sends the QR code information or manually entered information to the server. The server then stores the received product information in a database and prepares related operating instructions and manuals.

[1984] When a user selects a specific appliance within the application and enters a question about how to use it or troubleshooting, the question is sent to the server. The server uses a natural language processing engine to analyze the question and understand the user's intent. For example, in response to the question, "Where is the start button on my microwave?", the server searches for information about the start button and generates an appropriate answer. This answer is sent to the information terminal in text or image format and displayed to the user.

[1985] The present invention also includes an AI support chatbot that can automatically provide appropriate answers when a user inputs a question through the chat interface of the application. For example, in response to the question, "What does the E03 error on my washing machine mean?", the chatbot will search the database and provide the meaning of the error code and how to deal with it.

[1986] Furthermore, it also has a function that uses augmented reality technology to display operating procedures and guidelines for home appliances through the information terminal's camera. When a user scans an appliance using the camera, the information terminal recognizes specific parts of the product and uses AR technology to display virtual operating procedures and guidelines. This allows the user to operate the product appropriately while visually checking the operating procedures.

[1987] Specific examples

[1988] Example 1: Getting instructions for operating a microwave

[1989] The user scans the QR code to register the microwave's information in the app. Then, they enter "Where is the start button on the microwave?" in the app. The server analyzes the question and searches for the location of the start button. The server sends text and an image explaining the location of the start button to the device, which displays it. The user checks the answer, finds the start button on the microwave, and operates it.

[1990] Example 2: Dealing with washing machine error codes

[1991] The user scans the washing machine's QR code to register the product information. Then, within the app, they ask, "What does the E03 error code mean?" The server analyzes the question and searches for troubleshooting information related to the E03 error. The server then sends the meaning of the error code and the solution to the device, which displays it. The user can then follow the solution to resolve the error.

[1992] In this way, the present invention allows users to easily and quickly obtain operating instructions and troubleshooting information for home appliances.

[1993] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1994] Step 1: Install the app and create an account

[1995] User: Install the dedicated application on your smartphone or tablet. Once installation is complete, launch the app for the first time and proceed to the account creation screen. Enter basic information such as your name, email address, and password, and press the "Create Account" button.

[1996] Input: Name, Email Address, Password

[1997] Output: Encrypted account information

[1998] Terminal: Basic information entered by the user is encrypted and sent to the server.

[1999] Input: Encrypted account information

[2000] Output: Data sent to the server

[2001] Server: Receives the encrypted account information, stores it in the database as a new account, and returns a registration completion response to the device.

[2002] Input: Encrypted user information

[2003] Output: Registration complete response

[2004] Step 2: Enter your appliance information

[2005] User: Scans a QR code on a home appliance's manual with a smartphone camera or manually enters the product's model number and serial number into designated fields in the app.

[2006] Input: QR code information or model number, serial number

[2007] Output: Product information submission request

[2008] Terminal: Analyzes the information in the QR code scanned with the camera or the product information entered manually, converts the analysis results into JSON format, and sends them to the server.

[2009] Input: QR code information or manually entered product information

[2010] Output: Product information in JSON format

[2011] Server: Analyzes the received product information and searches for a corresponding database entry for the home appliance. If a matching product is found, it registers the product's details and manual data in the database.

[2012] Input: Product Information

[2013] Output: Database registration results

[2014] Step 3: Enter your operating instructions

[2015] User: Selects a specific appliance within the app and types a how-to or troubleshooting question into the text field. For example, "Where is the start button on my microwave?"

[2016] Input: Question text

[2017] Output: Request to send question data

[2018] Terminal: The entered question is sent to the server as text data.

[2019] Input: Question text

[2020] Output: Query data to the server

[2021] Step 4: Parsing the Question

[2022] Server: The received question text is sent to a natural language processing (NLP) engine for semantic analysis. For example, keywords such as "start button" and "microwave" are extracted.

[2023] Input: Question text

[2024] Output: Parsed keywords

[2025] Step 5: Generate an answer

[2026] Server: Searches the database for appropriate operation methods and troubleshooting information. Based on the analysis results of the NLP engine, generates the most appropriate response and sends it to the device as text and image data.

[2027] Input: Parsed keyword

[2028] Output: Generate text and image data

[2029] Device: Analyzes the data received from the server and displays it in the app in a format that is easy for the user to view.

[2030] Input: Text and image data

[2031] Output: Data displayed in the app

[2032] Step 6: View your answers

[2033] User: Review the in-app instructions and troubleshooting information and use the information provided to operate the home appliance appropriately.

[2034] Input: The information shown

[2035] Output: User understanding and operation of home appliances

[2036] Step 7: Use a chatbot

[2037] User: Open the in-app chat interface and type a how-to or troubleshooting question. For example, "What does error E03 on my washing machine mean?"

[2038] Input: Chat question

[2039] Output: Request to send question data

[2040] Terminal: Sends the question to the server as text data.

[2041] Input: Chat question

[2042] Output: Query data to the server

[2043] Server: The chatbot AI analyzes the received questions and generates appropriate answers, which are then sent to the device in text format.

[2044] Input: Chat question

[2045] Output: Textual response

[2046] On your device: Display the received response on the chat screen.

[2047] Input: Text response

[2048] Output: Chat screen display data

[2049] User: Checks the chatbot's response and operates or troubleshoots the appliance based on it.

[2050] Enter: the displayed answer

[2051] Output: Operate or troubleshoot home appliances

[2052] Step 8: AR operation guidance

[2053] User: Uses the app's AR feature to activate their smartphone camera and scan home appliances.

[2054] Input: Video data from home appliances

[2055] Output: Scan data

[2056] Terminal: Analyzes video of home appliances in real time and recognizes specific parts. Based on the recognition results, it uses AR technology to display operating procedures and guidelines.

[2057] Input: Video data

[2058] Output: AR display data

[2059] Server: Generates any additional information needed and sends it to the device.

[2060] Input: Request information

[2061] output: additional information data

[2062] Device: AR guidelines and operating procedures are superimposed on the image captured by the camera.

[2063] Input: Additional information data

[2064] Output: Display data overlaid on camera image

[2065] User: Check the AR guidelines and follow the instructions to operate the home appliance.

[2066] Input: AR guidelines

[2067] Output: Control of home appliances

[2068] (Application example 1)

[2069] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2070] It is difficult for users to quickly and accurately obtain operating instructions and troubleshooting information for home appliances. Conventional systems can take time to provide operating instructions and troubleshooting, potentially reducing user satisfaction. Furthermore, elderly users and those unfamiliar with technology have difficulty operating products and resolving problems.

[2071] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2072] In this invention, the server includes means for receiving information about the home appliance input by the user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through the camera of the information terminal using augmented reality technology, and means for displaying the generated answers on the information terminal, thereby enabling the user to quickly and accurately obtain operating methods and troubleshooting information for the home appliance.

[2073] (definition statement)

[2074] "Information terminal" is a general term for an electronic device that a user owns and uses to obtain operating instructions and troubleshooting information for home appliances.

[2075] "Natural language processing" is a technology that analyzes user questions, understands their intent, and generates appropriate operating procedures and troubleshooting information.

[2076] "Augmented reality technology" is a technology that uses the camera of an information terminal to display operating procedures and guidelines for home appliances overlaid on real-world images.

[2077] A "generative AI model" is an artificial intelligence technology that generates answers and guidelines to user questions.

[2078] A "prompt" is a question or instruction phrase that is input to a generative AI model.

[2079] A "QR code" is a two-dimensional code that contains information about a home appliance, and is an identification means that can be read by scanning it with the camera of an information terminal.

[2080] "AR display" refers to the use of augmented reality technology to display virtual operating procedures and guidelines on the screen of an information terminal.

[2081] "Home appliances" is a general term for electrical appliances used in ordinary households that may require instruction on operation or troubleshooting.

[2082] A "database" is a system for accumulating information about home appliances, operating procedures, and troubleshooting information, and storing them in a searchable format.

[2083] A "chat interface" is an operation screen that allows users to input questions in natural language and receive answers in an interactive format.

[2084] The present invention provides a system that allows users to easily obtain operating instructions and troubleshooting information for home appliances via their own information terminals. This system begins when the user installs a dedicated application on their smartphone. Specific embodiments of this system are described below.

[2085] System configuration

[2086] The system consists of the following main components:

[2087] 1. Information terminal: Electronic device such as a smartphone.

[2088] 2. Server: A remote computer containing a database and a natural language processing engine.

[2089] 3. Generative AI model: An artificial intelligence technology that generates appropriate answers to user questions.

[2090] Program processing

[2091] 1. Registering Home Appliances

[2092] Users install a dedicated application on their smartphone and create an account when they first start the app.

[2093] Users can either scan the QR code on the home appliance they purchased with their smartphone camera or manually enter the product's model number and serial number into the app.

[2094] The terminal analyzes the QR code or manually entered product information and sends the information to the server.

[2095] The server stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating procedures on the server.

[2096] 2. Obtaining operation instructions

[2097] Users select registered home appliances within the app and enter specific operating instructions and troubleshooting questions.

[2098] The terminal transmits the entered question to the server.

[2099] The server analyzes the question using a natural language processing (NLP) engine to understand the user's intent.

[2100] The server generates an appropriate response based on the information retrieved from the database and sends it to the terminal.

[2101] The device displays the generated answer on the screen within the app.

[2102] 3. Use AI support chatbots

[2103] Users use the app's chat interface to type in their how-to and troubleshooting questions.

[2104] The terminal sends a query to the server.

[2105] The server uses a chatbot AI to analyze the question and search for the appropriate answer from a database.

[2106] The server sends the answers generated by the chatbot AI to the terminal, where they are displayed.

[2107] 4. Providing operation guidance using AR

[2108] Users use their smartphone camera to scan home appliances.

[2109] The device analyzes the scanned image and recognizes specific parts of the home appliance.

[2110] Based on the recognized operation locations, the device uses AR technology to generate virtual operation procedures and guidelines, and requests additional information from the server if necessary.

[2111] The server generates additional information and sends it to the terminal.

[2112] The device displays operating procedures and guidelines as AR displays on the smartphone screen.

[2113] Specific examples

[2114] Example 1: Getting instructions for operating a microwave

[2115] 1. The user scans the QR code to register the microwave's information in the app.

[2116] 2. The user types in the app, "Where is the start button on the microwave?"

[2117] 3. The server analyzes the question and searches for the location of the start button.

[2118] 4. The server sends text and an image to the device explaining the location of the start button.

[2119] 5. The device will display the answer.

[2120] 6. The user checks their answer, finds the start button on the microwave, and operates it.

[2121] Example 2: Dealing with washing machine error codes

[2122] 1. The user scans the QR code on the washing machine to register the product information.

[2123] 2. The user asks within the app, "What does the E03 error code mean?"

[2124] 3. The server parses the question and searches for troubleshooting information about the E03 error.

[2125] 4. The server sends the meaning of the error code and how to deal with it to the terminal.

[2126] 5. The device will display the answer.

[2127] 6. The user follows the instructions to resolve the error.

[2128] Prompt Sentence Examples

[2129] "Please tell me how to use the microwave."

[2130] The system aims to help users operate home appliances quickly and efficiently, and is especially useful for elderly and less technologically savvy users.

[2131] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2132] Step 1:

[2133] Users install a dedicated application on their smartphone and create an account the first time they start it. At this time, the user enters basic information such as an email address and password. The information entered is stored in the application's database and used for future authentication.

[2134] Step 2:

[2135] Users can scan the QR code of a purchased home appliance with their smartphone camera or manually enter the product's model number or serial number into the app. The entered information (QR code or text) is parsed by the device and product information is extracted, allowing the home appliance to be identified.

[2136] Step 3:

[2137] The device then sends the analyzed product information to the server, which then stores the received product information in a database and prepares and stores the corresponding product manuals and operating procedures. During this process, the product information and manuals are linked.

[2138] Step 4:

[2139] Users select a registered home appliance within the app and enter specific operating instructions or troubleshooting questions. The entered questions (text) are sent from the device to the server, where a generative AI model is used to answer them.

[2140] Step 5:

[2141] The server analyzes the question using a natural language processing (NLP) engine to understand the user's intent. The server extracts keywords from the question and uses them to search a database for relevant operating procedures and troubleshooting information. The analysis results are then confirmed by a generative AI model.

[2142] Step 6:

[2143] The server generates an appropriate answer based on the information retrieved from the database, and the answer (text or image) is sent to the device.

[2144] Step 7:

[2145] The device displays the received response on the app screen, where the user can review the displayed information to understand how to operate the appliance and troubleshoot the problem.

[2146] Step 8:

[2147] Users use the app's chat interface to type in any follow-up instructions or troubleshooting questions, which are also sent from the device to the server.

[2148] Step 9:

[2149] The server uses chatbot AI to analyze the question and generate an appropriate answer. A generative AI model is used in this process to ensure the accuracy of the answer. The generated answer is then sent to the device.

[2150] Step 10:

[2151] The device displays the answer on the chat interface, and the user can check the answer and operate the home appliance as needed.

[2152] Step 11:

[2153] Users scan home appliances using their smartphone cameras, and the scanned images (image data) are analyzed by the device to identify specific parts of the appliance.

[2154] Step 12:

[2155] The device uses AR technology to generate virtual operating procedures and guidelines based on the recognized operation points, using data provided by the generative AI model, and requests additional information from the server if necessary.

[2156] Step 13:

[2157] The server generates additional information and sends it to the device. The device displays operation procedures and guidelines as AR on the smartphone screen. The user follows the AR guidelines displayed through the camera to perform the appropriate operation.

[2158] Example prompt: "Tell me how to use a microwave."

[2159] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2160] The present invention combines an emotion engine with a system for obtaining operating instructions and troubleshooting information for home appliances via a user's information terminal. This system has the function of analyzing a user's questions and providing appropriate operating instructions and troubleshooting information, as well as the function of recognizing the user's emotional state and generating a response accordingly. Specific embodiments of this system are described below.

[2161] Program processing flow (natural language explanation)

[2162] 1. Registering Home Appliances

[2163] Step 1.1: Install and register the app

[2164] Users: Install the dedicated application on their smartphone and create an account by entering personal information on the account creation screen when first starting the application.

[2165] Step 1.2: Enter your appliance information

[2166] User: Opens the app and scans the QR code of the purchased appliance with their smartphone camera or manually enters the product's model number or serial number.

[2167] Terminal: Scans the QR code, analyzes the product information (or manually entered information), and sends it to the server.

[2168] Step 1.3: Save product information

[2169] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions.

[2170] 2. Obtaining operation instructions

[2171] Step 1: Enter your question

[2172] User: Selects a registered appliance within the app and enters a specific instruction or troubleshooting question in the text box.

[2173] Terminal: Sends the entered question to the server.

[2174] Step 2: Parse the question

[2175] Server: Analyzes the question using a natural language processing (NLP) engine to understand the intent of the user's question. For example, it analyzes the question "Where is the start button on the microwave?" and identifies the location of the start button.

[2176] Step 3: Emotion recognition with the emotion engine

[2177] Device: Using a camera and microphone, the device analyzes the user's facial expressions and tone of voice, and uses an emotion engine to recognize the user's emotional state.

[2178] Step 4: Generate an answer

[2179] Server: Generates appropriate responses based on information retrieved from the database, and adjusts the tone and content of the responses depending on the emotional state recognized by the emotion engine.

[2180] Server: Sends the generated answer to the device.

[2181] Step 5: View your answers

[2182] Device: The received answer is displayed on the screen within the app.

[2183] User: Check the displayed information, understand how to operate the appliance, and operate it correctly.

[2184] 3. Use AI support chatbots

[2185] Step 1: Chat Interface

[2186] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[2187] Terminal: Sends a question to the server.

[2188] Step 2: Processing questions

[2189] Server: The chatbot AI analyzes the question and searches the database for the appropriate answer. The emotion engine also analyzes the user's emotions.

[2190] Step 3: Chatbot response

[2191] Server: The server adjusts the answers generated by the chatbot AI according to the user's emotional state and sends them to the device.

[2192] On your device: Show the response in the chat screen.

[2193] User: Review the answers provided and take action as needed.

[2194] 4. Providing operation guidance using AR

[2195] Step 1: Use the camera

[2196] User: Scans home appliances using the smartphone camera.

[2197] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[2198] Step 2: Identify the operation location

[2199] Terminal: Using AR technology, it generates virtual operating procedures and guidelines based on the recognized operation locations, and requests additional information from the server as needed.

[2200] Step 3: AR display

[2201] Server: Generates any additional information needed and sends it to the device.

[2202] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen.

[2203] User: Follow the AR guidelines displayed through the camera to operate the home appliance appropriately.

[2204] Specific examples

[2205] Example 1: Getting instructions for operating a microwave

[2206] 1. User: Scans the QR code to register the microwave's information in the app.

[2207] 2. User: In the app, type "Where is the start button on the microwave?"

[2208] 3. Server: Analyzes the question and searches for the location of the start button.

[2209] 4. Device: Uses the camera and microphone to identify the user's emotions and prepares a quick, concise response if it determines, for example, that the user is feeling impatient.

[2210] 5. Server: Sends text and an image to the device explaining the location of the start button.

[2211] 6. Terminal: Display the answer.

[2212] 7. User: Check your answers and find and operate the start button on the microwave.

[2213] Example 2: Dealing with washing machine error codes

[2214] 1. User: Scan the QR code on the washing machine to register the product information.

[2215] 2. User: Asks in the app, "What does the E03 error code mean?"

[2216] 3. Server: Parse the question and find troubleshooting information for the E03 error.

[2217] 4. Device: The emotion engine analyzes the user's emotions and determines whether they are feeling anxious or impatient.

[2218] 5. Server: Depending on the user's emotional state, the server creates a message in a calm tone explaining the meaning of the error code and how to deal with it, and sends it to the device.

[2219] 6. Terminal: Display the answer.

[2220] 7. User: Follow the solutions to resolve the error.

[2221] This system allows users to easily access their appliances' instruction manuals and troubleshooting information, and also responds to their emotional state, improving the user experience and enabling them to operate their appliances more effectively.

[2222] The processing flow will be explained below.

[2223] Specific processing flow for the home appliance instruction manual platform

[2224] 1. Registering Home Appliances

[2225] Step 1: Install and register the app

[2226] Users: Install the dedicated application on their smartphone and create an account by entering personal information on the account creation screen when first starting the application.

[2227] Step 2: Enter your appliance information

[2228] User: Opens the app and scans the QR code of the purchased appliance with their smartphone camera or manually enters the product's model number or serial number.

[2229] Terminal: Scans the QR code, analyzes the product information (or manually entered information), and sends it to the server.

[2230] Step 3: Save your product information

[2231] Server: Stores the received information about the home appliances in a database, and prepares and stores the relevant product manuals and operating instructions.

[2232] 2. Obtaining operation instructions

[2233] Step 1: Enter your question

[2234] User: Selects a registered appliance within the app and enters a specific instruction or troubleshooting question in the text box.

[2235] Terminal: Sends the entered question to the server.

[2236] Step 2: Parsing the Question

[2237] Server: Analyzes the question using a natural language processing (NLP) engine to understand the intent of the user's question. For example, it analyzes the question "Where is the start button on the microwave?" and identifies the location of the start button.

[2238] Step 3: Emotion recognition by the emotion engine

[2239] On the device: Using a camera and microphone, the device analyzes the user's facial expressions and tone of voice, and uses an emotion engine to recognize the user's emotional state. For example, emotions such as smile, surprise, and anxiety are identified and an index is generated.

[2240] Step 4: Generate an answer

[2241] Server: Generates appropriate answers based on information retrieved from a database, and adjusts the tone and content of the answers based on the user's perceived emotional state, for example, explaining things in a calmer tone to an anxious user.

[2242] Server: Sends the generated answer to the device.

[2243] Step 5: View your answers

[2244] Device: The received answer is displayed on the screen within the app, with an appropriate interface based on the emotion.

[2245] User: Check the displayed information, understand how to operate the appliance, and operate it correctly.

[2246] 3. Use AI support chatbots

[2247] Step 1: Chat Interface

[2248] Users: Use the app's chat interface to type in how-to and troubleshooting questions.

[2249] Terminal: Sends a question to the server.

[2250] Step 2: Processing questions

[2251] Server: The chatbot AI analyzes the question and searches for the appropriate answer from the database. The emotion engine also analyzes the user's emotions.

[2252] Step 3: Chatbot response

[2253] Server: The server adjusts the answers generated by the chatbot AI according to the user's emotional state and sends them to the device.

[2254] Device: Responses will be displayed on the chat screen, and responses to specific emotional states may be highlighted with color coding or icons.

[2255] User: Review the answers provided and take action as needed.

[2256] 4. Providing operation guidance using AR

[2257] Step 1: Using the camera

[2258] User: Scans home appliances using the smartphone camera.

[2259] Terminal: Analyzes the scanned image, recognizes specific parts of the home appliance, and obtains their location information.

[2260] Step 2: Identify the operation location

[2261] Terminal: Using AR technology, it generates virtual operating procedures and guidelines based on the recognized operation locations, and requests additional information from the server as needed.

[2262] Step 3: AR display

[2263] Server: Generates any additional information needed and sends it to the device.

[2264] Device: Operating procedures and guidelines are displayed in real time as AR on the smartphone screen.

[2265] User: Follow the AR guidelines displayed through the camera to operate the home appliance appropriately.

[2266] Specific examples

[2267] Example 1: Getting instructions for operating a microwave

[2268] 1. User: Scans the QR code to register the microwave's information in the app.

[2269] 2. User: In the app, type "Where is the start button on the microwave?"

[2270] 3. Server: Analyzes the question and searches for the location of the start button.

[2271] 4. Device: Uses the camera and microphone to identify the user's emotions and prepares a quick, concise response if it determines, for example, that the user is feeling impatient.

[2272] 5. Server: Sends text and an image to the device explaining the location of the start button.

[2273] 6. Terminal: Display the answer.

[2274] 7. User: Check your answers and find and operate the start button on the microwave.

[2275] Example 2: Dealing with washing machine error codes

[2276] 1. User: Scan the QR code on the washing machine to register the product information.

[2277] 2. User: Asks in the app, "What does the E03 error code mean?"

[2278] 3. Server: Parse the question and find troubleshooting information for the E03 error.

[2279] 4. Device: The emotion engine analyzes the user's emotions and determines whether they are feeling anxious or impatient.

[2280] 5. Server: Depending on the user's emotional state, the server creates a message in a calm tone explaining the meaning of the error code and how to deal with it, and sends it to the device.

[2281] 6. Terminal: Display the answer.

[2282] 7. User: Follow the solutions to resolve the error.

[2283] This system allows users to easily access their appliances' instruction manuals and troubleshooting information, and also responds to their emotional state, improving the user experience and enabling them to operate their appliances more effectively.

[2284] Example 2

[2285] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2286] Conventional systems that provide operating instructions and troubleshooting for home appliances often prevent users from properly obtaining the information they need. They also lack the ability to respond to user emotions, making it difficult to improve the user experience. Furthermore, entering specific product information and answering questions can be cumbersome, making intuitive operation difficult.

[2287] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2288] In this invention, the server includes means for receiving information about the home appliance input by the user, means for analyzing the user's question using natural language processing and generating corresponding operating procedures and troubleshooting information, means for displaying operating procedures and guidelines for the home appliance through the camera of the information terminal using augmented reality technology, emotion analysis means for recognizing the emotional state of the user, and means for generating a response according to the emotional state recognized by the emotion analysis means. This allows the user to intuitively obtain the information they need and enables responses according to their emotions, which is expected to improve the user experience.

[2289] The "information terminal owned by the user" refers to a portable electronic device that the user personally owns and can operate, such as a smartphone or tablet.

[2290] "Home appliance information" refers to basic information about the home appliance, such as the product name, model number, serial number, and QR code.

[2291] "Natural language processing" is a technology that uses computers to analyze and automatically process human language, including techniques for interpreting the meaning of text and answering questions.

[2292] "Augmented reality technology" is a technology that displays virtual information overlaid on the real environment, and includes, for example, a method of displaying operation guides on the surface of home appliances that can be seen through a camera.

[2293] "Emotion analysis means" refers to technology that analyzes a user's facial expressions and tone of voice to recognize the user's emotional state, and includes, for example, a method of determining a user's emotions in real time using a camera or microphone.

[2294] A "generative AI model" is a model that uses artificial intelligence technology to automatically generate answers to user questions, and includes, for example, large-scale natural language generation models.

[2295] A "database" is a collection of structured data that systematically stores large amounts of data and makes it easy to search and update. This system stores operating procedures and troubleshooting information for home appliances.

[2296] The "means for analyzing a question" refers to a technique for receiving a question from a user and understanding its content, and includes, for example, analyzing the intent of the question using a natural language processing engine.

[2297] The "means for displaying operating procedures and guidelines" refers to a technology that uses augmented reality technology to visually display specific operating procedures and guidelines for home appliances on the screen of an information terminal.

[2298] "Means for generating answers according to the emotional state" refers to technology that generates answers with appropriate tone and content based on the user's emotional state, and includes, for example, a method for adjusting the expression of the answer based on the results of emotional analysis.

[2299] The present invention is a system for obtaining operation methods and troubleshooting information for home appliances via an information terminal owned...

Claims

1. A system for obtaining operation methods and troubleshooting information for home appliances via an information terminal owned by a user, means for receiving information about home appliances input by a user; means for analyzing a user's question using natural language processing and generating corresponding operating procedures and troubleshooting information; A method for displaying operating procedures and guidelines for home appliances through the camera of an information terminal using augmented reality technology; A system including:

2. The system of claim 1 , further comprising means for providing an AI support chatbot to provide appropriate answers to user questions.

3. The system of claim 1 , further comprising means for obtaining product information by scanning a QR code of a home appliance product.

Images